starting February 28, 2025 at 06:00:53 AM UTC
Creating 3 downstairs regions
Creating region directory at: /var/tmp/dsc/region
Creating dsc directory at: /tmp/dsc
Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.34060928
Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.021993108
Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.020925635
Created 3 regions
Starting 3 downstairs
Using existing output directory "/tmp/dsc"
Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
start ds: 8810
start ds: 8820
start ds: 8830
start access at:127.0.0.1:9998
Starting downstairs at port 8810
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
Make output file at "/tmp/dsc/downstairs-8810.txt"
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8810][0] initial start wait reports Starting
[8820][1] initial start wait reports Starting
[8830][2] initial start wait reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
[8810][0] initial start wait reports Running
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
[8820][1] initial start wait reports Running
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1203)
[8830][2] initial start wait reports Running
All downstairs are running
Feb 28 06:00:54.078 INFO listening, local_addr: 127.0.0.1:9998
Control access at:127.0.0.1:9998
Running initial fill
Feb 28 06:00:59.160 INFO accepted connection, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
Feb 28 06:00:59.162 INFO request completed, latency_us: 394, response_code: 200, uri: /regioninfo, method: GET, req_id: a60912e9-076f-486f-b7b3-7e4b826f914e, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
Feb 28 06:00:59.164 INFO request completed, latency_us: 114, response_code: 200, uri: /regioncount, method: GET, req_id: 44e823ac-c66f-4956-8967-28bc7ef7cdf8, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.164150873Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"task":"crutest"}
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.164680966Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"task":"crutest"}
Feb 28 06:00:59.165 INFO request completed, latency_us: 175, response_code: 200, uri: /port/cid/0, method: GET, req_id: a266326f-09b3-44da-a78e-1042881ce6a6, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
Feb 28 06:00:59.165 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: a244afb5-bfbc-479c-aeae-06d19a06f7de, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
Feb 28 06:00:59.166 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: 4a775af0-69e9-41a3-bf87-499abb342f8c, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.16637808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.16679448Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0c103f9ae2997264530dbd443fadbdf83878dac4\",\n git_commit_timestamp: \"2025-02-28T05:45:10.000000000Z\",\n git_branch: \"renovate/uuid-1.x-lockfile\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.166940625Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.166968628Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.166988403Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
{"msg":"Crucible 46604218-9055-4498-b8c8-dba792c5ec55 has session id: 03892bbc-2ae7-40f4-9218-bbfee5a2de00","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.167805588Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Upstairs opts: Upstairs UUID: 46604218-9055-4498-b8c8-dba792c5ec55, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.167845348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Crucible stats registered with UUID: 46604218-9055-4498-b8c8-dba792c5ec55","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.167859644Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:00:59.167951913Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.16929808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"task":"crutest"}
{"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169330892Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
{"msg":"46604218-9055-4498-b8c8-dba792c5ec55 active request set","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.16938206Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169404664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169468049Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169497452Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169525455Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169551109Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169576463Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169594969Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169609555Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169623192Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51786)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169654574Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36704)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169681628Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43639)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.169708301Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"io task","client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.171303949Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.171332062Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.171347009Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.172313747Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.172348799Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.172376802Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173338981Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173364875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.17347044Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173492734Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173507881Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173521648Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173534664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173548601Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173561948Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173575185Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173588711Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173601099Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173613856Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173626473Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.17363895Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"":"downstairs","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173651667Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173663484Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"0","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.17367788Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"1","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173690607Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"client":"2","":"downstairs","session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173703664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173715601Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"46604218-9055-4498-b8c8-dba792c5ec55 is now active with session: 03892bbc-2ae7-40f4-9218-bbfee5a2de00","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173728428Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","Activate successful
pid":Wait for a query_work_queue command to finish before sending IO
1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173746244Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
{"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173758471Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Fill test
Read and Verify all blocks (0..5000 range:false)
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Read 256 Done Done Done false
1022 Acked Read 256 Done Done Done false
1023 Acked Read 256 Done Done Done false
1024 Acked Read 256 Done Done Done false
1025 Acked Read 256 Done Done Done false
1026 Acked Read 256 Done Done Done false
1027 Acked Read 256 Done Done Done false
1028 Acked Read 256 Done Done Done false
1029 Acked Read 256 Done Done Done false
1030 Acked Read 256 Done Done Done false
1031 Acked Read 256 Done Done Done false
1032 Acked Read 256 Done Done Done false
1033 Acked Read 256 Done Done Done false
1034 Acked Read 256 Done Done Done false
1035 Acked Read 256 Done Done Done false
1036 Acked Read 256 Done Done Done false
1037 Acked Read 256 Done Done Done false
1038 Acked Read 136 Done Done Done false
1039 Acked Read 256 Done Done Done false
1040 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1040 1039 1038 1037 1036
CLIENT: Up:0 ds:20 act:3
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1041 1040 1039 1038 1037
Upstairs last five completed: 1041 1040 1039 1038 1037
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:03.764987232Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1207}
Replay loop starts now February 28, 2025 at 06:01:03 AM UTC
Feb 28 06:01:03.835 INFO accepted connection, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
Feb 28 06:01:03.835 INFO request completed, latency_us: 171, response_code: 200, uri: /regioninfo, method: GET, req_id: f2655093-b468-485f-930b-437639575d5a, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.836587962Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"task":"crutest"}
Feb 28 06:01:03.836 INFO request completed, latency_us: 84, response_code: 200, uri: /regioncount, method: GET, req_id: 911ef9b1-3eb1-48dc-8c06-0436bf3b8e33, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.837102949Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"task":"crutest"}
Feb 28 06:01:03.837 INFO request completed, latency_us: 128, response_code: 200, uri: /port/cid/0, method: GET, req_id: 8dc0fcdc-de28-44cd-9e56-0f2049c4a1a3, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
Feb 28 06:01:03.838 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/1, method: GET, req_id: 350d8791-8487-4934-953b-e7537df3f574, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
Feb 28 06:01:03.838 INFO request completed, latency_us: 84, response_code: 200, uri: /port/cid/2, method: GET, req_id: f0fe630b-bf01-4186-a72d-9ff45252fb31, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.838709304Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.838884542Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0c103f9ae2997264530dbd443fadbdf83878dac4\",\n git_commit_timestamp: \"2025-02-28T05:45:10.000000000Z\",\n git_branch: \"renovate/uuid-1.x-lockfile\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.838920733Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.83893426Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.838948306Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
{"msg":"Crucible 1ce0952d-43d2-414d-849d-64dd1aa825bb has session id: 969fa3b2-3050-4644-bbdf-c5076af6b813","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.839428371Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Upstairs opts: Upstairs UUID: 1ce0952d-43d2-414d-849d-64dd1aa825bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.839466512Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Crucible stats registered with UUID: 1ce0952d-43d2-414d-849d-64dd1aa825bb","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.839494146Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:03.839514031Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840429531Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"task":"crutest"}
{"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840463883Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
{"msg":"1ce0952d-43d2-414d-849d-64dd1aa825bb active request set","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840487828Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840535856Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840575496Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.84060333Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840629603Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840654617Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840681971Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840705875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840732179Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.840757963Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53702)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.841011532Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60632)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.841043124Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57554)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.84106234Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.841489507Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.841513102Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.841531437Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842152269Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842180552Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842198727Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842911077Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842936371Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.842996886Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843016122Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843030808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843044145Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843057142Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843075117Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843088754Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843101861Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Activate successful
,Wait for a query_work_queue command to finish before sending IO
"time":"2025-02-28T06:01:03.843115408Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843133713Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.84314667Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843159277Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843171854Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843184561Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843196088Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843208805Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843221462Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843234039Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843245776Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"1ce0952d-43d2-414d-849d-64dd1aa825bb is now active with session: 969fa3b2-3050-4644-bbdf-c5076af6b813","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843258173Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.84327249Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.843284357Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Loading write count information from file "/tmp/test_replay_verify.log"
Feb 28 06:01:03.876 INFO accepted connection, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:01:03.877 INFO request completed, latency_us: 195, response_code: 204, uri: /stop/cid/1, method: POST, req_id: fb1fceb5-7be1-43c4-8e1c-8c89ea75b5a8, remote_addr: 127.0.0.1:63908, local_addr[8820] Got stop action so:false kr:false
: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:03.877716215Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:03.877806544Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:03.877830598Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:03.877888624Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:01:03.878 INFO request completed, latency_us: 302, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6b654463-e477-4552-ab05-0be23952096d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:01:07.879 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6a168064-2039-465a-8b09-3c2748560d7e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 3859 len 12288 data: 2 2 2
002/300 Write block 971 len 4096 data: 2
003/300 Write block 378 len 36864 data: 2 2 2 2 2 2 2 2 2
004/300 Write block 1950 len 40960 data: 2 2 2 2 2 2 2 2 2 2
005/300 Flush
006/300 Read block 1359 len 40960
007/300 Read block 3799 len 8192
008/300 Write block 3 len 40960 data: 2 2 2 2 2 2 2 2 2 2
009/300 Flush
010/300 Read block 3654 len 32768
011/300 Flush
012/300 Read block 3941 len 36864
013/300 Read block 985 len 4096
014/300 Read block 1597 len 8192
015/300 Read block 1891 len 32768
016/300 Read block 2009 len 36864
017/300 Read block 4051 len 12288
018/300 Read block 445 len 36864
019/300 Write block 2597 len 20480 data: 2 2 2 2 2
020/300 Read block 1277 len 4096
021/300 Write block 4020 len 24576 data: 2 2 2 2 2 2
022/300 Write block 413 len 24576 data: 2 2 2 2 2 2
023/300 Read block 4891 len 8192
024/300 Write block 3781 len 32768 data: 2 2 2 2 2 2 2 2
025/300 Read block 3259 len 24576
026/300 Read block 2979 len 36864
027/300 Write block 4717 len 16384 data: 2 2 2 2
028/300 Write block 1851 len 8192 data: 2 2
029/300 Read block 1660 len 12288
030/300 Write block 800 len 8192 data: 2 2
031/300 Write block 4352 len 32768 data: 2 2 2 2 2 2 2 2
032/300 Read block 4630 len 12288
033/300 Flush
034/300 Write block 2375 len 16384 data: 2 2 2 2
035/300 Write block 1347 len 36864 data: 2 2 2 2 2 2 2 2 2
036/300 Write block 1688 len 40960 data: 2 2 2 2 2 2 2 2 2 2
037/300 Read block 3420 len 36864
038/300 Read block 749 len 16384
039/300 Write block 3522 len 32768 data: 2 2 2 2 2 2 2 2
040/300 Write block 690 len 12288 data: 2 2 2
041/300 Read block 3326 len 16384
042/300 Read block 3156 len 32768
043/300 Write block 1521 len 40960 data: 2 2 2 2 2 2 2 2 2 2
044/300 Read block 2115 len 8192
045/300 Read block 941 len 8192
046/300 Read block 2173 len 24576
047/300 Write block 4039 len 4096 data: 2
048/300 Write block 4888 len 36864 data: 2 2 2 2 2 2 2 2 2
049/300 Write block 4542 len 40960 data: 2 2 2 2 2 2 2 2 2 2
050/300 Flush
051/300 Flush
052/300 Read block 1071 len 20480
053/300 Read block 1498 len 36864
054/300 Write block 41 len 8192 data: 2 2
055/300 Write block 1517 len 24576 data: 2 2 2 2 3 3
056/300 Read block 227 len 32768
057/300 Write block 4267 len 24576 data: 2 2 2 2 2 2
058/300 Write block 4407 len 28672 data: 2 2 2 2 2 2 2
059/300 Flush
060/300 Write block 1291 len 16384 data: 2 2 2 2
061/300 Write block 1668 len 36864 data: 2 2 2 2 2 2 2 2 2
062/300 Write block 2938 len 24576 data: 2 2 2 2 2 2
063/300 Read block 728 len 40960
064/300 Read block 4960 len 4096
065/300 Write block 2961 len 40960 data: 2 2 2 2 2 2 2 2 2 2
066/300 Read block 3906 len 32768
067/300 Write block 2273 len 8192 data: 2 2
068/300 Flush
069/300 Flush
070/300 Write block 4888 len 16384 data: 3 3 3 3
071/300 Read block 3141 len 4096
072/300 Write block 221 len 28672 data: 2 2 2 2 2 2 2
073/300 Read block 4057 len 32768
074/300 Write block 75 len 20480 data: 2 2 2 2 2
075/300 Read block 1837 len 32768
076/300 Write block 3416 len 8192 data: 2 2
077/300 Read block 4701 len 12288
078/300 Write block 1613 len 32768 data: 2 2 2 2 2 2 2 2
079/300 Read block 79 len 8192
080/300 Read block 2569 len 16384
081/300 Write block 4209 len 20480 data: 2 2 2 2 2
082/300 Write block 4190 len 40960 data: 2 2 2 2 2 2 2 2 2 2
083/300 Write block 749 len 40960 data: 2 2 2 2 2 2 2 2 2 2
084/300 Read block 3852 len 16384
085/300 Write block 3589 len 12288 data: 2 2 2
086/300 Read block 568 len 40960
087/300 Read block 3786 len 32768
088/300 Write block 337 len 8192 data: 2 2
089/300 Write block 857 len 32768 data: 2 2 2 2 2 2 2 2
090/300 Write block 841 len 24576 data: 2 2 2 2 2 2
091/300 Flush
092/300 Write block 2783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
093/300 Read block 4955 len 20480
094/300 Flush
095/300 Write block 4405 len 32768 data: 2 2 3 3 3 3 3 3
096/300 Read block 947 len 16384
097/300 Read block 4255 len 8192
098/300 Read block 3446 len 32768
099/300 Read block 2158 len 8192
100/300 Write block 4375 len 28672 data: 2 2 2 2 2 2 2
101/300 Read block 1539 len 20480
102/300 Read block 3711 len 32768
103/300 Read block 1577 len 24576
104/300 Flush
105/300 Write block 4764 len 36864 data: 2 2 2 2 2 2 2 2 2
106/300 Read block 636 len 24576
107/300 Write block 829 len 28672 data: 2 2 2 2 2 2 2
108/300 Read block 3901 len 4096
109/300 Write block 1142 len 20480 data: 2 2 2 2 2
110/300 Read block 4611 len 12288
111/300 Read block 2879 len 28672
112/300 Write block 2234 len 28672 data: 2 2 2 2 2 2 2
113/300 Read block 953 len 24576
114/300 Flush
115/300 Read block 1584 len 16384
116/300 Write block 3180 len 8192 data: 2 2
117/300 Write block 144 len 28672 data: 2 2 2 2 2 2 2
118/300 Flush
119/300 Write block 2392 len 8192 data: 2 2
120/300 Read block 4345 len 4096
121/300 Flush
122/300 Read block 4300 len 4096
123/300 Write block 3417 len 8192 data: 3 2
124/300 Read block 4402 len 28672
125/300 Read block 4524 len 40960
126/300 Flush
127/300 Read block 2144 len 20480
128/300 Read block 4599 len 8192
129/300 Write block 107 len 24576 data: 2 2 2 2 2 2
130/300 Read block 2414 len 36864
131/300 Flush
132/300 Read block 3088 len 8192
133/300 Read block 642 len 8192
134/300 Write block 866 len 20480 data: 2 2 2 2 2
135/300 Read block 4069 len 28672
136/300 Read block 3238 len 28672
137/300 Read block 1694 len 4096
138/300 Read block 76 len 16384
139/300 Read block 4023 len 4096
140/300 Read block 194 len 36864
141/300 Flush
142/300 Read block 4901 len 8192
143/300 Read block 1505 len 20480
144/300 Read block 4790 len 4096
145/300 Flush
146/300 Write block 938 len 40960 data: 2 2 2 2 2 2 2 2 2 2
147/300 Read block 3806 len 8192
148/300 Write block 4271 len 36864 data: 3 3 2 2 2 2 2 2 2
149/300 Read block 4887 len 36864
150/300 Write block 2353 len 4096 data: 2
151/300 Flush
152/300 Write block 1501 len 4096 data: 2
153/300 Write block 178 len 4096 data: 2
154/300 Read block 3084 len 36864
155/300 Write block 1674 len 28672 data: 3 3 3 2 2 2 2
156/300 Read block 4190 len 4096
157/300 Write block 1676 len 20480 data: 4 3 3 3 3
158/300 Flush
159/300 Write block 3688 len 4096 data: 2
160/300 Write block 678 len 16384 data: 2 2 2 2
161/300 Read block 1421 len 32768
162/300 Write block 2730 len 4096 data: 2
163/300 Read block 1274 len 28672
164/300 Read block 3719 len 8192
165/300 Read block 563 len 12288
166/300 Read block 1900 len 8192
167/300 Write block 1179 len 32768 data: 2 2 2 2 2 2 2 2
168/300 Read block 689 len 36864
169/300 Read block 3086 len 8192
170/300 Write block 2363 len 16384 data: 2 2 2 2
171/300 Write block 4984 len 16384 data: 2 2 2 2
172/300 Read block 3894 len 40960
173/300 Read block 2882 len 32768
174/300 Read block 1216 len 16384
175/300 Write block 4963 len 20480 data: 2 2 2 2 2
176/300 Write block 4238 len 24576 data: 2 2 2 2 2 2
177/300 Read block 226 len 4096
178/300 Read block 896 len 32768
179/300 Write block 4005 len 28672 data: 2 2 2 2 2 2 2
180/300 Read block 1586 len 4096
181/300 Write block 4842 len 4096 data: 2
182/300 Read block 4078 len 16384
183/300 Write block 428 len 24576 data: 2 2 2 2 2 2
184/300 Write block 3194 len 24576 data: 2 2 2 2 2 2
185/300 Write block 2146 len 28672 data: 2 2 2 2 2 2 2
186/300 Read block 2769 len 36864
187/300 Write block 625 len 24576 data: 2 2 2 2 2 2
188/300 Flush
189/300 Read block 1185 len 12288
190/300 Write block 4951 len 16384 data: 2 2 2 2
191/300 Read block 1825 len 32768
192/300 Write block 1568 len 40960 data: 2 2 2 2 2 2 2 2 2 2
193/300 Write block 4831 len 8192 data: 2 2
194/300 Read block 2709 len 8192
195/300 Write block 3364 len 12288 data: 2 2 2
196/300 Read block 3531 len 40960
197/300 Write block 1459 len 24576 data: 2 2 2 2 2 2
198/300 Write block 736 len 4096 data: 2
199/300 Flush
200/300 Read block 3456 len 24576
201/300 Write block 4828 len 16384 data: 2 2 2 3
202/300 Write block 3053 len 16384 data: 2 2 2 2
203/300 Read block 3927 len 4096
204/300 Read block 438 len 36864
205/300 Read block 3626 len 4096
206/300 Read block 3036 len 16384
207/300 Read block 1246 len 12288
208/300 Write block 887 len 8192 data: 2 2
209/300 Read block 1801 len 16384
210/300 Write block 1409 len 24576 data: 2 2 2 2 2 2
211/300 Write block 234 len 20480 data: 2 2 2 2 2
212/300 Read block 1261 len 4096
213/300 Read block 330 len 4096
214/300 Write block 3090 len 20480 data: 2 2 2 2 2
215/300 Write block 3687 len 8192 data: 2 3
216/300 Flush
217/300 Read block 1497 len 36864
218/300 Flush
219/300 Write block 3114 len 32768 data: 2 2 2 2 2 2 2 2
220/300 Read block 2951 len 28672
221/300 Write block 214 len 32768 data: 2 2 2 2 2 2 2 3
222/300 Read block 1877 len 20480
223/300 Flush
224/300 Read block 4978 len 32768
225/300 Write block 1527 len 40960 data: 3 3 3 3 2 2 2 2 2 2
226/300 Read block 459 len 24576
227/300 Write block 2259 len 8192 data: 2 2
228/300 Write block 1359 len 12288 data: 2 2 2
229/300 Read block 4895 len 16384
230/300 Write block 1037 len 40960 data: 2 2 2 2 2 2 2 2 2 2
231/300 Read block 1145 len 4096
232/300 Write block 4190 len 16384 data: 3 3 3 3
233/300 Read block 590 len 12288
234/300 Read block 1415 len 24576
235/300 Read block 2871 len 4096
236/300 Read block 4530 len 36864
237/300 Read block 718 len 36864
238/300 Flush
239/300 Read block 617 len 28672
240/300 Read block 4387 len 36864
241/300 Write block 2168 len 4096 data: 2
242/300 Read block 966 len 24576
243/300 Read block 1486 len 12288
244/300 Read block 1295 len 28672
245/300 Write block 2863 len 4096 data: 2
246/300 Flush
247/300 Write block 1148 len 40960 data: 2 2 2 2 2 2 2 2 2 2
248/300 Read block 1199 len 40960
249/300 Flush
250/300 Read block 909 len 28672
251/300 Flush
252/300 Write block 144 len 24576 data: 3 3 3 3 3 3
253/300 Write block 2420 len 32768 data: 2 2 2 2 2 2 2 2
254/300 Flush
255/300 Read block 3472 len 20480
256/300 Write block 521 len 16384 data: 2 2 2 2
257/300 Write block 2544 len 32768 data: 2 2 2 2 2 2 2 2
258/300 Write block 1489 len 16384 data: 2 2 2 2
259/300 Read block 1064 len 4096
260/300 Read block 1082 len 12288
261/300 Read block 1817 len 16384
262/300 Write block 419 len 16384 data: 2 2 2 2
263/300 Read block 1733 len 4096
264/300 Write block 3110 len 40960 data: 2 2 2 2 3 3 3 3 3 3
265/300 Read block 4947 len 28672
266/300 Read block 1569 len 12288
267/300 Read block 296 len 32768
268/300 Read block 3837 len 36864
269/300 Write block 3795 len 28672 data: 2 2 2 2 2 2 2
270/300 Read block 790 len 24576
271/300 Write block 1078 len 24576 data: 2 2 2 2 2 2
272/300 Flush
273/300 Write block 812 len 12288 data: 2 2 2
274/300 Read block 2483 len 4096
275/300 Read block 2144 len 40960
276/300 Write block 2195 len 12288 data: 2 2 2
277/300 Read block 1859 len 36864
278/300 Read block 4964 len 36864
279/300 Read block 1441 len 16384
280/300 Read block 3921 len 4096
281/300 Read block 334 len 8192
282/300 Read block 4391 len 40960
283/300 Write block 2021 len 4096 data: 2
284/300 Read block 1420 len 28672
285/300 Read block 4465 len 12288
286/300 Flush
287/300 Flush
288/300 Read block 4698 len 16384
289/300 Read block 3340 len 24576
290/300 Read block 4972 len 4096
291/300 Read block 2234 len 8192
292/300 Read block 3759 len 40960
293/300 Read block 2757 len 20480
294/300 Read block 305 len 32768
295/300 Read block 3827 len 20480
296/300 Read block 1541 len 36864
297/300 Read block 4476 len 36864
298/300 Read block 1822 len 4096
299/300 Read block 4638 len 24576
300/300 Read block 4917 len 8192
301/300 Flush
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:01:08.121 INFO request completed, latency_us: 248, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5c66957c-1a76-4416-b394-effd9308bc25, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 3 Done Sent Done false
1001 Acked Write 1 Done Sent Done false
1002 Acked Write 9 Done Sent Done false
1003 Acked Write 10 Done Sent Done false
1004 Acked Flush 0 Done Sent Done false
1005 Acked Read 10 Done Sent Done false
1006 Acked Read 2 Done Sent Done false
1007 Acked Write 10 Done Sent Done false
1008 Acked Flush 0 Done Sent Done false
1009 Acked Read 8 Done Sent Done false
1010 Acked Flush 0 Done Sent Done false
1011 Acked Read 9 Done Sent Done false
1012 Acked Read 1 Done Sent Done false
1013 Acked Read 2 Done Sent Done false
1014 Acked Read 8 Done Sent Done false
1015 Acked Read 9 Done Sent Done false
1016 Acked Read 3 Done Sent Done false
1017 Acked Read 9 Done Sent Done false
1018 Acked Write 5 Done Sent Done false
1019 Acked Read 1 Done Sent Done false
1020 Acked Write 6 Done Sent Done false
1021 Acked Write 6 Done Sent Done false
1022 Acked Read 2 Done Sent Done false
1023 Acked Write 8 Done Sent Done false
1024 Acked Read 6 Done Sent Done false
1025 Acked Read 9 Done Sent Done false
1026 Acked Write 4 Done Sent Done false
1027 Acked Write 2 Done Sent Done false
1028 Acked Read 3 Done Sent Done false
1029 Acked Write 2 Done Sent Done false
1030 Acked Write 8 Done Sent Done false
1031 Acked Read 3 Done Sent Done false
1032 Acked Flush 0 Done Sent Done false
1033 Acked Write 4 Done Sent Done false
1034 Acked Write 9 Done Sent Done false
1035 Acked Write 10 Done Sent Done false
1036 Acked Read 9 Done Sent Done false
1037 Acked Read 4 Done Sent Done false
1038 Acked Write 8 Done Sent Done false
1039 Acked Write 3 Done Sent Done false
1040 Acked Read 4 Done Sent Done false
1041 Acked Read 8 Done Sent Done false
1042 Acked Write 10 Done Sent Done false
1043 Acked Read 2 Done Sent Done false
1044 Acked Read 2 Done Sent Done false
1045 Acked Read 6 Done Sent Done false
1046 Acked Write 1 Done Sent Done false
1047 Acked Write 9 Done Sent Done false
1048 Acked Write 10 Done Sent Done false
1049 Acked Flush 0 Done Sent Done false
1050 Acked Flush 0 Done Sent Done false
1051 Acked Read 5 Done Sent Done false
1052 Acked Read 9 Done Sent Done false
1053 Acked Write 2 Done Sent Done false
1054 Acked Write 6 Done Sent Done false
1055 Acked Read 8 Done Sent Done false
1056 Acked Write 6 Done Sent Done false
1057 Acked Write 7 Done Sent Done false
1058 Acked Flush 0 Done Sent Done false
1059 Acked Write 4 Done Sent Done false
1060 Acked Write 9 Done Sent Done false
1061 Acked Write 6 Done Sent Done false
1062 Acked Read 10 Done Sent Done false
1063 Acked Read 1 Done Sent Done false
1064 Acked Write 10 Done Sent Done false
1065 Acked Read 8 Done Sent Done false
1066 Acked Write 2 Done Sent Done false
1067 Acked Flush 0 Done Sent Done false
1068 Acked Flush 0 Done Sent Done false
1069 Acked Write 4 Done Sent Done false
1070 Acked Read 1 Done Sent Done false
1071 Acked Write 7 Done Sent Done false
1072 Acked Read 8 Done Sent Done false
1073 Acked Write 5 Done Sent Done false
1074 Acked Read 8 Done Sent Done false
1075 Acked Write 2 Done Sent Done false
1076 Acked Read 3 Done Sent Done false
1077 Acked Write 8 Done Sent Done false
1078 Acked Read 2 Done Sent Done false
1079 Acked Read 4 Done Sent Done false
1080 Acked Write 5 Done Sent Done false
1081 Acked Write 10 Done Sent Done false
1082 Acked Write 10 Done Sent Done false
1083 Acked Read 4 Done Sent Done false
1084 Acked Write 3 Done Sent Done false
1085 Acked Read 10 Done Sent Done false
1086 Acked Read 8 Done Sent Done false
1087 Acked Write 2 Done Sent Done false
1088 Acked Write 8 Done Sent Done false
1089 Acked Write 6 Done Sent Done false
1090 Acked Flush 0 Done Sent Done false
1091 Acked Write 10 Done Sent Done false
1092 Acked Read 5 Done Sent Done false
1093 Acked Flush 0 Done Sent Done false
1094 Acked Write 8 Done Sent Done false
1095 Acked Read 4 Done Sent Done false
1096 Acked Read 2 Done Sent Done false
1097 Acked Read 8 Done Sent Done false
1098 Acked Read 2 Done Sent Done false
1099 Acked Write 7 Done Sent Done false
1100 Acked Read 5 Done Sent Done false
1101 Acked Read 8 Done Sent Done false
1102 Acked Read 6 Done Sent Done false
1103 Acked Flush 0 Done Sent Done false
1104 Acked Write 9 Done Sent Done false
1105 Acked Read 6 Done Sent Done false
1106 Acked Write 7 Done Sent Done false
1107 Acked Read 1 Done Sent Done false
1108 Acked Write 5 Done Sent Done false
1109 Acked Read 3 Done Sent Done false
1110 Acked Read 7 Done Sent Done false
1111 Acked Write 7 Done Sent Done false
1112 Acked Read 6 Done Sent Done false
1113 Acked Flush 0 Done Sent Done false
1114 Acked Read 4 Done Sent Done false
1115 Acked Write 2 Done Sent Done false
1116 Acked Write 7 Done Sent Done false
1117 Acked Flush 0 Done Sent Done false
1118 Acked Write 2 Done Sent Done false
1119 Acked Read 1 Done Sent Done false
1120 Acked Flush 0 Done Sent Done false
1121 Acked Read 1 Done Sent Done false
1122 Acked Write 2 Done Sent Done false
1123 Acked Read 7 Done Sent Done false
1124 Acked Read 10 Done Sent Done false
1125 Acked Flush 0 Done Sent Done false
1126 Acked Read 5 Done Sent Done false
1127 Acked Read 2 Done Sent Done false
1128 Acked Write 6 Done Sent Done false
1129 Acked Read 9 Done Sent Done false
1130 Acked Flush 0 Done Sent Done false
1131 Acked Read 2 Done Sent Done false
1132 Acked Read 2 Done Sent Done false
1133 Acked Write 5 Done Sent Done false
1134 Acked Read 7 Done Sent Done false
1135 Acked Read 7 Done Sent Done false
1136 Acked Read 1 Done Sent Done false
1137 Acked Read 4 Done Sent Done false
1138 Acked Read 1 Done Sent Done false
1139 Acked Read 9 Done Sent Done false
1140 Acked Flush 0 Done Sent Done false
1141 Acked Read 2 Done Sent Done false
1142 Acked Read 5 Done Sent Done false
1143 Acked Read 1 Done Sent Done false
1144 Acked Flush 0 Done Sent Done false
1145 Acked Write 10 Done Sent Done false
1146 Acked Read 2 Done Sent Done false
1147 Acked Write 9 Done Sent Done false
1148 Acked Read 9 Done Sent Done false
1149 Acked Write 1 Done Sent Done false
1150 Acked Flush 0 Done Sent Done false
1151 Acked Write 1 Done Sent Done false
1152 Acked Write 1 Done Sent Done false
1153 Acked Read 9 Done Sent Done false
1154 Acked Write 7 Done Sent Done false
1155 Acked Read 1 Done Sent Done false
1156 Acked Write 5 Done Sent Done false
1157 Acked Flush 0 Done Sent Done false
1158 Acked Write 1 Done Sent Done false
1159 Acked Write 4 Done Sent Done false
1160 Acked Read 8 Done Sent Done false
1161 Acked Write 1 Done Sent Done false
1162 Acked Read 7 Done Sent Done false
1163 Acked Read 2 Done Sent Done false
1164 Acked Read 3 Done Sent Done false
1165 Acked Read 2 Done Sent Done false
1166 Acked Write 8 Done Sent Done false
1167 Acked Read 9 Done Sent Done false
1168 Acked Read 2 Done Sent Done false
1169 Acked Write 4 Done Sent Done false
1170 Acked Write 4 Done Sent Done false
1171 Acked Read 10 Done Sent Done false
1172 Acked Read 8 Done Sent Done false
1173 Acked Read 4 Done Sent Done false
1174 Acked Write 5 Done Sent Done false
1175 Acked Write 6 Done Sent Done false
1176 Acked Read 1 Done Sent Done false
1177 Acked Read 8 Done Sent Done false
1178 Acked Write 7 Done Sent Done false
1179 Acked Read 1 Done Sent Done false
1180 Acked Write 1 Done Sent Done false
1181 Acked Read 4 Done Sent Done false
1182 Acked Write 6 Done Sent Done false
1183 Acked Write 6 Done Sent Done false
1184 Acked Write 7 Done Sent Done false
1185 Acked Read 9 Done Sent Done false
1186 Acked Write 6 Done Sent Done false
1187 Acked Flush 0 Done Sent Done false
1188 Acked Read 3 Done Sent Done false
1189 Acked Write 4 Done Sent Done false
1190 Acked Read 8 Done Sent Done false
1191 Acked Write 10 Done Sent Done false
1192 Acked Write 2 Done Sent Done false
1193 Acked Read 2 Done Sent Done false
1194 Acked Write 3 Done Sent Done false
1195 Acked Read 10 Done Sent Done false
1196 Acked Write 6 Done Sent Done false
1197 Acked Write 1 Done Sent Done false
1198 Acked Flush 0 Done Sent Done false
1199 Acked Read 6 Done Sent Done false
1200 Acked Write 4 Done Sent Done false
1201 Acked Write 4 Done Sent Done false
1202 Acked Read 1 Done Sent Done false
1203 Acked Read 9 Done Sent Done false
1204 Acked Read 1 Done Sent Done false
1205 Acked Read 4 Done Sent Done false
1206 Acked Read 3 Done Sent Done false
1207 Acked Write 2 Done Sent Done false
1208 Acked Read 4 Done Sent Done false
1209 Acked Write 6 Done Sent Done false
1210 Acked Write 5 Done Sent Done false
1211 Acked Read 1 Done Sent Done false
1212 Acked Read 1 Done Sent Done false
1213 Acked Write 5 Done Sent Done false
1214 Acked Write 2 Done Sent Done false
1215 Acked Flush 0 Done Sent Done false
1216 Acked Read 9 Done Sent Done false
1217 Acked Flush 0 Done Sent Done false
1218 Acked Write 8 Done Sent Done false
1219 Acked Read 7 Done Sent Done false
1220 Acked Write 8 Done Sent Done false
1221 Acked Read 5 Done Sent Done false
1222 Acked Flush 0 Done Sent Done false
1223 Acked Read 8 Done Sent Done false
1224 Acked Write 10 Done Sent Done false
1225 Acked Read 6 Done Sent Done false
1226 Acked Write 2 Done Sent Done false
1227 Acked Write 3 Done Sent Done false
1228 Acked Read 4 Done Sent Done false
1229 Acked Write 10 Done Sent Done false
1230 Acked Read 1 Done Sent Done false
1231 Acked Write 4 Done Sent Done false
1232 Acked Read 3 Done Sent Done false
1233 Acked Read 6 Done Sent Done false
1234 Acked Read 1 Done Sent Done false
1235 Acked Read 9 Done Sent Done false
1236 Acked Read 9 Done Sent Done false
1237 Acked Flush 0 Done Sent Done false
1238 Acked Read 7 Done Sent Done false
1239 Acked Read 9 Done Sent Done false
1240 Acked Write 1 Done Sent Done false
1241 Acked Read 6 Done Sent Done false
1242 Acked Read 3 Done Sent Done false
1243 Acked Read 7 Done Sent Done false
1244 Acked Write 1 Done Sent Done false
1245 Acked Flush 0 Done Sent Done false
1246 Acked Write 10 Done Sent Done false
1247 Acked Read 10 Done Sent Done false
1248 Acked Flush 0 Done Sent Done false
1249 Acked Read 7 Done Sent Done false
1250 Acked Flush 0 Done Sent Done false
1251 Acked Write 6 Done Sent Done false
1252 Acked Write 8 Done Sent Done false
1253 Acked Flush 0 Done Sent Done false
1254 Acked Read 5 Done Sent Done false
1255 Acked Write 4 Done Sent Done false
1256 Acked Write 8 Done Sent Done false
1257 Acked Write 4 Done Sent Done false
1258 Acked Read 1 Done Sent Done false
1259 Acked Read 3 Done Sent Done false
1260 Acked Read 4 Done Sent Done false
1261 Acked Write 4 Done Sent Done false
1262 Acked Read 1 Done Sent Done false
1263 Acked Write 10 Done Sent Done false
1264 Acked Read 7 Done Sent Done false
1265 Acked Read 3 Done Sent Done false
1266 Acked Read 8 Done Sent Done false
1267 Acked Read 9 Done Sent Done false
1268 Acked Write 7 Done Sent Done false
1269 Acked Read 6 Done Sent Done false
1270 Acked Write 6 Done Sent Done false
1271 Acked Flush 0 Done Sent Done false
1272 Acked Write 3 Done Sent Done false
1273 Acked Read 1 Done Sent Done false
1274 Acked Read 10 Done Sent Done false
1275 Acked Write 3 Done Sent Done false
1276 Acked Read 9 Done Sent Done false
1277 Acked Read 9 Done Sent Done false
1278 Acked Read 4 Done Sent Done false
1279 Acked Read 1 Done Sent Done false
1280 Acked Read 2 Done Sent Done false
1281 Acked Read 10 Done Sent Done false
1282 Acked Write 1 Done Sent Done false
1283 Acked Read 7 Done Sent Done false
1284 Acked Read 3 Done Sent Done false
1285 Acked Flush 0 Done Sent Done false
1286 Acked Flush 0 Done Sent Done false
1287 Acked Read 4 Done Sent Done false
1288 Acked Read 6 Done Sent Done false
1289 Acked Read 1 Done Sent Done false
1290 Acked Read 2 Done Sent Done false
1291 Acked Read 10 Done Sent Done false
1292 Acked Read 5 Done Sent Done false
1293 Acked Read 8 Done Sent Done false
1294 Acked Read 5 Done Sent Done false
1295 Acked Read 9 Done Sent Done false
1296 Acked Read 9 Done Sent Done false
1297 Acked Read 1 Done Sent Done false
1298 Acked Read 6 Done Sent Done false
1299 Acked Read 2 Done Sent Done false
1300 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1300)) None Some(JobId(1300))
Downstairs last five completed:
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:08.623455298Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:09.350087046Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:10.352289471Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:11.354506423Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 3 Done Sent Done false
1001 Acked Write 1 Done Sent Done false
1002 Acked Write 9 Done Sent Done false
1003 Acked Write 10 Done Sent Done false
1004 Acked Flush 0 Done Sent Done false
1005 Acked Read 10 Done Sent Done false
1006 Acked Read 2 Done Sent Done false
1007 Acked Write 10 Done Sent Done false
1008 Acked Flush 0 Done Sent Done false
1009 Acked Read 8 Done Sent Done false
1010 Acked Flush 0 Done Sent Done false
1011 Acked Read 9 Done Sent Done false
1012 Acked Read 1 Done Sent Done false
1013 Acked Read 2 Done Sent Done false
1014 Acked Read 8 Done Sent Done false
1015 Acked Read 9 Done Sent Done false
1016 Acked Read 3 Done Sent Done false
1017 Acked Read 9 Done Sent Done false
1018 Acked Write 5 Done Sent Done false
1019 Acked Read 1 Done Sent Done false
1020 Acked Write 6 Done Sent Done false
1021 Acked Write 6 Done Sent Done false
1022 Acked Read 2 Done Sent Done false
1023 Acked Write 8 Done Sent Done false
1024 Acked Read 6 Done Sent Done false
1025 Acked Read 9 Done Sent Done false
1026 Acked Write 4 Done Sent Done false
1027 Acked Write 2 Done Sent Done false
1028 Acked Read 3 Done Sent Done false
1029 Acked Write 2 Done Sent Done false
1030 Acked Write 8 Done Sent Done false
1031 Acked Read 3 Done Sent Done false
1032 Acked Flush 0 Done Sent Done false
1033 Acked Write 4 Done Sent Done false
1034 Acked Write 9 Done Sent Done false
1035 Acked Write 10 Done Sent Done false
1036 Acked Read 9 Done Sent Done false
1037 Acked Read 4 Done Sent Done false
1038 Acked Write 8 Done Sent Done false
1039 Acked Write 3 Done Sent Done false
1040 Acked Read 4 Done Sent Done false
1041 Acked Read 8 Done Sent Done false
1042 Acked Write 10 Done Sent Done false
1043 Acked Read 2 Done Sent Done false
1044 Acked Read 2 Done Sent Done false
1045 Acked Read 6 Done Sent Done false
1046 Acked Write 1 Done Sent Done false
1047 Acked Write 9 Done Sent Done false
1048 Acked Write 10 Done Sent Done false
1049 Acked Flush 0 Done Sent Done false
1050 Acked Flush 0 Done Sent Done false
1051 Acked Read 5 Done Sent Done false
1052 Acked Read 9 Done Sent Done false
1053 Acked Write 2 Done Sent Done false
1054 Acked Write 6 Done Sent Done false
1055 Acked Read 8 Done Sent Done false
1056 Acked Write 6 Done Sent Done false
1057 Acked Write 7 Done Sent Done false
1058 Acked Flush 0 Done Sent Done false
1059 Acked Write 4 Done Sent Done false
1060 Acked Write 9 Done Sent Done false
1061 Acked Write 6 Done Sent Done false
1062 Acked Read 10 Done Sent Done false
1063 Acked Read 1 Done Sent Done false
1064 Acked Write 10 Done Sent Done false
1065 Acked Read 8 Done Sent Done false
1066 Acked Write 2 Done Sent Done false
1067 Acked Flush 0 Done Sent Done false
1068 Acked Flush 0 Done Sent Done false
1069 Acked Write 4 Done Sent Done false
1070 Acked Read 1 Done Sent Done false
1071 Acked Write 7 Done Sent Done false
1072 Acked Read 8 Done Sent Done false
1073 Acked Write 5 Done Sent Done false
1074 Acked Read 8 Done Sent Done false
1075 Acked Write 2 Done Sent Done false
1076 Acked Read 3 Done Sent Done false
1077 Acked Write 8 Done Sent Done false
1078 Acked Read 2 Done Sent Done false
1079 Acked Read 4 Done Sent Done false
1080 Acked Write 5 Done Sent Done false
1081 Acked Write 10 Done Sent Done false
1082 Acked Write 10 Done Sent Done false
1083 Acked Read 4 Done Sent Done false
1084 Acked Write 3 Done Sent Done false
1085 Acked Read 10 Done Sent Done false
1086 Acked Read 8 Done Sent Done false
1087 Acked Write 2 Done Sent Done false
1088 Acked Write 8 Done Sent Done false
1089 Acked Write 6 Done Sent Done false
1090 Acked Flush 0 Done Sent Done false
1091 Acked Write 10 Done Sent Done false
1092 Acked Read 5 Done Sent Done false
1093 Acked Flush 0 Done Sent Done false
1094 Acked Write 8 Done Sent Done false
1095 Acked Read 4 Done Sent Done false
1096 Acked Read 2 Done Sent Done false
1097 Acked Read 8 Done Sent Done false
1098 Acked Read 2 Done Sent Done false
1099 Acked Write 7 Done Sent Done false
1100 Acked Read 5 Done Sent Done false
1101 Acked Read 8 Done Sent Done false
1102 Acked Read 6 Done Sent Done false
1103 Acked Flush 0 Done Sent Done false
1104 Acked Write 9 Done Sent Done false
1105 Acked Read 6 Done Sent Done false
1106 Acked Write 7 Done Sent Done false
1107 Acked Read 1 Done Sent Done false
1108 Acked Write 5 Done Sent Done false
1109 Acked Read 3 Done Sent Done false
1110 Acked Read 7 Done Sent Done false
1111 Acked Write 7 Done Sent Done false
1112 Acked Read 6 Done Sent Done false
1113 Acked Flush 0 Done Sent Done false
1114 Acked Read 4 Done Sent Done false
1115 Acked Write 2 Done Sent Done false
1116 Acked Write 7 Done Sent Done false
1117 Acked Flush 0 Done Sent Done false
1118 Acked Write 2 Done Sent Done false
1119 Acked Read 1 Done Sent Done false
1120 Acked Flush 0 Done Sent Done false
1121 Acked Read 1 Done Sent Done false
1122 Acked Write 2 Done Sent Done false
1123 Acked Read 7 Done Sent Done false
1124 Acked Read 10 Done Sent Done false
1125 Acked Flush 0 Done Sent Done false
1126 Acked Read 5 Done Sent Done false
1127 Acked Read 2 Done Sent Done false
1128 Acked Write 6 Done Sent Done false
1129 Acked Read 9 Done Sent Done false
1130 Acked Flush 0 Done Sent Done false
1131 Acked Read 2 Done Sent Done false
1132 Acked Read 2 Done Sent Done false
1133 Acked Write 5 Done Sent Done false
1134 Acked Read 7 Done Sent Done false
1135 Acked Read 7 Done Sent Done false
1136 Acked Read 1 Done Sent Done false
1137 Acked Read 4 Done Sent Done false
1138 Acked Read 1 Done Sent Done false
1139 Acked Read 9 Done Sent Done false
1140 Acked Flush 0 Done Sent Done false
1141 Acked Read 2 Done Sent Done false
1142 Acked Read 5 Done Sent Done false
1143 Acked Read 1 Done Sent Done false
1144 Acked Flush 0 Done Sent Done false
1145 Acked Write 10 Done Sent Done false
1146 Acked Read 2 Done Sent Done false
1147 Acked Write 9 Done Sent Done false
1148 Acked Read 9 Done Sent Done false
1149 Acked Write 1 Done Sent Done false
1150 Acked Flush 0 Done Sent Done false
1151 Acked Write 1 Done Sent Done false
1152 Acked Write 1 Done Sent Done false
1153 Acked Read 9 Done Sent Done false
1154 Acked Write 7 Done Sent Done false
1155 Acked Read 1 Done Sent Done false
1156 Acked Write 5 Done Sent Done false
1157 Acked Flush 0 Done Sent Done false
1158 Acked Write 1 Done Sent Done false
1159 Acked Write 4 Done Sent Done false
1160 Acked Read 8 Done Sent Done false
1161 Acked Write 1 Done Sent Done false
1162 Acked Read 7 Done Sent Done false
1163 Acked Read 2 Done Sent Done false
1164 Acked Read 3 Done Sent Done false
1165 Acked Read 2 Done Sent Done false
1166 Acked Write 8 Done Sent Done false
1167 Acked Read 9 Done Sent Done false
1168 Acked Read 2 Done Sent Done false
1169 Acked Write 4 Done Sent Done false
1170 Acked Write 4 Done Sent Done false
1171 Acked Read 10 Done Sent Done false
1172 Acked Read 8 Done Sent Done false
1173 Acked Read 4 Done Sent Done false
1174 Acked Write 5 Done Sent Done false
1175 Acked Write 6 Done Sent Done false
1176 Acked Read 1 Done Sent Done false
1177 Acked Read 8 Done Sent Done false
1178 Acked Write 7 Done Sent Done false
1179 Acked Read 1 Done Sent Done false
1180 Acked Write 1 Done Sent Done false
1181 Acked Read 4 Done Sent Done false
1182 Acked Write 6 Done Sent Done false
1183 Acked Write 6 Done Sent Done false
1184 Acked Write 7 Done Sent Done false
1185 Acked Read 9 Done Sent Done false
1186 Acked Write 6 Done Sent Done false
1187 Acked Flush 0 Done Sent Done false
1188 Acked Read 3 Done Sent Done false
1189 Acked Write 4 Done Sent Done false
1190 Acked Read 8 Done Sent Done false
1191 Acked Write 10 Done Sent Done false
1192 Acked Write 2 Done Sent Done false
1193 Acked Read 2 Done Sent Done false
1194 Acked Write 3 Done Sent Done false
1195 Acked Read 10 Done Sent Done false
1196 Acked Write 6 Done Sent Done false
1197 Acked Write 1 Done Sent Done false
1198 Acked Flush 0 Done Sent Done false
1199 Acked Read 6 Done Sent Done false
1200 Acked Write 4 Done Sent Done false
1201 Acked Write 4 Done Sent Done false
1202 Acked Read 1 Done Sent Done false
1203 Acked Read 9 Done Sent Done false
1204 Acked Read 1 Done Sent Done false
1205 Acked Read 4 Done Sent Done false
1206 Acked Read 3 Done Sent Done false
1207 Acked Write 2 Done Sent Done false
1208 Acked Read 4 Done Sent Done false
1209 Acked Write 6 Done Sent Done false
1210 Acked Write 5 Done Sent Done false
1211 Acked Read 1 Done Sent Done false
1212 Acked Read 1 Done Sent Done false
1213 Acked Write 5 Done Sent Done false
1214 Acked Write 2 Done Sent Done false
1215 Acked Flush 0 Done Sent Done false
1216 Acked Read 9 Done Sent Done false
1217 Acked Flush 0 Done Sent Done false
1218 Acked Write 8 Done Sent Done false
1219 Acked Read 7 Done Sent Done false
1220 Acked Write 8 Done Sent Done false
1221 Acked Read 5 Done Sent Done false
1222 Acked Flush 0 Done Sent Done false
1223 Acked Read 8 Done Sent Done false
1224 Acked Write 10 Done Sent Done false
1225 Acked Read 6 Done Sent Done false
1226 Acked Write 2 Done Sent Done false
1227 Acked Write 3 Done Sent Done false
1228 Acked Read 4 Done Sent Done false
1229 Acked Write 10 Done Sent Done false
1230 Acked Read 1 Done Sent Done false
1231 Acked Write 4 Done Sent Done false
1232 Acked Read 3 Done Sent Done false
1233 Acked Read 6 Done Sent Done false
1234 Acked Read 1 Done Sent Done false
1235 Acked Read 9 Done Sent Done false
1236 Acked Read 9 Done Sent Done false
1237 Acked Flush 0 Done Sent Done false
1238 Acked Read 7 Done Sent Done false
1239 Acked Read 9 Done Sent Done false
1240 Acked Write 1 Done Sent Done false
1241 Acked Read 6 Done Sent Done false
1242 Acked Read 3 Done Sent Done false
1243 Acked Read 7 Done Sent Done false
1244 Acked Write 1 Done Sent Done false
1245 Acked Flush 0 Done Sent Done false
1246 Acked Write 10 Done Sent Done false
1247 Acked Read 10 Done Sent Done false
1248 Acked Flush 0 Done Sent Done false
1249 Acked Read 7 Done Sent Done false
1250 Acked Flush 0 Done Sent Done false
1251 Acked Write 6 Done Sent Done false
1252 Acked Write 8 Done Sent Done false
1253 Acked Flush 0 Done Sent Done false
1254 Acked Read 5 Done Sent Done false
1255 Acked Write 4 Done Sent Done false
1256 Acked Write 8 Done Sent Done false
1257 Acked Write 4 Done Sent Done false
1258 Acked Read 1 Done Sent Done false
1259 Acked Read 3 Done Sent Done false
1260 Acked Read 4 Done Sent Done false
1261 Acked Write 4 Done Sent Done false
1262 Acked Read 1 Done Sent Done false
1263 Acked Write 10 Done Sent Done false
1264 Acked Read 7 Done Sent Done false
1265 Acked Read 3 Done Sent Done false
1266 Acked Read 8 Done Sent Done false
1267 Acked Read 9 Done Sent Done false
1268 Acked Write 7 Done Sent Done false
1269 Acked Read 6 Done Sent Done false
1270 Acked Write 6 Done Sent Done false
1271 Acked Flush 0 Done Sent Done false
1272 Acked Write 3 Done Sent Done false
1273 Acked Read 1 Done Sent Done false
1274 Acked Read 10 Done Sent Done false
1275 Acked Write 3 Done Sent Done false
1276 Acked Read 9 Done Sent Done false
1277 Acked Read 9 Done Sent Done false
1278 Acked Read 4 Done Sent Done false
1279 Acked Read 1 Done Sent Done false
1280 Acked Read 2 Done Sent Done false
1281 Acked Read 10 Done Sent Done false
1282 Acked Write 1 Done Sent Done false
1283 Acked Read 7 Done Sent Done false
1284 Acked Read 3 Done Sent Done false
1285 Acked Flush 0 Done Sent Done false
1286 Acked Flush 0 Done Sent Done false
1287 Acked Read 4 Done Sent Done false
1288 Acked Read 6 Done Sent Done false
1289 Acked Read 1 Done Sent Done false
1290 Acked Read 2 Done Sent Done false
1291 Acked Read 10 Done Sent Done false
1292 Acked Read 5 Done Sent Done false
1293 Acked Read 8 Done Sent Done false
1294 Acked Read 5 Done Sent Done false
1295 Acked Read 9 Done Sent Done false
1296 Acked Read 9 Done Sent Done false
1297 Acked Read 1 Done Sent Done false
1298 Acked Read 6 Done Sent Done false
1299 Acked Read 2 Done Sent Done false
1300 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1300)) None Some(JobId(1300))
Downstairs last five completed:
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:12.625530303Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:13.358536073Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:13.879152654Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:13.879387808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:13.879414911Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:13.879439565Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:13.879521526Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:14.380646968Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:15.361018145Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 3 Done Sent Done false
1001 Acked Write 1 Done Sent Done false
1002 Acked Write 9 Done Sent Done false
1003 Acked Write 10 Done Sent Done false
1004 Acked Flush 0 Done Sent Done false
1005 Acked Read 10 Done Sent Done false
1006 Acked Read 2 Done Sent Done false
1007 Acked Write 10 Done Sent Done false
1008 Acked Flush 0 Done Sent Done false
1009 Acked Read 8 Done Sent Done false
1010 Acked Flush 0 Done Sent Done false
1011 Acked Read 9 Done Sent Done false
1012 Acked Read 1 Done Sent Done false
1013 Acked Read 2 Done Sent Done false
1014 Acked Read 8 Done Sent Done false
1015 Acked Read 9 Done Sent Done false
1016 Acked Read 3 Done Sent Done false
1017 Acked Read 9 Done Sent Done false
1018 Acked Write 5 Done Sent Done false
1019 Acked Read 1 Done Sent Done false
1020 Acked Write 6 Done Sent Done false
1021 Acked Write 6 Done Sent Done false
1022 Acked Read 2 Done Sent Done false
1023 Acked Write 8 Done Sent Done false
1024 Acked Read 6 Done Sent Done false
1025 Acked Read 9 Done Sent Done false
1026 Acked Write 4 Done Sent Done false
1027 Acked Write 2 Done Sent Done false
1028 Acked Read 3 Done Sent Done false
1029 Acked Write 2 Done Sent Done false
1030 Acked Write 8 Done Sent Done false
1031 Acked Read 3 Done Sent Done false
1032 Acked Flush 0 Done Sent Done false
1033 Acked Write 4 Done Sent Done false
1034 Acked Write 9 Done Sent Done false
1035 Acked Write 10 Done Sent Done false
1036 Acked Read 9 Done Sent Done false
1037 Acked Read 4 Done Sent Done false
1038 Acked Write 8 Done Sent Done false
1039 Acked Write 3 Done Sent Done false
1040 Acked Read 4 Done Sent Done false
1041 Acked Read 8 Done Sent Done false
1042 Acked Write 10 Done Sent Done false
1043 Acked Read 2 Done Sent Done false
1044 Acked Read 2 Done Sent Done false
1045 Acked Read 6 Done Sent Done false
1046 Acked Write 1 Done Sent Done false
1047 Acked Write 9 Done Sent Done false
1048 Acked Write 10 Done Sent Done false
1049 Acked Flush 0 Done Sent Done false
1050 Acked Flush 0 Done Sent Done false
1051 Acked Read 5 Done Sent Done false
1052 Acked Read 9 Done Sent Done false
1053 Acked Write 2 Done Sent Done false
1054 Acked Write 6 Done Sent Done false
1055 Acked Read 8 Done Sent Done false
1056 Acked Write 6 Done Sent Done false
1057 Acked Write 7 Done Sent Done false
1058 Acked Flush 0 Done Sent Done false
1059 Acked Write 4 Done Sent Done false
1060 Acked Write 9 Done Sent Done false
1061 Acked Write 6 Done Sent Done false
1062 Acked Read 10 Done Sent Done false
1063 Acked Read 1 Done Sent Done false
1064 Acked Write 10 Done Sent Done false
1065 Acked Read 8 Done Sent Done false
1066 Acked Write 2 Done Sent Done false
1067 Acked Flush 0 Done Sent Done false
1068 Acked Flush 0 Done Sent Done false
1069 Acked Write 4 Done Sent Done false
1070 Acked Read 1 Done Sent Done false
1071 Acked Write 7 Done Sent Done false
1072 Acked Read 8 Done Sent Done false
1073 Acked Write 5 Done Sent Done false
1074 Acked Read 8 Done Sent Done false
1075 Acked Write 2 Done Sent Done false
1076 Acked Read 3 Done Sent Done false
1077 Acked Write 8 Done Sent Done false
1078 Acked Read 2 Done Sent Done false
1079 Acked Read 4 Done Sent Done false
1080 Acked Write 5 Done Sent Done false
1081 Acked Write 10 Done Sent Done false
1082 Acked Write 10 Done Sent Done false
1083 Acked Read 4 Done Sent Done false
1084 Acked Write 3 Done Sent Done false
1085 Acked Read 10 Done Sent Done false
1086 Acked Read 8 Done Sent Done false
1087 Acked Write 2 Done Sent Done false
1088 Acked Write 8 Done Sent Done false
1089 Acked Write 6 Done Sent Done false
1090 Acked Flush 0 Done Sent Done false
1091 Acked Write 10 Done Sent Done false
1092 Acked Read 5 Done Sent Done false
1093 Acked Flush 0 Done Sent Done false
1094 Acked Write 8 Done Sent Done false
1095 Acked Read 4 Done Sent Done false
1096 Acked Read 2 Done Sent Done false
1097 Acked Read 8 Done Sent Done false
1098 Acked Read 2 Done Sent Done false
1099 Acked Write 7 Done Sent Done false
1100 Acked Read 5 Done Sent Done false
1101 Acked Read 8 Done Sent Done false
1102 Acked Read 6 Done Sent Done false
1103 Acked Flush 0 Done Sent Done false
1104 Acked Write 9 Done Sent Done false
1105 Acked Read 6 Done Sent Done false
1106 Acked Write 7 Done Sent Done false
1107 Acked Read 1 Done Sent Done false
1108 Acked Write 5 Done Sent Done false
1109 Acked Read 3 Done Sent Done false
1110 Acked Read 7 Done Sent Done false
1111 Acked Write 7 Done Sent Done false
1112 Acked Read 6 Done Sent Done false
1113 Acked Flush 0 Done Sent Done false
1114 Acked Read 4 Done Sent Done false
1115 Acked Write 2 Done Sent Done false
1116 Acked Write 7 Done Sent Done false
1117 Acked Flush 0 Done Sent Done false
1118 Acked Write 2 Done Sent Done false
1119 Acked Read 1 Done Sent Done false
1120 Acked Flush 0 Done Sent Done false
1121 Acked Read 1 Done Sent Done false
1122 Acked Write 2 Done Sent Done false
1123 Acked Read 7 Done Sent Done false
1124 Acked Read 10 Done Sent Done false
1125 Acked Flush 0 Done Sent Done false
1126 Acked Read 5 Done Sent Done false
1127 Acked Read 2 Done Sent Done false
1128 Acked Write 6 Done Sent Done false
1129 Acked Read 9 Done Sent Done false
1130 Acked Flush 0 Done Sent Done false
1131 Acked Read 2 Done Sent Done false
1132 Acked Read 2 Done Sent Done false
1133 Acked Write 5 Done Sent Done false
1134 Acked Read 7 Done Sent Done false
1135 Acked Read 7 Done Sent Done false
1136 Acked Read 1 Done Sent Done false
1137 Acked Read 4 Done Sent Done false
1138 Acked Read 1 Done Sent Done false
1139 Acked Read 9 Done Sent Done false
1140 Acked Flush 0 Done Sent Done false
1141 Acked Read 2 Done Sent Done false
1142 Acked Read 5 Done Sent Done false
1143 Acked Read 1 Done Sent Done false
1144 Acked Flush 0 Done Sent Done false
1145 Acked Write 10 Done Sent Done false
1146 Acked Read 2 Done Sent Done false
1147 Acked Write 9 Done Sent Done false
1148 Acked Read 9 Done Sent Done false
1149 Acked Write 1 Done Sent Done false
1150 Acked Flush 0 Done Sent Done false
1151 Acked Write 1 Done Sent Done false
1152 Acked Write 1 Done Sent Done false
1153 Acked Read 9 Done Sent Done false
1154 Acked Write 7 Done Sent Done false
1155 Acked Read 1 Done Sent Done false
1156 Acked Write 5 Done Sent Done false
1157 Acked Flush 0 Done Sent Done false
1158 Acked Write 1 Done Sent Done false
1159 Acked Write 4 Done Sent Done false
1160 Acked Read 8 Done Sent Done false
1161 Acked Write 1 Done Sent Done false
1162 Acked Read 7 Done Sent Done false
1163 Acked Read 2 Done Sent Done false
1164 Acked Read 3 Done Sent Done false
1165 Acked Read 2 Done Sent Done false
1166 Acked Write 8 Done Sent Done false
1167 Acked Read 9 Done Sent Done false
1168 Acked Read 2 Done Sent Done false
1169 Acked Write 4 Done Sent Done false
1170 Acked Write 4 Done Sent Done false
1171 Acked Read 10 Done Sent Done false
1172 Acked Read 8 Done Sent Done false
1173 Acked Read 4 Done Sent Done false
1174 Acked Write 5 Done Sent Done false
1175 Acked Write 6 Done Sent Done false
1176 Acked Read 1 Done Sent Done false
1177 Acked Read 8 Done Sent Done false
1178 Acked Write 7 Done Sent Done false
1179 Acked Read 1 Done Sent Done false
1180 Acked Write 1 Done Sent Done false
1181 Acked Read 4 Done Sent Done false
1182 Acked Write 6 Done Sent Done false
1183 Acked Write 6 Done Sent Done false
1184 Acked Write 7 Done Sent Done false
1185 Acked Read 9 Done Sent Done false
1186 Acked Write 6 Done Sent Done false
1187 Acked Flush 0 Done Sent Done false
1188 Acked Read 3 Done Sent Done false
1189 Acked Write 4 Done Sent Done false
1190 Acked Read 8 Done Sent Done false
1191 Acked Write 10 Done Sent Done false
1192 Acked Write 2 Done Sent Done false
1193 Acked Read 2 Done Sent Done false
1194 Acked Write 3 Done Sent Done false
1195 Acked Read 10 Done Sent Done false
1196 Acked Write 6 Done Sent Done false
1197 Acked Write 1 Done Sent Done false
1198 Acked Flush 0 Done Sent Done false
1199 Acked Read 6 Done Sent Done false
1200 Acked Write 4 Done Sent Done false
1201 Acked Write 4 Done Sent Done false
1202 Acked Read 1 Done Sent Done false
1203 Acked Read 9 Done Sent Done false
1204 Acked Read 1 Done Sent Done false
1205 Acked Read 4 Done Sent Done false
1206 Acked Read 3 Done Sent Done false
1207 Acked Write 2 Done Sent Done false
1208 Acked Read 4 Done Sent Done false
1209 Acked Write 6 Done Sent Done false
1210 Acked Write 5 Done Sent Done false
1211 Acked Read 1 Done Sent Done false
1212 Acked Read 1 Done Sent Done false
1213 Acked Write 5 Done Sent Done false
1214 Acked Write 2 Done Sent Done false
1215 Acked Flush 0 Done Sent Done false
1216 Acked Read 9 Done Sent Done false
1217 Acked Flush 0 Done Sent Done false
1218 Acked Write 8 Done Sent Done false
1219 Acked Read 7 Done Sent Done false
1220 Acked Write 8 Done Sent Done false
1221 Acked Read 5 Done Sent Done false
1222 Acked Flush 0 Done Sent Done false
1223 Acked Read 8 Done Sent Done false
1224 Acked Write 10 Done Sent Done false
1225 Acked Read 6 Done Sent Done false
1226 Acked Write 2 Done Sent Done false
1227 Acked Write 3 Done Sent Done false
1228 Acked Read 4 Done Sent Done false
1229 Acked Write 10 Done Sent Done false
1230 Acked Read 1 Done Sent Done false
1231 Acked Write 4 Done Sent Done false
1232 Acked Read 3 Done Sent Done false
1233 Acked Read 6 Done Sent Done false
1234 Acked Read 1 Done Sent Done false
1235 Acked Read 9 Done Sent Done false
1236 Acked Read 9 Done Sent Done false
1237 Acked Flush 0 Done Sent Done false
1238 Acked Read 7 Done Sent Done false
1239 Acked Read 9 Done Sent Done false
1240 Acked Write 1 Done Sent Done false
1241 Acked Read 6 Done Sent Done false
1242 Acked Read 3 Done Sent Done false
1243 Acked Read 7 Done Sent Done false
1244 Acked Write 1 Done Sent Done false
1245 Acked Flush 0 Done Sent Done false
1246 Acked Write 10 Done Sent Done false
1247 Acked Read 10 Done Sent Done false
1248 Acked Flush 0 Done Sent Done false
1249 Acked Read 7 Done Sent Done false
1250 Acked Flush 0 Done Sent Done false
1251 Acked Write 6 Done Sent Done false
1252 Acked Write 8 Done Sent Done false
1253 Acked Flush 0 Done Sent Done false
1254 Acked Read 5 Done Sent Done false
1255 Acked Write 4 Done Sent Done false
1256 Acked Write 8 Done Sent Done false
1257 Acked Write 4 Done Sent Done false
1258 Acked Read 1 Done Sent Done false
1259 Acked Read 3 Done Sent Done false
1260 Acked Read 4 Done Sent Done false
1261 Acked Write 4 Done Sent Done false
1262 Acked Read 1 Done Sent Done false
1263 Acked Write 10 Done Sent Done false
1264 Acked Read 7 Done Sent Done false
1265 Acked Read 3 Done Sent Done false
1266 Acked Read 8 Done Sent Done false
1267 Acked Read 9 Done Sent Done false
1268 Acked Write 7 Done Sent Done false
1269 Acked Read 6 Done Sent Done false
1270 Acked Write 6 Done Sent Done false
1271 Acked Flush 0 Done Sent Done false
1272 Acked Write 3 Done Sent Done false
1273 Acked Read 1 Done Sent Done false
1274 Acked Read 10 Done Sent Done false
1275 Acked Write 3 Done Sent Done false
1276 Acked Read 9 Done Sent Done false
1277 Acked Read 9 Done Sent Done false
1278 Acked Read 4 Done Sent Done false
1279 Acked Read 1 Done Sent Done false
1280 Acked Read 2 Done Sent Done false
1281 Acked Read 10 Done Sent Done false
1282 Acked Write 1 Done Sent Done false
1283 Acked Read 7 Done Sent Done false
1284 Acked Read 3 Done Sent Done false
1285 Acked Flush 0 Done Sent Done false
1286 Acked Flush 0 Done Sent Done false
1287 Acked Read 4 Done Sent Done false
1288 Acked Read 6 Done Sent Done false
1289 Acked Read 1 Done Sent Done false
1290 Acked Read 2 Done Sent Done false
1291 Acked Read 10 Done Sent Done false
1292 Acked Read 5 Done Sent Done false
1293 Acked Read 8 Done Sent Done false
1294 Acked Read 5 Done Sent Done false
1295 Acked Read 9 Done Sent Done false
1296 Acked Read 9 Done Sent Done false
1297 Acked Read 1 Done Sent Done false
1298 Acked Read 6 Done Sent Done false
1299 Acked Read 2 Done Sent Done false
1300 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1300)) None Some(JobId(1300))
Downstairs last five completed:
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:16.627169894Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:17.364555515Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:18.365750192Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:19.36777459Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 3 Done Sent Done false
1001 Acked Write 1 Done Sent Done false
1002 Acked Write 9 Done Sent Done false
1003 Acked Write 10 Done Sent Done false
1004 Acked Flush 0 Done Sent Done false
1005 Acked Read 10 Done Sent Done false
1006 Acked Read 2 Done Sent Done false
1007 Acked Write 10 Done Sent Done false
1008 Acked Flush 0 Done Sent Done false
1009 Acked Read 8 Done Sent Done false
1010 Acked Flush 0 Done Sent Done false
1011 Acked Read 9 Done Sent Done false
1012 Acked Read 1 Done Sent Done false
1013 Acked Read 2 Done Sent Done false
1014 Acked Read 8 Done Sent Done false
1015 Acked Read 9 Done Sent Done false
1016 Acked Read 3 Done Sent Done false
1017 Acked Read 9 Done Sent Done false
1018 Acked Write 5 Done Sent Done false
1019 Acked Read 1 Done Sent Done false
1020 Acked Write 6 Done Sent Done false
1021 Acked Write 6 Done Sent Done false
1022 Acked Read 2 Done Sent Done false
1023 Acked Write 8 Done Sent Done false
1024 Acked Read 6 Done Sent Done false
1025 Acked Read 9 Done Sent Done false
1026 Acked Write 4 Done Sent Done false
1027 Acked Write 2 Done Sent Done false
1028 Acked Read 3 Done Sent Done false
1029 Acked Write 2 Done Sent Done false
1030 Acked Write 8 Done Sent Done false
1031 Acked Read 3 Done Sent Done false
1032 Acked Flush 0 Done Sent Done false
1033 Acked Write 4 Done Sent Done false
1034 Acked Write 9 Done Sent Done false
1035 Acked Write 10 Done Sent Done false
1036 Acked Read 9 Done Sent Done false
1037 Acked Read 4 Done Sent Done false
1038 Acked Write 8 Done Sent Done false
1039 Acked Write 3 Done Sent Done false
1040 Acked Read 4 Done Sent Done false
1041 Acked Read 8 Done Sent Done false
1042 Acked Write 10 Done Sent Done false
1043 Acked Read 2 Done Sent Done false
1044 Acked Read 2 Done Sent Done false
1045 Acked Read 6 Done Sent Done false
1046 Acked Write 1 Done Sent Done false
1047 Acked Write 9 Done Sent Done false
1048 Acked Write 10 Done Sent Done false
1049 Acked Flush 0 Done Sent Done false
1050 Acked Flush 0 Done Sent Done false
1051 Acked Read 5 Done Sent Done false
1052 Acked Read 9 Done Sent Done false
1053 Acked Write 2 Done Sent Done false
1054 Acked Write 6 Done Sent Done false
1055 Acked Read 8 Done Sent Done false
1056 Acked Write 6 Done Sent Done false
1057 Acked Write 7 Done Sent Done false
1058 Acked Flush 0 Done Sent Done false
1059 Acked Write 4 Done Sent Done false
1060 Acked Write 9 Done Sent Done false
1061 Acked Write 6 Done Sent Done false
1062 Acked Read 10 Done Sent Done false
1063 Acked Read 1 Done Sent Done false
1064 Acked Write 10 Done Sent Done false
1065 Acked Read 8 Done Sent Done false
1066 Acked Write 2 Done Sent Done false
1067 Acked Flush 0 Done Sent Done false
1068 Acked Flush 0 Done Sent Done false
1069 Acked Write 4 Done Sent Done false
1070 Acked Read 1 Done Sent Done false
1071 Acked Write 7 Done Sent Done false
1072 Acked Read 8 Done Sent Done false
1073 Acked Write 5 Done Sent Done false
1074 Acked Read 8 Done Sent Done false
1075 Acked Write 2 Done Sent Done false
1076 Acked Read 3 Done Sent Done false
1077 Acked Write 8 Done Sent Done false
1078 Acked Read 2 Done Sent Done false
1079 Acked Read 4 Done Sent Done false
1080 Acked Write 5 Done Sent Done false
1081 Acked Write 10 Done Sent Done false
1082 Acked Write 10 Done Sent Done false
1083 Acked Read 4 Done Sent Done false
1084 Acked Write 3 Done Sent Done false
1085 Acked Read 10 Done Sent Done false
1086 Acked Read 8 Done Sent Done false
1087 Acked Write 2 Done Sent Done false
1088 Acked Write 8 Done Sent Done false
1089 Acked Write 6 Done Sent Done false
1090 Acked Flush 0 Done Sent Done false
1091 Acked Write 10 Done Sent Done false
1092 Acked Read 5 Done Sent Done false
1093 Acked Flush 0 Done Sent Done false
1094 Acked Write 8 Done Sent Done false
1095 Acked Read 4 Done Sent Done false
1096 Acked Read 2 Done Sent Done false
1097 Acked Read 8 Done Sent Done false
1098 Acked Read 2 Done Sent Done false
1099 Acked Write 7 Done Sent Done false
1100 Acked Read 5 Done Sent Done false
1101 Acked Read 8 Done Sent Done false
1102 Acked Read 6 Done Sent Done false
1103 Acked Flush 0 Done Sent Done false
1104 Acked Write 9 Done Sent Done false
1105 Acked Read 6 Done Sent Done false
1106 Acked Write 7 Done Sent Done false
1107 Acked Read 1 Done Sent Done false
1108 Acked Write 5 Done Sent Done false
1109 Acked Read 3 Done Sent Done false
1110 Acked Read 7 Done Sent Done false
1111 Acked Write 7 Done Sent Done false
1112 Acked Read 6 Done Sent Done false
1113 Acked Flush 0 Done Sent Done false
1114 Acked Read 4 Done Sent Done false
1115 Acked Write 2 Done Sent Done false
1116 Acked Write 7 Done Sent Done false
1117 Acked Flush 0 Done Sent Done false
1118 Acked Write 2 Done Sent Done false
1119 Acked Read 1 Done Sent Done false
1120 Acked Flush 0 Done Sent Done false
1121 Acked Read 1 Done Sent Done false
1122 Acked Write 2 Done Sent Done false
1123 Acked Read 7 Done Sent Done false
1124 Acked Read 10 Done Sent Done false
1125 Acked Flush 0 Done Sent Done false
1126 Acked Read 5 Done Sent Done false
1127 Acked Read 2 Done Sent Done false
1128 Acked Write 6 Done Sent Done false
1129 Acked Read 9 Done Sent Done false
1130 Acked Flush 0 Done Sent Done false
1131 Acked Read 2 Done Sent Done false
1132 Acked Read 2 Done Sent Done false
1133 Acked Write 5 Done Sent Done false
1134 Acked Read 7 Done Sent Done false
1135 Acked Read 7 Done Sent Done false
1136 Acked Read 1 Done Sent Done false
1137 Acked Read 4 Done Sent Done false
1138 Acked Read 1 Done Sent Done false
1139 Acked Read 9 Done Sent Done false
1140 Acked Flush 0 Done Sent Done false
1141 Acked Read 2 Done Sent Done false
1142 Acked Read 5 Done Sent Done false
1143 Acked Read 1 Done Sent Done false
1144 Acked Flush 0 Done Sent Done false
1145 Acked Write 10 Done Sent Done false
1146 Acked Read 2 Done Sent Done false
1147 Acked Write 9 Done Sent Done false
1148 Acked Read 9 Done Sent Done false
1149 Acked Write 1 Done Sent Done false
1150 Acked Flush 0 Done Sent Done false
1151 Acked Write 1 Done Sent Done false
1152 Acked Write 1 Done Sent Done false
1153 Acked Read 9 Done Sent Done false
1154 Acked Write 7 Done Sent Done false
1155 Acked Read 1 Done Sent Done false
1156 Acked Write 5 Done Sent Done false
1157 Acked Flush 0 Done Sent Done false
1158 Acked Write 1 Done Sent Done false
1159 Acked Write 4 Done Sent Done false
1160 Acked Read 8 Done Sent Done false
1161 Acked Write 1 Done Sent Done false
1162 Acked Read 7 Done Sent Done false
1163 Acked Read 2 Done Sent Done false
1164 Acked Read 3 Done Sent Done false
1165 Acked Read 2 Done Sent Done false
1166 Acked Write 8 Done Sent Done false
1167 Acked Read 9 Done Sent Done false
1168 Acked Read 2 Done Sent Done false
1169 Acked Write 4 Done Sent Done false
1170 Acked Write 4 Done Sent Done false
1171 Acked Read 10 Done Sent Done false
1172 Acked Read 8 Done Sent Done false
1173 Acked Read 4 Done Sent Done false
1174 Acked Write 5 Done Sent Done false
1175 Acked Write 6 Done Sent Done false
1176 Acked Read 1 Done Sent Done false
1177 Acked Read 8 Done Sent Done false
1178 Acked Write 7 Done Sent Done false
1179 Acked Read 1 Done Sent Done false
1180 Acked Write 1 Done Sent Done false
1181 Acked Read 4 Done Sent Done false
1182 Acked Write 6 Done Sent Done false
1183 Acked Write 6 Done Sent Done false
1184 Acked Write 7 Done Sent Done false
1185 Acked Read 9 Done Sent Done false
1186 Acked Write 6 Done Sent Done false
1187 Acked Flush 0 Done Sent Done false
1188 Acked Read 3 Done Sent Done false
1189 Acked Write 4 Done Sent Done false
1190 Acked Read 8 Done Sent Done false
1191 Acked Write 10 Done Sent Done false
1192 Acked Write 2 Done Sent Done false
1193 Acked Read 2 Done Sent Done false
1194 Acked Write 3 Done Sent Done false
1195 Acked Read 10 Done Sent Done false
1196 Acked Write 6 Done Sent Done false
1197 Acked Write 1 Done Sent Done false
1198 Acked Flush 0 Done Sent Done false
1199 Acked Read 6 Done Sent Done false
1200 Acked Write 4 Done Sent Done false
1201 Acked Write 4 Done Sent Done false
1202 Acked Read 1 Done Sent Done false
1203 Acked Read 9 Done Sent Done false
1204 Acked Read 1 Done Sent Done false
1205 Acked Read 4 Done Sent Done false
1206 Acked Read 3 Done Sent Done false
1207 Acked Write 2 Done Sent Done false
1208 Acked Read 4 Done Sent Done false
1209 Acked Write 6 Done Sent Done false
1210 Acked Write 5 Done Sent Done false
1211 Acked Read 1 Done Sent Done false
1212 Acked Read 1 Done Sent Done false
1213 Acked Write 5 Done Sent Done false
1214 Acked Write 2 Done Sent Done false
1215 Acked Flush 0 Done Sent Done false
1216 Acked Read 9 Done Sent Done false
1217 Acked Flush 0 Done Sent Done false
1218 Acked Write 8 Done Sent Done false
1219 Acked Read 7 Done Sent Done false
1220 Acked Write 8 Done Sent Done false
1221 Acked Read 5 Done Sent Done false
1222 Acked Flush 0 Done Sent Done false
1223 Acked Read 8 Done Sent Done false
1224 Acked Write 10 Done Sent Done false
1225 Acked Read 6 Done Sent Done false
1226 Acked Write 2 Done Sent Done false
1227 Acked Write 3 Done Sent Done false
1228 Acked Read 4 Done Sent Done false
1229 Acked Write 10 Done Sent Done false
1230 Acked Read 1 Done Sent Done false
1231 Acked Write 4 Done Sent Done false
1232 Acked Read 3 Done Sent Done false
1233 Acked Read 6 Done Sent Done false
1234 Acked Read 1 Done Sent Done false
1235 Acked Read 9 Done Sent Done false
1236 Acked Read 9 Done Sent Done false
1237 Acked Flush 0 Done Sent Done false
1238 Acked Read 7 Done Sent Done false
1239 Acked Read 9 Done Sent Done false
1240 Acked Write 1 Done Sent Done false
1241 Acked Read 6 Done Sent Done false
1242 Acked Read 3 Done Sent Done false
1243 Acked Read 7 Done Sent Done false
1244 Acked Write 1 Done Sent Done false
1245 Acked Flush 0 Done Sent Done false
1246 Acked Write 10 Done Sent Done false
1247 Acked Read 10 Done Sent Done false
1248 Acked Flush 0 Done Sent Done false
1249 Acked Read 7 Done Sent Done false
1250 Acked Flush 0 Done Sent Done false
1251 Acked Write 6 Done Sent Done false
1252 Acked Write 8 Done Sent Done false
1253 Acked Flush 0 Done Sent Done false
1254 Acked Read 5 Done Sent Done false
1255 Acked Write 4 Done Sent Done false
1256 Acked Write 8 Done Sent Done false
1257 Acked Write 4 Done Sent Done false
1258 Acked Read 1 Done Sent Done false
1259 Acked Read 3 Done Sent Done false
1260 Acked Read 4 Done Sent Done false
1261 Acked Write 4 Done Sent Done false
1262 Acked Read 1 Done Sent Done false
1263 Acked Write 10 Done Sent Done false
1264 Acked Read 7 Done Sent Done false
1265 Acked Read 3 Done Sent Done false
1266 Acked Read 8 Done Sent Done false
1267 Acked Read 9 Done Sent Done false
1268 Acked Write 7 Done Sent Done false
1269 Acked Read 6 Done Sent Done false
1270 Acked Write 6 Done Sent Done false
1271 Acked Flush 0 Done Sent Done false
1272 Acked Write 3 Done Sent Done false
1273 Acked Read 1 Done Sent Done false
1274 Acked Read 10 Done Sent Done false
1275 Acked Write 3 Done Sent Done false
1276 Acked Read 9 Done Sent Done false
1277 Acked Read 9 Done Sent Done false
1278 Acked Read 4 Done Sent Done false
1279 Acked Read 1 Done Sent Done false
1280 Acked Read 2 Done Sent Done false
1281 Acked Read 10 Done Sent Done false
1282 Acked Write 1 Done Sent Done false
1283 Acked Read 7 Done Sent Done false
1284 Acked Read 3 Done Sent Done false
1285 Acked Flush 0 Done Sent Done false
1286 Acked Flush 0 Done Sent Done false
1287 Acked Read 4 Done Sent Done false
1288 Acked Read 6 Done Sent Done false
1289 Acked Read 1 Done Sent Done false
1290 Acked Read 2 Done Sent Done false
1291 Acked Read 10 Done Sent Done false
1292 Acked Read 5 Done Sent Done false
1293 Acked Read 8 Done Sent Done false
1294 Acked Read 5 Done Sent Done false
1295 Acked Read 9 Done Sent Done false
1296 Acked Read 9 Done Sent Done false
1297 Acked Read 1 Done Sent Done false
1298 Acked Read 6 Done Sent Done false
1299 Acked Read 2 Done Sent Done false
1300 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1300)) None Some(JobId(1300))
Downstairs last five completed:
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:20.630215198Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:21.369976718Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:22.372315201Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:23.374340858Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.879942918Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56891)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.880137041Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881231399Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881740707Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881765961Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: None","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881783397Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881799573Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush None","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.881815939Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: None","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:23.88218747Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:01:24.131 INFO request completed, latency_us: 341, response_code: 204, uri: /stop/cid/1, method: POST, req_id: aeca7b05-817c-4d05-a394-dc815cffbc58, [8820] Got stop action so:false kr:false
remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:24.132287012Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:24.13233683Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:24.13242102Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:01:24.132 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/1, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:24.13250304Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
GET, req_id: 04e5f861-8a7e-458d-bacc-a34495637078, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:01:28.135 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1fc4a83e-2897-4f27-94a6-bf0cce55d4f4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 1436 len 8192
002/300 Read block 143 len 24576
003/300 Write block 1280 len 20480 data: 2 2 2 2 2
004/300 Read block 1579 len 24576
005/300 Read block 2440 len 16384
006/300 Read block 4182 len 16384
007/300 Flush
008/300 Write block 3297 len 12288 data: 2 2 2
009/300 Write block 1546 len 24576 data: 2 2 2 2 2 2
010/300 Flush
011/300 Read block 204 len 16384
012/300 Read block 331 len 32768
013/300 Read block 1734 len 12288
014/300 Read block 1638 len 24576
015/300 Write block 3071 len 16384 data: 2 2 2 2
016/300 Write block 4528 len 16384 data: 2 2 2 2
017/300 Read block 3602 len 32768
018/300 Flush
019/300 Write block 4678 len 16384 data: 2 2 2 2
020/300 Write block 1581 len 40960 data: 2 2 2 2 2 2 2 2 2 2
021/300 Read block 145 len 4096
022/300 Read block 1661 len 20480
023/300 Write block 580 len 28672 data: 2 2 2 2 2 2 2
024/300 Write block 2764 len 4096 data: 2
025/300 Read block 4476 len 36864
026/300 Read block 227 len 40960
027/300 Write block 1465 len 20480 data: 2 2 2 2 2
028/300 Write block 3244 len 28672 data: 2 2 2 2 2 2 2
029/300 Read block 4324 len 8192
030/300 Flush
031/300 Read block 422 len 8192
032/300 Read block 180 len 40960
033/300 Read block 271 len 40960
034/300 Write block 438 len 12288 data: 2 2 2
035/300 Flush
036/300 Read block 2334 len 12288
037/300 Read block 3872 len 16384
038/300 Flush
039/300 Write block 4887 len 20480 data: 2 4 4 4 4
040/300 Write block 1932 len 24576 data: 2 2 2 2 2 2
041/300 Flush
042/300 Read block 1422 len 20480
043/300 Write block 1537 len 8192 data: 2 2
044/300 Read block 537 len 28672
045/300 Read block 3927 len 16384
046/300 Read block 102 len 28672
047/300 Write block 3158 len 12288 data: 2 2 2
048/300 Flush
049/300 Read block 3050 len 36864
050/300 Write block 905 len 16384 data: 2 2 2 2
051/300 Read block 4672 len 8192
052/300 Read block 2450 len 16384
053/300 Write block 1176 len 16384 data: 2 2 2 3
054/300 Write block 2583 len 36864 data: 2 2 2 2 2 2 2 2 2
055/300 Write block 500 len 32768 data: 2 2 2 2 2 2 2 2
056/300 Flush
057/300 Read block 1495 len 20480
058/300 Read block 3094 len 28672
059/300 Write block 3780 len 16384 data: 2 3 3 3
060/300 Write block 2190 len 36864 data: 2 2 2 2 2 3 3 3 2
061/300 Read block 4066 len 28672
062/300 Flush
063/300 Read block 123 len 36864
064/300 Flush
065/300 Write block 3996 len 28672 data: 2 2 2 2 2 2 2
066/300 Read block 742 len 12288
067/300 Read block 4198 len 4096
068/300 Write block 4280 len 32768 data: 2 2 2 2 2 2 2 2
069/300 Read block 4207 len 40960
070/300 Write block 2937 len 24576 data: 2 3 3 3 3 3
071/300 Read block 2377 len 12288
072/300 Read block 776 len 24576
073/300 Flush
074/300 Read block 3435 len 12288
075/300 Read block 4274 len 16384
076/300 Read block 3268 len 32768
077/300 Read block 2720 len 16384
078/300 Read block 848 len 28672
079/300 Read block 2409 len 24576
080/300 Write block 1402 len 40960 data: 2 2 2 2 2 2 2 3 3 3
081/300 Write block 2599 len 32768 data: 3 3 3 2 2 2 2 2
082/300 Read block 313 len 8192
083/300 Read block 3820 len 40960
084/300 Read block 1899 len 12288
085/300 Read block 2199 len 16384
086/300 Read block 3386 len 16384
087/300 Read block 4553 len 16384
088/300 Write block 3794 len 4096 data: 2
089/300 Read block 78 len 32768
090/300 Read block 4397 len 20480
091/300 Flush
092/300 Read block 4927 len 8192
093/300 Read block 2374 len 24576
094/300 Read block 3477 len 36864
095/300 Write block 237 len 8192 data: 3 3
096/300 Read block 286 len 12288
097/300 Write block 3306 len 16384 data: 2 2 2 2
098/300 Read block 2634 len 40960
099/300 Read block 2545 len 36864
100/300 Write block 2521 len 36864 data: 2 2 2 2 2 2 2 2 2
101/300 Read block 3325 len 8192
102/300 Write block 3925 len 8192 data: 2 2
103/300 Read block 585 len 20480
104/300 Write block 2199 len 24576 data: 2 2 2 2 2 2
105/300 Read block 1257 len 16384
106/300 Read block 3974 len 12288
107/300 Flush
108/300 Read block 573 len 40960
109/300 Read block 1897 len 32768
110/300 Write block 3628 len 8192 data: 2 2
111/300 Write block 1761 len 24576 data: 2 2 2 2 2 2
112/300 Write block 2713 len 24576 data: 2 2 2 2 2 2
113/300 Read block 612 len 28672
114/300 Read block 3073 len 4096
115/300 Read block 3358 len 20480
116/300 Write block 2670 len 28672 data: 2 2 2 2 2 2 2
117/300 Write block 1334 len 40960 data: 2 2 2 2 2 2 2 2 2 2
118/300 Read block 3137 len 40960
119/300 Write block 4897 len 16384 data: 2 2 2 2
120/300 Read block 4241 len 36864
121/300 Read block 249 len 20480
122/300 Write block 3336 len 20480 data: 2 2 2 2 2
123/300 Read block 3823 len 16384
124/300 Write block 2227 len 32768 data: 2 2 2 2 2 2 2 3
125/300 Read block 1498 len 24576
126/300 Write block 2511 len 40960 data: 2 2 2 2 2 2 2 2 2 2
127/300 Read block 2187 len 28672
128/300 Write block 3282 len 4096 data: 2
129/300 Read block 3868 len 16384
130/300 Flush
131/300 Read block 1935 len 4096
132/300 Flush
133/300 Read block 4970 len 36864
134/300 Read block 4406 len 24576
135/300 Write block 4776 len 24576 data: 2 2 2 2 2 2
136/300 Write block 4178 len 40960 data: 2 2 2 2 2 2 2 2 2 2
137/300 Write block 32 len 24576 data: 2 2 2 2 2 2
138/300 Read block 3874 len 40960
139/300 Read block 1272 len 16384
140/300 Read block 3031 len 28672
141/300 Write block 2624 len 4096 data: 2
142/300 Flush
143/300 Read block 4482 len 4096
144/300 Read block 763 len 20480
145/300 Write block 4729 len 32768 data: 2 2 2 2 2 2 2 2
146/300 Read block 1530 len 28672
147/300 Read block 3034 len 32768
148/300 Write block 1891 len 40960 data: 2 2 2 2 2 2 2 2 2 2
149/300 Flush
150/300 Read block 774 len 8192
151/300 Read block 2727 len 8192
152/300 Write block 75 len 20480 data: 3 3 3 3 3
153/300 Read block 2777 len 8192
154/300 Read block 270 len 4096
155/300 Read block 2140 len 32768
156/300 Write block 2117 len 40960 data: 2 2 2 2 2 2 2 2 2 2
157/300 Read block 1344 len 28672
158/300 Flush
159/300 Read block 1828 len 36864
160/300 Read block 1224 len 28672
161/300 Flush
162/300 Read block 2577 len 28672
163/300 Read block 644 len 4096
164/300 Write block 1578 len 40960 data: 2 2 2 3 3 3 3 3 3 3
165/300 Read block 4744 len 16384
166/300 Write block 4518 len 16384 data: 2 2 2 2
167/300 Read block 791 len 8192
168/300 Write block 961 len 4096 data: 2
169/300 Read block 197 len 36864
170/300 Read block 3877 len 36864
171/300 Write block 4780 len 40960 data: 3 3 2 2 2 2 2 2 2 2
172/300 Write block 4004 len 20480 data: 2 3 3 3 3
173/300 Read block 818 len 12288
174/300 Read block 3643 len 8192
175/300 Read block 555 len 8192
176/300 Write block 1758 len 32768 data: 2 2 2 3 3 3 3 3
177/300 Write block 1682 len 24576 data: 2 2 2 2 2 2
178/300 Write block 2253 len 12288 data: 2 2 2
179/300 Read block 4050 len 20480
180/300 Read block 3693 len 12288
181/300 Write block 3100 len 24576 data: 2 2 2 2 2 2
182/300 Read block 431 len 32768
183/300 Write block 281 len 40960 data: 2 2 2 2 2 2 2 2 2 2
184/300 Read block 2466 len 32768
185/300 Write block 3335 len 28672 data: 2 3 3 3 3 3 2
186/300 Write block 271 len 32768 data: 2 2 2 2 2 2 2 2
187/300 Read block 2616 len 8192
188/300 Read block 2690 len 28672
189/300 Write block 2231 len 16384 data: 3 3 3 4
190/300 Write block 3939 len 32768 data: 2 2 2 2 2 2 2 2
191/300 Read block 4443 len 16384
192/300 Read block 3643 len 4096
193/300 Read block 242 len 40960
194/300 Flush
195/300 Read block 3074 len 12288
196/300 Write block 551 len 12288 data: 2 2 2
197/300 Read block 872 len 8192
198/300 Write block 2439 len 8192 data: 2 2
199/300 Write block 1271 len 16384 data: 2 2 2 2
200/300 Flush
201/300 Read block 112 len 24576
202/300 Write block 552 len 8192 data: 3 3
203/300 Write block 2799 len 40960 data: 2 2 2 2 2 2 2 2 2 2
204/300 Read block 72 len 40960
205/300 Write block 3156 len 28672 data: 2 2 3 3 3 2 2
206/300 Write block 2750 len 8192 data: 2 2
207/300 Write block 1000 len 36864 data: 2 2 2 2 2 2 2 2 2
208/300 Read block 224 len 20480
209/300 Flush
210/300 Read block 1132 len 4096
211/300 Write block 4535 len 24576 data: 2 2 2 2 2 2
212/300 Write block 1049 len 24576 data: 2 2 2 2 2 2
213/300 Read block 4445 len 28672
214/300 Read block 4535 len 12288
215/300 Read block 4962 len 40960
216/300 Read block 4129 len 12288
217/300 Write block 2317 len 12288 data: 2 2 2
218/300 Write block 3302 len 24576 data: 2 2 2 2 3 3
219/300 Read block 1036 len 16384
220/300 Read block 2972 len 32768
221/300 Write block 2990 len 12288 data: 2 2 2
222/300 Write block 3024 len 4096 data: 2
223/300 Read block 797 len 20480
224/300 Read block 1677 len 12288
225/300 Write block 2336 len 36864 data: 2 2 2 2 2 2 2 2 2
226/300 Write block 4840 len 24576 data: 2 2 3 2 2 2
227/300 Read block 2009 len 24576
228/300 Read block 2551 len 20480
229/300 Write block 779 len 28672 data: 2 2 2 2 2 2 2
230/300 Read block 1561 len 40960
231/300 Read block 3825 len 8192
232/300 Read block 2984 len 4096
233/300 Write block 910 len 24576 data: 2 2 2 2 2 2
234/300 Read block 3459 len 28672
235/300 Read block 1409 len 12288
236/300 Read block 3321 len 36864
237/300 Read block 4946 len 32768
238/300 Read block 470 len 36864
239/300 Read block 1357 len 24576
240/300 Write block 2603 len 8192 data: 3 3
241/300 Read block 3473 len 8192
242/300 Flush
243/300 Read block 1658 len 32768
244/300 Read block 1641 len 28672
245/300 Read block 2690 len 24576
246/300 Flush
247/300 Write block 2805 len 12288 data: 3 3 3
248/300 Read block 3509 len 24576
249/300 Read block 1548 len 4096
250/300 Read block 4038 len 16384
251/300 Write block 3873 len 16384 data: 2 2 2 2
252/300 Write block 948 len 16384 data: 2 2 2 2
253/300 Flush
254/300 Read block 367 len 12288
255/300 Write block 929 len 16384 data: 2 2 2 2
256/300 Read block 1331 len 40960
257/300 Read block 1689 len 8192
258/300 Read block 4174 len 4096
259/300 Read block 4155 len 20480
260/300 Read block 1276 len 32768
261/300 Write block 1604 len 12288 data: 2 2 2
262/300 Write block 3341 len 28672 data: 3 2 2 2 2 2 2
263/300 Read block 1929 len 8192
264/300 Read block 3950 len 16384
265/300 Read block 4891 len 12288
266/300 Read block 25 len 4096
267/300 Write block 1350 len 16384 data: 3 3 3 3
268/300 Read block 2230 len 32768
269/300 Write block 62 len 12288 data: 2 2 2
270/300 Read block 2953 len 36864
271/300 Read block 4344 len 32768
272/300 Read block 4121 len 20480
273/300 Write block 122 len 40960 data: 2 2 2 2 2 2 2 2 2 2
274/300 Read block 1463 len 4096
275/300 Read block 1361 len 8192
276/300 Flush
277/300 Read block 2745 len 4096
278/300 Write block 2124 len 16384 data: 3 3 3 2
279/300 Write block 3883 len 16384 data: 2 2 2 2
280/300 Read block 3494 len 8192
281/300 Read block 440 len 28672
282/300 Read block 2047 len 12288
283/300 Write block 4408 len 20480 data: 4 4 4 4 4
284/300 Read block 4108 len 20480
285/300 Read block 3188 len 4096
286/300 Read block 2558 len 12288
287/300 Write block 569 len 20480 data: 2 2 2 2 2
288/300 Write block 1448 len 28672 data: 2 2 2 2 2 2 2
289/300 Read block 1535 len 4096
290/300 Read block 4468 len 40960
291/300 Read block 3677 len 12288
292/300 Read block 2618 len 4096
293/300 Read block 4226 len 40960
294/300 Read block 4266 len 36864
295/300 Read block 3018 len 28672
296/300 Flush
297/300 Read block 4630 len 4096
298/300 Write block 373 len 32768 data: 2 2 2 2 2 3 3 3
299/300 Write block 1691 len 8192 data: 3 3
300/300 Read block 3953 len 24576
301/300 Write block 3681 len 16384 data: 2 2 2 2
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:01:28.383 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/1, method: POST, req_id: 91251bca-0438-4afd-9141-a832189e6d2f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1301 Acked Read 2 Done Sent Done false
1302 Acked Read 6 Done Sent Done false
1303 Acked Write 5 Done Sent Done false
1304 Acked Read 6 Done Sent Done false
1305 Acked Read 4 Done Sent Done false
1306 Acked Read 4 Done Sent Done false
1307 Acked Flush 0 Done Sent Done false
1308 Acked Write 3 Done Sent Done false
1309 Acked Write 6 Done Sent Done false
1310 Acked Flush 0 Done Sent Done false
1311 Acked Read 4 Done Sent Done false
1312 Acked Read 8 Done Sent Done false
1313 Acked Read 3 Done Sent Done false
1314 Acked Read 6 Done Sent Done false
1315 Acked Write 4 Done Sent Done false
1316 Acked Write 4 Done Sent Done false
1317 Acked Read 8 Done Sent Done false
1318 Acked Flush 0 Done Sent Done false
1319 Acked Write 4 Done Sent Done false
1320 Acked Write 10 Done Sent Done false
1321 Acked Read 1 Done Sent Done false
1322 Acked Read 5 Done Sent Done false
1323 Acked Write 7 Done Sent Done false
1324 Acked Write 1 Done Sent Done false
1325 Acked Read 9 Done Sent Done false
1326 Acked Read 10 Done Sent Done false
1327 Acked Write 5 Done Sent Done false
1328 Acked Write 7 Done Sent Done false
1329 Acked Read 2 Done Sent Done false
1330 Acked Flush 0 Done Sent Done false
1331 Acked Read 2 Done Sent Done false
1332 Acked Read 10 Done Sent Done false
1333 Acked Read 10 Done Sent Done false
1334 Acked Write 3 Done Sent Done false
1335 Acked Flush 0 Done Sent Done false
1336 Acked Read 3 Done Sent Done false
1337 Acked Read 4 Done Sent Done false
1338 Acked Flush 0 Done Sent Done false
1339 Acked Write 5 Done Sent Done false
1340 Acked Write 6 Done Sent Done false
1341 Acked Flush 0 Done Sent Done false
1342 Acked Read 5 Done Sent Done false
1343 Acked Write 2 Done Sent Done false
1344 Acked Read 7 Done Sent Done false
1345 Acked Read 4 Done Sent Done false
1346 Acked Read 7 Done Sent Done false
1347 Acked Write 3 Done Sent Done false
1348 Acked Flush 0 Done Sent Done false
1349 Acked Read 9 Done Sent Done false
1350 Acked Write 4 Done Sent Done false
1351 Acked Read 2 Done Sent Done false
1352 Acked Read 4 Done Sent Done false
1353 Acked Write 4 Done Sent Done false
1354 Acked Write 9 Done Sent Done false
1355 Acked Write 8 Done Sent Done false
1356 Acked Flush 0 Done Sent Done false
1357 Acked Read 5 Done Sent Done false
1358 Acked Read 7 Done Sent Done false
1359 Acked Write 4 Done Sent Done false
1360 Acked Write 9 Done Sent Done false
1361 Acked Read 7 Done Sent Done false
1362 Acked Flush 0 Done Sent Done false
1363 Acked Read 9 Done Sent Done false
1364 Acked Flush 0 Done Sent Done false
1365 Acked Write 7 Done Sent Done false
1366 Acked Read 3 Done Sent Done false
1367 Acked Read 1 Done Sent Done false
1368 Acked Write 8 Done Sent Done false
1369 Acked Read 10 Done Sent Done false
1370 Acked Write 6 Done Sent Done false
1371 Acked Read 3 Done Sent Done false
1372 Acked Read 6 Done Sent Done false
1373 Acked Flush 0 Done Sent Done false
1374 Acked Read 3 Done Sent Done false
1375 Acked Read 4 Done Sent Done false
1376 Acked Read 8 Done Sent Done false
1377 Acked Read 4 Done Sent Done false
1378 Acked Read 7 Done Sent Done false
1379 Acked Read 6 Done Sent Done false
1380 Acked Write 10 Done Sent Done false
1381 Acked Write 8 Done Sent Done false
1382 Acked Read 2 Done Sent Done false
1383 Acked Read 10 Done Sent Done false
1384 Acked Read 3 Done Sent Done false
1385 Acked Read 4 Done Sent Done false
1386 Acked Read 4 Done Sent Done false
1387 Acked Read 4 Done Sent Done false
1388 Acked Write 1 Done Sent Done false
1389 Acked Read 8 Done Sent Done false
1390 Acked Read 5 Done Sent Done false
1391 Acked Flush 0 Done Sent Done false
1392 Acked Read 2 Done Sent Done false
1393 Acked Read 6 Done Sent Done false
1394 Acked Read 9 Done Sent Done false
1395 Acked Write 2 Done Sent Done false
1396 Acked Read 3 Done Sent Done false
1397 Acked Write 4 Done Sent Done false
1398 Acked Read 10 Done Sent Done false
1399 Acked Read 9 Done Sent Done false
1400 Acked Write 9 Done Sent Done false
1401 Acked Read 2 Done Sent Done false
1402 Acked Write 2 Done Sent Done false
1403 Acked Read 5 Done Sent Done false
1404 Acked Write 6 Done Sent Done false
1405 Acked Read 4 Done Sent Done false
1406 Acked Read 3 Done Sent Done false
1407 Acked Flush 0 Done Sent Done false
1408 Acked Read 10 Done Sent Done false
1409 Acked Read 8 Done Sent Done false
1410 Acked Write 2 Done Sent Done false
1411 Acked Write 6 Done Sent Done false
1412 Acked Write 6 Done Sent Done false
1413 Acked Read 7 Done Sent Done false
1414 Acked Read 1 Done Sent Done false
1415 Acked Read 5 Done Sent Done false
1416 Acked Write 7 Done Sent Done false
1417 Acked Write 10 Done Sent Done false
1418 Acked Read 10 Done Sent Done false
1419 Acked Write 4 Done Sent Done false
1420 Acked Read 9 Done Sent Done false
1421 Acked Read 5 Done Sent Done false
1422 Acked Write 5 Done Sent Done false
1423 Acked Read 4 Done Sent Done false
1424 Acked Write 8 Done Sent Done false
1425 Acked Read 6 Done Sent Done false
1426 Acked Write 10 Done Sent Done false
1427 Acked Read 7 Done Sent Done false
1428 Acked Write 1 Done Sent Done false
1429 Acked Read 4 Done Sent Done false
1430 Acked Flush 0 Done Sent Done false
1431 Acked Read 1 Done Sent Done false
1432 Acked Flush 0 Done Sent Done false
1433 Acked Read 9 Done Sent Done false
1434 Acked Read 6 Done Sent Done false
1435 Acked Write 6 Done Sent Done false
1436 Acked Write 10 Done Sent Done false
1437 Acked Write 6 Done Sent Done false
1438 Acked Read 10 Done Sent Done false
1439 Acked Read 4 Done Sent Done false
1440 Acked Read 7 Done Sent Done false
1441 Acked Write 1 Done Sent Done false
1442 Acked Flush 0 Done Sent Done false
1443 Acked Read 1 Done Sent Done false
1444 Acked Read 5 Done Sent Done false
1445 Acked Write 8 Done Sent Done false
1446 Acked Read 7 Done Sent Done false
1447 Acked Read 8 Done Sent Done false
1448 Acked Write 10 Done Sent Done false
1449 Acked Flush 0 Done Sent Done false
1450 Acked Read 2 Done Sent Done false
1451 Acked Read 2 Done Sent Done false
1452 Acked Write 5 Done Sent Done false
1453 Acked Read 2 Done Sent Done false
1454 Acked Read 1 Done Sent Done false
1455 Acked Read 8 Done Sent Done false
1456 Acked Write 10 Done Sent Done false
1457 Acked Read 7 Done Sent Done false
1458 Acked Flush 0 Done Sent Done false
1459 Acked Read 9 Done Sent Done false
1460 Acked Read 7 Done Sent Done false
1461 Acked Flush 0 Done Sent Done false
1462 Acked Read 7 Done Sent Done false
1463 Acked Read 1 Done Sent Done false
1464 Acked Write 10 Done Sent Done false
1465 Acked Read 4 Done Sent Done false
1466 Acked Write 4 Done Sent Done false
1467 Acked Read 2 Done Sent Done false
1468 Acked Write 1 Done Sent Done false
1469 Acked Read 9 Done Sent Done false
1470 Acked Read 9 Done Sent Done false
1471 Acked Write 10 Done Sent Done false
1472 Acked Write 5 Done Sent Done false
1473 Acked Read 3 Done Sent Done false
1474 Acked Read 2 Done Sent Done false
1475 Acked Read 2 Done Sent Done false
1476 Acked Write 8 Done Sent Done false
1477 Acked Write 6 Done Sent Done false
1478 Acked Write 3 Done Sent Done false
1479 Acked Read 5 Done Sent Done false
1480 Acked Read 3 Done Sent Done false
1481 Acked Write 6 Done Sent Done false
1482 Acked Read 8 Done Sent Done false
1483 Acked Write 10 Done Sent Done false
1484 Acked Read 8 Done Sent Done false
1485 Acked Write 7 Done Sent Done false
1486 Acked Write 8 Done Sent Done false
1487 Acked Read 2 Done Sent Done false
1488 Acked Read 7 Done Sent Done false
1489 Acked Write 4 Done Sent Done false
1490 Acked Write 8 Done Sent Done false
1491 Acked Read 4 Done Sent Done false
1492 Acked Read 1 Done Sent Done false
1493 Acked Read 10 Done Sent Done false
1494 Acked Flush 0 Done Sent Done false
1495 Acked Read 3 Done Sent Done false
1496 Acked Write 3 Done Sent Done false
1497 Acked Read 2 Done Sent Done false
1498 Acked Write 2 Done Sent Done false
1499 Acked Write 4 Done Sent Done false
1500 Acked Flush 0 Done Sent Done false
1501 Acked Read 6 Done Sent Done false
1502 Acked Write 2 Done Sent Done false
1503 Acked Write 10 Done Sent Done false
1504 Acked Read 10 Done Sent Done false
1505 Acked Write 7 Done Sent Done false
1506 Acked Write 2 Done Sent Done false
1507 Acked Write 9 Done Sent Done false
1508 Acked Read 5 Done Sent Done false
1509 Acked Flush 0 Done Sent Done false
1510 Acked Read 1 Done Sent Done false
1511 Acked Write 6 Done Sent Done false
1512 Acked Write 6 Done Sent Done false
1513 Acked Read 7 Done Sent Done false
1514 Acked Read 3 Done Sent Done false
1515 Acked Read 10 Done Sent Done false
1516 Acked Read 3 Done Sent Done false
1517 Acked Write 3 Done Sent Done false
1518 Acked Write 6 Done Sent Done false
1519 Acked Read 4 Done Sent Done false
1520 Acked Read 8 Done Sent Done false
1521 Acked Write 3 Done Sent Done false
1522 Acked Write 1 Done Sent Done false
1523 Acked Read 5 Done Sent Done false
1524 Acked Read 3 Done Sent Done false
1525 Acked Write 9 Done Sent Done false
1526 Acked Write 6 Done Sent Done false
1527 Acked Read 6 Done Sent Done false
1528 Acked Read 5 Done Sent Done false
1529 Acked Write 7 Done Sent Done false
1530 Acked Read 10 Done Sent Done false
1531 Acked Read 2 Done Sent Done false
1532 Acked Read 1 Done Sent Done false
1533 Acked Write 6 Done Sent Done false
1534 Acked Read 7 Done Sent Done false
1535 Acked Read 3 Done Sent Done false
1536 Acked Read 9 Done Sent Done false
1537 Acked Read 8 Done Sent Done false
1538 Acked Read 9 Done Sent Done false
1539 Acked Read 6 Done Sent Done false
1540 Acked Write 2 Done Sent Done false
1541 Acked Read 2 Done Sent Done false
1542 Acked Flush 0 Done Sent Done false
1543 Acked Read 8 Done Sent Done false
1544 Acked Read 7 Done Sent Done false
1545 Acked Read 6 Done Sent Done false
1546 Acked Flush 0 Done Sent Done false
1547 Acked Write 3 Done Sent Done false
1548 Acked Read 6 Done Sent Done false
1549 Acked Read 1 Done Sent Done false
1550 Acked Read 4 Done Sent Done false
1551 Acked Write 4 Done Sent Done false
1552 Acked Write 4 Done Sent Done false
1553 Acked Flush 0 Done Sent Done false
1554 Acked Read 3 Done Sent Done false
1555 Acked Write 4 Done Sent Done false
1556 Acked Read 10 Done Sent Done false
1557 Acked Read 2 Done Sent Done false
1558 Acked Read 1 Done Sent Done false
1559 Acked Read 5 Done Sent Done false
1560 Acked Read 8 Done Sent Done false
1561 Acked Write 3 Done Sent Done false
1562 Acked Write 7 Done Sent Done false
1563 Acked Read 2 Done Sent Done false
1564 Acked Read 4 Done Sent Done false
1565 Acked Read 3 Done Sent Done false
1566 Acked Read 1 Done Sent Done false
1567 Acked Write 4 Done Sent Done false
1568 Acked Read 8 Done Sent Done false
1569 Acked Write 3 Done Sent Done false
1570 Acked Read 9 Done Sent Done false
1571 Acked Read 8 Done Sent Done false
1572 Acked Read 5 Done Sent Done false
1573 Acked Write 10 Done Sent Done false
1574 Acked Read 1 Done Sent Done false
1575 Acked Read 2 Done Sent Done false
1576 Acked Flush 0 Done Sent Done false
1577 Acked Read 1 Done Sent Done false
1578 Acked Write 4 Done Sent Done false
1579 Acked Write 4 Done Sent Done false
1580 Acked Read 2 Done Sent Done false
1581 Acked Read 7 Done Sent Done false
1582 Acked Read 3 Done Sent Done false
1583 Acked Write 5 Done Sent Done false
1584 Acked Read 5 Done Sent Done false
1585 Acked Read 1 Done Sent Done false
1586 Acked Read 3 Done Sent Done false
1587 Acked Write 5 Done Sent Done false
1588 Acked Write 7 Done Sent Done false
1589 Acked Read 1 Done Sent Done false
1590 Acked Read 10 Done Sent Done false
1591 Acked Read 3 Done Sent Done false
1592 Acked Read 1 Done Sent Done false
1593 Acked Read 10 Done Sent Done false
1594 Acked Read 9 Done Sent Done false
1595 Acked Read 7 Done Sent Done false
1596 Acked Flush 0 Done Sent Done false
1597 Acked Read 1 Done Sent Done false
1598 Acked Write 8 Done Sent Done false
1599 Acked Write 2 Done Sent Done false
1600 Acked Read 6 Done Sent Done false
1601 Acked Write 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 1601 1600 1599 1598 1597
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:29.382692446Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:30.383829636Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:31.385045668Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1301 Acked Read 2 Done Sent Done false
1302 Acked Read 6 Done Sent Done false
1303 Acked Write 5 Done Sent Done false
1304 Acked Read 6 Done Sent Done false
1305 Acked Read 4 Done Sent Done false
1306 Acked Read 4 Done Sent Done false
1307 Acked Flush 0 Done Sent Done false
1308 Acked Write 3 Done Sent Done false
1309 Acked Write 6 Done Sent Done false
1310 Acked Flush 0 Done Sent Done false
1311 Acked Read 4 Done Sent Done false
1312 Acked Read 8 Done Sent Done false
1313 Acked Read 3 Done Sent Done false
1314 Acked Read 6 Done Sent Done false
1315 Acked Write 4 Done Sent Done false
1316 Acked Write 4 Done Sent Done false
1317 Acked Read 8 Done Sent Done false
1318 Acked Flush 0 Done Sent Done false
1319 Acked Write 4 Done Sent Done false
1320 Acked Write 10 Done Sent Done false
1321 Acked Read 1 Done Sent Done false
1322 Acked Read 5 Done Sent Done false
1323 Acked Write 7 Done Sent Done false
1324 Acked Write 1 Done Sent Done false
1325 Acked Read 9 Done Sent Done false
1326 Acked Read 10 Done Sent Done false
1327 Acked Write 5 Done Sent Done false
1328 Acked Write 7 Done Sent Done false
1329 Acked Read 2 Done Sent Done false
1330 Acked Flush 0 Done Sent Done false
1331 Acked Read 2 Done Sent Done false
1332 Acked Read 10 Done Sent Done false
1333 Acked Read 10 Done Sent Done false
1334 Acked Write 3 Done Sent Done false
1335 Acked Flush 0 Done Sent Done false
1336 Acked Read 3 Done Sent Done false
1337 Acked Read 4 Done Sent Done false
1338 Acked Flush 0 Done Sent Done false
1339 Acked Write 5 Done Sent Done false
1340 Acked Write 6 Done Sent Done false
1341 Acked Flush 0 Done Sent Done false
1342 Acked Read 5 Done Sent Done false
1343 Acked Write 2 Done Sent Done false
1344 Acked Read 7 Done Sent Done false
1345 Acked Read 4 Done Sent Done false
1346 Acked Read 7 Done Sent Done false
1347 Acked Write 3 Done Sent Done false
1348 Acked Flush 0 Done Sent Done false
1349 Acked Read 9 Done Sent Done false
1350 Acked Write 4 Done Sent Done false
1351 Acked Read 2 Done Sent Done false
1352 Acked Read 4 Done Sent Done false
1353 Acked Write 4 Done Sent Done false
1354 Acked Write 9 Done Sent Done false
1355 Acked Write 8 Done Sent Done false
1356 Acked Flush 0 Done Sent Done false
1357 Acked Read 5 Done Sent Done false
1358 Acked Read 7 Done Sent Done false
1359 Acked Write 4 Done Sent Done false
1360 Acked Write 9 Done Sent Done false
1361 Acked Read 7 Done Sent Done false
1362 Acked Flush 0 Done Sent Done false
1363 Acked Read 9 Done Sent Done false
1364 Acked Flush 0 Done Sent Done false
1365 Acked Write 7 Done Sent Done false
1366 Acked Read 3 Done Sent Done false
1367 Acked Read 1 Done Sent Done false
1368 Acked Write 8 Done Sent Done false
1369 Acked Read 10 Done Sent Done false
1370 Acked Write 6 Done Sent Done false
1371 Acked Read 3 Done Sent Done false
1372 Acked Read 6 Done Sent Done false
1373 Acked Flush 0 Done Sent Done false
1374 Acked Read 3 Done Sent Done false
1375 Acked Read 4 Done Sent Done false
1376 Acked Read 8 Done Sent Done false
1377 Acked Read 4 Done Sent Done false
1378 Acked Read 7 Done Sent Done false
1379 Acked Read 6 Done Sent Done false
1380 Acked Write 10 Done Sent Done false
1381 Acked Write 8 Done Sent Done false
1382 Acked Read 2 Done Sent Done false
1383 Acked Read 10 Done Sent Done false
1384 Acked Read 3 Done Sent Done false
1385 Acked Read 4 Done Sent Done false
1386 Acked Read 4 Done Sent Done false
1387 Acked Read 4 Done Sent Done false
1388 Acked Write 1 Done Sent Done false
1389 Acked Read 8 Done Sent Done false
1390 Acked Read 5 Done Sent Done false
1391 Acked Flush 0 Done Sent Done false
1392 Acked Read 2 Done Sent Done false
1393 Acked Read 6 Done Sent Done false
1394 Acked Read 9 Done Sent Done false
1395 Acked Write 2 Done Sent Done false
1396 Acked Read 3 Done Sent Done false
1397 Acked Write 4 Done Sent Done false
1398 Acked Read 10 Done Sent Done false
1399 Acked Read 9 Done Sent Done false
1400 Acked Write 9 Done Sent Done false
1401 Acked Read 2 Done Sent Done false
1402 Acked Write 2 Done Sent Done false
1403 Acked Read 5 Done Sent Done false
1404 Acked Write 6 Done Sent Done false
1405 Acked Read 4 Done Sent Done false
1406 Acked Read 3 Done Sent Done false
1407 Acked Flush 0 Done Sent Done false
1408 Acked Read 10 Done Sent Done false
1409 Acked Read 8 Done Sent Done false
1410 Acked Write 2 Done Sent Done false
1411 Acked Write 6 Done Sent Done false
1412 Acked Write 6 Done Sent Done false
1413 Acked Read 7 Done Sent Done false
1414 Acked Read 1 Done Sent Done false
1415 Acked Read 5 Done Sent Done false
1416 Acked Write 7 Done Sent Done false
1417 Acked Write 10 Done Sent Done false
1418 Acked Read 10 Done Sent Done false
1419 Acked Write 4 Done Sent Done false
1420 Acked Read 9 Done Sent Done false
1421 Acked Read 5 Done Sent Done false
1422 Acked Write 5 Done Sent Done false
1423 Acked Read 4 Done Sent Done false
1424 Acked Write 8 Done Sent Done false
1425 Acked Read 6 Done Sent Done false
1426 Acked Write 10 Done Sent Done false
1427 Acked Read 7 Done Sent Done false
1428 Acked Write 1 Done Sent Done false
1429 Acked Read 4 Done Sent Done false
1430 Acked Flush 0 Done Sent Done false
1431 Acked Read 1 Done Sent Done false
1432 Acked Flush 0 Done Sent Done false
1433 Acked Read 9 Done Sent Done false
1434 Acked Read 6 Done Sent Done false
1435 Acked Write 6 Done Sent Done false
1436 Acked Write 10 Done Sent Done false
1437 Acked Write 6 Done Sent Done false
1438 Acked Read 10 Done Sent Done false
1439 Acked Read 4 Done Sent Done false
1440 Acked Read 7 Done Sent Done false
1441 Acked Write 1 Done Sent Done false
1442 Acked Flush 0 Done Sent Done false
1443 Acked Read 1 Done Sent Done false
1444 Acked Read 5 Done Sent Done false
1445 Acked Write 8 Done Sent Done false
1446 Acked Read 7 Done Sent Done false
1447 Acked Read 8 Done Sent Done false
1448 Acked Write 10 Done Sent Done false
1449 Acked Flush 0 Done Sent Done false
1450 Acked Read 2 Done Sent Done false
1451 Acked Read 2 Done Sent Done false
1452 Acked Write 5 Done Sent Done false
1453 Acked Read 2 Done Sent Done false
1454 Acked Read 1 Done Sent Done false
1455 Acked Read 8 Done Sent Done false
1456 Acked Write 10 Done Sent Done false
1457 Acked Read 7 Done Sent Done false
1458 Acked Flush 0 Done Sent Done false
1459 Acked Read 9 Done Sent Done false
1460 Acked Read 7 Done Sent Done false
1461 Acked Flush 0 Done Sent Done false
1462 Acked Read 7 Done Sent Done false
1463 Acked Read 1 Done Sent Done false
1464 Acked Write 10 Done Sent Done false
1465 Acked Read 4 Done Sent Done false
1466 Acked Write 4 Done Sent Done false
1467 Acked Read 2 Done Sent Done false
1468 Acked Write 1 Done Sent Done false
1469 Acked Read 9 Done Sent Done false
1470 Acked Read 9 Done Sent Done false
1471 Acked Write 10 Done Sent Done false
1472 Acked Write 5 Done Sent Done false
1473 Acked Read 3 Done Sent Done false
1474 Acked Read 2 Done Sent Done false
1475 Acked Read 2 Done Sent Done false
1476 Acked Write 8 Done Sent Done false
1477 Acked Write 6 Done Sent Done false
1478 Acked Write 3 Done Sent Done false
1479 Acked Read 5 Done Sent Done false
1480 Acked Read 3 Done Sent Done false
1481 Acked Write 6 Done Sent Done false
1482 Acked Read 8 Done Sent Done false
1483 Acked Write 10 Done Sent Done false
1484 Acked Read 8 Done Sent Done false
1485 Acked Write 7 Done Sent Done false
1486 Acked Write 8 Done Sent Done false
1487 Acked Read 2 Done Sent Done false
1488 Acked Read 7 Done Sent Done false
1489 Acked Write 4 Done Sent Done false
1490 Acked Write 8 Done Sent Done false
1491 Acked Read 4 Done Sent Done false
1492 Acked Read 1 Done Sent Done false
1493 Acked Read 10 Done Sent Done false
1494 Acked Flush 0 Done Sent Done false
1495 Acked Read 3 Done Sent Done false
1496 Acked Write 3 Done Sent Done false
1497 Acked Read 2 Done Sent Done false
1498 Acked Write 2 Done Sent Done false
1499 Acked Write 4 Done Sent Done false
1500 Acked Flush 0 Done Sent Done false
1501 Acked Read 6 Done Sent Done false
1502 Acked Write 2 Done Sent Done false
1503 Acked Write 10 Done Sent Done false
1504 Acked Read 10 Done Sent Done false
1505 Acked Write 7 Done Sent Done false
1506 Acked Write 2 Done Sent Done false
1507 Acked Write 9 Done Sent Done false
1508 Acked Read 5 Done Sent Done false
1509 Acked Flush 0 Done Sent Done false
1510 Acked Read 1 Done Sent Done false
1511 Acked Write 6 Done Sent Done false
1512 Acked Write 6 Done Sent Done false
1513 Acked Read 7 Done Sent Done false
1514 Acked Read 3 Done Sent Done false
1515 Acked Read 10 Done Sent Done false
1516 Acked Read 3 Done Sent Done false
1517 Acked Write 3 Done Sent Done false
1518 Acked Write 6 Done Sent Done false
1519 Acked Read 4 Done Sent Done false
1520 Acked Read 8 Done Sent Done false
1521 Acked Write 3 Done Sent Done false
1522 Acked Write 1 Done Sent Done false
1523 Acked Read 5 Done Sent Done false
1524 Acked Read 3 Done Sent Done false
1525 Acked Write 9 Done Sent Done false
1526 Acked Write 6 Done Sent Done false
1527 Acked Read 6 Done Sent Done false
1528 Acked Read 5 Done Sent Done false
1529 Acked Write 7 Done Sent Done false
1530 Acked Read 10 Done Sent Done false
1531 Acked Read 2 Done Sent Done false
1532 Acked Read 1 Done Sent Done false
1533 Acked Write 6 Done Sent Done false
1534 Acked Read 7 Done Sent Done false
1535 Acked Read 3 Done Sent Done false
1536 Acked Read 9 Done Sent Done false
1537 Acked Read 8 Done Sent Done false
1538 Acked Read 9 Done Sent Done false
1539 Acked Read 6 Done Sent Done false
1540 Acked Write 2 Done Sent Done false
1541 Acked Read 2 Done Sent Done false
1542 Acked Flush 0 Done Sent Done false
1543 Acked Read 8 Done Sent Done false
1544 Acked Read 7 Done Sent Done false
1545 Acked Read 6 Done Sent Done false
1546 Acked Flush 0 Done Sent Done false
1547 Acked Write 3 Done Sent Done false
1548 Acked Read 6 Done Sent Done false
1549 Acked Read 1 Done Sent Done false
1550 Acked Read 4 Done Sent Done false
1551 Acked Write 4 Done Sent Done false
1552 Acked Write 4 Done Sent Done false
1553 Acked Flush 0 Done Sent Done false
1554 Acked Read 3 Done Sent Done false
1555 Acked Write 4 Done Sent Done false
1556 Acked Read 10 Done Sent Done false
1557 Acked Read 2 Done Sent Done false
1558 Acked Read 1 Done Sent Done false
1559 Acked Read 5 Done Sent Done false
1560 Acked Read 8 Done Sent Done false
1561 Acked Write 3 Done Sent Done false
1562 Acked Write 7 Done Sent Done false
1563 Acked Read 2 Done Sent Done false
1564 Acked Read 4 Done Sent Done false
1565 Acked Read 3 Done Sent Done false
1566 Acked Read 1 Done Sent Done false
1567 Acked Write 4 Done Sent Done false
1568 Acked Read 8 Done Sent Done false
1569 Acked Write 3 Done Sent Done false
1570 Acked Read 9 Done Sent Done false
1571 Acked Read 8 Done Sent Done false
1572 Acked Read 5 Done Sent Done false
1573 Acked Write 10 Done Sent Done false
1574 Acked Read 1 Done Sent Done false
1575 Acked Read 2 Done Sent Done false
1576 Acked Flush 0 Done Sent Done false
1577 Acked Read 1 Done Sent Done false
1578 Acked Write 4 Done Sent Done false
1579 Acked Write 4 Done Sent Done false
1580 Acked Read 2 Done Sent Done false
1581 Acked Read 7 Done Sent Done false
1582 Acked Read 3 Done Sent Done false
1583 Acked Write 5 Done Sent Done false
1584 Acked Read 5 Done Sent Done false
1585 Acked Read 1 Done Sent Done false
1586 Acked Read 3 Done Sent Done false
1587 Acked Write 5 Done Sent Done false
1588 Acked Write 7 Done Sent Done false
1589 Acked Read 1 Done Sent Done false
1590 Acked Read 10 Done Sent Done false
1591 Acked Read 3 Done Sent Done false
1592 Acked Read 1 Done Sent Done false
1593 Acked Read 10 Done Sent Done false
1594 Acked Read 9 Done Sent Done false
1595 Acked Read 7 Done Sent Done false
1596 Acked Flush 0 Done Sent Done false
1597 Acked Read 1 Done Sent Done false
1598 Acked Write 8 Done Sent Done false
1599 Acked Write 2 Done Sent Done false
1600 Acked Read 6 Done Sent Done false
1601 Acked Write 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 1601 1600 1599 1598 1597
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:32.887612078Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:33.389742239Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:34.133562258Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:34.133657075Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:34.133698285Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:34.133735227Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:34.13392899Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:34.635209225Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:35.393389622Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1301 Acked Read 2 Done Sent Done false
1302 Acked Read 6 Done Sent Done false
1303 Acked Write 5 Done Sent Done false
1304 Acked Read 6 Done Sent Done false
1305 Acked Read 4 Done Sent Done false
1306 Acked Read 4 Done Sent Done false
1307 Acked Flush 0 Done Sent Done false
1308 Acked Write 3 Done Sent Done false
1309 Acked Write 6 Done Sent Done false
1310 Acked Flush 0 Done Sent Done false
1311 Acked Read 4 Done Sent Done false
1312 Acked Read 8 Done Sent Done false
1313 Acked Read 3 Done Sent Done false
1314 Acked Read 6 Done Sent Done false
1315 Acked Write 4 Done Sent Done false
1316 Acked Write 4 Done Sent Done false
1317 Acked Read 8 Done Sent Done false
1318 Acked Flush 0 Done Sent Done false
1319 Acked Write 4 Done Sent Done false
1320 Acked Write 10 Done Sent Done false
1321 Acked Read 1 Done Sent Done false
1322 Acked Read 5 Done Sent Done false
1323 Acked Write 7 Done Sent Done false
1324 Acked Write 1 Done Sent Done false
1325 Acked Read 9 Done Sent Done false
1326 Acked Read 10 Done Sent Done false
1327 Acked Write 5 Done Sent Done false
1328 Acked Write 7 Done Sent Done false
1329 Acked Read 2 Done Sent Done false
1330 Acked Flush 0 Done Sent Done false
1331 Acked Read 2 Done Sent Done false
1332 Acked Read 10 Done Sent Done false
1333 Acked Read 10 Done Sent Done false
1334 Acked Write 3 Done Sent Done false
1335 Acked Flush 0 Done Sent Done false
1336 Acked Read 3 Done Sent Done false
1337 Acked Read 4 Done Sent Done false
1338 Acked Flush 0 Done Sent Done false
1339 Acked Write 5 Done Sent Done false
1340 Acked Write 6 Done Sent Done false
1341 Acked Flush 0 Done Sent Done false
1342 Acked Read 5 Done Sent Done false
1343 Acked Write 2 Done Sent Done false
1344 Acked Read 7 Done Sent Done false
1345 Acked Read 4 Done Sent Done false
1346 Acked Read 7 Done Sent Done false
1347 Acked Write 3 Done Sent Done false
1348 Acked Flush 0 Done Sent Done false
1349 Acked Read 9 Done Sent Done false
1350 Acked Write 4 Done Sent Done false
1351 Acked Read 2 Done Sent Done false
1352 Acked Read 4 Done Sent Done false
1353 Acked Write 4 Done Sent Done false
1354 Acked Write 9 Done Sent Done false
1355 Acked Write 8 Done Sent Done false
1356 Acked Flush 0 Done Sent Done false
1357 Acked Read 5 Done Sent Done false
1358 Acked Read 7 Done Sent Done false
1359 Acked Write 4 Done Sent Done false
1360 Acked Write 9 Done Sent Done false
1361 Acked Read 7 Done Sent Done false
1362 Acked Flush 0 Done Sent Done false
1363 Acked Read 9 Done Sent Done false
1364 Acked Flush 0 Done Sent Done false
1365 Acked Write 7 Done Sent Done false
1366 Acked Read 3 Done Sent Done false
1367 Acked Read 1 Done Sent Done false
1368 Acked Write 8 Done Sent Done false
1369 Acked Read 10 Done Sent Done false
1370 Acked Write 6 Done Sent Done false
1371 Acked Read 3 Done Sent Done false
1372 Acked Read 6 Done Sent Done false
1373 Acked Flush 0 Done Sent Done false
1374 Acked Read 3 Done Sent Done false
1375 Acked Read 4 Done Sent Done false
1376 Acked Read 8 Done Sent Done false
1377 Acked Read 4 Done Sent Done false
1378 Acked Read 7 Done Sent Done false
1379 Acked Read 6 Done Sent Done false
1380 Acked Write 10 Done Sent Done false
1381 Acked Write 8 Done Sent Done false
1382 Acked Read 2 Done Sent Done false
1383 Acked Read 10 Done Sent Done false
1384 Acked Read 3 Done Sent Done false
1385 Acked Read 4 Done Sent Done false
1386 Acked Read 4 Done Sent Done false
1387 Acked Read 4 Done Sent Done false
1388 Acked Write 1 Done Sent Done false
1389 Acked Read 8 Done Sent Done false
1390 Acked Read 5 Done Sent Done false
1391 Acked Flush 0 Done Sent Done false
1392 Acked Read 2 Done Sent Done false
1393 Acked Read 6 Done Sent Done false
1394 Acked Read 9 Done Sent Done false
1395 Acked Write 2 Done Sent Done false
1396 Acked Read 3 Done Sent Done false
1397 Acked Write 4 Done Sent Done false
1398 Acked Read 10 Done Sent Done false
1399 Acked Read 9 Done Sent Done false
1400 Acked Write 9 Done Sent Done false
1401 Acked Read 2 Done Sent Done false
1402 Acked Write 2 Done Sent Done false
1403 Acked Read 5 Done Sent Done false
1404 Acked Write 6 Done Sent Done false
1405 Acked Read 4 Done Sent Done false
1406 Acked Read 3 Done Sent Done false
1407 Acked Flush 0 Done Sent Done false
1408 Acked Read 10 Done Sent Done false
1409 Acked Read 8 Done Sent Done false
1410 Acked Write 2 Done Sent Done false
1411 Acked Write 6 Done Sent Done false
1412 Acked Write 6 Done Sent Done false
1413 Acked Read 7 Done Sent Done false
1414 Acked Read 1 Done Sent Done false
1415 Acked Read 5 Done Sent Done false
1416 Acked Write 7 Done Sent Done false
1417 Acked Write 10 Done Sent Done false
1418 Acked Read 10 Done Sent Done false
1419 Acked Write 4 Done Sent Done false
1420 Acked Read 9 Done Sent Done false
1421 Acked Read 5 Done Sent Done false
1422 Acked Write 5 Done Sent Done false
1423 Acked Read 4 Done Sent Done false
1424 Acked Write 8 Done Sent Done false
1425 Acked Read 6 Done Sent Done false
1426 Acked Write 10 Done Sent Done false
1427 Acked Read 7 Done Sent Done false
1428 Acked Write 1 Done Sent Done false
1429 Acked Read 4 Done Sent Done false
1430 Acked Flush 0 Done Sent Done false
1431 Acked Read 1 Done Sent Done false
1432 Acked Flush 0 Done Sent Done false
1433 Acked Read 9 Done Sent Done false
1434 Acked Read 6 Done Sent Done false
1435 Acked Write 6 Done Sent Done false
1436 Acked Write 10 Done Sent Done false
1437 Acked Write 6 Done Sent Done false
1438 Acked Read 10 Done Sent Done false
1439 Acked Read 4 Done Sent Done false
1440 Acked Read 7 Done Sent Done false
1441 Acked Write 1 Done Sent Done false
1442 Acked Flush 0 Done Sent Done false
1443 Acked Read 1 Done Sent Done false
1444 Acked Read 5 Done Sent Done false
1445 Acked Write 8 Done Sent Done false
1446 Acked Read 7 Done Sent Done false
1447 Acked Read 8 Done Sent Done false
1448 Acked Write 10 Done Sent Done false
1449 Acked Flush 0 Done Sent Done false
1450 Acked Read 2 Done Sent Done false
1451 Acked Read 2 Done Sent Done false
1452 Acked Write 5 Done Sent Done false
1453 Acked Read 2 Done Sent Done false
1454 Acked Read 1 Done Sent Done false
1455 Acked Read 8 Done Sent Done false
1456 Acked Write 10 Done Sent Done false
1457 Acked Read 7 Done Sent Done false
1458 Acked Flush 0 Done Sent Done false
1459 Acked Read 9 Done Sent Done false
1460 Acked Read 7 Done Sent Done false
1461 Acked Flush 0 Done Sent Done false
1462 Acked Read 7 Done Sent Done false
1463 Acked Read 1 Done Sent Done false
1464 Acked Write 10 Done Sent Done false
1465 Acked Read 4 Done Sent Done false
1466 Acked Write 4 Done Sent Done false
1467 Acked Read 2 Done Sent Done false
1468 Acked Write 1 Done Sent Done false
1469 Acked Read 9 Done Sent Done false
1470 Acked Read 9 Done Sent Done false
1471 Acked Write 10 Done Sent Done false
1472 Acked Write 5 Done Sent Done false
1473 Acked Read 3 Done Sent Done false
1474 Acked Read 2 Done Sent Done false
1475 Acked Read 2 Done Sent Done false
1476 Acked Write 8 Done Sent Done false
1477 Acked Write 6 Done Sent Done false
1478 Acked Write 3 Done Sent Done false
1479 Acked Read 5 Done Sent Done false
1480 Acked Read 3 Done Sent Done false
1481 Acked Write 6 Done Sent Done false
1482 Acked Read 8 Done Sent Done false
1483 Acked Write 10 Done Sent Done false
1484 Acked Read 8 Done Sent Done false
1485 Acked Write 7 Done Sent Done false
1486 Acked Write 8 Done Sent Done false
1487 Acked Read 2 Done Sent Done false
1488 Acked Read 7 Done Sent Done false
1489 Acked Write 4 Done Sent Done false
1490 Acked Write 8 Done Sent Done false
1491 Acked Read 4 Done Sent Done false
1492 Acked Read 1 Done Sent Done false
1493 Acked Read 10 Done Sent Done false
1494 Acked Flush 0 Done Sent Done false
1495 Acked Read 3 Done Sent Done false
1496 Acked Write 3 Done Sent Done false
1497 Acked Read 2 Done Sent Done false
1498 Acked Write 2 Done Sent Done false
1499 Acked Write 4 Done Sent Done false
1500 Acked Flush 0 Done Sent Done false
1501 Acked Read 6 Done Sent Done false
1502 Acked Write 2 Done Sent Done false
1503 Acked Write 10 Done Sent Done false
1504 Acked Read 10 Done Sent Done false
1505 Acked Write 7 Done Sent Done false
1506 Acked Write 2 Done Sent Done false
1507 Acked Write 9 Done Sent Done false
1508 Acked Read 5 Done Sent Done false
1509 Acked Flush 0 Done Sent Done false
1510 Acked Read 1 Done Sent Done false
1511 Acked Write 6 Done Sent Done false
1512 Acked Write 6 Done Sent Done false
1513 Acked Read 7 Done Sent Done false
1514 Acked Read 3 Done Sent Done false
1515 Acked Read 10 Done Sent Done false
1516 Acked Read 3 Done Sent Done false
1517 Acked Write 3 Done Sent Done false
1518 Acked Write 6 Done Sent Done false
1519 Acked Read 4 Done Sent Done false
1520 Acked Read 8 Done Sent Done false
1521 Acked Write 3 Done Sent Done false
1522 Acked Write 1 Done Sent Done false
1523 Acked Read 5 Done Sent Done false
1524 Acked Read 3 Done Sent Done false
1525 Acked Write 9 Done Sent Done false
1526 Acked Write 6 Done Sent Done false
1527 Acked Read 6 Done Sent Done false
1528 Acked Read 5 Done Sent Done false
1529 Acked Write 7 Done Sent Done false
1530 Acked Read 10 Done Sent Done false
1531 Acked Read 2 Done Sent Done false
1532 Acked Read 1 Done Sent Done false
1533 Acked Write 6 Done Sent Done false
1534 Acked Read 7 Done Sent Done false
1535 Acked Read 3 Done Sent Done false
1536 Acked Read 9 Done Sent Done false
1537 Acked Read 8 Done Sent Done false
1538 Acked Read 9 Done Sent Done false
1539 Acked Read 6 Done Sent Done false
1540 Acked Write 2 Done Sent Done false
1541 Acked Read 2 Done Sent Done false
1542 Acked Flush 0 Done Sent Done false
1543 Acked Read 8 Done Sent Done false
1544 Acked Read 7 Done Sent Done false
1545 Acked Read 6 Done Sent Done false
1546 Acked Flush 0 Done Sent Done false
1547 Acked Write 3 Done Sent Done false
1548 Acked Read 6 Done Sent Done false
1549 Acked Read 1 Done Sent Done false
1550 Acked Read 4 Done Sent Done false
1551 Acked Write 4 Done Sent Done false
1552 Acked Write 4 Done Sent Done false
1553 Acked Flush 0 Done Sent Done false
1554 Acked Read 3 Done Sent Done false
1555 Acked Write 4 Done Sent Done false
1556 Acked Read 10 Done Sent Done false
1557 Acked Read 2 Done Sent Done false
1558 Acked Read 1 Done Sent Done false
1559 Acked Read 5 Done Sent Done false
1560 Acked Read 8 Done Sent Done false
1561 Acked Write 3 Done Sent Done false
1562 Acked Write 7 Done Sent Done false
1563 Acked Read 2 Done Sent Done false
1564 Acked Read 4 Done Sent Done false
1565 Acked Read 3 Done Sent Done false
1566 Acked Read 1 Done Sent Done false
1567 Acked Write 4 Done Sent Done false
1568 Acked Read 8 Done Sent Done false
1569 Acked Write 3 Done Sent Done false
1570 Acked Read 9 Done Sent Done false
1571 Acked Read 8 Done Sent Done false
1572 Acked Read 5 Done Sent Done false
1573 Acked Write 10 Done Sent Done false
1574 Acked Read 1 Done Sent Done false
1575 Acked Read 2 Done Sent Done false
1576 Acked Flush 0 Done Sent Done false
1577 Acked Read 1 Done Sent Done false
1578 Acked Write 4 Done Sent Done false
1579 Acked Write 4 Done Sent Done false
1580 Acked Read 2 Done Sent Done false
1581 Acked Read 7 Done Sent Done false
1582 Acked Read 3 Done Sent Done false
1583 Acked Write 5 Done Sent Done false
1584 Acked Read 5 Done Sent Done false
1585 Acked Read 1 Done Sent Done false
1586 Acked Read 3 Done Sent Done false
1587 Acked Write 5 Done Sent Done false
1588 Acked Write 7 Done Sent Done false
1589 Acked Read 1 Done Sent Done false
1590 Acked Read 10 Done Sent Done false
1591 Acked Read 3 Done Sent Done false
1592 Acked Read 1 Done Sent Done false
1593 Acked Read 10 Done Sent Done false
1594 Acked Read 9 Done Sent Done false
1595 Acked Read 7 Done Sent Done false
1596 Acked Flush 0 Done Sent Done false
1597 Acked Read 1 Done Sent Done false
1598 Acked Write 8 Done Sent Done false
1599 Acked Write 2 Done Sent Done false
1600 Acked Read 6 Done Sent Done false
1601 Acked Write 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 1601 1600 1599 1598 1597
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:36.889796928Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:37.396632952Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:38.3975706Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:39.399462921Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1301 Acked Read 2 Done Sent Done false
1302 Acked Read 6 Done Sent Done false
1303 Acked Write 5 Done Sent Done false
1304 Acked Read 6 Done Sent Done false
1305 Acked Read 4 Done Sent Done false
1306 Acked Read 4 Done Sent Done false
1307 Acked Flush 0 Done Sent Done false
1308 Acked Write 3 Done Sent Done false
1309 Acked Write 6 Done Sent Done false
1310 Acked Flush 0 Done Sent Done false
1311 Acked Read 4 Done Sent Done false
1312 Acked Read 8 Done Sent Done false
1313 Acked Read 3 Done Sent Done false
1314 Acked Read 6 Done Sent Done false
1315 Acked Write 4 Done Sent Done false
1316 Acked Write 4 Done Sent Done false
1317 Acked Read 8 Done Sent Done false
1318 Acked Flush 0 Done Sent Done false
1319 Acked Write 4 Done Sent Done false
1320 Acked Write 10 Done Sent Done false
1321 Acked Read 1 Done Sent Done false
1322 Acked Read 5 Done Sent Done false
1323 Acked Write 7 Done Sent Done false
1324 Acked Write 1 Done Sent Done false
1325 Acked Read 9 Done Sent Done false
1326 Acked Read 10 Done Sent Done false
1327 Acked Write 5 Done Sent Done false
1328 Acked Write 7 Done Sent Done false
1329 Acked Read 2 Done Sent Done false
1330 Acked Flush 0 Done Sent Done false
1331 Acked Read 2 Done Sent Done false
1332 Acked Read 10 Done Sent Done false
1333 Acked Read 10 Done Sent Done false
1334 Acked Write 3 Done Sent Done false
1335 Acked Flush 0 Done Sent Done false
1336 Acked Read 3 Done Sent Done false
1337 Acked Read 4 Done Sent Done false
1338 Acked Flush 0 Done Sent Done false
1339 Acked Write 5 Done Sent Done false
1340 Acked Write 6 Done Sent Done false
1341 Acked Flush 0 Done Sent Done false
1342 Acked Read 5 Done Sent Done false
1343 Acked Write 2 Done Sent Done false
1344 Acked Read 7 Done Sent Done false
1345 Acked Read 4 Done Sent Done false
1346 Acked Read 7 Done Sent Done false
1347 Acked Write 3 Done Sent Done false
1348 Acked Flush 0 Done Sent Done false
1349 Acked Read 9 Done Sent Done false
1350 Acked Write 4 Done Sent Done false
1351 Acked Read 2 Done Sent Done false
1352 Acked Read 4 Done Sent Done false
1353 Acked Write 4 Done Sent Done false
1354 Acked Write 9 Done Sent Done false
1355 Acked Write 8 Done Sent Done false
1356 Acked Flush 0 Done Sent Done false
1357 Acked Read 5 Done Sent Done false
1358 Acked Read 7 Done Sent Done false
1359 Acked Write 4 Done Sent Done false
1360 Acked Write 9 Done Sent Done false
1361 Acked Read 7 Done Sent Done false
1362 Acked Flush 0 Done Sent Done false
1363 Acked Read 9 Done Sent Done false
1364 Acked Flush 0 Done Sent Done false
1365 Acked Write 7 Done Sent Done false
1366 Acked Read 3 Done Sent Done false
1367 Acked Read 1 Done Sent Done false
1368 Acked Write 8 Done Sent Done false
1369 Acked Read 10 Done Sent Done false
1370 Acked Write 6 Done Sent Done false
1371 Acked Read 3 Done Sent Done false
1372 Acked Read 6 Done Sent Done false
1373 Acked Flush 0 Done Sent Done false
1374 Acked Read 3 Done Sent Done false
1375 Acked Read 4 Done Sent Done false
1376 Acked Read 8 Done Sent Done false
1377 Acked Read 4 Done Sent Done false
1378 Acked Read 7 Done Sent Done false
1379 Acked Read 6 Done Sent Done false
1380 Acked Write 10 Done Sent Done false
1381 Acked Write 8 Done Sent Done false
1382 Acked Read 2 Done Sent Done false
1383 Acked Read 10 Done Sent Done false
1384 Acked Read 3 Done Sent Done false
1385 Acked Read 4 Done Sent Done false
1386 Acked Read 4 Done Sent Done false
1387 Acked Read 4 Done Sent Done false
1388 Acked Write 1 Done Sent Done false
1389 Acked Read 8 Done Sent Done false
1390 Acked Read 5 Done Sent Done false
1391 Acked Flush 0 Done Sent Done false
1392 Acked Read 2 Done Sent Done false
1393 Acked Read 6 Done Sent Done false
1394 Acked Read 9 Done Sent Done false
1395 Acked Write 2 Done Sent Done false
1396 Acked Read 3 Done Sent Done false
1397 Acked Write 4 Done Sent Done false
1398 Acked Read 10 Done Sent Done false
1399 Acked Read 9 Done Sent Done false
1400 Acked Write 9 Done Sent Done false
1401 Acked Read 2 Done Sent Done false
1402 Acked Write 2 Done Sent Done false
1403 Acked Read 5 Done Sent Done false
1404 Acked Write 6 Done Sent Done false
1405 Acked Read 4 Done Sent Done false
1406 Acked Read 3 Done Sent Done false
1407 Acked Flush 0 Done Sent Done false
1408 Acked Read 10 Done Sent Done false
1409 Acked Read 8 Done Sent Done false
1410 Acked Write 2 Done Sent Done false
1411 Acked Write 6 Done Sent Done false
1412 Acked Write 6 Done Sent Done false
1413 Acked Read 7 Done Sent Done false
1414 Acked Read 1 Done Sent Done false
1415 Acked Read 5 Done Sent Done false
1416 Acked Write 7 Done Sent Done false
1417 Acked Write 10 Done Sent Done false
1418 Acked Read 10 Done Sent Done false
1419 Acked Write 4 Done Sent Done false
1420 Acked Read 9 Done Sent Done false
1421 Acked Read 5 Done Sent Done false
1422 Acked Write 5 Done Sent Done false
1423 Acked Read 4 Done Sent Done false
1424 Acked Write 8 Done Sent Done false
1425 Acked Read 6 Done Sent Done false
1426 Acked Write 10 Done Sent Done false
1427 Acked Read 7 Done Sent Done false
1428 Acked Write 1 Done Sent Done false
1429 Acked Read 4 Done Sent Done false
1430 Acked Flush 0 Done Sent Done false
1431 Acked Read 1 Done Sent Done false
1432 Acked Flush 0 Done Sent Done false
1433 Acked Read 9 Done Sent Done false
1434 Acked Read 6 Done Sent Done false
1435 Acked Write 6 Done Sent Done false
1436 Acked Write 10 Done Sent Done false
1437 Acked Write 6 Done Sent Done false
1438 Acked Read 10 Done Sent Done false
1439 Acked Read 4 Done Sent Done false
1440 Acked Read 7 Done Sent Done false
1441 Acked Write 1 Done Sent Done false
1442 Acked Flush 0 Done Sent Done false
1443 Acked Read 1 Done Sent Done false
1444 Acked Read 5 Done Sent Done false
1445 Acked Write 8 Done Sent Done false
1446 Acked Read 7 Done Sent Done false
1447 Acked Read 8 Done Sent Done false
1448 Acked Write 10 Done Sent Done false
1449 Acked Flush 0 Done Sent Done false
1450 Acked Read 2 Done Sent Done false
1451 Acked Read 2 Done Sent Done false
1452 Acked Write 5 Done Sent Done false
1453 Acked Read 2 Done Sent Done false
1454 Acked Read 1 Done Sent Done false
1455 Acked Read 8 Done Sent Done false
1456 Acked Write 10 Done Sent Done false
1457 Acked Read 7 Done Sent Done false
1458 Acked Flush 0 Done Sent Done false
1459 Acked Read 9 Done Sent Done false
1460 Acked Read 7 Done Sent Done false
1461 Acked Flush 0 Done Sent Done false
1462 Acked Read 7 Done Sent Done false
1463 Acked Read 1 Done Sent Done false
1464 Acked Write 10 Done Sent Done false
1465 Acked Read 4 Done Sent Done false
1466 Acked Write 4 Done Sent Done false
1467 Acked Read 2 Done Sent Done false
1468 Acked Write 1 Done Sent Done false
1469 Acked Read 9 Done Sent Done false
1470 Acked Read 9 Done Sent Done false
1471 Acked Write 10 Done Sent Done false
1472 Acked Write 5 Done Sent Done false
1473 Acked Read 3 Done Sent Done false
1474 Acked Read 2 Done Sent Done false
1475 Acked Read 2 Done Sent Done false
1476 Acked Write 8 Done Sent Done false
1477 Acked Write 6 Done Sent Done false
1478 Acked Write 3 Done Sent Done false
1479 Acked Read 5 Done Sent Done false
1480 Acked Read 3 Done Sent Done false
1481 Acked Write 6 Done Sent Done false
1482 Acked Read 8 Done Sent Done false
1483 Acked Write 10 Done Sent Done false
1484 Acked Read 8 Done Sent Done false
1485 Acked Write 7 Done Sent Done false
1486 Acked Write 8 Done Sent Done false
1487 Acked Read 2 Done Sent Done false
1488 Acked Read 7 Done Sent Done false
1489 Acked Write 4 Done Sent Done false
1490 Acked Write 8 Done Sent Done false
1491 Acked Read 4 Done Sent Done false
1492 Acked Read 1 Done Sent Done false
1493 Acked Read 10 Done Sent Done false
1494 Acked Flush 0 Done Sent Done false
1495 Acked Read 3 Done Sent Done false
1496 Acked Write 3 Done Sent Done false
1497 Acked Read 2 Done Sent Done false
1498 Acked Write 2 Done Sent Done false
1499 Acked Write 4 Done Sent Done false
1500 Acked Flush 0 Done Sent Done false
1501 Acked Read 6 Done Sent Done false
1502 Acked Write 2 Done Sent Done false
1503 Acked Write 10 Done Sent Done false
1504 Acked Read 10 Done Sent Done false
1505 Acked Write 7 Done Sent Done false
1506 Acked Write 2 Done Sent Done false
1507 Acked Write 9 Done Sent Done false
1508 Acked Read 5 Done Sent Done false
1509 Acked Flush 0 Done Sent Done false
1510 Acked Read 1 Done Sent Done false
1511 Acked Write 6 Done Sent Done false
1512 Acked Write 6 Done Sent Done false
1513 Acked Read 7 Done Sent Done false
1514 Acked Read 3 Done Sent Done false
1515 Acked Read 10 Done Sent Done false
1516 Acked Read 3 Done Sent Done false
1517 Acked Write 3 Done Sent Done false
1518 Acked Write 6 Done Sent Done false
1519 Acked Read 4 Done Sent Done false
1520 Acked Read 8 Done Sent Done false
1521 Acked Write 3 Done Sent Done false
1522 Acked Write 1 Done Sent Done false
1523 Acked Read 5 Done Sent Done false
1524 Acked Read 3 Done Sent Done false
1525 Acked Write 9 Done Sent Done false
1526 Acked Write 6 Done Sent Done false
1527 Acked Read 6 Done Sent Done false
1528 Acked Read 5 Done Sent Done false
1529 Acked Write 7 Done Sent Done false
1530 Acked Read 10 Done Sent Done false
1531 Acked Read 2 Done Sent Done false
1532 Acked Read 1 Done Sent Done false
1533 Acked Write 6 Done Sent Done false
1534 Acked Read 7 Done Sent Done false
1535 Acked Read 3 Done Sent Done false
1536 Acked Read 9 Done Sent Done false
1537 Acked Read 8 Done Sent Done false
1538 Acked Read 9 Done Sent Done false
1539 Acked Read 6 Done Sent Done false
1540 Acked Write 2 Done Sent Done false
1541 Acked Read 2 Done Sent Done false
1542 Acked Flush 0 Done Sent Done false
1543 Acked Read 8 Done Sent Done false
1544 Acked Read 7 Done Sent Done false
1545 Acked Read 6 Done Sent Done false
1546 Acked Flush 0 Done Sent Done false
1547 Acked Write 3 Done Sent Done false
1548 Acked Read 6 Done Sent Done false
1549 Acked Read 1 Done Sent Done false
1550 Acked Read 4 Done Sent Done false
1551 Acked Write 4 Done Sent Done false
1552 Acked Write 4 Done Sent Done false
1553 Acked Flush 0 Done Sent Done false
1554 Acked Read 3 Done Sent Done false
1555 Acked Write 4 Done Sent Done false
1556 Acked Read 10 Done Sent Done false
1557 Acked Read 2 Done Sent Done false
1558 Acked Read 1 Done Sent Done false
1559 Acked Read 5 Done Sent Done false
1560 Acked Read 8 Done Sent Done false
1561 Acked Write 3 Done Sent Done false
1562 Acked Write 7 Done Sent Done false
1563 Acked Read 2 Done Sent Done false
1564 Acked Read 4 Done Sent Done false
1565 Acked Read 3 Done Sent Done false
1566 Acked Read 1 Done Sent Done false
1567 Acked Write 4 Done Sent Done false
1568 Acked Read 8 Done Sent Done false
1569 Acked Write 3 Done Sent Done false
1570 Acked Read 9 Done Sent Done false
1571 Acked Read 8 Done Sent Done false
1572 Acked Read 5 Done Sent Done false
1573 Acked Write 10 Done Sent Done false
1574 Acked Read 1 Done Sent Done false
1575 Acked Read 2 Done Sent Done false
1576 Acked Flush 0 Done Sent Done false
1577 Acked Read 1 Done Sent Done false
1578 Acked Write 4 Done Sent Done false
1579 Acked Write 4 Done Sent Done false
1580 Acked Read 2 Done Sent Done false
1581 Acked Read 7 Done Sent Done false
1582 Acked Read 3 Done Sent Done false
1583 Acked Write 5 Done Sent Done false
1584 Acked Read 5 Done Sent Done false
1585 Acked Read 1 Done Sent Done false
1586 Acked Read 3 Done Sent Done false
1587 Acked Write 5 Done Sent Done false
1588 Acked Write 7 Done Sent Done false
1589 Acked Read 1 Done Sent Done false
1590 Acked Read 10 Done Sent Done false
1591 Acked Read 3 Done Sent Done false
1592 Acked Read 1 Done Sent Done false
1593 Acked Read 10 Done Sent Done false
1594 Acked Read 9 Done Sent Done false
1595 Acked Read 7 Done Sent Done false
1596 Acked Flush 0 Done Sent Done false
1597 Acked Read 1 Done Sent Done false
1598 Acked Write 8 Done Sent Done false
1599 Acked Write 2 Done Sent Done false
1600 Acked Read 6 Done Sent Done false
1601 Acked Write 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 1601 1600 1599 1598 1597
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:40.891038654Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:41.402719266Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:42.403872473Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:43.405160807Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.134868388Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47964)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.135073349Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136124087Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136673796Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.13669558Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(1300))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136712326Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136728292Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(1300))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136744368Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(1300))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:44.136956748Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1597 Acked Read 1 Done Done Done true
1598 Acked Write 8 Done Done Done true
1599 Acked Write 2 Done Done Done true
1600 Acked Read 6 Done Done Done true
1601 Acked Write 4 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 5 5 5 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1596)) Some(JobId(1596)) Some(JobId(1596))
Downstairs last five completed: 1596 1595 1594 1593 1592
Upstairs last five completed: 1601 1600 1599 1598 1597
CLIENT: Up:0 ds:5 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1602 1601 1600 1599 1598
Upstairs last five completed: 1602 1601 1600 1599 1598
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:01:48.393 INFO request completed, latency_us: 363, [8820] Got stop action so:false kr:false
response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2cbcbefd-ad05-4664-8466-bf54009c63d9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:48.393702831Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:48.393756448Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:48.393778912Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:01:48.393 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7f6b3f5f-ae30-40ae-b541-595af776e11a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:48.393888306Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:01:52.396 INFO request completed, latency_us: 353, response_code: 200, uri: /state/cid/1, method: GET, req_id: 21708c3b-f2ef-44e9-a1e6-8770bbd43775, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 4415 len 36864 data: 2 2 2 2 2 2 2 2 2
002/300 Write block 3108 len 40960 data: 2 2 3 3 3 3 4 4 4 4
003/300 Write block 4647 len 4096 data: 2
004/300 Write block 978 len 24576 data: 2 2 2 2 2 2
005/300 Read block 2590 len 32768
006/300 Read block 1008 len 24576
007/300 Read block 4306 len 4096
008/300 Write block 2257 len 24576 data: 2 2 3 3 2 2
009/300 Write block 3665 len 24576 data: 2 2 2 2 2 2
010/300 Read block 87 len 36864
011/300 Write block 1554 len 32768 data: 2 2 2 2 2 2 2 2
012/300 Write block 3623 len 20480 data: 2 2 2 2 2
013/300 Write block 3907 len 20480 data: 2 2 2 2 2
014/300 Write block 3027 len 32768 data: 2 2 2 2 2 2 2 2
015/300 Read block 767 len 8192
016/300 Flush
017/300 Flush
018/300 Write block 4475 len 20480 data: 2 2 2 2 2
019/300 Read block 2841 len 40960
020/300 Read block 3723 len 40960
021/300 Read block 659 len 12288
022/300 Read block 3468 len 36864
023/300 Read block 1850 len 12288
024/300 Read block 2820 len 40960
025/300 Write block 2364 len 4096 data: 3
026/300 Read block 3191 len 4096
027/300 Write block 2128 len 28672 data: 2 2 2 2 2 2 2
028/300 Read block 3329 len 12288
029/300 Write block 4405 len 40960 data: 3 3 4 5 5 5 5 5 3 2
030/300 Write block 2209 len 40960 data: 2 2 2 2 2 2 2 2 2 2
031/300 Read block 2200 len 36864
032/300 Read block 3284 len 32768
033/300 Read block 2740 len 8192
034/300 Flush
035/300 Write block 3611 len 28672 data: 2 2 2 2 2 2 2
036/300 Flush
037/300 Read block 2444 len 8192
038/300 Read block 1060 len 12288
039/300 Flush
040/300 Write block 422 len 12288 data: 3 2 2
041/300 Read block 231 len 12288
042/300 Write block 2630 len 12288 data: 2 2 2
043/300 Write block 2108 len 28672 data: 2 2 2 2 2 2 2
044/300 Read block 1554 len 24576
045/300 Flush
046/300 Read block 2554 len 12288
047/300 Write block 151 len 20480 data: 2 2 2 2 2
048/300 Write block 4295 len 12288 data: 2 2 2
049/300 Write block 3751 len 12288 data: 2 2 2
050/300 Read block 3313 len 12288
051/300 Flush
052/300 Flush
053/300 Read block 2928 len 24576
054/300 Write block 1324 len 32768 data: 2 2 2 2 2 2 2 2
055/300 Read block 1057 len 24576
056/300 Write block 4808 len 16384 data: 2 2 2 2
057/300 Flush
058/300 Flush
059/300 Write block 172 len 24576 data: 2 2 2 2 2 2
060/300 Read block 4040 len 32768
061/300 Read block 1105 len 28672
062/300 Write block 2275 len 36864 data: 2 2 2 2 2 2 2 2 2
063/300 Flush
064/300 Read block 252 len 32768
065/300 Write block 2622 len 40960 data: 2 2 3 2 2 2 2 2 3 3
066/300 Write block 1125 len 12288 data: 2 2 2
067/300 Write block 4493 len 36864 data: 2 2 2 2 2 2 2 2 2
068/300 Read block 1429 len 32768
069/300 Write block 1706 len 40960 data: 2 2 2 2 2 2 2 2 2 2
070/300 Read block 685 len 36864
071/300 Write block 3430 len 4096 data: 2
072/300 Read block 4669 len 40960
073/300 Write block 785 len 12288 data: 3 2 2
074/300 Write block 3648 len 24576 data: 2 2 2 2 2 2
075/300 Read block 1615 len 16384
076/300 Read block 4655 len 36864
077/300 Read block 2011 len 8192
078/300 Read block 1066 len 24576
079/300 Flush
080/300 Read block 1165 len 24576
081/300 Read block 806 len 12288
082/300 Write block 4112 len 24576 data: 2 2 2 2 2 2
083/300 Read block 4307 len 24576
084/300 Write block 3269 len 16384 data: 2 2 2 2
085/300 Write block 3852 len 24576 data: 2 2 2 2 2 2
086/300 Write block 1308 len 8192 data: 2 2
087/300 Read block 4429 len 28672
088/300 Read block 2590 len 24576
089/300 Read block 3866 len 32768
090/300 Write block 2794 len 32768 data: 2 2 2 2 2 3 3 3
091/300 Write block 4122 len 20480 data: 2 2 2 2 2
092/300 Write block 1620 len 8192 data: 3 2
093/300 Write block 402 len 36864 data: 2 2 2 2 2 2 2 2 2
094/300 Write block 1561 len 8192 data: 3 2
095/300 Write block 2740 len 36864 data: 2 2 2 2 2 2 2 2 2
096/300 Read block 4567 len 8192
097/300 Write block 902 len 4096 data: 2
098/300 Write block 2107 len 8192 data: 2 3
099/300 Write block 81 len 12288 data: 2 2 2
100/300 Read block 2928 len 8192
101/300 Read block 3396 len 12288
102/300 Write block 2064 len 36864 data: 2 2 2 2 2 2 2 2 2
103/300 Write block 2362 len 40960 data: 2 3 4 3 3 2 2 2 2 2
104/300 Read block 685 len 40960
105/300 Read block 4860 len 16384
106/300 Write block 4613 len 12288 data: 2 2 2
107/300 Read block 1420 len 16384
108/300 Read block 4968 len 40960
109/300 Write block 816 len 16384 data: 2 2 2 2
110/300 Write block 4441 len 20480 data: 2 2 2 2 2
111/300 Write block 3029 len 20480 data: 3 3 3 3 3
112/300 Write block 2981 len 16384 data: 2 2 2 2
113/300 Write block 455 len 8192 data: 2 2
114/300 Flush
115/300 Flush
116/300 Write block 3869 len 40960 data: 2 2 2 2 3 3 3 3 2 2
117/300 Write block 3480 len 40960 data: 2 2 2 2 2 2 2 2 2 2
118/300 Flush
119/300 Write block 2376 len 36864 data: 3 3 3 2 2 2 2 2 2
120/300 Read block 658 len 28672
121/300 Read block 4249 len 24576
122/300 Write block 4079 len 4096 data: 2
123/300 Write block 2563 len 8192 data: 2 2
124/300 Write block 3800 len 28672 data: 3 3 2 2 2 2 2
125/300 Read block 1044 len 4096
126/300 Read block 2928 len 36864
127/300 Read block 4261 len 24576
128/300 Flush
129/300 Write block 4048 len 12288 data: 2 2 2
130/300 Read block 1799 len 36864
131/300 Write block 1624 len 32768 data: 2 2 2 2 2 2 2 2
132/300 Flush
133/300 Write block 3741 len 20480 data: 2 2 2 2 2
134/300 Read block 1481 len 40960
135/300 Write block 3605 len 24576 data: 2 2 2 2 2 2
136/300 Read block 4106 len 4096
137/300 Read block 2367 len 24576
138/300 Read block 814 len 24576
139/300 Read block 974 len 40960
140/300 Write block 1624 len 28672 data: 3 3 3 3 3 3 3
141/300 Write block 442 len 40960 data: 2 2 2 2 2 2 2 2 2 2
142/300 Write block 470 len 36864 data: 2 2 2 2 2 2 2 2 2
143/300 Flush
144/300 Read block 2397 len 24576
145/300 Flush
146/300 Write block 1319 len 20480 data: 2 2 2 2 2
147/300 Read block 2524 len 28672
148/300 Read block 3043 len 40960
149/300 Read block 3185 len 28672
150/300 Write block 2230 len 8192 data: 3 4
151/300 Write block 4364 len 40960 data: 2 2 2 2 2 2 2 2 2 2
152/300 Write block 2687 len 20480 data: 2 2 2 2 2
153/300 Read block 3149 len 12288
154/300 Read block 3239 len 20480
155/300 Flush
156/300 Write block 2962 len 32768 data: 3 3 3 3 3 3 3 3
157/300 Flush
158/300 Read block 3406 len 32768
159/300 Read block 4930 len 24576
160/300 Write block 4631 len 16384 data: 2 2 2 2
161/300 Read block 1751 len 4096
162/300 Read block 1777 len 12288
163/300 Write block 469 len 8192 data: 2 3
164/300 Write block 2689 len 40960 data: 3 3 3 2 2 2 2 2 2 2
165/300 Write block 436 len 40960 data: 2 2 3 3 3 2 3 3 3 3
166/300 Read block 1904 len 28672
167/300 Read block 2988 len 8192
168/300 Write block 3341 len 24576 data: 4 3 3 3 3 3
169/300 Flush
170/300 Read block 1653 len 32768
171/300 Read block 4797 len 40960
172/300 Write block 2871 len 16384 data: 2 2 2 2
173/300 Write block 2662 len 24576 data: 2 2 2 2 2 2
174/300 Flush
175/300 Flush
176/300 Read block 1746 len 32768
177/300 Write block 2516 len 40960 data: 3 3 3 3 3 3 3 3 3 3
178/300 Read block 3668 len 20480
179/300 Write block 1040 len 24576 data: 3 3 3 3 3 3
180/300 Read block 454 len 4096
181/300 Write block 1016 len 28672 data: 2 2 2 2 2 2 2
182/300 Read block 345 len 40960
183/300 Read block 3468 len 20480
184/300 Read block 4046 len 24576
185/300 Read block 4199 len 8192
186/300 Read block 3564 len 28672
187/300 Read block 4606 len 4096
188/300 Read block 4911 len 20480
189/300 Read block 3705 len 12288
190/300 Write block 4145 len 28672 data: 2 2 2 2 2 2 2
191/300 Read block 4865 len 8192
192/300 Write block 2356 len 36864 data: 2 2 2 2 2 2 3 4 5
193/300 Read block 134 len 20480
194/300 Read block 4959 len 28672
195/300 Flush
196/300 Write block 667 len 36864 data: 2 2 2 2 2 2 2 2 2
197/300 Write block 4095 len 4096 data: 2
198/300 Write block 4062 len 32768 data: 2 2 2 2 2 2 2 2
199/300 Flush
200/300 Read block 2004 len 28672
201/300 Flush
202/300 Read block 133 len 28672
203/300 Write block 1399 len 28672 data: 2 2 2 3 3 3 3
204/300 Write block 2510 len 20480 data: 2 3 3 3 3
205/300 Read block 2930 len 12288
206/300 Write block 532 len 12288 data: 2 2 2
207/300 Flush
208/300 Write block 4068 len 20480 data: 3 3 2 2 2
209/300 Read block 1027 len 24576
210/300 Read block 3736 len 36864
211/300 Write block 547 len 40960 data: 2 2 2 2 3 4 4 2 2 2
212/300 Flush
213/300 Write block 1261 len 20480 data: 2 2 2 2 2
214/300 Read block 1382 len 28672
215/300 Write block 1894 len 8192 data: 3 3
216/300 Write block 667 len 32768 data: 3 3 3 3 3 3 3 3
217/300 Read block 549 len 20480
218/300 Read block 4521 len 16384
219/300 Read block 1288 len 20480
220/300 Write block 796 len 12288 data: 2 2 2
221/300 Read block 2508 len 24576
222/300 Write block 1578 len 32768 data: 3 3 3 4 4 4 4 4
223/300 Read block 1666 len 4096
224/300 Read block 2010 len 32768
225/300 Read block 3964 len 16384
226/300 Read block 4755 len 40960
227/300 Read block 2324 len 28672
228/300 Write block 1988 len 20480 data: 2 2 2 2 2
229/300 Write block 1654 len 4096 data: 2
230/300 Read block 1057 len 36864
231/300 Write block 885 len 32768 data: 2 2 3 3 2 2 2 2
232/300 Write block 156 len 4096 data: 2
233/300 Read block 2137 len 4096
234/300 Write block 4930 len 24576 data: 2 2 2 2 2 2
235/300 Read block 3196 len 8192
236/300 Read block 659 len 40960
237/300 Read block 1626 len 4096
238/300 Read block 3715 len 40960
239/300 Read block 1225 len 24576
240/300 Write block 4625 len 20480 data: 2 2 2 2 2
241/300 Write block 4127 len 20480 data: 2 2 2 2 2
242/300 Read block 3771 len 8192
243/300 Write block 1063 len 4096 data: 2
244/300 Read block 4975 len 32768
245/300 Read block 4167 len 8192
246/300 Write block 2179 len 20480 data: 2 2 2 2 2
247/300 Write block 4769 len 12288 data: 3 3 3
248/300 Read block 1202 len 16384
249/300 Write block 3633 len 28672 data: 2 2 2 2 2 2 2
250/300 Write block 377 len 28672 data: 3 4 4 4 3 3 3
251/300 Read block 3663 len 12288
252/300 Write block 341 len 40960 data: 2 2 2 2 2 2 2 2 2 2
253/300 Write block 430 len 28672 data: 3 3 3 3 2 2 3
254/300 Read block 1946 len 28672
255/300 Read block 4362 len 8192
256/300 Write block 924 len 20480 data: 2 2 2 2 2
257/300 Write block 4623 len 8192 data: 2 2
258/300 Read block 4555 len 12288
259/300 Read block 2782 len 8192
260/300 Read block 4397 len 16384
261/300 Write block 805 len 36864 data: 2 2 2 2 2 2 2 3 3
262/300 Flush
263/300 Write block 2316 len 32768 data: 2 3 3 3 2 2 2 2
264/300 Flush
265/300 Read block 1304 len 32768
266/300 Flush
267/300 Read block 977 len 24576
268/300 Read block 2392 len 28672
269/300 Read block 4580 len 28672
270/300 Write block 2646 len 24576 data: 2 2 2 2 2 2
271/300 Read block 2216 len 40960
272/300 Write block 2019 len 24576 data: 2 2 3 2 2 2
273/300 Flush
274/300 Write block 2885 len 32768 data: 2 2 2 2 2 2 2 2
275/300 Write block 3452 len 32768 data: 2 2 2 2 2 2 2 2
276/300 Write block 930 len 24576 data: 3 3 3 2 2 2
277/300 Write block 983 len 36864 data: 3 2 2 2 2 2 2 2 2
278/300 Read block 1507 len 12288
279/300 Read block 2443 len 16384
280/300 Read block 2667 len 16384
281/300 Read block 2691 len 28672
282/300 Read block 4994 len 16384
283/300 Read block 4918 len 36864
284/300 Write block 1694 len 32768 data: 3 3 3 3 2 2 2 2
285/300 Write block 853 len 12288 data: 2 2 2
286/300 Read block 4291 len 4096
287/300 Read block 1536 len 8192
288/300 Write block 468 len 12288 data: 2 3 4
289/300 Read block 2397 len 4096
290/300 Write block 4495 len 28672 data: 3 3 3 3 3 3 3
291/300 Read block 3284 len 40960
292/300 Flush
293/300 Flush
294/300 Read block 3778 len 28672
295/300 Write block 274 len 28672 data: 3 3 3 3 3 2 2
296/300 Flush
297/300 Write block 4255 len 36864 data: 2 2 2 2 2 2 2 2 2
298/300 Read block 276 len 24576
299/300 Write block 1176 len 24576 data: 3 3 3 4 3 3
300/300 Write block 3902 len 24576 data: 2 2 2 2 2 3
301/300 Flush
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:01:52.633 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5230b673-1158-4b66-8fc1-6277212ce6b3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1603 Acked Write 9 Done Sent Done false
1604 Acked Write 10 Done Sent Done false
1605 Acked Write 1 Done Sent Done false
1606 Acked Write 6 Done Sent Done false
1607 Acked Read 8 Done Sent Done false
1608 Acked Read 6 Done Sent Done false
1609 Acked Read 1 Done Sent Done false
1610 Acked Write 6 Done Sent Done false
1611 Acked Write 6 Done Sent Done false
1612 Acked Read 9 Done Sent Done false
1613 Acked Write 8 Done Sent Done false
1614 Acked Write 5 Done Sent Done false
1615 Acked Write 5 Done Sent Done false
1616 Acked Write 8 Done Sent Done false
1617 Acked Read 2 Done Sent Done false
1618 Acked Flush 0 Done Sent Done false
1619 Acked Flush 0 Done Sent Done false
1620 Acked Write 5 Done Sent Done false
1621 Acked Read 10 Done Sent Done false
1622 Acked Read 10 Done Sent Done false
1623 Acked Read 3 Done Sent Done false
1624 Acked Read 9 Done Sent Done false
1625 Acked Read 3 Done Sent Done false
1626 Acked Read 10 Done Sent Done false
1627 Acked Write 1 Done Sent Done false
1628 Acked Read 1 Done Sent Done false
1629 Acked Write 7 Done Sent Done false
1630 Acked Read 3 Done Sent Done false
1631 Acked Write 10 Done Sent Done false
1632 Acked Write 10 Done Sent Done false
1633 Acked Read 9 Done Sent Done false
1634 Acked Read 8 Done Sent Done false
1635 Acked Read 2 Done Sent Done false
1636 Acked Flush 0 Done Sent Done false
1637 Acked Write 7 Done Sent Done false
1638 Acked Flush 0 Done Sent Done false
1639 Acked Read 2 Done Sent Done false
1640 Acked Read 3 Done Sent Done false
1641 Acked Flush 0 Done Sent Done false
1642 Acked Write 3 Done Sent Done false
1643 Acked Read 3 Done Sent Done false
1644 Acked Write 3 Done Sent Done false
1645 Acked Write 7 Done Sent Done false
1646 Acked Read 6 Done Sent Done false
1647 Acked Flush 0 Done Sent Done false
1648 Acked Read 3 Done Sent Done false
1649 Acked Write 5 Done Sent Done false
1650 Acked Write 3 Done Sent Done false
1651 Acked Write 3 Done Sent Done false
1652 Acked Read 3 Done Sent Done false
1653 Acked Flush 0 Done Sent Done false
1654 Acked Flush 0 Done Sent Done false
1655 Acked Read 6 Done Sent Done false
1656 Acked Write 8 Done Sent Done false
1657 Acked Read 6 Done Sent Done false
1658 Acked Write 4 Done Sent Done false
1659 Acked Flush 0 Done Sent Done false
1660 Acked Flush 0 Done Sent Done false
1661 Acked Write 6 Done Sent Done false
1662 Acked Read 8 Done Sent Done false
1663 Acked Read 7 Done Sent Done false
1664 Acked Write 9 Done Sent Done false
1665 Acked Flush 0 Done Sent Done false
1666 Acked Read 8 Done Sent Done false
1667 Acked Write 10 Done Sent Done false
1668 Acked Write 3 Done Sent Done false
1669 Acked Write 9 Done Sent Done false
1670 Acked Read 8 Done Sent Done false
1671 Acked Write 10 Done Sent Done false
1672 Acked Read 9 Done Sent Done false
1673 Acked Write 1 Done Sent Done false
1674 Acked Read 10 Done Sent Done false
1675 Acked Write 3 Done Sent Done false
1676 Acked Write 6 Done Sent Done false
1677 Acked Read 4 Done Sent Done false
1678 Acked Read 9 Done Sent Done false
1679 Acked Read 2 Done Sent Done false
1680 Acked Read 6 Done Sent Done false
1681 Acked Flush 0 Done Sent Done false
1682 Acked Read 6 Done Sent Done false
1683 Acked Read 3 Done Sent Done false
1684 Acked Write 6 Done Sent Done false
1685 Acked Read 6 Done Sent Done false
1686 Acked Write 4 Done Sent Done false
1687 Acked Write 6 Done Sent Done false
1688 Acked Write 2 Done Sent Done false
1689 Acked Read 7 Done Sent Done false
1690 Acked Read 6 Done Sent Done false
1691 Acked Read 8 Done Sent Done false
1692 Acked Write 8 Done Sent Done false
1693 Acked Write 5 Done Sent Done false
1694 Acked Write 2 Done Sent Done false
1695 Acked Write 9 Done Sent Done false
1696 Acked Write 2 Done Sent Done false
1697 Acked Write 9 Done Sent Done false
1698 Acked Read 2 Done Sent Done false
1699 Acked Write 1 Done Sent Done false
1700 Acked Write 2 Done Sent Done false
1701 Acked Write 3 Done Sent Done false
1702 Acked Read 2 Done Sent Done false
1703 Acked Read 3 Done Sent Done false
1704 Acked Write 9 Done Sent Done false
1705 Acked Write 10 Done Sent Done false
1706 Acked Read 10 Done Sent Done false
1707 Acked Read 4 Done Sent Done false
1708 Acked Write 3 Done Sent Done false
1709 Acked Read 4 Done Sent Done false
1710 Acked Read 10 Done Sent Done false
1711 Acked Write 4 Done Sent Done false
1712 Acked Write 5 Done Sent Done false
1713 Acked Write 5 Done Sent Done false
1714 Acked Write 4 Done Sent Done false
1715 Acked Write 2 Done Sent Done false
1716 Acked Flush 0 Done Sent Done false
1717 Acked Flush 0 Done Sent Done false
1718 Acked Write 10 Done Sent Done false
1719 Acked Write 10 Done Sent Done false
1720 Acked Flush 0 Done Sent Done false
1721 Acked Write 9 Done Sent Done false
1722 Acked Read 7 Done Sent Done false
1723 Acked Read 6 Done Sent Done false
1724 Acked Write 1 Done Sent Done false
1725 Acked Write 2 Done Sent Done false
1726 Acked Write 7 Done Sent Done false
1727 Acked Read 1 Done Sent Done false
1728 Acked Read 9 Done Sent Done false
1729 Acked Read 6 Done Sent Done false
1730 Acked Flush 0 Done Sent Done false
1731 Acked Write 3 Done Sent Done false
1732 Acked Read 9 Done Sent Done false
1733 Acked Write 8 Done Sent Done false
1734 Acked Flush 0 Done Sent Done false
1735 Acked Write 5 Done Sent Done false
1736 Acked Read 10 Done Sent Done false
1737 Acked Write 6 Done Sent Done false
1738 Acked Read 1 Done Sent Done false
1739 Acked Read 6 Done Sent Done false
1740 Acked Read 6 Done Sent Done false
1741 Acked Read 10 Done Sent Done false
1742 Acked Write 7 Done Sent Done false
1743 Acked Write 10 Done Sent Done false
1744 Acked Write 9 Done Sent Done false
1745 Acked Flush 0 Done Sent Done false
1746 Acked Read 6 Done Sent Done false
1747 Acked Flush 0 Done Sent Done false
1748 Acked Write 5 Done Sent Done false
1749 Acked Read 7 Done Sent Done false
1750 Acked Read 10 Done Sent Done false
1751 Acked Read 7 Done Sent Done false
1752 Acked Write 2 Done Sent Done false
1753 Acked Write 10 Done Sent Done false
1754 Acked Write 5 Done Sent Done false
1755 Acked Read 3 Done Sent Done false
1756 Acked Read 5 Done Sent Done false
1757 Acked Flush 0 Done Sent Done false
1758 Acked Write 8 Done Sent Done false
1759 Acked Flush 0 Done Sent Done false
1760 Acked Read 8 Done Sent Done false
1761 Acked Read 6 Done Sent Done false
1762 Acked Write 4 Done Sent Done false
1763 Acked Read 1 Done Sent Done false
1764 Acked Read 3 Done Sent Done false
1765 Acked Write 2 Done Sent Done false
1766 Acked Write 10 Done Sent Done false
1767 Acked Write 10 Done Sent Done false
1768 Acked Read 7 Done Sent Done false
1769 Acked Read 2 Done Sent Done false
1770 Acked Write 6 Done Sent Done false
1771 Acked Flush 0 Done Sent Done false
1772 Acked Read 8 Done Sent Done false
1773 Acked Read 10 Done Sent Done false
1774 Acked Write 4 Done Sent Done false
1775 Acked Write 6 Done Sent Done false
1776 Acked Flush 0 Done Sent Done false
1777 Acked Flush 0 Done Sent Done false
1778 Acked Read 8 Done Sent Done false
1779 Acked Write 10 Done Sent Done false
1780 Acked Read 5 Done Sent Done false
1781 Acked Write 6 Done Sent Done false
1782 Acked Read 1 Done Sent Done false
1783 Acked Write 7 Done Sent Done false
1784 Acked Read 10 Done Sent Done false
1785 Acked Read 5 Done Sent Done false
1786 Acked Read 6 Done Sent Done false
1787 Acked Read 2 Done Sent Done false
1788 Acked Read 7 Done Sent Done false
1789 Acked Read 1 Done Sent Done false
1790 Acked Read 5 Done Sent Done false
1791 Acked Read 3 Done Sent Done false
1792 Acked Write 7 Done Sent Done false
1793 Acked Read 2 Done Sent Done false
1794 Acked Write 9 Done Sent Done false
1795 Acked Read 5 Done Sent Done false
1796 Acked Read 7 Done Sent Done false
1797 Acked Flush 0 Done Sent Done false
1798 Acked Write 9 Done Sent Done false
1799 Acked Write 1 Done Sent Done false
1800 Acked Write 8 Done Sent Done false
1801 Acked Flush 0 Done Sent Done false
1802 Acked Read 7 Done Sent Done false
1803 Acked Flush 0 Done Sent Done false
1804 Acked Read 7 Done Sent Done false
1805 Acked Write 7 Done Sent Done false
1806 Acked Write 5 Done Sent Done false
1807 Acked Read 3 Done Sent Done false
1808 Acked Write 3 Done Sent Done false
1809 Acked Flush 0 Done Sent Done false
1810 Acked Write 5 Done Sent Done false
1811 Acked Read 6 Done Sent Done false
1812 Acked Read 9 Done Sent Done false
1813 Acked Write 10 Done Sent Done false
1814 Acked Flush 0 Done Sent Done false
1815 Acked Write 5 Done Sent Done false
1816 Acked Read 7 Done Sent Done false
1817 Acked Write 2 Done Sent Done false
1818 Acked Write 8 Done Sent Done false
1819 Acked Read 5 Done Sent Done false
1820 Acked Read 4 Done Sent Done false
1821 Acked Read 5 Done Sent Done false
1822 Acked Write 3 Done Sent Done false
1823 Acked Read 6 Done Sent Done false
1824 Acked Write 8 Done Sent Done false
1825 Acked Read 1 Done Sent Done false
1826 Acked Read 8 Done Sent Done false
1827 Acked Read 4 Done Sent Done false
1828 Acked Read 10 Done Sent Done false
1829 Acked Read 7 Done Sent Done false
1830 Acked Write 5 Done Sent Done false
1831 Acked Write 1 Done Sent Done false
1832 Acked Read 9 Done Sent Done false
1833 Acked Write 8 Done Sent Done false
1834 Acked Write 1 Done Sent Done false
1835 Acked Read 1 Done Sent Done false
1836 Acked Write 6 Done Sent Done false
1837 Acked Read 2 Done Sent Done false
1838 Acked Read 10 Done Sent Done false
1839 Acked Read 1 Done Sent Done false
1840 Acked Read 10 Done Sent Done false
1841 Acked Read 6 Done Sent Done false
1842 Acked Write 5 Done Sent Done false
1843 Acked Write 5 Done Sent Done false
1844 Acked Read 2 Done Sent Done false
1845 Acked Write 1 Done Sent Done false
1846 Acked Read 8 Done Sent Done false
1847 Acked Read 2 Done Sent Done false
1848 Acked Write 5 Done Sent Done false
1849 Acked Write 3 Done Sent Done false
1850 Acked Read 4 Done Sent Done false
1851 Acked Write 7 Done Sent Done false
1852 Acked Write 7 Done Sent Done false
1853 Acked Read 3 Done Sent Done false
1854 Acked Write 10 Done Sent Done false
1855 Acked Write 7 Done Sent Done false
1856 Acked Read 7 Done Sent Done false
1857 Acked Read 2 Done Sent Done false
1858 Acked Write 5 Done Sent Done false
1859 Acked Write 2 Done Sent Done false
1860 Acked Read 3 Done Sent Done false
1861 Acked Read 2 Done Sent Done false
1862 Acked Read 4 Done Sent Done false
1863 Acked Write 9 Done Sent Done false
1864 Acked Flush 0 Done Sent Done false
1865 Acked Write 8 Done Sent Done false
1866 Acked Flush 0 Done Sent Done false
1867 Acked Read 8 Done Sent Done false
1868 Acked Flush 0 Done Sent Done false
1869 Acked Read 6 Done Sent Done false
1870 Acked Read 7 Done Sent Done false
1871 Acked Read 7 Done Sent Done false
1872 Acked Write 6 Done Sent Done false
1873 Acked Read 10 Done Sent Done false
1874 Acked Write 6 Done Sent Done false
1875 Acked Flush 0 Done Sent Done false
1876 Acked Write 8 Done Sent Done false
1877 Acked Write 8 Done Sent Done false
1878 Acked Write 6 Done Sent Done false
1879 Acked Write 9 Done Sent Done false
1880 Acked Read 3 Done Sent Done false
1881 Acked Read 4 Done Sent Done false
1882 Acked Read 4 Done Sent Done false
1883 Acked Read 7 Done Sent Done false
1884 Acked Read 4 Done Sent Done false
1885 Acked Read 9 Done Sent Done false
1886 Acked Write 8 Done Sent Done false
1887 Acked Write 3 Done Sent Done false
1888 Acked Read 1 Done Sent Done false
1889 Acked Read 2 Done Sent Done false
1890 Acked Write 3 Done Sent Done false
1891 Acked Read 1 Done Sent Done false
1892 Acked Write 7 Done Sent Done false
1893 Acked Read 10 Done Sent Done false
1894 Acked Flush 0 Done Sent Done false
1895 Acked Flush 0 Done Sent Done false
1896 Acked Read 7 Done Sent Done false
1897 Acked Write 7 Done Sent Done false
1898 Acked Flush 0 Done Sent Done false
1899 Acked Write 9 Done Sent Done false
1900 Acked Read 6 Done Sent Done false
1901 Acked Write 6 Done Sent Done false
1902 Acked Write 6 Done Sent Done false
1903 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1903)) Some(JobId(1602)) Some(JobId(1903))
Downstairs last five completed: 1602 1601 1600 1599 1598
Upstairs last five completed: 1903 1902 1901 1900 1899
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:53.422461265Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:54.424111203Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:55.42626299Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:56.428026181Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1603 Acked Write 9 Done Sent Done false
1604 Acked Write 10 Done Sent Done false
1605 Acked Write 1 Done Sent Done false
1606 Acked Write 6 Done Sent Done false
1607 Acked Read 8 Done Sent Done false
1608 Acked Read 6 Done Sent Done false
1609 Acked Read 1 Done Sent Done false
1610 Acked Write 6 Done Sent Done false
1611 Acked Write 6 Done Sent Done false
1612 Acked Read 9 Done Sent Done false
1613 Acked Write 8 Done Sent Done false
1614 Acked Write 5 Done Sent Done false
1615 Acked Write 5 Done Sent Done false
1616 Acked Write 8 Done Sent Done false
1617 Acked Read 2 Done Sent Done false
1618 Acked Flush 0 Done Sent Done false
1619 Acked Flush 0 Done Sent Done false
1620 Acked Write 5 Done Sent Done false
1621 Acked Read 10 Done Sent Done false
1622 Acked Read 10 Done Sent Done false
1623 Acked Read 3 Done Sent Done false
1624 Acked Read 9 Done Sent Done false
1625 Acked Read 3 Done Sent Done false
1626 Acked Read 10 Done Sent Done false
1627 Acked Write 1 Done Sent Done false
1628 Acked Read 1 Done Sent Done false
1629 Acked Write 7 Done Sent Done false
1630 Acked Read 3 Done Sent Done false
1631 Acked Write 10 Done Sent Done false
1632 Acked Write 10 Done Sent Done false
1633 Acked Read 9 Done Sent Done false
1634 Acked Read 8 Done Sent Done false
1635 Acked Read 2 Done Sent Done false
1636 Acked Flush 0 Done Sent Done false
1637 Acked Write 7 Done Sent Done false
1638 Acked Flush 0 Done Sent Done false
1639 Acked Read 2 Done Sent Done false
1640 Acked Read 3 Done Sent Done false
1641 Acked Flush 0 Done Sent Done false
1642 Acked Write 3 Done Sent Done false
1643 Acked Read 3 Done Sent Done false
1644 Acked Write 3 Done Sent Done false
1645 Acked Write 7 Done Sent Done false
1646 Acked Read 6 Done Sent Done false
1647 Acked Flush 0 Done Sent Done false
1648 Acked Read 3 Done Sent Done false
1649 Acked Write 5 Done Sent Done false
1650 Acked Write 3 Done Sent Done false
1651 Acked Write 3 Done Sent Done false
1652 Acked Read 3 Done Sent Done false
1653 Acked Flush 0 Done Sent Done false
1654 Acked Flush 0 Done Sent Done false
1655 Acked Read 6 Done Sent Done false
1656 Acked Write 8 Done Sent Done false
1657 Acked Read 6 Done Sent Done false
1658 Acked Write 4 Done Sent Done false
1659 Acked Flush 0 Done Sent Done false
1660 Acked Flush 0 Done Sent Done false
1661 Acked Write 6 Done Sent Done false
1662 Acked Read 8 Done Sent Done false
1663 Acked Read 7 Done Sent Done false
1664 Acked Write 9 Done Sent Done false
1665 Acked Flush 0 Done Sent Done false
1666 Acked Read 8 Done Sent Done false
1667 Acked Write 10 Done Sent Done false
1668 Acked Write 3 Done Sent Done false
1669 Acked Write 9 Done Sent Done false
1670 Acked Read 8 Done Sent Done false
1671 Acked Write 10 Done Sent Done false
1672 Acked Read 9 Done Sent Done false
1673 Acked Write 1 Done Sent Done false
1674 Acked Read 10 Done Sent Done false
1675 Acked Write 3 Done Sent Done false
1676 Acked Write 6 Done Sent Done false
1677 Acked Read 4 Done Sent Done false
1678 Acked Read 9 Done Sent Done false
1679 Acked Read 2 Done Sent Done false
1680 Acked Read 6 Done Sent Done false
1681 Acked Flush 0 Done Sent Done false
1682 Acked Read 6 Done Sent Done false
1683 Acked Read 3 Done Sent Done false
1684 Acked Write 6 Done Sent Done false
1685 Acked Read 6 Done Sent Done false
1686 Acked Write 4 Done Sent Done false
1687 Acked Write 6 Done Sent Done false
1688 Acked Write 2 Done Sent Done false
1689 Acked Read 7 Done Sent Done false
1690 Acked Read 6 Done Sent Done false
1691 Acked Read 8 Done Sent Done false
1692 Acked Write 8 Done Sent Done false
1693 Acked Write 5 Done Sent Done false
1694 Acked Write 2 Done Sent Done false
1695 Acked Write 9 Done Sent Done false
1696 Acked Write 2 Done Sent Done false
1697 Acked Write 9 Done Sent Done false
1698 Acked Read 2 Done Sent Done false
1699 Acked Write 1 Done Sent Done false
1700 Acked Write 2 Done Sent Done false
1701 Acked Write 3 Done Sent Done false
1702 Acked Read 2 Done Sent Done false
1703 Acked Read 3 Done Sent Done false
1704 Acked Write 9 Done Sent Done false
1705 Acked Write 10 Done Sent Done false
1706 Acked Read 10 Done Sent Done false
1707 Acked Read 4 Done Sent Done false
1708 Acked Write 3 Done Sent Done false
1709 Acked Read 4 Done Sent Done false
1710 Acked Read 10 Done Sent Done false
1711 Acked Write 4 Done Sent Done false
1712 Acked Write 5 Done Sent Done false
1713 Acked Write 5 Done Sent Done false
1714 Acked Write 4 Done Sent Done false
1715 Acked Write 2 Done Sent Done false
1716 Acked Flush 0 Done Sent Done false
1717 Acked Flush 0 Done Sent Done false
1718 Acked Write 10 Done Sent Done false
1719 Acked Write 10 Done Sent Done false
1720 Acked Flush 0 Done Sent Done false
1721 Acked Write 9 Done Sent Done false
1722 Acked Read 7 Done Sent Done false
1723 Acked Read 6 Done Sent Done false
1724 Acked Write 1 Done Sent Done false
1725 Acked Write 2 Done Sent Done false
1726 Acked Write 7 Done Sent Done false
1727 Acked Read 1 Done Sent Done false
1728 Acked Read 9 Done Sent Done false
1729 Acked Read 6 Done Sent Done false
1730 Acked Flush 0 Done Sent Done false
1731 Acked Write 3 Done Sent Done false
1732 Acked Read 9 Done Sent Done false
1733 Acked Write 8 Done Sent Done false
1734 Acked Flush 0 Done Sent Done false
1735 Acked Write 5 Done Sent Done false
1736 Acked Read 10 Done Sent Done false
1737 Acked Write 6 Done Sent Done false
1738 Acked Read 1 Done Sent Done false
1739 Acked Read 6 Done Sent Done false
1740 Acked Read 6 Done Sent Done false
1741 Acked Read 10 Done Sent Done false
1742 Acked Write 7 Done Sent Done false
1743 Acked Write 10 Done Sent Done false
1744 Acked Write 9 Done Sent Done false
1745 Acked Flush 0 Done Sent Done false
1746 Acked Read 6 Done Sent Done false
1747 Acked Flush 0 Done Sent Done false
1748 Acked Write 5 Done Sent Done false
1749 Acked Read 7 Done Sent Done false
1750 Acked Read 10 Done Sent Done false
1751 Acked Read 7 Done Sent Done false
1752 Acked Write 2 Done Sent Done false
1753 Acked Write 10 Done Sent Done false
1754 Acked Write 5 Done Sent Done false
1755 Acked Read 3 Done Sent Done false
1756 Acked Read 5 Done Sent Done false
1757 Acked Flush 0 Done Sent Done false
1758 Acked Write 8 Done Sent Done false
1759 Acked Flush 0 Done Sent Done false
1760 Acked Read 8 Done Sent Done false
1761 Acked Read 6 Done Sent Done false
1762 Acked Write 4 Done Sent Done false
1763 Acked Read 1 Done Sent Done false
1764 Acked Read 3 Done Sent Done false
1765 Acked Write 2 Done Sent Done false
1766 Acked Write 10 Done Sent Done false
1767 Acked Write 10 Done Sent Done false
1768 Acked Read 7 Done Sent Done false
1769 Acked Read 2 Done Sent Done false
1770 Acked Write 6 Done Sent Done false
1771 Acked Flush 0 Done Sent Done false
1772 Acked Read 8 Done Sent Done false
1773 Acked Read 10 Done Sent Done false
1774 Acked Write 4 Done Sent Done false
1775 Acked Write 6 Done Sent Done false
1776 Acked Flush 0 Done Sent Done false
1777 Acked Flush 0 Done Sent Done false
1778 Acked Read 8 Done Sent Done false
1779 Acked Write 10 Done Sent Done false
1780 Acked Read 5 Done Sent Done false
1781 Acked Write 6 Done Sent Done false
1782 Acked Read 1 Done Sent Done false
1783 Acked Write 7 Done Sent Done false
1784 Acked Read 10 Done Sent Done false
1785 Acked Read 5 Done Sent Done false
1786 Acked Read 6 Done Sent Done false
1787 Acked Read 2 Done Sent Done false
1788 Acked Read 7 Done Sent Done false
1789 Acked Read 1 Done Sent Done false
1790 Acked Read 5 Done Sent Done false
1791 Acked Read 3 Done Sent Done false
1792 Acked Write 7 Done Sent Done false
1793 Acked Read 2 Done Sent Done false
1794 Acked Write 9 Done Sent Done false
1795 Acked Read 5 Done Sent Done false
1796 Acked Read 7 Done Sent Done false
1797 Acked Flush 0 Done Sent Done false
1798 Acked Write 9 Done Sent Done false
1799 Acked Write 1 Done Sent Done false
1800 Acked Write 8 Done Sent Done false
1801 Acked Flush 0 Done Sent Done false
1802 Acked Read 7 Done Sent Done false
1803 Acked Flush 0 Done Sent Done false
1804 Acked Read 7 Done Sent Done false
1805 Acked Write 7 Done Sent Done false
1806 Acked Write 5 Done Sent Done false
1807 Acked Read 3 Done Sent Done false
1808 Acked Write 3 Done Sent Done false
1809 Acked Flush 0 Done Sent Done false
1810 Acked Write 5 Done Sent Done false
1811 Acked Read 6 Done Sent Done false
1812 Acked Read 9 Done Sent Done false
1813 Acked Write 10 Done Sent Done false
1814 Acked Flush 0 Done Sent Done false
1815 Acked Write 5 Done Sent Done false
1816 Acked Read 7 Done Sent Done false
1817 Acked Write 2 Done Sent Done false
1818 Acked Write 8 Done Sent Done false
1819 Acked Read 5 Done Sent Done false
1820 Acked Read 4 Done Sent Done false
1821 Acked Read 5 Done Sent Done false
1822 Acked Write 3 Done Sent Done false
1823 Acked Read 6 Done Sent Done false
1824 Acked Write 8 Done Sent Done false
1825 Acked Read 1 Done Sent Done false
1826 Acked Read 8 Done Sent Done false
1827 Acked Read 4 Done Sent Done false
1828 Acked Read 10 Done Sent Done false
1829 Acked Read 7 Done Sent Done false
1830 Acked Write 5 Done Sent Done false
1831 Acked Write 1 Done Sent Done false
1832 Acked Read 9 Done Sent Done false
1833 Acked Write 8 Done Sent Done false
1834 Acked Write 1 Done Sent Done false
1835 Acked Read 1 Done Sent Done false
1836 Acked Write 6 Done Sent Done false
1837 Acked Read 2 Done Sent Done false
1838 Acked Read 10 Done Sent Done false
1839 Acked Read 1 Done Sent Done false
1840 Acked Read 10 Done Sent Done false
1841 Acked Read 6 Done Sent Done false
1842 Acked Write 5 Done Sent Done false
1843 Acked Write 5 Done Sent Done false
1844 Acked Read 2 Done Sent Done false
1845 Acked Write 1 Done Sent Done false
1846 Acked Read 8 Done Sent Done false
1847 Acked Read 2 Done Sent Done false
1848 Acked Write 5 Done Sent Done false
1849 Acked Write 3 Done Sent Done false
1850 Acked Read 4 Done Sent Done false
1851 Acked Write 7 Done Sent Done false
1852 Acked Write 7 Done Sent Done false
1853 Acked Read 3 Done Sent Done false
1854 Acked Write 10 Done Sent Done false
1855 Acked Write 7 Done Sent Done false
1856 Acked Read 7 Done Sent Done false
1857 Acked Read 2 Done Sent Done false
1858 Acked Write 5 Done Sent Done false
1859 Acked Write 2 Done Sent Done false
1860 Acked Read 3 Done Sent Done false
1861 Acked Read 2 Done Sent Done false
1862 Acked Read 4 Done Sent Done false
1863 Acked Write 9 Done Sent Done false
1864 Acked Flush 0 Done Sent Done false
1865 Acked Write 8 Done Sent Done false
1866 Acked Flush 0 Done Sent Done false
1867 Acked Read 8 Done Sent Done false
1868 Acked Flush 0 Done Sent Done false
1869 Acked Read 6 Done Sent Done false
1870 Acked Read 7 Done Sent Done false
1871 Acked Read 7 Done Sent Done false
1872 Acked Write 6 Done Sent Done false
1873 Acked Read 10 Done Sent Done false
1874 Acked Write 6 Done Sent Done false
1875 Acked Flush 0 Done Sent Done false
1876 Acked Write 8 Done Sent Done false
1877 Acked Write 8 Done Sent Done false
1878 Acked Write 6 Done Sent Done false
1879 Acked Write 9 Done Sent Done false
1880 Acked Read 3 Done Sent Done false
1881 Acked Read 4 Done Sent Done false
1882 Acked Read 4 Done Sent Done false
1883 Acked Read 7 Done Sent Done false
1884 Acked Read 4 Done Sent Done false
1885 Acked Read 9 Done Sent Done false
1886 Acked Write 8 Done Sent Done false
1887 Acked Write 3 Done Sent Done false
1888 Acked Read 1 Done Sent Done false
1889 Acked Read 2 Done Sent Done false
1890 Acked Write 3 Done Sent Done false
1891 Acked Read 1 Done Sent Done false
1892 Acked Write 7 Done Sent Done false
1893 Acked Read 10 Done Sent Done false
1894 Acked Flush 0 Done Sent Done false
1895 Acked Flush 0 Done Sent Done false
1896 Acked Read 7 Done Sent Done false
1897 Acked Write 7 Done Sent Done false
1898 Acked Flush 0 Done Sent Done false
1899 Acked Write 9 Done Sent Done false
1900 Acked Read 6 Done Sent Done false
1901 Acked Write 6 Done Sent Done false
1902 Acked Write 6 Done Sent Done false
1903 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(1903)) Some(JobId(1602)) Some(JobId(1903))
Downstairs last five completed: 1602 1601 1600 1599 1598
Upstairs last five completed: 1903 1902 1901 1900 1899
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:01:57.430552569Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.394958823Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42781)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.395160615Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.396301091Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.396828335Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.396857408Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(1602))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.396876173Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.39689292Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(1602))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.396909396Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(1602))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:58.397194587Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 1903 1902 1901 1900 1899
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:02:00.638 INFO request completed, latency_us: 364[8820] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4cd98aba-6d45-460f-9782-52a99cc40e48, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:02:00.638 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/1, method: GET, req_id: c7e8827e-ef2d-43fa-b753-54738213e2d2, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:00.638790814Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:00.638866876Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:00.638895709Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:00.639009701Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:02:04.641 INFO request completed, latency_us: 338, response_code: 200, uri: /state/cid/1, method: GET, req_id: 291837d2-ee4c-4dc8-b126-d9922c4a647e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 447 len 40960 data: 3 3 3 3 3 2 2 2 3 3
002/300 Read block 1603 len 24576
003/300 Write block 1383 len 40960 data: 2 2 2 2 2 2 2 2 2 2
004/300 Read block 1082 len 12288
005/300 Read block 1936 len 12288
006/300 Read block 1784 len 28672
007/300 Read block 4869 len 28672
008/300 Write block 991 len 16384 data: 3 2 2 2
009/300 Flush
010/300 Read block 758 len 8192
011/300 Write block 1869 len 4096 data: 2
012/300 Read block 4065 len 32768
013/300 Read block 1447 len 24576
014/300 Read block 3430 len 16384
015/300 Read block 1564 len 28672
016/300 Read block 4976 len 16384
017/300 Read block 2985 len 32768
018/300 Write block 291 len 36864 data: 2 2 2 2 2 2 2 2 2
019/300 Read block 1434 len 40960
020/300 Read block 4365 len 16384
021/300 Flush
022/300 Write block 191 len 16384 data: 2 2 2 2
023/300 Write block 1512 len 12288 data: 2 2 2
024/300 Read block 2101 len 16384
025/300 Write block 2283 len 16384 data: 3 2 2 2
026/300 Read block 3808 len 36864
027/300 Read block 681 len 8192
028/300 Write block 2234 len 12288 data: 5 3 3
029/300 Read block 391 len 8192
030/300 Read block 4387 len 40960
031/300 Read block 3311 len 12288
032/300 Write block 2186 len 28672 data: 2 2 2 2 3 3 3
033/300 Read block 4975 len 4096
034/300 Read block 4529 len 20480
035/300 Read block 616 len 12288
036/300 Read block 4876 len 28672
037/300 Flush
038/300 Write block 2521 len 20480 data: 4 4 4 4 4
039/300 Read block 4406 len 12288
040/300 Write block 4694 len 16384 data: 2 2 2 2
041/300 Read block 1984 len 24576
042/300 Write block 2319 len 16384 data: 4 3 3 3
043/300 Write block 1398 len 20480 data: 2 3 3 3 4
044/300 Read block 2593 len 24576
045/300 Read block 2129 len 20480
046/300 Write block 4897 len 24576 data: 3 3 3 3 2 2
047/300 Write block 135 len 24576 data: 2 2 2 2 2 2
048/300 Write block 675 len 40960 data: 3 2 2 3 3 3 3 2 2 2
049/300 Write block 2445 len 12288 data: 2 2 2
050/300 Flush
051/300 Write block 3731 len 28672 data: 2 2 2 2 2 2 2
052/300 Write block 4492 len 24576 data: 2 3 3 4 4 4
053/300 Write block 3648 len 40960 data: 3 3 3 3 3 3 2 2 2 2
054/300 Write block 897 len 28672 data: 2 2 2 2 2 3 2
055/300 Write block 4898 len 32768 data: 4 4 4 3 3 2 2 2
056/300 Read block 830 len 8192
057/300 Write block 3122 len 16384 data: 2 2 2 2
058/300 Read block 2651 len 24576
059/300 Read block 2561 len 16384
060/300 Read block 1019 len 28672
061/300 Write block 844 len 8192 data: 3 3
062/300 Read block 3508 len 16384
063/300 Write block 679 len 28672 data: 4 4 4 3 3 3 2
064/300 Read block 523 len 40960
065/300 Write block 2083 len 28672 data: 2 2 2 2 2 2 2
066/300 Write block 3441 len 32768 data: 2 2 2 2 2 2 2 2
067/300 Write block 537 len 16384 data: 2 2 2 2
068/300 Read block 4415 len 12288
069/300 Read block 4718 len 32768
070/300 Read block 2219 len 28672
071/300 Write block 3475 len 4096 data: 2
072/300 Write block 679 len 4096 data: 5
073/300 Write block 2824 len 12288 data: 2 2 2
074/300 Read block 2 len 24576
075/300 Write block 2212 len 32768 data: 3 3 3 3 3 3 3 2
076/300 Write block 446 len 36864 data: 3 4 4 4 4 4 3 3 3
077/300 Write block 4336 len 16384 data: 2 2 2 2
078/300 Read block 3148 len 40960
079/300 Write block 4333 len 32768 data: 2 2 2 3 3 3 3 2
080/300 Write block 319 len 20480 data: 2 2 2 2 2
081/300 Flush
082/300 Flush
083/300 Read block 1761 len 40960
084/300 Read block 3469 len 24576
085/300 Flush
086/300 Read block 978 len 36864
087/300 Read block 15 len 40960
088/300 Read block 234 len 40960
089/300 Read block 1175 len 20480
090/300 Write block 38 len 20480 data: 2 2 2 3 3
091/300 Flush
092/300 Read block 4382 len 20480
093/300 Read block 2485 len 12288
094/300 Read block 1454 len 36864
095/300 Write block 2724 len 40960 data: 2 2 2 2 2 2 3 2 2 2
096/300 Flush
097/300 Read block 2751 len 40960
098/300 Flush
099/300 Flush
100/300 Write block 577 len 28672 data: 2 2 2 3 3 3 3
101/300 Read block 116 len 16384
102/300 Read block 1186 len 24576
103/300 Flush
104/300 Read block 3322 len 32768
105/300 Write block 133 len 20480 data: 2 2 3 3 3
106/300 Read block 4358 len 36864
107/300 Read block 2042 len 28672
108/300 Write block 4141 len 40960 data: 2 2 2 2 3 3 3 3 3 3
109/300 Write block 1864 len 40960 data: 2 2 2 2 2 3 2 2 2 2
110/300 Write block 4499 len 36864 data: 4 4 4 2 2 2 2 2 2
111/300 Write block 2701 len 32768 data: 2 2 2 2 2 2 2 2
112/300 Write block 4498 len 36864 data: 4 5 5 5 3 3 3 3 3
113/300 Read block 4105 len 12288
114/300 Write block 2653 len 28672 data: 2 2 2 2 2 2 2
115/300 Read block 2783 len 4096
116/300 Read block 3112 len 40960
117/300 Read block 2095 len 28672
118/300 Read block 3226 len 28672
119/300 Read block 2882 len 16384
120/300 Write block 3296 len 28672 data: 2 3 3 3 2 2 3
121/300 Read block 235 len 40960
122/300 Write block 4290 len 12288 data: 2 2 2
123/300 Write block 1160 len 40960 data: 2 2 2 2 2 2 2 2 2 2
124/300 Flush
125/300 Read block 2059 len 8192
126/300 Read block 266 len 24576
127/300 Write block 283 len 8192 data: 3 3
128/300 Flush
129/300 Write block 1632 len 24576 data: 2 2 2 2 2 2
130/300 Write block 2273 len 24576 data: 3 3 3 3 3 3
131/300 Read block 1590 len 16384
132/300 Read block 3795 len 32768
133/300 Flush
134/300 Read block 1776 len 32768
135/300 Read block 4247 len 4096
136/300 Write block 2209 len 40960 data: 3 3 3 4 4 4 4 4 4 4
137/300 Read block 1906 len 28672
138/300 Write block 2594 len 4096 data: 2
139/300 Flush
140/300 Write block 1891 len 24576 data: 3 3 3 4 4 3
141/300 Read block 1156 len 24576
142/300 Write block 2656 len 20480 data: 3 3 3 3 2
143/300 Read block 2671 len 32768
144/300 Write block 4054 len 28672 data: 2 2 2 2 2 2 2
145/300 Read block 3396 len 8192
146/300 Flush
147/300 Write block 3101 len 36864 data: 3 3 3 3 3 2 2 3 3
148/300 Read block 4071 len 40960
149/300 Write block 3882 len 28672 data: 2 3 3 3 3 2 2
150/300 Write block 4184 len 16384 data: 3 3 3 3
151/300 Flush
152/300 Flush
153/300 Read block 4970 len 24576
154/300 Flush
155/300 Read block 209 len 8192
156/300 Read block 3482 len 28672
157/300 Read block 619 len 36864
158/300 Flush
159/300 Read block 3 len 20480
160/300 Read block 1667 len 8192
161/300 Write block 1838 len 36864 data: 2 2 2 2 2 2 2 2 2
162/300 Read block 2191 len 8192
163/300 Write block 2137 len 4096 data: 2
164/300 Read block 1068 len 20480
165/300 Write block 3897 len 20480 data: 2 2 2 2 2
166/300 Read block 913 len 20480
167/300 Write block 2985 len 32768 data: 2 2 2 2 2 3 3 3
168/300 Read block 765 len 28672
169/300 Read block 1825 len 8192
170/300 Write block 4830 len 8192 data: 3 4
171/300 Flush
172/300 Write block 1656 len 20480 data: 2 2 2 2 2
173/300 Read block 1912 len 20480
174/300 Read block 325 len 8192
175/300 Read block 2686 len 8192
176/300 Write block 3685 len 8192 data: 2 2
177/300 Flush
178/300 Read block 4569 len 32768
179/300 Read block 1226 len 4096
180/300 Read block 988 len 36864
181/300 Write block 2236 len 12288 data: 4 3 3
182/300 Read block 4677 len 36864
183/300 Write block 2705 len 36864 data: 3 3 3 3 2 2 2 2 3
184/300 Write block 3376 len 4096 data: 2
185/300 Write block 149 len 24576 data: 4 3 3 3 3 3
186/300 Write block 21 len 20480 data: 2 2 2 2 2
187/300 Read block 404 len 24576
188/300 Read block 2412 len 12288
189/300 Flush
190/300 Write block 133 len 28672 data: 3 3 4 4 4 3 3
191/300 Flush
192/300 Read block 1980 len 4096
193/300 Write block 1901 len 4096 data: 2
194/300 Read block 4723 len 12288
195/300 Read block 2175 len 28672
196/300 Write block 3434 len 24576 data: 2 2 2 2 2 2
197/300 Flush
198/300 Write block 2223 len 28672 data: 2 2 2 2 3 3 3
199/300 Read block 1174 len 24576
200/300 Write block 2709 len 36864 data: 3 3 3 3 4 3 3 3 3
201/300 Flush
202/300 Flush
203/300 Read block 3275 len 20480
204/300 Write block 3435 len 16384 data: 3 3 3 3
205/300 Read block 1149 len 8192
206/300 Flush
207/300 Flush
208/300 Read block 1580 len 24576
209/300 Write block 3938 len 16384 data: 2 3 3 3
210/300 Write block 752 len 12288 data: 3 3 3
211/300 Write block 2374 len 12288 data: 2 3 4
212/300 Read block 1143 len 4096
213/300 Write block 4878 len 40960 data: 2 2 2 2 2 2 2 2 2 3
214/300 Write block 730 len 36864 data: 2 2 2 2 2 2 3 2 2
215/300 Write block 823 len 4096 data: 2
216/300 Read block 4663 len 24576
217/300 Read block 3646 len 4096
218/300 Write block 2789 len 4096 data: 3
219/300 Read block 4795 len 20480
220/300 Read block 2870 len 16384
221/300 Read block 3023 len 12288
222/300 Write block 1739 len 12288 data: 2 2 2
223/300 Write block 3479 len 8192 data: 2 3
224/300 Write block 3497 len 28672 data: 2 2 2 2 2 2 2
225/300 Read block 4236 len 4096
226/300 Write block 2779 len 36864 data: 2 2 2 2 3 3 3 3 3
227/300 Read block 1366 len 8192
228/300 Write block 4978 len 8192 data: 2 2
229/300 Flush
230/300 Write block 2512 len 36864 data: 4 4 4 3 4 4 4 4 4
231/300 Read block 4288 len 20480
232/300 Write block 3018 len 24576 data: 2 2 2 2 2 2
233/300 Read block 3107 len 28672
234/300 Read block 3822 len 16384
235/300 Read block 2185 len 16384
236/300 Flush
237/300 Flush
238/300 Flush
239/300 Read block 4699 len 4096
240/300 Write block 3527 len 8192 data: 3 3
241/300 Write block 2678 len 32768 data: 2 2 2 2 2 2 2 2
242/300 Write block 437 len 36864 data: 3 4 4 4 3 4 4 4 4
243/300 Write block 4070 len 28672 data: 3 3 3 2 2 2 2
244/300 Read block 70 len 40960
245/300 Read block 2355 len 20480
246/300 Write block 443 len 40960 data: 5 5 5 4 5 5 5 5 5 4
247/300 Write block 911 len 12288 data: 3 3 3
248/300 Read block 844 len 16384
249/300 Read block 2231 len 4096
250/300 Read block 1921 len 32768
251/300 Write block 3738 len 32768 data: 2 2 2 3 3 3 3 3
252/300 Flush
253/300 Read block 1580 len 36864
254/300 Read block 2153 len 32768
255/300 Write block 2619 len 4096 data: 2
256/300 Flush
257/300 Read block 1679 len 8192
258/300 Write block 336 len 32768 data: 2 3 3 2 2 3 3 3
259/300 Read block 798 len 28672
260/300 Write block 1080 len 8192 data: 3 3
261/300 Read block 162 len 24576
262/300 Write block 4925 len 8192 data: 2 2
263/300 Read block 3931 len 20480
264/300 Read block 4491 len 32768
265/300 Read block 3211 len 16384
266/300 Read block 4647 len 28672
267/300 Read block 2932 len 28672
268/300 Write block 3073 len 28672 data: 3 3 2 2 2 2 2
269/300 Write block 4436 len 12288 data: 2 2 2
270/300 Write block 1696 len 32768 data: 4 4 3 3 3 3 2 2
271/300 Read block 335 len 24576
272/300 Read block 1844 len 36864
273/300 Read block 3055 len 12288
274/300 Read block 973 len 12288
275/300 Flush
276/300 Read block 31 len 8192
277/300 Flush
278/300 Write block 3400 len 12288 data: 2 2 2
279/300 Write block 1332 len 24576 data: 2 2 3 3 3 3
280/300 Read block 375 len 36864
281/300 Write block 1538 len 24576 data: 3 2 2 2 2 2
282/300 Read block 3192 len 28672
283/300 Read block 2084 len 4096
284/300 Flush
285/300 Write block 452 len 24576 data: 5 4 4 4 4 2
286/300 Write block 2947 len 40960 data: 2 2 2 2 2 2 2 2 2 2
287/300 Read block 3466 len 4096
288/300 Write block 237 len 12288 data: 4 4 2
289/300 Write block 4165 len 16384 data: 2 2 2 2
290/300 Read block 243 len 16384
291/300 Read block 739 len 8192
292/300 Read block 309 len 16384
293/300 Write block 3934 len 36864 data: 2 2 2 2 3 4 4 4 3
294/300 Read block 1068 len 20480
295/300 Write block 2307 len 32768 data: 2 2 2 2 2 2 2 2
296/300 Read block 3529 len 12288
297/300 Write block 3067 len 4096 data: 2
298/300 Read block 1163 len 24576
299/300 Flush
300/300 Read block 3138 len 40960
301/300 Read block 4645 len 8192
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:02:04.876 INFO request completed, latency_us: 182, response_code: 204, uri: /start/cid/1, method: POST, req_id: 94672950-8000-408d-b70c-2c271212ce6d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1904 Acked Write 10 Done Sent Done false
1905 Acked Read 6 Done Sent Done false
1906 Acked Write 10 Done Sent Done false
1907 Acked Read 3 Done Sent Done false
1908 Acked Read 3 Done Sent Done false
1909 Acked Read 7 Done Sent Done false
1910 Acked Read 7 Done Sent Done false
1911 Acked Write 4 Done Sent Done false
1912 Acked Flush 0 Done Sent Done false
1913 Acked Read 2 Done Sent Done false
1914 Acked Write 1 Done Sent Done false
1915 Acked Read 8 Done Sent Done false
1916 Acked Read 6 Done Sent Done false
1917 Acked Read 4 Done Sent Done false
1918 Acked Read 7 Done Sent Done false
1919 Acked Read 4 Done Sent Done false
1920 Acked Read 8 Done Sent Done false
1921 Acked Write 9 Done Sent Done false
1922 Acked Read 10 Done Sent Done false
1923 Acked Read 4 Done Sent Done false
1924 Acked Flush 0 Done Sent Done false
1925 Acked Write 4 Done Sent Done false
1926 Acked Write 3 Done Sent Done false
1927 Acked Read 4 Done Sent Done false
1928 Acked Write 4 Done Sent Done false
1929 Acked Read 9 Done Sent Done false
1930 Acked Read 2 Done Sent Done false
1931 Acked Write 3 Done Sent Done false
1932 Acked Read 2 Done Sent Done false
1933 Acked Read 10 Done Sent Done false
1934 Acked Read 3 Done Sent Done false
1935 Acked Write 7 Done Sent Done false
1936 Acked Read 1 Done Sent Done false
1937 Acked Read 5 Done Sent Done false
1938 Acked Read 3 Done Sent Done false
1939 Acked Read 7 Done Sent Done false
1940 Acked Flush 0 Done Sent Done false
1941 Acked Write 5 Done Sent Done false
1942 Acked Read 3 Done Sent Done false
1943 Acked Write 4 Done Sent Done false
1944 Acked Read 6 Done Sent Done false
1945 Acked Write 4 Done Sent Done false
1946 Acked Write 5 Done Sent Done false
1947 Acked Read 6 Done Sent Done false
1948 Acked Read 5 Done Sent Done false
1949 Acked Write 6 Done Sent Done false
1950 Acked Write 6 Done Sent Done false
1951 Acked Write 10 Done Sent Done false
1952 Acked Write 3 Done Sent Done false
1953 Acked Flush 0 Done Sent Done false
1954 Acked Write 7 Done Sent Done false
1955 Acked Write 6 Done Sent Done false
1956 Acked Write 10 Done Sent Done false
1957 Acked Write 7 Done Sent Done false
1958 Acked Write 8 Done Sent Done false
1959 Acked Read 2 Done Sent Done false
1960 Acked Write 4 Done Sent Done false
1961 Acked Read 6 Done Sent Done false
1962 Acked Read 4 Done Sent Done false
1963 Acked Read 7 Done Sent Done false
1964 Acked Write 2 Done Sent Done false
1965 Acked Read 4 Done Sent Done false
1966 Acked Write 7 Done Sent Done false
1967 Acked Read 10 Done Sent Done false
1968 Acked Write 7 Done Sent Done false
1969 Acked Write 8 Done Sent Done false
1970 Acked Write 4 Done Sent Done false
1971 Acked Read 3 Done Sent Done false
1972 Acked Read 8 Done Sent Done false
1973 Acked Read 7 Done Sent Done false
1974 Acked Write 1 Done Sent Done false
1975 Acked Write 1 Done Sent Done false
1976 Acked Write 3 Done Sent Done false
1977 Acked Read 6 Done Sent Done false
1978 Acked Write 8 Done Sent Done false
1979 Acked Write 9 Done Sent Done false
1980 Acked Write 4 Done Sent Done false
1981 Acked Read 10 Done Sent Done false
1982 Acked Write 8 Done Sent Done false
1983 Acked Write 5 Done Sent Done false
1984 Acked Flush 0 Done Sent Done false
1985 Acked Flush 0 Done Sent Done false
1986 Acked Read 10 Done Sent Done false
1987 Acked Read 6 Done Sent Done false
1988 Acked Flush 0 Done Sent Done false
1989 Acked Read 9 Done Sent Done false
1990 Acked Read 10 Done Sent Done false
1991 Acked Read 10 Done Sent Done false
1992 Acked Read 5 Done Sent Done false
1993 Acked Write 5 Done Sent Done false
1994 Acked Flush 0 Done Sent Done false
1995 Acked Read 5 Done Sent Done false
1996 Acked Read 3 Done Sent Done false
1997 Acked Read 9 Done Sent Done false
1998 Acked Write 10 Done Sent Done false
1999 Acked Flush 0 Done Sent Done false
2000 Acked Read 10 Done Sent Done false
2001 Acked Flush 0 Done Sent Done false
2002 Acked Flush 0 Done Sent Done false
2003 Acked Write 7 Done Sent Done false
2004 Acked Read 4 Done Sent Done false
2005 Acked Read 6 Done Sent Done false
2006 Acked Flush 0 Done Sent Done false
2007 Acked Read 8 Done Sent Done false
2008 Acked Write 5 Done Sent Done false
2009 Acked Read 9 Done Sent Done false
2010 Acked Read 7 Done Sent Done false
2011 Acked Write 10 Done Sent Done false
2012 Acked Write 10 Done Sent Done false
2013 Acked Write 9 Done Sent Done false
2014 Acked Write 8 Done Sent Done false
2015 Acked Write 9 Done Sent Done false
2016 Acked Read 3 Done Sent Done false
2017 Acked Write 7 Done Sent Done false
2018 Acked Read 1 Done Sent Done false
2019 Acked Read 10 Done Sent Done false
2020 Acked Read 7 Done Sent Done false
2021 Acked Read 7 Done Sent Done false
2022 Acked Read 4 Done Sent Done false
2023 Acked Write 7 Done Sent Done false
2024 Acked Read 10 Done Sent Done false
2025 Acked Write 3 Done Sent Done false
2026 Acked Write 10 Done Sent Done false
2027 Acked Flush 0 Done Sent Done false
2028 Acked Read 2 Done Sent Done false
2029 Acked Read 6 Done Sent Done false
2030 Acked Write 2 Done Sent Done false
2031 Acked Flush 0 Done Sent Done false
2032 Acked Write 6 Done Sent Done false
2033 Acked Write 6 Done Sent Done false
2034 Acked Read 4 Done Sent Done false
2035 Acked Read 8 Done Sent Done false
2036 Acked Flush 0 Done Sent Done false
2037 Acked Read 8 Done Sent Done false
2038 Acked Read 1 Done Sent Done false
2039 Acked Write 10 Done Sent Done false
2040 Acked Read 7 Done Sent Done false
2041 Acked Write 1 Done Sent Done false
2042 Acked Flush 0 Done Sent Done false
2043 Acked Write 6 Done Sent Done false
2044 Acked Read 6 Done Sent Done false
2045 Acked Write 5 Done Sent Done false
2046 Acked Read 8 Done Sent Done false
2047 Acked Write 7 Done Sent Done false
2048 Acked Read 2 Done Sent Done false
2049 Acked Flush 0 Done Sent Done false
2050 Acked Write 9 Done Sent Done false
2051 Acked Read 10 Done Sent Done false
2052 Acked Write 7 Done Sent Done false
2053 Acked Write 4 Done Sent Done false
2054 Acked Flush 0 Done Sent Done false
2055 Acked Flush 0 Done Sent Done false
2056 Acked Read 6 Done Sent Done false
2057 Acked Flush 0 Done Sent Done false
2058 Acked Read 2 Done Sent Done false
2059 Acked Read 7 Done Sent Done false
2060 Acked Read 9 Done Sent Done false
2061 Acked Flush 0 Done Sent Done false
2062 Acked Read 5 Done Sent Done false
2063 Acked Read 2 Done Sent Done false
2064 Acked Write 9 Done Sent Done false
2065 Acked Read 2 Done Sent Done false
2066 Acked Write 1 Done Sent Done false
2067 Acked Read 5 Done Sent Done false
2068 Acked Write 5 Done Sent Done false
2069 Acked Read 5 Done Sent Done false
2070 Acked Write 8 Done Sent Done false
2071 Acked Read 7 Done Sent Done false
2072 Acked Read 2 Done Sent Done false
2073 Acked Write 2 Done Sent Done false
2074 Acked Flush 0 Done Sent Done false
2075 Acked Write 5 Done Sent Done false
2076 Acked Read 5 Done Sent Done false
2077 Acked Read 2 Done Sent Done false
2078 Acked Read 2 Done Sent Done false
2079 Acked Write 2 Done Sent Done false
2080 Acked Flush 0 Done Sent Done false
2081 Acked Read 8 Done Sent Done false
2082 Acked Read 1 Done Sent Done false
2083 Acked Read 9 Done Sent Done false
2084 Acked Write 3 Done Sent Done false
2085 Acked Read 9 Done Sent Done false
2086 Acked Write 9 Done Sent Done false
2087 Acked Write 1 Done Sent Done false
2088 Acked Write 6 Done Sent Done false
2089 Acked Write 5 Done Sent Done false
2090 Acked Read 6 Done Sent Done false
2091 Acked Read 3 Done Sent Done false
2092 Acked Flush 0 Done Sent Done false
2093 Acked Write 7 Done Sent Done false
2094 Acked Flush 0 Done Sent Done false
2095 Acked Read 1 Done Sent Done false
2096 Acked Write 1 Done Sent Done false
2097 Acked Read 3 Done Sent Done false
2098 Acked Read 7 Done Sent Done false
2099 Acked Write 6 Done Sent Done false
2100 Acked Flush 0 Done Sent Done false
2101 Acked Write 7 Done Sent Done false
2102 Acked Read 6 Done Sent Done false
2103 Acked Write 9 Done Sent Done false
2104 Acked Flush 0 Done Sent Done false
2105 Acked Flush 0 Done Sent Done false
2106 Acked Read 5 Done Sent Done false
2107 Acked Write 4 Done Sent Done false
2108 Acked Read 2 Done Sent Done false
2109 Acked Flush 0 Done Sent Done false
2110 Acked Flush 0 Done Sent Done false
2111 Acked Read 6 Done Sent Done false
2112 Acked Write 4 Done Sent Done false
2113 Acked Write 3 Done Sent Done false
2114 Acked Write 3 Done Sent Done false
2115 Acked Read 1 Done Sent Done false
2116 Acked Write 10 Done Sent Done false
2117 Acked Write 9 Done Sent Done false
2118 Acked Write 1 Done Sent Done false
2119 Acked Read 6 Done Sent Done false
2120 Acked Read 1 Done Sent Done false
2121 Acked Write 1 Done Sent Done false
2122 Acked Read 5 Done Sent Done false
2123 Acked Read 4 Done Sent Done false
2124 Acked Read 3 Done Sent Done false
2125 Acked Write 3 Done Sent Done false
2126 Acked Write 2 Done Sent Done false
2127 Acked Write 7 Done Sent Done false
2128 Acked Read 1 Done Sent Done false
2129 Acked Write 9 Done Sent Done false
2130 Acked Read 2 Done Sent Done false
2131 Acked Write 2 Done Sent Done false
2132 Acked Flush 0 Done Sent Done false
2133 Acked Write 9 Done Sent Done false
2134 Acked Read 5 Done Sent Done false
2135 Acked Write 6 Done Sent Done false
2136 Acked Read 7 Done Sent Done false
2137 Acked Read 4 Done Sent Done false
2138 Acked Read 4 Done Sent Done false
2139 Acked Flush 0 Done Sent Done false
2140 Acked Flush 0 Done Sent Done false
2141 Acked Flush 0 Done Sent Done false
2142 Acked Read 1 Done Sent Done false
2143 Acked Write 2 Done Sent Done false
2144 Acked Write 8 Done Sent Done false
2145 Acked Write 9 Done Sent Done false
2146 Acked Write 7 Done Sent Done false
2147 Acked Read 10 Done Sent Done false
2148 Acked Read 5 Done Sent Done false
2149 Acked Write 10 Done Sent Done false
2150 Acked Write 3 Done Sent Done false
2151 Acked Read 4 Done Sent Done false
2152 Acked Read 1 Done Sent Done false
2153 Acked Read 8 Done Sent Done false
2154 Acked Write 8 Done Sent Done false
2155 Acked Flush 0 Done Sent Done false
2156 Acked Read 9 Done Sent Done false
2157 Acked Read 8 Done Sent Done false
2158 Acked Write 1 Done Sent Done false
2159 Acked Flush 0 Done Sent Done false
2160 Acked Read 2 Done Sent Done false
2161 Acked Write 8 Done Sent Done false
2162 Acked Read 7 Done Sent Done false
2163 Acked Write 2 Done Sent Done false
2164 Acked Read 6 Done Sent Done false
2165 Acked Write 2 Done Sent Done false
2166 Acked Read 5 Done Sent Done false
2167 Acked Read 8 Done Sent Done false
2168 Acked Read 4 Done Sent Done false
2169 Acked Read 7 Done Sent Done false
2170 Acked Read 7 Done Sent Done false
2171 Acked Write 7 Done Sent Done false
2172 Acked Write 3 Done Sent Done false
2173 Acked Write 8 Done Sent Done false
2174 Acked Read 6 Done Sent Done false
2175 Acked Read 9 Done Sent Done false
2176 Acked Read 3 Done Sent Done false
2177 Acked Read 3 Done Sent Done false
2178 Acked Flush 0 Done Sent Done false
2179 Acked Read 2 Done Sent Done false
2180 Acked Flush 0 Done Sent Done false
2181 Acked Write 3 Done Sent Done false
2182 Acked Write 6 Done Sent Done false
2183 Acked Read 9 Done Sent Done false
2184 Acked Write 6 Done Sent Done false
2185 Acked Read 7 Done Sent Done false
2186 Acked Read 1 Done Sent Done false
2187 Acked Flush 0 Done Sent Done false
2188 Acked Write 6 Done Sent Done false
2189 Acked Write 10 Done Sent Done false
2190 Acked Read 1 Done Sent Done false
2191 Acked Write 3 Done Sent Done false
2192 Acked Write 4 Done Sent Done false
2193 Acked Read 4 Done Sent Done false
2194 Acked Read 2 Done Sent Done false
2195 Acked Read 4 Done Sent Done false
2196 Acked Write 9 Done Sent Done false
2197 Acked Read 5 Done Sent Done false
2198 Acked Write 8 Done Sent Done false
2199 Acked Read 3 Done Sent Done false
2200 Acked Write 1 Done Sent Done false
2201 Acked Read 6 Done Sent Done false
2202 Acked Flush 0 Done Sent Done false
2203 Acked Read 10 Done Sent Done false
2204 Acked Read 2 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2202)) Some(JobId(1903)) Some(JobId(2202))
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 2204 2203 2202 2201 2200
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:05.443839352Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:06.443800185Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1215)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:07.445799619Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:08.445983348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1904 Acked Write 10 Done Sent Done false
1905 Acked Read 6 Done Sent Done false
1906 Acked Write 10 Done Sent Done false
1907 Acked Read 3 Done Sent Done false
1908 Acked Read 3 Done Sent Done false
1909 Acked Read 7 Done Sent Done false
1910 Acked Read 7 Done Sent Done false
1911 Acked Write 4 Done Sent Done false
1912 Acked Flush 0 Done Sent Done false
1913 Acked Read 2 Done Sent Done false
1914 Acked Write 1 Done Sent Done false
1915 Acked Read 8 Done Sent Done false
1916 Acked Read 6 Done Sent Done false
1917 Acked Read 4 Done Sent Done false
1918 Acked Read 7 Done Sent Done false
1919 Acked Read 4 Done Sent Done false
1920 Acked Read 8 Done Sent Done false
1921 Acked Write 9 Done Sent Done false
1922 Acked Read 10 Done Sent Done false
1923 Acked Read 4 Done Sent Done false
1924 Acked Flush 0 Done Sent Done false
1925 Acked Write 4 Done Sent Done false
1926 Acked Write 3 Done Sent Done false
1927 Acked Read 4 Done Sent Done false
1928 Acked Write 4 Done Sent Done false
1929 Acked Read 9 Done Sent Done false
1930 Acked Read 2 Done Sent Done false
1931 Acked Write 3 Done Sent Done false
1932 Acked Read 2 Done Sent Done false
1933 Acked Read 10 Done Sent Done false
1934 Acked Read 3 Done Sent Done false
1935 Acked Write 7 Done Sent Done false
1936 Acked Read 1 Done Sent Done false
1937 Acked Read 5 Done Sent Done false
1938 Acked Read 3 Done Sent Done false
1939 Acked Read 7 Done Sent Done false
1940 Acked Flush 0 Done Sent Done false
1941 Acked Write 5 Done Sent Done false
1942 Acked Read 3 Done Sent Done false
1943 Acked Write 4 Done Sent Done false
1944 Acked Read 6 Done Sent Done false
1945 Acked Write 4 Done Sent Done false
1946 Acked Write 5 Done Sent Done false
1947 Acked Read 6 Done Sent Done false
1948 Acked Read 5 Done Sent Done false
1949 Acked Write 6 Done Sent Done false
1950 Acked Write 6 Done Sent Done false
1951 Acked Write 10 Done Sent Done false
1952 Acked Write 3 Done Sent Done false
1953 Acked Flush 0 Done Sent Done false
1954 Acked Write 7 Done Sent Done false
1955 Acked Write 6 Done Sent Done false
1956 Acked Write 10 Done Sent Done false
1957 Acked Write 7 Done Sent Done false
1958 Acked Write 8 Done Sent Done false
1959 Acked Read 2 Done Sent Done false
1960 Acked Write 4 Done Sent Done false
1961 Acked Read 6 Done Sent Done false
1962 Acked Read 4 Done Sent Done false
1963 Acked Read 7 Done Sent Done false
1964 Acked Write 2 Done Sent Done false
1965 Acked Read 4 Done Sent Done false
1966 Acked Write 7 Done Sent Done false
1967 Acked Read 10 Done Sent Done false
1968 Acked Write 7 Done Sent Done false
1969 Acked Write 8 Done Sent Done false
1970 Acked Write 4 Done Sent Done false
1971 Acked Read 3 Done Sent Done false
1972 Acked Read 8 Done Sent Done false
1973 Acked Read 7 Done Sent Done false
1974 Acked Write 1 Done Sent Done false
1975 Acked Write 1 Done Sent Done false
1976 Acked Write 3 Done Sent Done false
1977 Acked Read 6 Done Sent Done false
1978 Acked Write 8 Done Sent Done false
1979 Acked Write 9 Done Sent Done false
1980 Acked Write 4 Done Sent Done false
1981 Acked Read 10 Done Sent Done false
1982 Acked Write 8 Done Sent Done false
1983 Acked Write 5 Done Sent Done false
1984 Acked Flush 0 Done Sent Done false
1985 Acked Flush 0 Done Sent Done false
1986 Acked Read 10 Done Sent Done false
1987 Acked Read 6 Done Sent Done false
1988 Acked Flush 0 Done Sent Done false
1989 Acked Read 9 Done Sent Done false
1990 Acked Read 10 Done Sent Done false
1991 Acked Read 10 Done Sent Done false
1992 Acked Read 5 Done Sent Done false
1993 Acked Write 5 Done Sent Done false
1994 Acked Flush 0 Done Sent Done false
1995 Acked Read 5 Done Sent Done false
1996 Acked Read 3 Done Sent Done false
1997 Acked Read 9 Done Sent Done false
1998 Acked Write 10 Done Sent Done false
1999 Acked Flush 0 Done Sent Done false
2000 Acked Read 10 Done Sent Done false
2001 Acked Flush 0 Done Sent Done false
2002 Acked Flush 0 Done Sent Done false
2003 Acked Write 7 Done Sent Done false
2004 Acked Read 4 Done Sent Done false
2005 Acked Read 6 Done Sent Done false
2006 Acked Flush 0 Done Sent Done false
2007 Acked Read 8 Done Sent Done false
2008 Acked Write 5 Done Sent Done false
2009 Acked Read 9 Done Sent Done false
2010 Acked Read 7 Done Sent Done false
2011 Acked Write 10 Done Sent Done false
2012 Acked Write 10 Done Sent Done false
2013 Acked Write 9 Done Sent Done false
2014 Acked Write 8 Done Sent Done false
2015 Acked Write 9 Done Sent Done false
2016 Acked Read 3 Done Sent Done false
2017 Acked Write 7 Done Sent Done false
2018 Acked Read 1 Done Sent Done false
2019 Acked Read 10 Done Sent Done false
2020 Acked Read 7 Done Sent Done false
2021 Acked Read 7 Done Sent Done false
2022 Acked Read 4 Done Sent Done false
2023 Acked Write 7 Done Sent Done false
2024 Acked Read 10 Done Sent Done false
2025 Acked Write 3 Done Sent Done false
2026 Acked Write 10 Done Sent Done false
2027 Acked Flush 0 Done Sent Done false
2028 Acked Read 2 Done Sent Done false
2029 Acked Read 6 Done Sent Done false
2030 Acked Write 2 Done Sent Done false
2031 Acked Flush 0 Done Sent Done false
2032 Acked Write 6 Done Sent Done false
2033 Acked Write 6 Done Sent Done false
2034 Acked Read 4 Done Sent Done false
2035 Acked Read 8 Done Sent Done false
2036 Acked Flush 0 Done Sent Done false
2037 Acked Read 8 Done Sent Done false
2038 Acked Read 1 Done Sent Done false
2039 Acked Write 10 Done Sent Done false
2040 Acked Read 7 Done Sent Done false
2041 Acked Write 1 Done Sent Done false
2042 Acked Flush 0 Done Sent Done false
2043 Acked Write 6 Done Sent Done false
2044 Acked Read 6 Done Sent Done false
2045 Acked Write 5 Done Sent Done false
2046 Acked Read 8 Done Sent Done false
2047 Acked Write 7 Done Sent Done false
2048 Acked Read 2 Done Sent Done false
2049 Acked Flush 0 Done Sent Done false
2050 Acked Write 9 Done Sent Done false
2051 Acked Read 10 Done Sent Done false
2052 Acked Write 7 Done Sent Done false
2053 Acked Write 4 Done Sent Done false
2054 Acked Flush 0 Done Sent Done false
2055 Acked Flush 0 Done Sent Done false
2056 Acked Read 6 Done Sent Done false
2057 Acked Flush 0 Done Sent Done false
2058 Acked Read 2 Done Sent Done false
2059 Acked Read 7 Done Sent Done false
2060 Acked Read 9 Done Sent Done false
2061 Acked Flush 0 Done Sent Done false
2062 Acked Read 5 Done Sent Done false
2063 Acked Read 2 Done Sent Done false
2064 Acked Write 9 Done Sent Done false
2065 Acked Read 2 Done Sent Done false
2066 Acked Write 1 Done Sent Done false
2067 Acked Read 5 Done Sent Done false
2068 Acked Write 5 Done Sent Done false
2069 Acked Read 5 Done Sent Done false
2070 Acked Write 8 Done Sent Done false
2071 Acked Read 7 Done Sent Done false
2072 Acked Read 2 Done Sent Done false
2073 Acked Write 2 Done Sent Done false
2074 Acked Flush 0 Done Sent Done false
2075 Acked Write 5 Done Sent Done false
2076 Acked Read 5 Done Sent Done false
2077 Acked Read 2 Done Sent Done false
2078 Acked Read 2 Done Sent Done false
2079 Acked Write 2 Done Sent Done false
2080 Acked Flush 0 Done Sent Done false
2081 Acked Read 8 Done Sent Done false
2082 Acked Read 1 Done Sent Done false
2083 Acked Read 9 Done Sent Done false
2084 Acked Write 3 Done Sent Done false
2085 Acked Read 9 Done Sent Done false
2086 Acked Write 9 Done Sent Done false
2087 Acked Write 1 Done Sent Done false
2088 Acked Write 6 Done Sent Done false
2089 Acked Write 5 Done Sent Done false
2090 Acked Read 6 Done Sent Done false
2091 Acked Read 3 Done Sent Done false
2092 Acked Flush 0 Done Sent Done false
2093 Acked Write 7 Done Sent Done false
2094 Acked Flush 0 Done Sent Done false
2095 Acked Read 1 Done Sent Done false
2096 Acked Write 1 Done Sent Done false
2097 Acked Read 3 Done Sent Done false
2098 Acked Read 7 Done Sent Done false
2099 Acked Write 6 Done Sent Done false
2100 Acked Flush 0 Done Sent Done false
2101 Acked Write 7 Done Sent Done false
2102 Acked Read 6 Done Sent Done false
2103 Acked Write 9 Done Sent Done false
2104 Acked Flush 0 Done Sent Done false
2105 Acked Flush 0 Done Sent Done false
2106 Acked Read 5 Done Sent Done false
2107 Acked Write 4 Done Sent Done false
2108 Acked Read 2 Done Sent Done false
2109 Acked Flush 0 Done Sent Done false
2110 Acked Flush 0 Done Sent Done false
2111 Acked Read 6 Done Sent Done false
2112 Acked Write 4 Done Sent Done false
2113 Acked Write 3 Done Sent Done false
2114 Acked Write 3 Done Sent Done false
2115 Acked Read 1 Done Sent Done false
2116 Acked Write 10 Done Sent Done false
2117 Acked Write 9 Done Sent Done false
2118 Acked Write 1 Done Sent Done false
2119 Acked Read 6 Done Sent Done false
2120 Acked Read 1 Done Sent Done false
2121 Acked Write 1 Done Sent Done false
2122 Acked Read 5 Done Sent Done false
2123 Acked Read 4 Done Sent Done false
2124 Acked Read 3 Done Sent Done false
2125 Acked Write 3 Done Sent Done false
2126 Acked Write 2 Done Sent Done false
2127 Acked Write 7 Done Sent Done false
2128 Acked Read 1 Done Sent Done false
2129 Acked Write 9 Done Sent Done false
2130 Acked Read 2 Done Sent Done false
2131 Acked Write 2 Done Sent Done false
2132 Acked Flush 0 Done Sent Done false
2133 Acked Write 9 Done Sent Done false
2134 Acked Read 5 Done Sent Done false
2135 Acked Write 6 Done Sent Done false
2136 Acked Read 7 Done Sent Done false
2137 Acked Read 4 Done Sent Done false
2138 Acked Read 4 Done Sent Done false
2139 Acked Flush 0 Done Sent Done false
2140 Acked Flush 0 Done Sent Done false
2141 Acked Flush 0 Done Sent Done false
2142 Acked Read 1 Done Sent Done false
2143 Acked Write 2 Done Sent Done false
2144 Acked Write 8 Done Sent Done false
2145 Acked Write 9 Done Sent Done false
2146 Acked Write 7 Done Sent Done false
2147 Acked Read 10 Done Sent Done false
2148 Acked Read 5 Done Sent Done false
2149 Acked Write 10 Done Sent Done false
2150 Acked Write 3 Done Sent Done false
2151 Acked Read 4 Done Sent Done false
2152 Acked Read 1 Done Sent Done false
2153 Acked Read 8 Done Sent Done false
2154 Acked Write 8 Done Sent Done false
2155 Acked Flush 0 Done Sent Done false
2156 Acked Read 9 Done Sent Done false
2157 Acked Read 8 Done Sent Done false
2158 Acked Write 1 Done Sent Done false
2159 Acked Flush 0 Done Sent Done false
2160 Acked Read 2 Done Sent Done false
2161 Acked Write 8 Done Sent Done false
2162 Acked Read 7 Done Sent Done false
2163 Acked Write 2 Done Sent Done false
2164 Acked Read 6 Done Sent Done false
2165 Acked Write 2 Done Sent Done false
2166 Acked Read 5 Done Sent Done false
2167 Acked Read 8 Done Sent Done false
2168 Acked Read 4 Done Sent Done false
2169 Acked Read 7 Done Sent Done false
2170 Acked Read 7 Done Sent Done false
2171 Acked Write 7 Done Sent Done false
2172 Acked Write 3 Done Sent Done false
2173 Acked Write 8 Done Sent Done false
2174 Acked Read 6 Done Sent Done false
2175 Acked Read 9 Done Sent Done false
2176 Acked Read 3 Done Sent Done false
2177 Acked Read 3 Done Sent Done false
2178 Acked Flush 0 Done Sent Done false
2179 Acked Read 2 Done Sent Done false
2180 Acked Flush 0 Done Sent Done false
2181 Acked Write 3 Done Sent Done false
2182 Acked Write 6 Done Sent Done false
2183 Acked Read 9 Done Sent Done false
2184 Acked Write 6 Done Sent Done false
2185 Acked Read 7 Done Sent Done false
2186 Acked Read 1 Done Sent Done false
2187 Acked Flush 0 Done Sent Done false
2188 Acked Write 6 Done Sent Done false
2189 Acked Write 10 Done Sent Done false
2190 Acked Read 1 Done Sent Done false
2191 Acked Write 3 Done Sent Done false
2192 Acked Write 4 Done Sent Done false
2193 Acked Read 4 Done Sent Done false
2194 Acked Read 2 Done Sent Done false
2195 Acked Read 4 Done Sent Done false
2196 Acked Write 9 Done Sent Done false
2197 Acked Read 5 Done Sent Done false
2198 Acked Write 8 Done Sent Done false
2199 Acked Read 3 Done Sent Done false
2200 Acked Write 1 Done Sent Done false
2201 Acked Read 6 Done Sent Done false
2202 Acked Flush 0 Done Sent Done false
2203 Acked Read 10 Done Sent Done false
2204 Acked Read 2 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2202)) Some(JobId(1903)) Some(JobId(2202))
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 2204 2203 2202 2201 2200
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:09.447739551Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:10.44796736Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.640308524Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37128)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.640547356Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.641553795Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642078269Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642101664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(1903))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642129367Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642143904Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(1903))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642157211Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(1903))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:10.642389945Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2205 2204 2203 2202 2201
Upstairs last five completed: 2205 2204 2203 2202 2201
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:02:12.882 INFO[8820] Got stop action so:false kr:false
request completed, latency_us: 342, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3086fd52-3f58-48fe-b182-c7de404c4107, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:02:12.882 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: b06365b1-cc7d-4a4e-a62d-7f2b3b7f6801, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:12.882818241Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:12.882886255Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:12.882913468Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:12.882951249Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:02:16.885 INFO request completed, latency_us: 399, response_code: 200, uri: /state/cid/1, method: GET, req_id: 275ec45e-eee3-40e0-b3bb-c67ecf225d73, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 4967 len 16384 data: 3 2 2 2
002/300 Read block 3779 len 32768
003/300 Read block 209 len 40960
004/300 Read block 4935 len 32768
005/300 Write block 3351 len 24576 data: 2 2 2 2 2 2
006/300 Flush
007/300 Write block 1198 len 4096 data: 2
008/300 Read block 3251 len 36864
009/300 Write block 3974 len 36864 data: 2 2 2 2 2 2 2 2 2
010/300 Read block 3960 len 24576
011/300 Read block 4832 len 28672
012/300 Write block 303 len 24576 data: 2 2 2 2 2 2
013/300 Read block 4476 len 36864
014/300 Read block 4976 len 20480
015/300 Write block 80 len 28672 data: 2 3 3 3 2 2 2
016/300 Read block 4976 len 24576
017/300 Write block 1084 len 24576 data: 2 2 2 2 2 2
018/300 Read block 2357 len 16384
019/300 Write block 493 len 28672 data: 2 2 2 2 2 2 2
020/300 Read block 1783 len 16384
021/300 Read block 98 len 32768
022/300 Write block 51 len 32768 data: 2 2 2 2 2 2 2 2
023/300 Read block 4041 len 4096
024/300 Read block 2425 len 8192
025/300 Write block 174 len 20480 data: 3 3 3 3 3
026/300 Read block 4124 len 36864
027/300 Read block 1878 len 24576
028/300 Write block 4524 len 28672 data: 2 2 2 2 3 3 3
029/300 Read block 1383 len 28672
030/300 Write block 3873 len 32768 data: 4 4 4 4 3 3 2 2
031/300 Flush
032/300 Read block 4882 len 8192
033/300 Flush
034/300 Read block 2739 len 4096
035/300 Write block 2213 len 28672 data: 5 5 5 5 5 5 3
036/300 Flush
037/300 Read block 2218 len 28672
038/300 Flush
039/300 Write block 4043 len 16384 data: 2 2 2 2
040/300 Write block 4630 len 16384 data: 2 3 3 3
041/300 Write block 2306 len 16384 data: 2 3 3 3
042/300 Write block 1473 len 8192 data: 2 2
043/300 Read block 1873 len 16384
044/300 Write block 2000 len 16384 data: 2 2 2 2
045/300 Read block 1179 len 12288
046/300 Write block 3743 len 36864 data: 4 4 4 2 2 2 2 2 3
047/300 Write block 4796 len 12288 data: 2 2 2
048/300 Write block 1161 len 4096 data: 3
049/300 Write block 449 len 24576 data: 6 6 6 6 5 5
050/300 Read block 2163 len 12288
051/300 Write block 4777 len 12288 data: 3 3 3
052/300 Write block 3791 len 8192 data: 2 2
053/300 Read block 1529 len 28672
054/300 Flush
055/300 Read block 3042 len 40960
056/300 Read block 2927 len 4096
057/300 Read block 4734 len 4096
058/300 Write block 160 len 12288 data: 2 2 2
059/300 Write block 1052 len 20480 data: 3 3 3 2 2
060/300 Write block 251 len 12288 data: 2 2 2
061/300 Flush
062/300 Read block 4402 len 40960
063/300 Read block 839 len 20480
064/300 Flush
065/300 Write block 1983 len 28672 data: 2 2 2 2 2 3 3
066/300 Read block 1562 len 28672
067/300 Read block 1914 len 40960
068/300 Write block 1084 len 32768 data: 3 3 3 3 3 3 2 2
069/300 Read block 4398 len 16384
070/300 Write block 358 len 36864 data: 2 2 2 2 2 2 2 2 2
071/300 Write block 4825 len 32768 data: 2 2 2 3 3 4 5 3
072/300 Write block 188 len 36864 data: 2 2 2 3 3 3 3 2 2
073/300 Read block 3622 len 8192
074/300 Write block 3410 len 8192 data: 2 2
075/300 Read block 3750 len 24576
076/300 Read block 3873 len 36864
077/300 Read block 2555 len 40960
078/300 Read block 3781 len 36864
079/300 Read block 664 len 8192
080/300 Read block 3678 len 36864
081/300 Read block 1804 len 16384
082/300 Write block 1440 len 36864 data: 2 2 2 2 2 2 2 2 3
083/300 Read block 1817 len 12288
084/300 Read block 264 len 40960
085/300 Write block 2256 len 8192 data: 2 3
086/300 Write block 2811 len 40960 data: 2 2 2 2 2 2 2 2 2 2
087/300 Flush
088/300 Read block 4321 len 12288
089/300 Write block 2957 len 12288 data: 2 2 2
090/300 Read block 1109 len 16384
091/300 Read block 4648 len 8192
092/300 Write block 4280 len 12288 data: 3 3 3
093/300 Read block 3114 len 20480
094/300 Read block 844 len 32768
095/300 Read block 181 len 12288
096/300 Read block 1397 len 4096
097/300 Read block 4678 len 24576
098/300 Write block 1636 len 8192 data: 3 3
099/300 Write block 4699 len 20480 data: 2 2 2 2 2
100/300 Write block 2471 len 36864 data: 2 2 2 2 2 2 2 2 2
101/300 Read block 431 len 12288
102/300 Write block 1613 len 20480 data: 3 3 3 3 3
103/300 Read block 4113 len 32768
104/300 Write block 1247 len 16384 data: 2 2 2 2
105/300 Read block 64 len 8192
106/300 Read block 2624 len 4096
107/300 Read block 419 len 24576
108/300 Read block 2259 len 32768
109/300 Read block 201 len 28672
110/300 Read block 1034 len 20480
111/300 Write block 4652 len 28672 data: 2 2 2 2 2 2 2
112/300 Write block 2189 len 8192 data: 3 4
113/300 Write block 472 len 28672 data: 3 3 3 3 3 3 3
114/300 Write block 3931 len 40960 data: 2 2 2 3 3 3 3 4 5 5
115/300 Read block 2130 len 40960
116/300 Write block 2318 len 8192 data: 4 5
117/300 Write block 4289 len 40960 data: 2 3 3 3 2 2 3 3 3 2
118/300 Write block 509 len 4096 data: 2
119/300 Read block 4708 len 16384
120/300 Read block 2836 len 20480
121/300 Write block 667 len 24576 data: 4 4 4 4 4 4
122/300 Read block 1374 len 40960
123/300 Write block 1660 len 40960 data: 3 2 2 2 2 2 2 2 3 3
124/300 Read block 869 len 24576
125/300 Read block 4338 len 24576
126/300 Write block 700 len 20480 data: 2 2 2 2 2
127/300 Write block 2178 len 20480 data: 2 3 3 3 3
128/300 Write block 2929 len 24576 data: 2 2 2 2 2 2
129/300 Read block 1298 len 12288
130/300 Flush
131/300 Read block 1912 len 16384
132/300 Flush
133/300 Write block 1621 len 4096 data: 3
134/300 Flush
135/300 Read block 4631 len 4096
136/300 Write block 345 len 36864 data: 3 3 3 3 3 3 2 2 2
137/300 Write block 1441 len 36864 data: 3 3 3 3 3 3 3 4 3
138/300 Read block 669 len 28672
139/300 Write block 4226 len 24576 data: 2 2 2 2 2 2
140/300 Write block 526 len 40960 data: 2 2 2 2 2 2 3 3 3 2
141/300 Read block 934 len 12288
142/300 Write block 4647 len 8192 data: 3 2
143/300 Read block 4465 len 12288
144/300 Read block 4403 len 20480
145/300 Write block 2179 len 20480 data: 4 4 4 4 3
146/300 Write block 3323 len 4096 data: 2
147/300 Write block 1077 len 16384 data: 2 3 3 4
148/300 Read block 1877 len 40960
149/300 Read block 3604 len 28672
150/300 Write block 4791 len 20480 data: 2 2 2 2 2
151/300 Read block 4975 len 8192
152/300 Write block 2757 len 28672 data: 2 2 2 2 2 2 2
153/300 Write block 295 len 28672 data: 3 3 3 3 3 2 2
154/300 Write block 4407 len 36864 data: 5 6 6 6 6 6 4 3 3
155/300 Read block 1220 len 20480
156/300 Write block 2595 len 8192 data: 2 2
157/300 Flush
158/300 Read block 3232 len 40960
159/300 Read block 4770 len 12288
160/300 Read block 1569 len 40960
161/300 Read block 1971 len 4096
162/300 Write block 4461 len 40960 data: 2 2 2 2 2 2 2 2 2 2
163/300 Read block 34 len 28672
164/300 Read block 4467 len 20480
165/300 Write block 4186 len 24576 data: 4 4 2 2 4 4
166/300 Read block 4162 len 40960
167/300 Write block 4383 len 16384 data: 2 2 2 2
168/300 Read block 4668 len 24576
169/300 Write block 414 len 40960 data: 3 3 3 3 3 3 3 3 4 3
170/300 Read block 4381 len 12288
171/300 Read block 2981 len 40960
172/300 Read block 3796 len 16384
173/300 Flush
174/300 Read block 3707 len 24576
175/300 Read block 3490 len 8192
176/300 Flush
177/300 Read block 2212 len 8192
178/300 Write block 3662 len 32768 data: 2 2 2 3 3 3 3 3
179/300 Flush
180/300 Write block 127 len 28672 data: 3 3 3 3 3 2 4
181/300 Write block 2323 len 32768 data: 3 2 2 2 2 2 2 2
182/300 Write block 2487 len 12288 data: 2 2 2
183/300 Write block 1329 len 16384 data: 3 3 3 3
184/300 Write block 3970 len 40960 data: 2 2 2 2 3 3 3 3 3 3
185/300 Write block 2547 len 24576 data: 3 3 3 3 3 2
186/300 Read block 4469 len 20480
187/300 Flush
188/300 Read block 1715 len 12288
189/300 Write block 4138 len 24576 data: 2 2 2 3 3 3
190/300 Write block 4557 len 36864 data: 2 2 2 2 2 2 2 2 2
191/300 Write block 2983 len 12288 data: 3 3 3
192/300 Write block 4906 len 40960 data: 2 2 2 2 2 2 2 2 2 2
193/300 Read block 3143 len 32768
194/300 Write block 4052 len 20480 data: 2 2 3 3 3
195/300 Write block 2045 len 36864 data: 2 2 2 2 2 2 2 2 2
196/300 Write block 1254 len 20480 data: 2 2 2 2 2
197/300 Write block 99 len 36864 data: 2 2 2 2 2 2 2 2 3
198/300 Write block 572 len 36864 data: 3 3 2 2 2 3 3 3 4
199/300 Read block 3109 len 4096
200/300 Write block 1220 len 20480 data: 2 2 2 2 2
201/300 Read block 1440 len 28672
202/300 Read block 4342 len 20480
203/300 Read block 3940 len 8192
204/300 Write block 1219 len 4096 data: 2
205/300 Read block 1550 len 16384
206/300 Read block 1074 len 40960
207/300 Write block 2747 len 4096 data: 3
208/300 Flush
209/300 Read block 1474 len 40960
210/300 Flush
211/300 Read block 4488 len 16384
212/300 Write block 236 len 28672 data: 3 5 5 3 2 2 2
213/300 Read block 3598 len 36864
214/300 Write block 2579 len 12288 data: 2 2 2
215/300 Write block 4788 len 40960 data: 3 3 2 3 3 3 3 3 3 3
216/300 Write block 1172 len 16384 data: 2 2 2 2
217/300 Write block 2278 len 16384 data: 4 3 3 3
218/300 Read block 4788 len 20480
219/300 Read block 2846 len 32768
220/300 Write block 2842 len 36864 data: 2 2 2 2 2 2 2 2 2
221/300 Flush
222/300 Flush
223/300 Read block 831 len 8192
224/300 Write block 4271 len 16384 data: 4 4 3 3
225/300 Read block 2268 len 20480
226/300 Write block 3848 len 28672 data: 2 2 2 2 3 3 3
227/300 Read block 171 len 4096
228/300 Read block 2657 len 28672
229/300 Read block 1905 len 4096
230/300 Write block 458 len 16384 data: 2 2 2 2
231/300 Write block 3718 len 28672 data: 2 2 2 2 2 2 2
232/300 Write block 4319 len 20480 data: 2 2 2 2 2
233/300 Read block 498 len 20480
234/300 Read block 4644 len 28672
235/300 Read block 1022 len 28672
236/300 Read block 1656 len 36864
237/300 Read block 4943 len 8192
238/300 Read block 2122 len 16384
239/300 Write block 1203 len 16384 data: 2 2 2 2
240/300 Read block 14 len 16384
241/300 Read block 3655 len 28672
242/300 Write block 3742 len 24576 data: 4 5 5 5 3 3
243/300 Write block 3411 len 16384 data: 3 2 2 2
244/300 Flush
245/300 Read block 4872 len 24576
246/300 Write block 870 len 32768 data: 3 2 2 2 2 2 2 2
247/300 Read block 3432 len 20480
248/300 Read block 2931 len 16384
249/300 Write block 3293 len 24576 data: 2 2 2 3 4 4
250/300 Read block 1332 len 32768
251/300 Read block 1930 len 20480
252/300 Write block 111 len 8192 data: 3 3
253/300 Write block 4971 len 16384 data: 2 2 2 2
254/300 Write block 2968 len 20480 data: 4 4 3 2 2
255/300 Write block 4737 len 36864 data: 2 2 2 2 2 2 2 2 2
256/300 Write block 598 len 24576 data: 2 2 2 2 2 2
257/300 Read block 3482 len 8192
258/300 Read block 2418 len 24576
259/300 Read block 3927 len 24576
260/300 Write block 1839 len 8192 data: 3 3
261/300 Flush
262/300 Write block 3520 len 40960 data: 2 2 3 3 3 3 3 4 4 3
263/300 Read block 718 len 36864
264/300 Write block 4593 len 12288 data: 2 2 2
265/300 Read block 3249 len 36864
266/300 Read block 2201 len 20480
267/300 Flush
268/300 Write block 1747 len 16384 data: 2 2 2 2
269/300 Write block 1534 len 36864 data: 3 3 3 3 4 3 3 3 3
270/300 Write block 2369 len 4096 data: 3
271/300 Read block 1995 len 28672
272/300 Write block 1451 len 28672 data: 3 3 3 3 2 2 2
273/300 Read block 1670 len 32768
274/300 Write block 791 len 20480 data: 2 2 2 2 2
275/300 Read block 285 len 28672
276/300 Read block 1260 len 20480
277/300 Write block 1186 len 40960 data: 3 2 2 2 2 2 2 2 2 2
278/300 Write block 4647 len 36864 data: 4 3 2 2 2 3 3 3 3
279/300 Write block 3533 len 8192 data: 2 2
280/300 Write block 2861 len 40960 data: 2 2 3 2 2 2 2 2 2 2
281/300 Read block 1285 len 16384
282/300 Write block 25 len 16384 data: 3 2 2 2
283/300 Read block 2482 len 4096
284/300 Read block 4752 len 8192
285/300 Read block 4265 len 20480
286/300 Read block 3101 len 32768
287/300 Read block 3075 len 8192
288/300 Read block 4797 len 20480
289/300 Read block 3618 len 40960
290/300 Write block 380 len 28672 data: 5 4 4 4 3 3 3
291/300 Write block 2749 len 32768 data: 2 3 3 2 2 2 2 2
292/300 Write block 477 len 8192 data: 4 4
293/300 Read block 2302 len 12288
294/300 Read block 4698 len 20480
295/300 Write block 2279 len 20480 data: 4 4 4 3 4
296/300 Write block 1889 len 20480 data: 2 2 4 4 4
297/300 Read block 4783 len 28672
298/300 Write block 1514 len 20480 data: 3 2 2 3 3
299/300 Write block 264 len 20480 data: 2 2 2 2 2
300/300 Write block 1936 len 36864 data: 3 3 2 2 2 2 2 2 2
301/300 Flush
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:02:17.123 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: 551c31d2-9831-41ff-bcfd-1f485fd0f484, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2206 Acked Write 4 Done Sent Done false
2207 Acked Read 8 Done Sent Done false
2208 Acked Read 10 Done Sent Done false
2209 Acked Read 8 Done Sent Done false
2210 Acked Write 6 Done Sent Done false
2211 Acked Flush 0 Done Sent Done false
2212 Acked Write 1 Done Sent Done false
2213 Acked Read 9 Done Sent Done false
2214 Acked Write 9 Done Sent Done false
2215 Acked Read 6 Done Sent Done false
2216 Acked Read 7 Done Sent Done false
2217 Acked Write 6 Done Sent Done false
2218 Acked Read 9 Done Sent Done false
2219 Acked Read 5 Done Sent Done false
2220 Acked Write 7 Done Sent Done false
2221 Acked Read 6 Done Sent Done false
2222 Acked Write 6 Done Sent Done false
2223 Acked Read 4 Done Sent Done false
2224 Acked Write 7 Done Sent Done false
2225 Acked Read 4 Done Sent Done false
2226 Acked Read 8 Done Sent Done false
2227 Acked Write 8 Done Sent Done false
2228 Acked Read 1 Done Sent Done false
2229 Acked Read 2 Done Sent Done false
2230 Acked Write 5 Done Sent Done false
2231 Acked Read 9 Done Sent Done false
2232 Acked Read 6 Done Sent Done false
2233 Acked Write 7 Done Sent Done false
2234 Acked Read 7 Done Sent Done false
2235 Acked Write 8 Done Sent Done false
2236 Acked Flush 0 Done Sent Done false
2237 Acked Read 2 Done Sent Done false
2238 Acked Flush 0 Done Sent Done false
2239 Acked Read 1 Done Sent Done false
2240 Acked Write 7 Done Sent Done false
2241 Acked Flush 0 Done Sent Done false
2242 Acked Read 7 Done Sent Done false
2243 Acked Flush 0 Done Sent Done false
2244 Acked Write 4 Done Sent Done false
2245 Acked Write 4 Done Sent Done false
2246 Acked Write 4 Done Sent Done false
2247 Acked Write 2 Done Sent Done false
2248 Acked Read 4 Done Sent Done false
2249 Acked Write 4 Done Sent Done false
2250 Acked Read 3 Done Sent Done false
2251 Acked Write 9 Done Sent Done false
2252 Acked Write 3 Done Sent Done false
2253 Acked Write 1 Done Sent Done false
2254 Acked Write 6 Done Sent Done false
2255 Acked Read 3 Done Sent Done false
2256 Acked Write 3 Done Sent Done false
2257 Acked Write 2 Done Sent Done false
2258 Acked Read 7 Done Sent Done false
2259 Acked Flush 0 Done Sent Done false
2260 Acked Read 10 Done Sent Done false
2261 Acked Read 1 Done Sent Done false
2262 Acked Read 1 Done Sent Done false
2263 Acked Write 3 Done Sent Done false
2264 Acked Write 5 Done Sent Done false
2265 Acked Write 3 Done Sent Done false
2266 Acked Flush 0 Done Sent Done false
2267 Acked Read 10 Done Sent Done false
2268 Acked Read 5 Done Sent Done false
2269 Acked Flush 0 Done Sent Done false
2270 Acked Write 7 Done Sent Done false
2271 Acked Read 7 Done Sent Done false
2272 Acked Read 10 Done Sent Done false
2273 Acked Write 8 Done Sent Done false
2274 Acked Read 4 Done Sent Done false
2275 Acked Write 9 Done Sent Done false
2276 Acked Write 8 Done Sent Done false
2277 Acked Write 9 Done Sent Done false
2278 Acked Read 2 Done Sent Done false
2279 Acked Write 2 Done Sent Done false
2280 Acked Read 6 Done Sent Done false
2281 Acked Read 9 Done Sent Done false
2282 Acked Read 10 Done Sent Done false
2283 Acked Read 9 Done Sent Done false
2284 Acked Read 2 Done Sent Done false
2285 Acked Read 9 Done Sent Done false
2286 Acked Read 4 Done Sent Done false
2287 Acked Write 9 Done Sent Done false
2288 Acked Read 3 Done Sent Done false
2289 Acked Read 10 Done Sent Done false
2290 Acked Write 2 Done Sent Done false
2291 Acked Write 10 Done Sent Done false
2292 Acked Flush 0 Done Sent Done false
2293 Acked Read 3 Done Sent Done false
2294 Acked Write 3 Done Sent Done false
2295 Acked Read 4 Done Sent Done false
2296 Acked Read 2 Done Sent Done false
2297 Acked Write 3 Done Sent Done false
2298 Acked Read 5 Done Sent Done false
2299 Acked Read 8 Done Sent Done false
2300 Acked Read 3 Done Sent Done false
2301 Acked Read 1 Done Sent Done false
2302 Acked Read 6 Done Sent Done false
2303 Acked Write 2 Done Sent Done false
2304 Acked Write 5 Done Sent Done false
2305 Acked Write 9 Done Sent Done false
2306 Acked Read 3 Done Sent Done false
2307 Acked Write 5 Done Sent Done false
2308 Acked Read 8 Done Sent Done false
2309 Acked Write 4 Done Sent Done false
2310 Acked Read 2 Done Sent Done false
2311 Acked Read 1 Done Sent Done false
2312 Acked Read 6 Done Sent Done false
2313 Acked Read 8 Done Sent Done false
2314 Acked Read 7 Done Sent Done false
2315 Acked Read 5 Done Sent Done false
2316 Acked Write 7 Done Sent Done false
2317 Acked Write 2 Done Sent Done false
2318 Acked Write 7 Done Sent Done false
2319 Acked Write 10 Done Sent Done false
2320 Acked Read 10 Done Sent Done false
2321 Acked Write 2 Done Sent Done false
2322 Acked Write 10 Done Sent Done false
2323 Acked Write 1 Done Sent Done false
2324 Acked Read 4 Done Sent Done false
2325 Acked Read 5 Done Sent Done false
2326 Acked Write 6 Done Sent Done false
2327 Acked Read 10 Done Sent Done false
2328 Acked Write 10 Done Sent Done false
2329 Acked Read 6 Done Sent Done false
2330 Acked Read 6 Done Sent Done false
2331 Acked Write 5 Done Sent Done false
2332 Acked Write 5 Done Sent Done false
2333 Acked Write 6 Done Sent Done false
2334 Acked Read 3 Done Sent Done false
2335 Acked Flush 0 Done Sent Done false
2336 Acked Read 4 Done Sent Done false
2337 Acked Flush 0 Done Sent Done false
2338 Acked Write 1 Done Sent Done false
2339 Acked Flush 0 Done Sent Done false
2340 Acked Read 1 Done Sent Done false
2341 Acked Write 9 Done Sent Done false
2342 Acked Write 9 Done Sent Done false
2343 Acked Read 7 Done Sent Done false
2344 Acked Write 6 Done Sent Done false
2345 Acked Write 10 Done Sent Done false
2346 Acked Read 3 Done Sent Done false
2347 Acked Write 2 Done Sent Done false
2348 Acked Read 3 Done Sent Done false
2349 Acked Read 5 Done Sent Done false
2350 Acked Write 5 Done Sent Done false
2351 Acked Write 1 Done Sent Done false
2352 Acked Write 4 Done Sent Done false
2353 Acked Read 10 Done Sent Done false
2354 Acked Read 7 Done Sent Done false
2355 Acked Write 5 Done Sent Done false
2356 Acked Read 2 Done Sent Done false
2357 Acked Write 7 Done Sent Done false
2358 Acked Write 7 Done Sent Done false
2359 Acked Write 9 Done Sent Done false
2360 Acked Read 5 Done Sent Done false
2361 Acked Write 2 Done Sent Done false
2362 Acked Flush 0 Done Sent Done false
2363 Acked Read 10 Done Sent Done false
2364 Acked Read 3 Done Sent Done false
2365 Acked Read 10 Done Sent Done false
2366 Acked Read 1 Done Sent Done false
2367 Acked Write 10 Done Sent Done false
2368 Acked Read 7 Done Sent Done false
2369 Acked Read 5 Done Sent Done false
2370 Acked Write 6 Done Sent Done false
2371 Acked Read 10 Done Sent Done false
2372 Acked Write 4 Done Sent Done false
2373 Acked Read 6 Done Sent Done false
2374 Acked Write 10 Done Sent Done false
2375 Acked Read 3 Done Sent Done false
2376 Acked Read 10 Done Sent Done false
2377 Acked Read 4 Done Sent Done false
2378 Acked Flush 0 Done Sent Done false
2379 Acked Read 6 Done Sent Done false
2380 Acked Read 2 Done Sent Done false
2381 Acked Flush 0 Done Sent Done false
2382 Acked Read 2 Done Sent Done false
2383 Acked Write 8 Done Sent Done false
2384 Acked Flush 0 Done Sent Done false
2385 Acked Write 7 Done Sent Done false
2386 Acked Write 8 Done Sent Done false
2387 Acked Write 3 Done Sent Done false
2388 Acked Write 4 Done Sent Done false
2389 Acked Write 10 Done Sent Done false
2390 Acked Write 6 Done Sent Done false
2391 Acked Read 5 Done Sent Done false
2392 Acked Flush 0 Done Sent Done false
2393 Acked Read 3 Done Sent Done false
2394 Acked Write 6 Done Sent Done false
2395 Acked Write 9 Done Sent Done false
2396 Acked Write 3 Done Sent Done false
2397 Acked Write 10 Done Sent Done false
2398 Acked Read 8 Done Sent Done false
2399 Acked Write 5 Done Sent Done false
2400 Acked Write 9 Done Sent Done false
2401 Acked Write 5 Done Sent Done false
2402 Acked Write 9 Done Sent Done false
2403 Acked Write 9 Done Sent Done false
2404 Acked Read 1 Done Sent Done false
2405 Acked Write 5 Done Sent Done false
2406 Acked Read 7 Done Sent Done false
2407 Acked Read 5 Done Sent Done false
2408 Acked Read 2 Done Sent Done false
2409 Acked Write 1 Done Sent Done false
2410 Acked Read 4 Done Sent Done false
2411 Acked Read 10 Done Sent Done false
2412 Acked Write 1 Done Sent Done false
2413 Acked Flush 0 Done Sent Done false
2414 Acked Read 10 Done Sent Done false
2415 Acked Flush 0 Done Sent Done false
2416 Acked Read 4 Done Sent Done false
2417 Acked Write 7 Done Sent Done false
2418 Acked Read 9 Done Sent Done false
2419 Acked Write 3 Done Sent Done false
2420 Acked Write 10 Done Sent Done false
2421 Acked Write 4 Done Sent Done false
2422 Acked Write 4 Done Sent Done false
2423 Acked Read 5 Done Sent Done false
2424 Acked Read 8 Done Sent Done false
2425 Acked Write 9 Done Sent Done false
2426 Acked Flush 0 Done Sent Done false
2427 Acked Flush 0 Done Sent Done false
2428 Acked Read 2 Done Sent Done false
2429 Acked Write 4 Done Sent Done false
2430 Acked Read 5 Done Sent Done false
2431 Acked Write 7 Done Sent Done false
2432 Acked Read 1 Done Sent Done false
2433 Acked Read 7 Done Sent Done false
2434 Acked Read 1 Done Sent Done false
2435 Acked Write 4 Done Sent Done false
2436 Acked Write 7 Done Sent Done false
2437 Acked Write 5 Done Sent Done false
2438 Acked Read 5 Done Sent Done false
2439 Acked Read 7 Done Sent Done false
2440 Acked Read 7 Done Sent Done false
2441 Acked Read 9 Done Sent Done false
2442 Acked Read 2 Done Sent Done false
2443 Acked Read 4 Done Sent Done false
2444 Acked Write 4 Done Sent Done false
2445 Acked Read 4 Done Sent Done false
2446 Acked Read 7 Done Sent Done false
2447 Acked Write 6 Done Sent Done false
2448 Acked Write 4 Done Sent Done false
2449 Acked Flush 0 Done Sent Done false
2450 Acked Read 6 Done Sent Done false
2451 Acked Write 8 Done Sent Done false
2452 Acked Read 5 Done Sent Done false
2453 Acked Read 4 Done Sent Done false
2454 Acked Write 6 Done Sent Done false
2455 Acked Read 8 Done Sent Done false
2456 Acked Read 5 Done Sent Done false
2457 Acked Write 2 Done Sent Done false
2458 Acked Write 4 Done Sent Done false
2459 Acked Write 5 Done Sent Done false
2460 Acked Write 9 Done Sent Done false
2461 Acked Write 6 Done Sent Done false
2462 Acked Read 2 Done Sent Done false
2463 Acked Read 6 Done Sent Done false
2464 Acked Read 6 Done Sent Done false
2465 Acked Write 2 Done Sent Done false
2466 Acked Flush 0 Done Sent Done false
2467 Acked Write 10 Done Sent Done false
2468 Acked Read 9 Done Sent Done false
2469 Acked Write 3 Done Sent Done false
2470 Acked Read 9 Done Sent Done false
2471 Acked Read 5 Done Sent Done false
2472 Acked Flush 0 Done Sent Done false
2473 Acked Write 4 Done Sent Done false
2474 Acked Write 9 Done Sent Done false
2475 Acked Write 1 Done Sent Done false
2476 Acked Read 7 Done Sent Done false
2477 Acked Write 7 Done Sent Done false
2478 Acked Read 8 Done Sent Done false
2479 Acked Write 5 Done Sent Done false
2480 Acked Read 7 Done Sent Done false
2481 Acked Read 5 Done Sent Done false
2482 Acked Write 10 Done Sent Done false
2483 Acked Write 9 Done Sent Done false
2484 Acked Write 2 Done Sent Done false
2485 Acked Write 10 Done Sent Done false
2486 Acked Read 4 Done Sent Done false
2487 Acked Write 4 Done Sent Done false
2488 Acked Read 1 Done Sent Done false
2489 Acked Read 2 Done Sent Done false
2490 Acked Read 5 Done Sent Done false
2491 Acked Read 8 Done Sent Done false
2492 Acked Read 2 Done Sent Done false
2493 Acked Read 5 Done Sent Done false
2494 Acked Read 10 Done Sent Done false
2495 Acked Write 7 Done Sent Done false
2496 Acked Write 8 Done Sent Done false
2497 Acked Write 2 Done Sent Done false
2498 Acked Read 3 Done Sent Done false
2499 Acked Read 5 Done Sent Done false
2500 Acked Write 5 Done Sent Done false
2501 Acked Write 5 Done Sent Done false
2502 Acked Read 7 Done Sent Done false
2503 Acked Write 5 Done Sent Done false
2504 Acked Write 5 Done Sent Done false
2505 Acked Write 9 Done Sent Done false
2506 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
Downstairs last five completed: 2205 2204 2203 2202 2201
Upstairs last five completed: 2506 2505 2504 2503 2502
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:17.625062951Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:18.462459283Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:19.463726133Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:20.465141497Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2206 Acked Write 4 Done Sent Done false
2207 Acked Read 8 Done Sent Done false
2208 Acked Read 10 Done Sent Done false
2209 Acked Read 8 Done Sent Done false
2210 Acked Write 6 Done Sent Done false
2211 Acked Flush 0 Done Sent Done false
2212 Acked Write 1 Done Sent Done false
2213 Acked Read 9 Done Sent Done false
2214 Acked Write 9 Done Sent Done false
2215 Acked Read 6 Done Sent Done false
2216 Acked Read 7 Done Sent Done false
2217 Acked Write 6 Done Sent Done false
2218 Acked Read 9 Done Sent Done false
2219 Acked Read 5 Done Sent Done false
2220 Acked Write 7 Done Sent Done false
2221 Acked Read 6 Done Sent Done false
2222 Acked Write 6 Done Sent Done false
2223 Acked Read 4 Done Sent Done false
2224 Acked Write 7 Done Sent Done false
2225 Acked Read 4 Done Sent Done false
2226 Acked Read 8 Done Sent Done false
2227 Acked Write 8 Done Sent Done false
2228 Acked Read 1 Done Sent Done false
2229 Acked Read 2 Done Sent Done false
2230 Acked Write 5 Done Sent Done false
2231 Acked Read 9 Done Sent Done false
2232 Acked Read 6 Done Sent Done false
2233 Acked Write 7 Done Sent Done false
2234 Acked Read 7 Done Sent Done false
2235 Acked Write 8 Done Sent Done false
2236 Acked Flush 0 Done Sent Done false
2237 Acked Read 2 Done Sent Done false
2238 Acked Flush 0 Done Sent Done false
2239 Acked Read 1 Done Sent Done false
2240 Acked Write 7 Done Sent Done false
2241 Acked Flush 0 Done Sent Done false
2242 Acked Read 7 Done Sent Done false
2243 Acked Flush 0 Done Sent Done false
2244 Acked Write 4 Done Sent Done false
2245 Acked Write 4 Done Sent Done false
2246 Acked Write 4 Done Sent Done false
2247 Acked Write 2 Done Sent Done false
2248 Acked Read 4 Done Sent Done false
2249 Acked Write 4 Done Sent Done false
2250 Acked Read 3 Done Sent Done false
2251 Acked Write 9 Done Sent Done false
2252 Acked Write 3 Done Sent Done false
2253 Acked Write 1 Done Sent Done false
2254 Acked Write 6 Done Sent Done false
2255 Acked Read 3 Done Sent Done false
2256 Acked Write 3 Done Sent Done false
2257 Acked Write 2 Done Sent Done false
2258 Acked Read 7 Done Sent Done false
2259 Acked Flush 0 Done Sent Done false
2260 Acked Read 10 Done Sent Done false
2261 Acked Read 1 Done Sent Done false
2262 Acked Read 1 Done Sent Done false
2263 Acked Write 3 Done Sent Done false
2264 Acked Write 5 Done Sent Done false
2265 Acked Write 3 Done Sent Done false
2266 Acked Flush 0 Done Sent Done false
2267 Acked Read 10 Done Sent Done false
2268 Acked Read 5 Done Sent Done false
2269 Acked Flush 0 Done Sent Done false
2270 Acked Write 7 Done Sent Done false
2271 Acked Read 7 Done Sent Done false
2272 Acked Read 10 Done Sent Done false
2273 Acked Write 8 Done Sent Done false
2274 Acked Read 4 Done Sent Done false
2275 Acked Write 9 Done Sent Done false
2276 Acked Write 8 Done Sent Done false
2277 Acked Write 9 Done Sent Done false
2278 Acked Read 2 Done Sent Done false
2279 Acked Write 2 Done Sent Done false
2280 Acked Read 6 Done Sent Done false
2281 Acked Read 9 Done Sent Done false
2282 Acked Read 10 Done Sent Done false
2283 Acked Read 9 Done Sent Done false
2284 Acked Read 2 Done Sent Done false
2285 Acked Read 9 Done Sent Done false
2286 Acked Read 4 Done Sent Done false
2287 Acked Write 9 Done Sent Done false
2288 Acked Read 3 Done Sent Done false
2289 Acked Read 10 Done Sent Done false
2290 Acked Write 2 Done Sent Done false
2291 Acked Write 10 Done Sent Done false
2292 Acked Flush 0 Done Sent Done false
2293 Acked Read 3 Done Sent Done false
2294 Acked Write 3 Done Sent Done false
2295 Acked Read 4 Done Sent Done false
2296 Acked Read 2 Done Sent Done false
2297 Acked Write 3 Done Sent Done false
2298 Acked Read 5 Done Sent Done false
2299 Acked Read 8 Done Sent Done false
2300 Acked Read 3 Done Sent Done false
2301 Acked Read 1 Done Sent Done false
2302 Acked Read 6 Done Sent Done false
2303 Acked Write 2 Done Sent Done false
2304 Acked Write 5 Done Sent Done false
2305 Acked Write 9 Done Sent Done false
2306 Acked Read 3 Done Sent Done false
2307 Acked Write 5 Done Sent Done false
2308 Acked Read 8 Done Sent Done false
2309 Acked Write 4 Done Sent Done false
2310 Acked Read 2 Done Sent Done false
2311 Acked Read 1 Done Sent Done false
2312 Acked Read 6 Done Sent Done false
2313 Acked Read 8 Done Sent Done false
2314 Acked Read 7 Done Sent Done false
2315 Acked Read 5 Done Sent Done false
2316 Acked Write 7 Done Sent Done false
2317 Acked Write 2 Done Sent Done false
2318 Acked Write 7 Done Sent Done false
2319 Acked Write 10 Done Sent Done false
2320 Acked Read 10 Done Sent Done false
2321 Acked Write 2 Done Sent Done false
2322 Acked Write 10 Done Sent Done false
2323 Acked Write 1 Done Sent Done false
2324 Acked Read 4 Done Sent Done false
2325 Acked Read 5 Done Sent Done false
2326 Acked Write 6 Done Sent Done false
2327 Acked Read 10 Done Sent Done false
2328 Acked Write 10 Done Sent Done false
2329 Acked Read 6 Done Sent Done false
2330 Acked Read 6 Done Sent Done false
2331 Acked Write 5 Done Sent Done false
2332 Acked Write 5 Done Sent Done false
2333 Acked Write 6 Done Sent Done false
2334 Acked Read 3 Done Sent Done false
2335 Acked Flush 0 Done Sent Done false
2336 Acked Read 4 Done Sent Done false
2337 Acked Flush 0 Done Sent Done false
2338 Acked Write 1 Done Sent Done false
2339 Acked Flush 0 Done Sent Done false
2340 Acked Read 1 Done Sent Done false
2341 Acked Write 9 Done Sent Done false
2342 Acked Write 9 Done Sent Done false
2343 Acked Read 7 Done Sent Done false
2344 Acked Write 6 Done Sent Done false
2345 Acked Write 10 Done Sent Done false
2346 Acked Read 3 Done Sent Done false
2347 Acked Write 2 Done Sent Done false
2348 Acked Read 3 Done Sent Done false
2349 Acked Read 5 Done Sent Done false
2350 Acked Write 5 Done Sent Done false
2351 Acked Write 1 Done Sent Done false
2352 Acked Write 4 Done Sent Done false
2353 Acked Read 10 Done Sent Done false
2354 Acked Read 7 Done Sent Done false
2355 Acked Write 5 Done Sent Done false
2356 Acked Read 2 Done Sent Done false
2357 Acked Write 7 Done Sent Done false
2358 Acked Write 7 Done Sent Done false
2359 Acked Write 9 Done Sent Done false
2360 Acked Read 5 Done Sent Done false
2361 Acked Write 2 Done Sent Done false
2362 Acked Flush 0 Done Sent Done false
2363 Acked Read 10 Done Sent Done false
2364 Acked Read 3 Done Sent Done false
2365 Acked Read 10 Done Sent Done false
2366 Acked Read 1 Done Sent Done false
2367 Acked Write 10 Done Sent Done false
2368 Acked Read 7 Done Sent Done false
2369 Acked Read 5 Done Sent Done false
2370 Acked Write 6 Done Sent Done false
2371 Acked Read 10 Done Sent Done false
2372 Acked Write 4 Done Sent Done false
2373 Acked Read 6 Done Sent Done false
2374 Acked Write 10 Done Sent Done false
2375 Acked Read 3 Done Sent Done false
2376 Acked Read 10 Done Sent Done false
2377 Acked Read 4 Done Sent Done false
2378 Acked Flush 0 Done Sent Done false
2379 Acked Read 6 Done Sent Done false
2380 Acked Read 2 Done Sent Done false
2381 Acked Flush 0 Done Sent Done false
2382 Acked Read 2 Done Sent Done false
2383 Acked Write 8 Done Sent Done false
2384 Acked Flush 0 Done Sent Done false
2385 Acked Write 7 Done Sent Done false
2386 Acked Write 8 Done Sent Done false
2387 Acked Write 3 Done Sent Done false
2388 Acked Write 4 Done Sent Done false
2389 Acked Write 10 Done Sent Done false
2390 Acked Write 6 Done Sent Done false
2391 Acked Read 5 Done Sent Done false
2392 Acked Flush 0 Done Sent Done false
2393 Acked Read 3 Done Sent Done false
2394 Acked Write 6 Done Sent Done false
2395 Acked Write 9 Done Sent Done false
2396 Acked Write 3 Done Sent Done false
2397 Acked Write 10 Done Sent Done false
2398 Acked Read 8 Done Sent Done false
2399 Acked Write 5 Done Sent Done false
2400 Acked Write 9 Done Sent Done false
2401 Acked Write 5 Done Sent Done false
2402 Acked Write 9 Done Sent Done false
2403 Acked Write 9 Done Sent Done false
2404 Acked Read 1 Done Sent Done false
2405 Acked Write 5 Done Sent Done false
2406 Acked Read 7 Done Sent Done false
2407 Acked Read 5 Done Sent Done false
2408 Acked Read 2 Done Sent Done false
2409 Acked Write 1 Done Sent Done false
2410 Acked Read 4 Done Sent Done false
2411 Acked Read 10 Done Sent Done false
2412 Acked Write 1 Done Sent Done false
2413 Acked Flush 0 Done Sent Done false
2414 Acked Read 10 Done Sent Done false
2415 Acked Flush 0 Done Sent Done false
2416 Acked Read 4 Done Sent Done false
2417 Acked Write 7 Done Sent Done false
2418 Acked Read 9 Done Sent Done false
2419 Acked Write 3 Done Sent Done false
2420 Acked Write 10 Done Sent Done false
2421 Acked Write 4 Done Sent Done false
2422 Acked Write 4 Done Sent Done false
2423 Acked Read 5 Done Sent Done false
2424 Acked Read 8 Done Sent Done false
2425 Acked Write 9 Done Sent Done false
2426 Acked Flush 0 Done Sent Done false
2427 Acked Flush 0 Done Sent Done false
2428 Acked Read 2 Done Sent Done false
2429 Acked Write 4 Done Sent Done false
2430 Acked Read 5 Done Sent Done false
2431 Acked Write 7 Done Sent Done false
2432 Acked Read 1 Done Sent Done false
2433 Acked Read 7 Done Sent Done false
2434 Acked Read 1 Done Sent Done false
2435 Acked Write 4 Done Sent Done false
2436 Acked Write 7 Done Sent Done false
2437 Acked Write 5 Done Sent Done false
2438 Acked Read 5 Done Sent Done false
2439 Acked Read 7 Done Sent Done false
2440 Acked Read 7 Done Sent Done false
2441 Acked Read 9 Done Sent Done false
2442 Acked Read 2 Done Sent Done false
2443 Acked Read 4 Done Sent Done false
2444 Acked Write 4 Done Sent Done false
2445 Acked Read 4 Done Sent Done false
2446 Acked Read 7 Done Sent Done false
2447 Acked Write 6 Done Sent Done false
2448 Acked Write 4 Done Sent Done false
2449 Acked Flush 0 Done Sent Done false
2450 Acked Read 6 Done Sent Done false
2451 Acked Write 8 Done Sent Done false
2452 Acked Read 5 Done Sent Done false
2453 Acked Read 4 Done Sent Done false
2454 Acked Write 6 Done Sent Done false
2455 Acked Read 8 Done Sent Done false
2456 Acked Read 5 Done Sent Done false
2457 Acked Write 2 Done Sent Done false
2458 Acked Write 4 Done Sent Done false
2459 Acked Write 5 Done Sent Done false
2460 Acked Write 9 Done Sent Done false
2461 Acked Write 6 Done Sent Done false
2462 Acked Read 2 Done Sent Done false
2463 Acked Read 6 Done Sent Done false
2464 Acked Read 6 Done Sent Done false
2465 Acked Write 2 Done Sent Done false
2466 Acked Flush 0 Done Sent Done false
2467 Acked Write 10 Done Sent Done false
2468 Acked Read 9 Done Sent Done false
2469 Acked Write 3 Done Sent Done false
2470 Acked Read 9 Done Sent Done false
2471 Acked Read 5 Done Sent Done false
2472 Acked Flush 0 Done Sent Done false
2473 Acked Write 4 Done Sent Done false
2474 Acked Write 9 Done Sent Done false
2475 Acked Write 1 Done Sent Done false
2476 Acked Read 7 Done Sent Done false
2477 Acked Write 7 Done Sent Done false
2478 Acked Read 8 Done Sent Done false
2479 Acked Write 5 Done Sent Done false
2480 Acked Read 7 Done Sent Done false
2481 Acked Read 5 Done Sent Done false
2482 Acked Write 10 Done Sent Done false
2483 Acked Write 9 Done Sent Done false
2484 Acked Write 2 Done Sent Done false
2485 Acked Write 10 Done Sent Done false
2486 Acked Read 4 Done Sent Done false
2487 Acked Write 4 Done Sent Done false
2488 Acked Read 1 Done Sent Done false
2489 Acked Read 2 Done Sent Done false
2490 Acked Read 5 Done Sent Done false
2491 Acked Read 8 Done Sent Done false
2492 Acked Read 2 Done Sent Done false
2493 Acked Read 5 Done Sent Done false
2494 Acked Read 10 Done Sent Done false
2495 Acked Write 7 Done Sent Done false
2496 Acked Write 8 Done Sent Done false
2497 Acked Write 2 Done Sent Done false
2498 Acked Read 3 Done Sent Done false
2499 Acked Read 5 Done Sent Done false
2500 Acked Write 5 Done Sent Done false
2501 Acked Write 5 Done Sent Done false
2502 Acked Read 7 Done Sent Done false
2503 Acked Write 5 Done Sent Done false
2504 Acked Write 5 Done Sent Done false
2505 Acked Write 9 Done Sent Done false
2506 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
Downstairs last five completed: 2205 2204 2203 2202 2201
Upstairs last five completed: 2506 2505 2504 2503 2502
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:21.627821503Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:22.468741261Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:22.884154644Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:22.884363874Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:22.88442013Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:22.884459161Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:22.884620062Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1216)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:23.46942319Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:24.471089514Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2206 Acked Write 4 Done Sent Done false
2207 Acked Read 8 Done Sent Done false
2208 Acked Read 10 Done Sent Done false
2209 Acked Read 8 Done Sent Done false
2210 Acked Write 6 Done Sent Done false
2211 Acked Flush 0 Done Sent Done false
2212 Acked Write 1 Done Sent Done false
2213 Acked Read 9 Done Sent Done false
2214 Acked Write 9 Done Sent Done false
2215 Acked Read 6 Done Sent Done false
2216 Acked Read 7 Done Sent Done false
2217 Acked Write 6 Done Sent Done false
2218 Acked Read 9 Done Sent Done false
2219 Acked Read 5 Done Sent Done false
2220 Acked Write 7 Done Sent Done false
2221 Acked Read 6 Done Sent Done false
2222 Acked Write 6 Done Sent Done false
2223 Acked Read 4 Done Sent Done false
2224 Acked Write 7 Done Sent Done false
2225 Acked Read 4 Done Sent Done false
2226 Acked Read 8 Done Sent Done false
2227 Acked Write 8 Done Sent Done false
2228 Acked Read 1 Done Sent Done false
2229 Acked Read 2 Done Sent Done false
2230 Acked Write 5 Done Sent Done false
2231 Acked Read 9 Done Sent Done false
2232 Acked Read 6 Done Sent Done false
2233 Acked Write 7 Done Sent Done false
2234 Acked Read 7 Done Sent Done false
2235 Acked Write 8 Done Sent Done false
2236 Acked Flush 0 Done Sent Done false
2237 Acked Read 2 Done Sent Done false
2238 Acked Flush 0 Done Sent Done false
2239 Acked Read 1 Done Sent Done false
2240 Acked Write 7 Done Sent Done false
2241 Acked Flush 0 Done Sent Done false
2242 Acked Read 7 Done Sent Done false
2243 Acked Flush 0 Done Sent Done false
2244 Acked Write 4 Done Sent Done false
2245 Acked Write 4 Done Sent Done false
2246 Acked Write 4 Done Sent Done false
2247 Acked Write 2 Done Sent Done false
2248 Acked Read 4 Done Sent Done false
2249 Acked Write 4 Done Sent Done false
2250 Acked Read 3 Done Sent Done false
2251 Acked Write 9 Done Sent Done false
2252 Acked Write 3 Done Sent Done false
2253 Acked Write 1 Done Sent Done false
2254 Acked Write 6 Done Sent Done false
2255 Acked Read 3 Done Sent Done false
2256 Acked Write 3 Done Sent Done false
2257 Acked Write 2 Done Sent Done false
2258 Acked Read 7 Done Sent Done false
2259 Acked Flush 0 Done Sent Done false
2260 Acked Read 10 Done Sent Done false
2261 Acked Read 1 Done Sent Done false
2262 Acked Read 1 Done Sent Done false
2263 Acked Write 3 Done Sent Done false
2264 Acked Write 5 Done Sent Done false
2265 Acked Write 3 Done Sent Done false
2266 Acked Flush 0 Done Sent Done false
2267 Acked Read 10 Done Sent Done false
2268 Acked Read 5 Done Sent Done false
2269 Acked Flush 0 Done Sent Done false
2270 Acked Write 7 Done Sent Done false
2271 Acked Read 7 Done Sent Done false
2272 Acked Read 10 Done Sent Done false
2273 Acked Write 8 Done Sent Done false
2274 Acked Read 4 Done Sent Done false
2275 Acked Write 9 Done Sent Done false
2276 Acked Write 8 Done Sent Done false
2277 Acked Write 9 Done Sent Done false
2278 Acked Read 2 Done Sent Done false
2279 Acked Write 2 Done Sent Done false
2280 Acked Read 6 Done Sent Done false
2281 Acked Read 9 Done Sent Done false
2282 Acked Read 10 Done Sent Done false
2283 Acked Read 9 Done Sent Done false
2284 Acked Read 2 Done Sent Done false
2285 Acked Read 9 Done Sent Done false
2286 Acked Read 4 Done Sent Done false
2287 Acked Write 9 Done Sent Done false
2288 Acked Read 3 Done Sent Done false
2289 Acked Read 10 Done Sent Done false
2290 Acked Write 2 Done Sent Done false
2291 Acked Write 10 Done Sent Done false
2292 Acked Flush 0 Done Sent Done false
2293 Acked Read 3 Done Sent Done false
2294 Acked Write 3 Done Sent Done false
2295 Acked Read 4 Done Sent Done false
2296 Acked Read 2 Done Sent Done false
2297 Acked Write 3 Done Sent Done false
2298 Acked Read 5 Done Sent Done false
2299 Acked Read 8 Done Sent Done false
2300 Acked Read 3 Done Sent Done false
2301 Acked Read 1 Done Sent Done false
2302 Acked Read 6 Done Sent Done false
2303 Acked Write 2 Done Sent Done false
2304 Acked Write 5 Done Sent Done false
2305 Acked Write 9 Done Sent Done false
2306 Acked Read 3 Done Sent Done false
2307 Acked Write 5 Done Sent Done false
2308 Acked Read 8 Done Sent Done false
2309 Acked Write 4 Done Sent Done false
2310 Acked Read 2 Done Sent Done false
2311 Acked Read 1 Done Sent Done false
2312 Acked Read 6 Done Sent Done false
2313 Acked Read 8 Done Sent Done false
2314 Acked Read 7 Done Sent Done false
2315 Acked Read 5 Done Sent Done false
2316 Acked Write 7 Done Sent Done false
2317 Acked Write 2 Done Sent Done false
2318 Acked Write 7 Done Sent Done false
2319 Acked Write 10 Done Sent Done false
2320 Acked Read 10 Done Sent Done false
2321 Acked Write 2 Done Sent Done false
2322 Acked Write 10 Done Sent Done false
2323 Acked Write 1 Done Sent Done false
2324 Acked Read 4 Done Sent Done false
2325 Acked Read 5 Done Sent Done false
2326 Acked Write 6 Done Sent Done false
2327 Acked Read 10 Done Sent Done false
2328 Acked Write 10 Done Sent Done false
2329 Acked Read 6 Done Sent Done false
2330 Acked Read 6 Done Sent Done false
2331 Acked Write 5 Done Sent Done false
2332 Acked Write 5 Done Sent Done false
2333 Acked Write 6 Done Sent Done false
2334 Acked Read 3 Done Sent Done false
2335 Acked Flush 0 Done Sent Done false
2336 Acked Read 4 Done Sent Done false
2337 Acked Flush 0 Done Sent Done false
2338 Acked Write 1 Done Sent Done false
2339 Acked Flush 0 Done Sent Done false
2340 Acked Read 1 Done Sent Done false
2341 Acked Write 9 Done Sent Done false
2342 Acked Write 9 Done Sent Done false
2343 Acked Read 7 Done Sent Done false
2344 Acked Write 6 Done Sent Done false
2345 Acked Write 10 Done Sent Done false
2346 Acked Read 3 Done Sent Done false
2347 Acked Write 2 Done Sent Done false
2348 Acked Read 3 Done Sent Done false
2349 Acked Read 5 Done Sent Done false
2350 Acked Write 5 Done Sent Done false
2351 Acked Write 1 Done Sent Done false
2352 Acked Write 4 Done Sent Done false
2353 Acked Read 10 Done Sent Done false
2354 Acked Read 7 Done Sent Done false
2355 Acked Write 5 Done Sent Done false
2356 Acked Read 2 Done Sent Done false
2357 Acked Write 7 Done Sent Done false
2358 Acked Write 7 Done Sent Done false
2359 Acked Write 9 Done Sent Done false
2360 Acked Read 5 Done Sent Done false
2361 Acked Write 2 Done Sent Done false
2362 Acked Flush 0 Done Sent Done false
2363 Acked Read 10 Done Sent Done false
2364 Acked Read 3 Done Sent Done false
2365 Acked Read 10 Done Sent Done false
2366 Acked Read 1 Done Sent Done false
2367 Acked Write 10 Done Sent Done false
2368 Acked Read 7 Done Sent Done false
2369 Acked Read 5 Done Sent Done false
2370 Acked Write 6 Done Sent Done false
2371 Acked Read 10 Done Sent Done false
2372 Acked Write 4 Done Sent Done false
2373 Acked Read 6 Done Sent Done false
2374 Acked Write 10 Done Sent Done false
2375 Acked Read 3 Done Sent Done false
2376 Acked Read 10 Done Sent Done false
2377 Acked Read 4 Done Sent Done false
2378 Acked Flush 0 Done Sent Done false
2379 Acked Read 6 Done Sent Done false
2380 Acked Read 2 Done Sent Done false
2381 Acked Flush 0 Done Sent Done false
2382 Acked Read 2 Done Sent Done false
2383 Acked Write 8 Done Sent Done false
2384 Acked Flush 0 Done Sent Done false
2385 Acked Write 7 Done Sent Done false
2386 Acked Write 8 Done Sent Done false
2387 Acked Write 3 Done Sent Done false
2388 Acked Write 4 Done Sent Done false
2389 Acked Write 10 Done Sent Done false
2390 Acked Write 6 Done Sent Done false
2391 Acked Read 5 Done Sent Done false
2392 Acked Flush 0 Done Sent Done false
2393 Acked Read 3 Done Sent Done false
2394 Acked Write 6 Done Sent Done false
2395 Acked Write 9 Done Sent Done false
2396 Acked Write 3 Done Sent Done false
2397 Acked Write 10 Done Sent Done false
2398 Acked Read 8 Done Sent Done false
2399 Acked Write 5 Done Sent Done false
2400 Acked Write 9 Done Sent Done false
2401 Acked Write 5 Done Sent Done false
2402 Acked Write 9 Done Sent Done false
2403 Acked Write 9 Done Sent Done false
2404 Acked Read 1 Done Sent Done false
2405 Acked Write 5 Done Sent Done false
2406 Acked Read 7 Done Sent Done false
2407 Acked Read 5 Done Sent Done false
2408 Acked Read 2 Done Sent Done false
2409 Acked Write 1 Done Sent Done false
2410 Acked Read 4 Done Sent Done false
2411 Acked Read 10 Done Sent Done false
2412 Acked Write 1 Done Sent Done false
2413 Acked Flush 0 Done Sent Done false
2414 Acked Read 10 Done Sent Done false
2415 Acked Flush 0 Done Sent Done false
2416 Acked Read 4 Done Sent Done false
2417 Acked Write 7 Done Sent Done false
2418 Acked Read 9 Done Sent Done false
2419 Acked Write 3 Done Sent Done false
2420 Acked Write 10 Done Sent Done false
2421 Acked Write 4 Done Sent Done false
2422 Acked Write 4 Done Sent Done false
2423 Acked Read 5 Done Sent Done false
2424 Acked Read 8 Done Sent Done false
2425 Acked Write 9 Done Sent Done false
2426 Acked Flush 0 Done Sent Done false
2427 Acked Flush 0 Done Sent Done false
2428 Acked Read 2 Done Sent Done false
2429 Acked Write 4 Done Sent Done false
2430 Acked Read 5 Done Sent Done false
2431 Acked Write 7 Done Sent Done false
2432 Acked Read 1 Done Sent Done false
2433 Acked Read 7 Done Sent Done false
2434 Acked Read 1 Done Sent Done false
2435 Acked Write 4 Done Sent Done false
2436 Acked Write 7 Done Sent Done false
2437 Acked Write 5 Done Sent Done false
2438 Acked Read 5 Done Sent Done false
2439 Acked Read 7 Done Sent Done false
2440 Acked Read 7 Done Sent Done false
2441 Acked Read 9 Done Sent Done false
2442 Acked Read 2 Done Sent Done false
2443 Acked Read 4 Done Sent Done false
2444 Acked Write 4 Done Sent Done false
2445 Acked Read 4 Done Sent Done false
2446 Acked Read 7 Done Sent Done false
2447 Acked Write 6 Done Sent Done false
2448 Acked Write 4 Done Sent Done false
2449 Acked Flush 0 Done Sent Done false
2450 Acked Read 6 Done Sent Done false
2451 Acked Write 8 Done Sent Done false
2452 Acked Read 5 Done Sent Done false
2453 Acked Read 4 Done Sent Done false
2454 Acked Write 6 Done Sent Done false
2455 Acked Read 8 Done Sent Done false
2456 Acked Read 5 Done Sent Done false
2457 Acked Write 2 Done Sent Done false
2458 Acked Write 4 Done Sent Done false
2459 Acked Write 5 Done Sent Done false
2460 Acked Write 9 Done Sent Done false
2461 Acked Write 6 Done Sent Done false
2462 Acked Read 2 Done Sent Done false
2463 Acked Read 6 Done Sent Done false
2464 Acked Read 6 Done Sent Done false
2465 Acked Write 2 Done Sent Done false
2466 Acked Flush 0 Done Sent Done false
2467 Acked Write 10 Done Sent Done false
2468 Acked Read 9 Done Sent Done false
2469 Acked Write 3 Done Sent Done false
2470 Acked Read 9 Done Sent Done false
2471 Acked Read 5 Done Sent Done false
2472 Acked Flush 0 Done Sent Done false
2473 Acked Write 4 Done Sent Done false
2474 Acked Write 9 Done Sent Done false
2475 Acked Write 1 Done Sent Done false
2476 Acked Read 7 Done Sent Done false
2477 Acked Write 7 Done Sent Done false
2478 Acked Read 8 Done Sent Done false
2479 Acked Write 5 Done Sent Done false
2480 Acked Read 7 Done Sent Done false
2481 Acked Read 5 Done Sent Done false
2482 Acked Write 10 Done Sent Done false
2483 Acked Write 9 Done Sent Done false
2484 Acked Write 2 Done Sent Done false
2485 Acked Write 10 Done Sent Done false
2486 Acked Read 4 Done Sent Done false
2487 Acked Write 4 Done Sent Done false
2488 Acked Read 1 Done Sent Done false
2489 Acked Read 2 Done Sent Done false
2490 Acked Read 5 Done Sent Done false
2491 Acked Read 8 Done Sent Done false
2492 Acked Read 2 Done Sent Done false
2493 Acked Read 5 Done Sent Done false
2494 Acked Read 10 Done Sent Done false
2495 Acked Write 7 Done Sent Done false
2496 Acked Write 8 Done Sent Done false
2497 Acked Write 2 Done Sent Done false
2498 Acked Read 3 Done Sent Done false
2499 Acked Read 5 Done Sent Done false
2500 Acked Write 5 Done Sent Done false
2501 Acked Write 5 Done Sent Done false
2502 Acked Read 7 Done Sent Done false
2503 Acked Write 5 Done Sent Done false
2504 Acked Write 5 Done Sent Done false
2505 Acked Write 9 Done Sent Done false
2506 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
Downstairs last five completed: 2205 2204 2203 2202 2201
Upstairs last five completed: 2506 2505 2504 2503 2502
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:25.630273707Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:26.475529136Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:26.97585215Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:27.477753357Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:28.479080302Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2206 Acked Write 4 Done Sent Done false
2207 Acked Read 8 Done Sent Done false
2208 Acked Read 10 Done Sent Done false
2209 Acked Read 8 Done Sent Done false
2210 Acked Write 6 Done Sent Done false
2211 Acked Flush 0 Done Sent Done false
2212 Acked Write 1 Done Sent Done false
2213 Acked Read 9 Done Sent Done false
2214 Acked Write 9 Done Sent Done false
2215 Acked Read 6 Done Sent Done false
2216 Acked Read 7 Done Sent Done false
2217 Acked Write 6 Done Sent Done false
2218 Acked Read 9 Done Sent Done false
2219 Acked Read 5 Done Sent Done false
2220 Acked Write 7 Done Sent Done false
2221 Acked Read 6 Done Sent Done false
2222 Acked Write 6 Done Sent Done false
2223 Acked Read 4 Done Sent Done false
2224 Acked Write 7 Done Sent Done false
2225 Acked Read 4 Done Sent Done false
2226 Acked Read 8 Done Sent Done false
2227 Acked Write 8 Done Sent Done false
2228 Acked Read 1 Done Sent Done false
2229 Acked Read 2 Done Sent Done false
2230 Acked Write 5 Done Sent Done false
2231 Acked Read 9 Done Sent Done false
2232 Acked Read 6 Done Sent Done false
2233 Acked Write 7 Done Sent Done false
2234 Acked Read 7 Done Sent Done false
2235 Acked Write 8 Done Sent Done false
2236 Acked Flush 0 Done Sent Done false
2237 Acked Read 2 Done Sent Done false
2238 Acked Flush 0 Done Sent Done false
2239 Acked Read 1 Done Sent Done false
2240 Acked Write 7 Done Sent Done false
2241 Acked Flush 0 Done Sent Done false
2242 Acked Read 7 Done Sent Done false
2243 Acked Flush 0 Done Sent Done false
2244 Acked Write 4 Done Sent Done false
2245 Acked Write 4 Done Sent Done false
2246 Acked Write 4 Done Sent Done false
2247 Acked Write 2 Done Sent Done false
2248 Acked Read 4 Done Sent Done false
2249 Acked Write 4 Done Sent Done false
2250 Acked Read 3 Done Sent Done false
2251 Acked Write 9 Done Sent Done false
2252 Acked Write 3 Done Sent Done false
2253 Acked Write 1 Done Sent Done false
2254 Acked Write 6 Done Sent Done false
2255 Acked Read 3 Done Sent Done false
2256 Acked Write 3 Done Sent Done false
2257 Acked Write 2 Done Sent Done false
2258 Acked Read 7 Done Sent Done false
2259 Acked Flush 0 Done Sent Done false
2260 Acked Read 10 Done Sent Done false
2261 Acked Read 1 Done Sent Done false
2262 Acked Read 1 Done Sent Done false
2263 Acked Write 3 Done Sent Done false
2264 Acked Write 5 Done Sent Done false
2265 Acked Write 3 Done Sent Done false
2266 Acked Flush 0 Done Sent Done false
2267 Acked Read 10 Done Sent Done false
2268 Acked Read 5 Done Sent Done false
2269 Acked Flush 0 Done Sent Done false
2270 Acked Write 7 Done Sent Done false
2271 Acked Read 7 Done Sent Done false
2272 Acked Read 10 Done Sent Done false
2273 Acked Write 8 Done Sent Done false
2274 Acked Read 4 Done Sent Done false
2275 Acked Write 9 Done Sent Done false
2276 Acked Write 8 Done Sent Done false
2277 Acked Write 9 Done Sent Done false
2278 Acked Read 2 Done Sent Done false
2279 Acked Write 2 Done Sent Done false
2280 Acked Read 6 Done Sent Done false
2281 Acked Read 9 Done Sent Done false
2282 Acked Read 10 Done Sent Done false
2283 Acked Read 9 Done Sent Done false
2284 Acked Read 2 Done Sent Done false
2285 Acked Read 9 Done Sent Done false
2286 Acked Read 4 Done Sent Done false
2287 Acked Write 9 Done Sent Done false
2288 Acked Read 3 Done Sent Done false
2289 Acked Read 10 Done Sent Done false
2290 Acked Write 2 Done Sent Done false
2291 Acked Write 10 Done Sent Done false
2292 Acked Flush 0 Done Sent Done false
2293 Acked Read 3 Done Sent Done false
2294 Acked Write 3 Done Sent Done false
2295 Acked Read 4 Done Sent Done false
2296 Acked Read 2 Done Sent Done false
2297 Acked Write 3 Done Sent Done false
2298 Acked Read 5 Done Sent Done false
2299 Acked Read 8 Done Sent Done false
2300 Acked Read 3 Done Sent Done false
2301 Acked Read 1 Done Sent Done false
2302 Acked Read 6 Done Sent Done false
2303 Acked Write 2 Done Sent Done false
2304 Acked Write 5 Done Sent Done false
2305 Acked Write 9 Done Sent Done false
2306 Acked Read 3 Done Sent Done false
2307 Acked Write 5 Done Sent Done false
2308 Acked Read 8 Done Sent Done false
2309 Acked Write 4 Done Sent Done false
2310 Acked Read 2 Done Sent Done false
2311 Acked Read 1 Done Sent Done false
2312 Acked Read 6 Done Sent Done false
2313 Acked Read 8 Done Sent Done false
2314 Acked Read 7 Done Sent Done false
2315 Acked Read 5 Done Sent Done false
2316 Acked Write 7 Done Sent Done false
2317 Acked Write 2 Done Sent Done false
2318 Acked Write 7 Done Sent Done false
2319 Acked Write 10 Done Sent Done false
2320 Acked Read 10 Done Sent Done false
2321 Acked Write 2 Done Sent Done false
2322 Acked Write 10 Done Sent Done false
2323 Acked Write 1 Done Sent Done false
2324 Acked Read 4 Done Sent Done false
2325 Acked Read 5 Done Sent Done false
2326 Acked Write 6 Done Sent Done false
2327 Acked Read 10 Done Sent Done false
2328 Acked Write 10 Done Sent Done false
2329 Acked Read 6 Done Sent Done false
2330 Acked Read 6 Done Sent Done false
2331 Acked Write 5 Done Sent Done false
2332 Acked Write 5 Done Sent Done false
2333 Acked Write 6 Done Sent Done false
2334 Acked Read 3 Done Sent Done false
2335 Acked Flush 0 Done Sent Done false
2336 Acked Read 4 Done Sent Done false
2337 Acked Flush 0 Done Sent Done false
2338 Acked Write 1 Done Sent Done false
2339 Acked Flush 0 Done Sent Done false
2340 Acked Read 1 Done Sent Done false
2341 Acked Write 9 Done Sent Done false
2342 Acked Write 9 Done Sent Done false
2343 Acked Read 7 Done Sent Done false
2344 Acked Write 6 Done Sent Done false
2345 Acked Write 10 Done Sent Done false
2346 Acked Read 3 Done Sent Done false
2347 Acked Write 2 Done Sent Done false
2348 Acked Read 3 Done Sent Done false
2349 Acked Read 5 Done Sent Done false
2350 Acked Write 5 Done Sent Done false
2351 Acked Write 1 Done Sent Done false
2352 Acked Write 4 Done Sent Done false
2353 Acked Read 10 Done Sent Done false
2354 Acked Read 7 Done Sent Done false
2355 Acked Write 5 Done Sent Done false
2356 Acked Read 2 Done Sent Done false
2357 Acked Write 7 Done Sent Done false
2358 Acked Write 7 Done Sent Done false
2359 Acked Write 9 Done Sent Done false
2360 Acked Read 5 Done Sent Done false
2361 Acked Write 2 Done Sent Done false
2362 Acked Flush 0 Done Sent Done false
2363 Acked Read 10 Done Sent Done false
2364 Acked Read 3 Done Sent Done false
2365 Acked Read 10 Done Sent Done false
2366 Acked Read 1 Done Sent Done false
2367 Acked Write 10 Done Sent Done false
2368 Acked Read 7 Done Sent Done false
2369 Acked Read 5 Done Sent Done false
2370 Acked Write 6 Done Sent Done false
2371 Acked Read 10 Done Sent Done false
2372 Acked Write 4 Done Sent Done false
2373 Acked Read 6 Done Sent Done false
2374 Acked Write 10 Done Sent Done false
2375 Acked Read 3 Done Sent Done false
2376 Acked Read 10 Done Sent Done false
2377 Acked Read 4 Done Sent Done false
2378 Acked Flush 0 Done Sent Done false
2379 Acked Read 6 Done Sent Done false
2380 Acked Read 2 Done Sent Done false
2381 Acked Flush 0 Done Sent Done false
2382 Acked Read 2 Done Sent Done false
2383 Acked Write 8 Done Sent Done false
2384 Acked Flush 0 Done Sent Done false
2385 Acked Write 7 Done Sent Done false
2386 Acked Write 8 Done Sent Done false
2387 Acked Write 3 Done Sent Done false
2388 Acked Write 4 Done Sent Done false
2389 Acked Write 10 Done Sent Done false
2390 Acked Write 6 Done Sent Done false
2391 Acked Read 5 Done Sent Done false
2392 Acked Flush 0 Done Sent Done false
2393 Acked Read 3 Done Sent Done false
2394 Acked Write 6 Done Sent Done false
2395 Acked Write 9 Done Sent Done false
2396 Acked Write 3 Done Sent Done false
2397 Acked Write 10 Done Sent Done false
2398 Acked Read 8 Done Sent Done false
2399 Acked Write 5 Done Sent Done false
2400 Acked Write 9 Done Sent Done false
2401 Acked Write 5 Done Sent Done false
2402 Acked Write 9 Done Sent Done false
2403 Acked Write 9 Done Sent Done false
2404 Acked Read 1 Done Sent Done false
2405 Acked Write 5 Done Sent Done false
2406 Acked Read 7 Done Sent Done false
2407 Acked Read 5 Done Sent Done false
2408 Acked Read 2 Done Sent Done false
2409 Acked Write 1 Done Sent Done false
2410 Acked Read 4 Done Sent Done false
2411 Acked Read 10 Done Sent Done false
2412 Acked Write 1 Done Sent Done false
2413 Acked Flush 0 Done Sent Done false
2414 Acked Read 10 Done Sent Done false
2415 Acked Flush 0 Done Sent Done false
2416 Acked Read 4 Done Sent Done false
2417 Acked Write 7 Done Sent Done false
2418 Acked Read 9 Done Sent Done false
2419 Acked Write 3 Done Sent Done false
2420 Acked Write 10 Done Sent Done false
2421 Acked Write 4 Done Sent Done false
2422 Acked Write 4 Done Sent Done false
2423 Acked Read 5 Done Sent Done false
2424 Acked Read 8 Done Sent Done false
2425 Acked Write 9 Done Sent Done false
2426 Acked Flush 0 Done Sent Done false
2427 Acked Flush 0 Done Sent Done false
2428 Acked Read 2 Done Sent Done false
2429 Acked Write 4 Done Sent Done false
2430 Acked Read 5 Done Sent Done false
2431 Acked Write 7 Done Sent Done false
2432 Acked Read 1 Done Sent Done false
2433 Acked Read 7 Done Sent Done false
2434 Acked Read 1 Done Sent Done false
2435 Acked Write 4 Done Sent Done false
2436 Acked Write 7 Done Sent Done false
2437 Acked Write 5 Done Sent Done false
2438 Acked Read 5 Done Sent Done false
2439 Acked Read 7 Done Sent Done false
2440 Acked Read 7 Done Sent Done false
2441 Acked Read 9 Done Sent Done false
2442 Acked Read 2 Done Sent Done false
2443 Acked Read 4 Done Sent Done false
2444 Acked Write 4 Done Sent Done false
2445 Acked Read 4 Done Sent Done false
2446 Acked Read 7 Done Sent Done false
2447 Acked Write 6 Done Sent Done false
2448 Acked Write 4 Done Sent Done false
2449 Acked Flush 0 Done Sent Done false
2450 Acked Read 6 Done Sent Done false
2451 Acked Write 8 Done Sent Done false
2452 Acked Read 5 Done Sent Done false
2453 Acked Read 4 Done Sent Done false
2454 Acked Write 6 Done Sent Done false
2455 Acked Read 8 Done Sent Done false
2456 Acked Read 5 Done Sent Done false
2457 Acked Write 2 Done Sent Done false
2458 Acked Write 4 Done Sent Done false
2459 Acked Write 5 Done Sent Done false
2460 Acked Write 9 Done Sent Done false
2461 Acked Write 6 Done Sent Done false
2462 Acked Read 2 Done Sent Done false
2463 Acked Read 6 Done Sent Done false
2464 Acked Read 6 Done Sent Done false
2465 Acked Write 2 Done Sent Done false
2466 Acked Flush 0 Done Sent Done false
2467 Acked Write 10 Done Sent Done false
2468 Acked Read 9 Done Sent Done false
2469 Acked Write 3 Done Sent Done false
2470 Acked Read 9 Done Sent Done false
2471 Acked Read 5 Done Sent Done false
2472 Acked Flush 0 Done Sent Done false
2473 Acked Write 4 Done Sent Done false
2474 Acked Write 9 Done Sent Done false
2475 Acked Write 1 Done Sent Done false
2476 Acked Read 7 Done Sent Done false
2477 Acked Write 7 Done Sent Done false
2478 Acked Read 8 Done Sent Done false
2479 Acked Write 5 Done Sent Done false
2480 Acked Read 7 Done Sent Done false
2481 Acked Read 5 Done Sent Done false
2482 Acked Write 10 Done Sent Done false
2483 Acked Write 9 Done Sent Done false
2484 Acked Write 2 Done Sent Done false
2485 Acked Write 10 Done Sent Done false
2486 Acked Read 4 Done Sent Done false
2487 Acked Write 4 Done Sent Done false
2488 Acked Read 1 Done Sent Done false
2489 Acked Read 2 Done Sent Done false
2490 Acked Read 5 Done Sent Done false
2491 Acked Read 8 Done Sent Done false
2492 Acked Read 2 Done Sent Done false
2493 Acked Read 5 Done Sent Done false
2494 Acked Read 10 Done Sent Done false
2495 Acked Write 7 Done Sent Done false
2496 Acked Write 8 Done Sent Done false
2497 Acked Write 2 Done Sent Done false
2498 Acked Read 3 Done Sent Done false
2499 Acked Read 5 Done Sent Done false
2500 Acked Write 5 Done Sent Done false
2501 Acked Write 5 Done Sent Done false
2502 Acked Read 7 Done Sent Done false
2503 Acked Write 5 Done Sent Done false
2504 Acked Write 5 Done Sent Done false
2505 Acked Write 9 Done Sent Done false
2506 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
Downstairs last five completed: 2205 2204 2203 2202 2201
Upstairs last five completed: 2506 2505 2504 2503 2502
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:29.632412139Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:30.483730175Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:31.483813088Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:32.486173296Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.88551842Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53284)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.885807721Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.887185651Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.88773278Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.887758284Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(2205))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.88777621Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.887797384Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(2205))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.8878143Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(2205))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:32.88802413Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 2506 2505 2504 2503 2502
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:02:33.133 [8810] Got stop action so:false kr:false
INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/0, method: POST, req_id: da63dc11-7cab-418a-b755-4aae559de780, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:33.134094468Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:33.134146505Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:33.134166041Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:33.134262687Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:02:33.134 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/0, method: GET, req_id: 25bf103d-fe57-4109-b016-3d6772ac5eed, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:02:37.135 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0859d4b5-1748-4d78-832e-7b0ed9bc9b76, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 4363 len 12288
002/300 Read block 3888 len 16384
003/300 Write block 795 len 24576 data: 3 3 3 3 2 3
004/300 Read block 2678 len 28672
005/300 Flush
006/300 Write block 1633 len 24576 data: 3 3 3 4 4 2
007/300 Read block 1099 len 20480
008/300 Write block 4390 len 32768 data: 2 2 2 2 2 2 2 2
009/300 Write block 1874 len 36864 data: 2 2 2 2 2 2 2 2 2
010/300 Write block 3717 len 32768 data: 2 3 3 3 3 3 3 3
011/300 Flush
012/300 Write block 3312 len 8192 data: 2 2
013/300 Write block 3098 len 32768 data: 2 2 3 4 4 4 4 4
014/300 Write block 3360 len 40960 data: 2 2 2 2 3 3 3 2 2 2
015/300 Read block 3772 len 20480
016/300 Write block 4018 len 8192 data: 2 2
017/300 Write block 4632 len 36864 data: 4 4 3 2 2 2 2 2 2
018/300 Read block 4085 len 16384
019/300 Write block 962 len 28672 data: 2 2 2 2 2 2 2
020/300 Flush
021/300 Write block 1456 len 12288 data: 3 3 2
022/300 Read block 3197 len 36864
023/300 Read block 2195 len 24576
024/300 Write block 4937 len 12288 data: 2 2 2
025/300 Flush
026/300 Write block 4621 len 32768 data: 2 2 3 3 3 3 3 3
027/300 Write block 1730 len 20480 data: 2 2 2 2 2
028/300 Read block 4399 len 40960
029/300 Write block 1969 len 28672 data: 2 2 2 2 2 2 2
030/300 Write block 3007 len 36864 data: 2 2 2 2 2 2 2 2 2
031/300 Read block 4295 len 16384
032/300 Write block 2115 len 40960 data: 2 2 3 3 3 3 3 3 3 4
033/300 Flush
034/300 Write block 844 len 12288 data: 4 4 3
035/300 Write block 3222 len 20480 data: 2 2 2 2 2
036/300 Read block 4047 len 40960
037/300 Read block 2508 len 28672
038/300 Write block 192 len 40960 data: 4 4 4 3 3 2 2 2 2 2
039/300 Read block 4385 len 8192
040/300 Read block 793 len 28672
041/300 Write block 115 len 4096 data: 2
042/300 Write block 4544 len 16384 data: 3 3 3 3
043/300 Write block 2611 len 12288 data: 2 2 2
044/300 Read block 1807 len 24576
045/300 Read block 1399 len 32768
046/300 Write block 4779 len 20480 data: 4 4 4 3 3
047/300 Write block 1795 len 28672 data: 2 2 2 2 2 2 2
048/300 Read block 502 len 12288
049/300 Read block 2410 len 12288
050/300 Write block 2455 len 16384 data: 2 2 2 2
051/300 Write block 153 len 16384 data: 4 4 3 3
052/300 Read block 3114 len 8192
053/300 Write block 1391 len 8192 data: 3 3
054/300 Write block 392 len 4096 data: 2
055/300 Write block 709 len 40960 data: 2 2 2 2 2 2 2 2 2 2
056/300 Write block 2783 len 4096 data: 4
057/300 Read block 553 len 16384
058/300 Write block 4386 len 24576 data: 3 2 2 2 3 3
059/300 Read block 4735 len 12288
060/300 Write block 4881 len 36864 data: 3 3 3 3 3 3 4 5 5
061/300 Write block 2470 len 12288 data: 2 3 3
062/300 Write block 2090 len 36864 data: 2 2 2 2 2 2 2 2 2
063/300 Read block 3737 len 16384
064/300 Write block 1727 len 24576 data: 2 2 2 3 3 3
065/300 Read block 2396 len 12288
066/300 Write block 1174 len 28672 data: 3 3 4 4 4 5 4
067/300 Read block 4102 len 24576
068/300 Write block 1700 len 20480 data: 4 4 3 3 2
069/300 Read block 4084 len 4096
070/300 Write block 1001 len 12288 data: 3 3 3
071/300 Write block 4427 len 20480 data: 2 2 2 2 2
072/300 Read block 1056 len 8192
073/300 Write block 827 len 20480 data: 2 2 3 3 3
074/300 Write block 4101 len 16384 data: 2 2 2 2
075/300 Write block 4630 len 8192 data: 3 4
076/300 Read block 1491 len 4096
077/300 Read block 3161 len 28672
078/300 Write block 3330 len 12288 data: 2 2 2
079/300 Read block 4159 len 36864
080/300 Flush
081/300 Flush
082/300 Write block 706 len 32768 data: 2 2 2 3 3 3 3 3
083/300 Read block 3446 len 28672
084/300 Flush
085/300 Write block 230 len 8192 data: 2 2
086/300 Write block 1373 len 28672 data: 2 2 2 2 2 2 2
087/300 Write block 3954 len 36864 data: 2 2 2 2 2 2 2 2 2
088/300 Read block 3672 len 40960
089/300 Write block 4159 len 40960 data: 2 2 2 2 2 2 3 3 3 3
090/300 Read block 3205 len 8192
091/300 Write block 4322 len 36864 data: 3 3 2 2 2 2 2 2 2
092/300 Read block 2384 len 28672
093/300 Read block 26 len 4096
094/300 Write block 1852 len 12288 data: 3 2 2
095/300 Read block 4888 len 40960
096/300 Read block 2522 len 36864
097/300 Read block 729 len 4096
098/300 Write block 4529 len 8192 data: 4 4
099/300 Read block 4753 len 12288
100/300 Read block 691 len 20480
101/300 Read block 1352 len 28672
102/300 Read block 1923 len 36864
103/300 Write block 4851 len 24576 data: 2 2 2 2 2 2
104/300 Write block 957 len 36864 data: 2 2 2 2 3 3 3 3 3
105/300 Write block 4076 len 24576 data: 3 2 2 3 2 2
106/300 Read block 4427 len 28672
107/300 Flush
108/300 Write block 4835 len 20480 data: 2 2 2 2 2
109/300 Write block 2626 len 4096 data: 3
110/300 Read block 3261 len 24576
111/300 Read block 1881 len 4096
112/300 Read block 1731 len 24576
113/300 Flush
114/300 Read block 2086 len 16384
115/300 Read block 3480 len 28672
116/300 Read block 1 len 20480
117/300 Read block 3559 len 40960
118/300 Write block 515 len 32768 data: 2 2 2 2 2 2 3 3
119/300 Read block 1729 len 12288
120/300 Read block 1127 len 20480
121/300 Read block 1087 len 12288
122/300 Read block 1000 len 20480
123/300 Write block 2202 len 4096 data: 3
124/300 Write block 3025 len 16384 data: 2 2 3 3
125/300 Read block 4594 len 20480
126/300 Read block 4011 len 32768
127/300 Write block 3308 len 40960 data: 3 3 2 2 3 3 2 2 2 2
128/300 Write block 3739 len 4096 data: 3
129/300 Read block 1681 len 4096
130/300 Write block 2776 len 36864 data: 2 2 2 3 3 3 3 5 4
131/300 Read block 77 len 32768
132/300 Write block 632 len 12288 data: 2 2 2
133/300 Write block 3510 len 28672 data: 2 2 2 2 2 2 2
134/300 Read block 727 len 8192
135/300 Read block 1001 len 4096
136/300 Write block 3724 len 8192 data: 4 2
137/300 Read block 1372 len 8192
138/300 Write block 4982 len 24576 data: 2 2 3 3 3 3
139/300 Read block 21 len 8192
140/300 Read block 2702 len 40960
141/300 Read block 33 len 36864
142/300 Write block 4313 len 36864 data: 2 2 2 2 2 2 3 3 3
143/300 Read block 3309 len 12288
144/300 Read block 1864 len 16384
145/300 Write block 1260 len 20480 data: 2 3 3 3 3
146/300 Write block 641 len 32768 data: 2 2 2 2 2 2 2 2
147/300 Flush
148/300 Write block 4110 len 4096 data: 2
149/300 Read block 4305 len 20480
150/300 Write block 620 len 32768 data: 2 2 2 2 2 3 3 3
151/300 Read block 3384 len 40960
152/300 Write block 1064 len 12288 data: 2 2 2
153/300 Read block 2518 len 16384
154/300 Read block 1961 len 28672
155/300 Write block 2852 len 28672 data: 2 2 2 2 2 2 2
156/300 Write block 1946 len 12288 data: 2 2 2
157/300 Read block 821 len 28672
158/300 Write block 3592 len 32768 data: 2 2 2 2 2 2 2 2
159/300 Write block 2690 len 36864 data: 4 4 3 3 3 3 3 3 3
160/300 Write block 427 len 40960 data: 2 3 3 4 4 4 4 3 3 4
161/300 Read block 3351 len 32768
162/300 Write block 3165 len 24576 data: 2 2 2 2 2 2
163/300 Flush
164/300 Read block 1170 len 40960
165/300 Write block 1066 len 20480 data: 3 2 2 2 2
166/300 Write block 3121 len 8192 data: 3 3
167/300 Read block 3632 len 20480
168/300 Write block 3027 len 12288 data: 4 4 4
169/300 Read block 69 len 20480
170/300 Read block 4374 len 8192
171/300 Read block 406 len 40960
172/300 Write block 926 len 4096 data: 3
173/300 Read block 4034 len 32768
174/300 Flush
175/300 Write block 2923 len 16384 data: 2 2 2 2
176/300 Write block 1358 len 32768 data: 2 3 3 3 2 2 2 2
177/300 Flush
178/300 Write block 4204 len 28672 data: 2 2 2 2 2 3 3
179/300 Read block 1182 len 24576
180/300 Read block 890 len 32768
181/300 Write block 4616 len 4096 data: 2
182/300 Read block 2275 len 28672
183/300 Read block 4755 len 4096
184/300 Read block 1547 len 36864
185/300 Read block 4208 len 24576
186/300 Read block 446 len 28672
187/300 Read block 3417 len 16384
188/300 Read block 4425 len 8192
189/300 Read block 1252 len 4096
190/300 Write block 3082 len 12288 data: 2 2 2
191/300 Read block 3154 len 36864
192/300 Flush
193/300 Write block 101 len 24576 data: 3 3 3 3 3 3
194/300 Write block 2866 len 28672 data: 3 3 3 3 3 3 3
195/300 Write block 4775 len 4096 data: 2
196/300 Read block 2190 len 16384
197/300 Flush
198/300 Read block 2070 len 12288
199/300 Read block 834 len 32768
200/300 Write block 3205 len 12288 data: 2 2 2
201/300 Write block 2917 len 8192 data: 2 2
202/300 Read block 862 len 16384
203/300 Write block 218 len 8192 data: 3 3
204/300 Read block 1957 len 16384
205/300 Write block 3461 len 4096 data: 2
206/300 Flush
207/300 Read block 3650 len 12288
208/300 Write block 4926 len 8192 data: 3 2
209/300 Write block 4756 len 36864 data: 2 2 2 2 2 2 2 2 3
210/300 Read block 3848 len 12288
211/300 Read block 3318 len 24576
212/300 Write block 161 len 24576 data: 3 3 2 2 2 2
213/300 Read block 3864 len 12288
214/300 Read block 4964 len 4096
215/300 Write block 1789 len 20480 data: 2 2 2 2 2
216/300 Read block 1509 len 16384
217/300 Write block 277 len 36864 data: 4 4 3 3 3 3 4 4 3
218/300 Read block 693 len 16384
219/300 Flush
220/300 Read block 3922 len 20480
221/300 Write block 4837 len 36864 data: 3 3 3 3 3 4 3 3 3
222/300 Write block 4980 len 32768 data: 2 2 3 3 4 4 4 4
223/300 Flush
224/300 Write block 893 len 8192 data: 2 2
225/300 Write block 2569 len 20480 data: 2 2 2 2 2
226/300 Read block 2429 len 20480
227/300 Write block 4853 len 16384 data: 3 3 3 3
228/300 Read block 1392 len 36864
229/300 Write block 1356 len 24576 data: 2 2 3 4 4 4
230/300 Write block 1278 len 24576 data: 2 2 3 3 3 3
231/300 Read block 332 len 12288
232/300 Write block 2454 len 8192 data: 2 3
233/300 Read block 3949 len 8192
234/300 Read block 2269 len 4096
235/300 Flush
236/300 Write block 3760 len 12288 data: 2 2 2
237/300 Read block 1757 len 20480
238/300 Flush
239/300 Write block 4903 len 16384 data: 3 3 3 3
240/300 Write block 4678 len 24576 data: 3 3 3 3 2 2
241/300 Write block 3912 len 20480 data: 2 2 2 2 2
242/300 Write block 2664 len 28672 data: 3 3 3 3 2 2 3
243/300 Read block 2081 len 12288
244/300 Write block 4426 len 16384 data: 2 3 3 3
245/300 Read block 1882 len 12288
246/300 Write block 1848 len 40960 data: 2 2 2 3 4 3 3 2 2 2
247/300 Write block 2028 len 24576 data: 2 2 2 2 2 2
248/300 Read block 4931 len 28672
249/300 Write block 3864 len 28672 data: 2 2 2 2 2 3 3
250/300 Write block 862 len 24576 data: 3 3 3 2 3 3
251/300 Write block 1178 len 16384 data: 5 6 5 4
252/300 Read block 2696 len 24576
253/300 Read block 4829 len 24576
254/300 Write block 416 len 24576 data: 4 4 4 4 4 4
255/300 Write block 1025 len 12288 data: 2 2 2
256/300 Read block 2899 len 12288
257/300 Read block 716 len 36864
258/300 Read block 2358 len 36864
259/300 Write block 4590 len 36864 data: 2 2 2 3 3 3 2 2 2
260/300 Read block 1953 len 4096
261/300 Write block 1821 len 36864 data: 2 2 2 2 2 2 2 2 2
262/300 Write block 3471 len 36864 data: 2 2 2 2 3 2 2 2 3
263/300 Flush
264/300 Write block 1995 len 24576 data: 2 2 2 2 2 3
265/300 Read block 2490 len 28672
266/300 Flush
267/300 Write block 2898 len 4096 data: 2
268/300 Write block 3297 len 4096 data: 5
269/300 Write block 2720 len 8192 data: 2 2
270/300 Write block 3212 len 40960 data: 2 2 2 2 2 2 2 2 2 2
271/300 Read block 2339 len 24576
272/300 Write block 53 len 32768 data: 3 3 3 3 3 3 2 2
273/300 Read block 101 len 32768
274/300 Read block 3167 len 28672
275/300 Read block 235 len 36864
276/300 Write block 3524 len 36864 data: 4 4 4 5 5 4 2 2 2
277/300 Read block 619 len 32768
278/300 Read block 1223 len 32768
279/300 Write block 647 len 36864 data: 3 3 2 2 2 2 2 2 2
280/300 Read block 615 len 4096
281/300 Read block 2701 len 16384
282/300 Read block 1176 len 12288
283/300 Read block 4450 len 28672
284/300 Write block 1261 len 40960 data: 4 4 4 4 3 2 2 2 2 2
285/300 Write block 2484 len 12288 data: 2 2 2
286/300 Write block 245 len 8192 data: 2 2
287/300 Write block 2957 len 24576 data: 3 3 3 2 3 4
288/300 Read block 500 len 24576
289/300 Read block 1210 len 40960
290/300 Read block 1981 len 28672
291/300 Read block 4863 len 36864
292/300 Read block 2617 len 32768
293/300 Write block 1340 len 24576 data: 3 3 3 3 2 2
294/300 Write block 2568 len 12288 data: 2 3 3
295/300 Read block 1861 len 32768
296/300 Write block 1723 len 32768 data: 2 2 2 2 3 3 3 4
297/300 Read block 1394 len 16384
298/300 Write block 1354 len 36864 data: 3 3 3 3 4 5 5 5 3
299/300 Write block 4249 len 12288 data: 2 2 2
300/300 Read block 1760 len 20480
301/300 Read block 1776 len 8192
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:02:37.367 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/0, method: POST, req_id: 243f4aa8-3818-4d51-a346-6757f84dfbb3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2507 Acked Read 3 Sent Done Done false
2508 Acked Read 4 Sent Done Done false
2509 Acked Write 6 Sent Done Done false
2510 Acked Read 7 Sent Done Done false
2511 Acked Flush 0 Sent Done Done false
2512 Acked Write 6 Sent Done Done false
2513 Acked Read 5 Sent Done Done false
2514 Acked Write 8 Sent Done Done false
2515 Acked Write 9 Sent Done Done false
2516 Acked Write 8 Sent Done Done false
2517 Acked Flush 0 Sent Done Done false
2518 Acked Write 2 Sent Done Done false
2519 Acked Write 8 Sent Done Done false
2520 Acked Write 10 Sent Done Done false
2521 Acked Read 5 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Write 9 Sent Done Done false
2524 Acked Read 4 Sent Done Done false
2525 Acked Write 7 Sent Done Done false
2526 Acked Flush 0 Sent Done Done false
2527 Acked Write 3 Sent Done Done false
2528 Acked Read 9 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Write 3 Sent Done Done false
2531 Acked Flush 0 Sent Done Done false
2532 Acked Write 8 Sent Done Done false
2533 Acked Write 5 Sent Done Done false
2534 Acked Read 10 Sent Done Done false
2535 Acked Write 7 Sent Done Done false
2536 Acked Write 9 Sent Done Done false
2537 Acked Read 4 Sent Done Done false
2538 Acked Write 10 Sent Done Done false
2539 Acked Flush 0 Sent Done Done false
2540 Acked Write 3 Sent Done Done false
2541 Acked Write 5 Sent Done Done false
2542 Acked Read 10 Sent Done Done false
2543 Acked Read 7 Sent Done Done false
2544 Acked Write 10 Sent Done Done false
2545 Acked Read 2 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 1 Sent Done Done false
2548 Acked Write 4 Sent Done Done false
2549 Acked Write 3 Sent Done Done false
2550 Acked Read 6 Sent Done Done false
2551 Acked Read 8 Sent Done Done false
2552 Acked Write 5 Sent Done Done false
2553 Acked Write 7 Sent Done Done false
2554 Acked Read 3 Sent Done Done false
2555 Acked Read 3 Sent Done Done false
2556 Acked Write 4 Sent Done Done false
2557 Acked Write 4 Sent Done Done false
2558 Acked Read 2 Sent Done Done false
2559 Acked Write 2 Sent Done Done false
2560 Acked Write 1 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Write 1 Sent Done Done false
2563 Acked Read 4 Sent Done Done false
2564 Acked Write 6 Sent Done Done false
2565 Acked Read 3 Sent Done Done false
2566 Acked Write 9 Sent Done Done false
2567 Acked Write 3 Sent Done Done false
2568 Acked Write 9 Sent Done Done false
2569 Acked Read 4 Sent Done Done false
2570 Acked Write 6 Sent Done Done false
2571 Acked Read 3 Sent Done Done false
2572 Acked Write 7 Sent Done Done false
2573 Acked Read 6 Sent Done Done false
2574 Acked Write 5 Sent Done Done false
2575 Acked Read 1 Sent Done Done false
2576 Acked Write 3 Sent Done Done false
2577 Acked Write 5 Sent Done Done false
2578 Acked Read 2 Sent Done Done false
2579 Acked Write 5 Sent Done Done false
2580 Acked Write 4 Sent Done Done false
2581 Acked Write 2 Sent Done Done false
2582 Acked Read 1 Sent Done Done false
2583 Acked Read 7 Sent Done Done false
2584 Acked Write 3 Sent Done Done false
2585 Acked Read 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Flush 0 Sent Done Done false
2588 Acked Write 8 Sent Done Done false
2589 Acked Read 7 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Write 2 Sent Done Done false
2592 Acked Write 7 Sent Done Done false
2593 Acked Write 9 Sent Done Done false
2594 Acked Read 10 Sent Done Done false
2595 Acked Write 10 Sent Done Done false
2596 Acked Read 2 Sent Done Done false
2597 Acked Write 9 Sent Done Done false
2598 Acked Read 7 Sent Done Done false
2599 Acked Read 1 Sent Done Done false
2600 Acked Write 3 Sent Done Done false
2601 Acked Read 10 Sent Done Done false
2602 Acked Read 9 Sent Done Done false
2603 Acked Read 1 Sent Done Done false
2604 Acked Write 2 Sent Done Done false
2605 Acked Read 3 Sent Done Done false
2606 Acked Read 5 Sent Done Done false
2607 Acked Read 7 Sent Done Done false
2608 Acked Read 9 Sent Done Done false
2609 Acked Write 6 Sent Done Done false
2610 Acked Write 9 Sent Done Done false
2611 Acked Write 6 Sent Done Done false
2612 Acked Read 7 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Write 5 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Read 6 Sent Done Done false
2617 Acked Read 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Flush 0 Sent Done Done false
2620 Acked Read 4 Sent Done Done false
2621 Acked Read 7 Sent Done Done false
2622 Acked Read 5 Sent Done Done false
2623 Acked Read 10 Sent Done Done false
2624 Acked Write 8 Sent Done Done false
2625 Acked Read 3 Sent Done Done false
2626 Acked Read 5 Sent Done Done false
2627 Acked Read 3 Sent Done Done false
2628 Acked Read 5 Sent Done Done false
2629 Acked Write 1 Sent Done Done false
2630 Acked Write 4 Sent Done Done false
2631 Acked Read 5 Sent Done Done false
2632 Acked Read 8 Sent Done Done false
2633 Acked Write 10 Sent Done Done false
2634 Acked Write 1 Sent Done Done false
2635 Acked Read 1 Sent Done Done false
2636 Acked Write 9 Sent Done Done false
2637 Acked Read 8 Sent Done Done false
2638 Acked Write 3 Sent Done Done false
2639 Acked Write 7 Sent Done Done false
2640 Acked Read 2 Sent Done Done false
2641 Acked Read 1 Sent Done Done false
2642 Acked Write 2 Sent Done Done false
2643 Acked Read 2 Sent Done Done false
2644 Acked Write 6 Sent Done Done false
2645 Acked Read 2 Sent Done Done false
2646 Acked Read 10 Sent Done Done false
2647 Acked Read 9 Sent Done Done false
2648 Acked Write 9 Sent Done Done false
2649 Acked Read 3 Sent Done Done false
2650 Acked Read 4 Sent Done Done false
2651 Acked Write 5 Sent Done Done false
2652 Acked Write 8 Sent Done Done false
2653 Acked Flush 0 Sent Done Done false
2654 Acked Write 1 Sent Done Done false
2655 Acked Read 5 Sent Done Done false
2656 Acked Write 8 Sent Done Done false
2657 Acked Read 10 Sent Done Done false
2658 Acked Write 3 Sent Done Done false
2659 Acked Read 4 Sent Done Done false
2660 Acked Read 7 Sent Done Done false
2661 Acked Write 7 Sent Done Done false
2662 Acked Write 3 Sent Done Done false
2663 Acked Read 7 Sent Done Done false
2664 Acked Write 8 Sent Done Done false
2665 Acked Write 9 Sent Done Done false
2666 Acked Write 10 Sent Done Done false
2667 Acked Read 8 Sent Done Done false
2668 Acked Write 6 Sent Done Done false
2669 Acked Flush 0 Sent Done Done false
2670 Acked Read 10 Sent Done Done false
2671 Acked Write 5 Sent Done Done false
2672 Acked Write 2 Sent Done Done false
2673 Acked Read 5 Sent Done Done false
2674 Acked Write 3 Sent Done Done false
2675 Acked Read 5 Sent Done Done false
2676 Acked Read 2 Sent Done Done false
2677 Acked Read 10 Sent Done Done false
2678 Acked Write 1 Sent Done Done false
2679 Acked Read 8 Sent Done Done false
2680 Acked Flush 0 Sent Done Done false
2681 Acked Write 4 Sent Done Done false
2682 Acked Write 8 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 7 Sent Done Done false
2685 Acked Read 6 Sent Done Done false
2686 Acked Read 8 Sent Done Done false
2687 Acked Write 1 Sent Done Done false
2688 Acked Read 7 Sent Done Done false
2689 Acked Read 1 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 6 Sent Done Done false
2692 Acked Read 7 Sent Done Done false
2693 Acked Read 4 Sent Done Done false
2694 Acked Read 2 Sent Done Done false
2695 Acked Read 1 Sent Done Done false
2696 Acked Write 3 Sent Done Done false
2697 Acked Read 9 Sent Done Done false
2698 Acked Flush 0 Sent Done Done false
2699 Acked Write 6 Sent Done Done false
2700 Acked Write 7 Sent Done Done false
2701 Acked Write 1 Sent Done Done false
2702 Acked Read 4 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Read 3 Sent Done Done false
2705 Acked Read 8 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Write 2 Sent Done Done false
2708 Acked Read 4 Sent Done Done false
2709 Acked Write 2 Sent Done Done false
2710 Acked Read 4 Sent Done Done false
2711 Acked Write 1 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Read 3 Sent Done Done false
2714 Acked Write 2 Sent Done Done false
2715 Acked Write 9 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 6 Sent Done Done false
2718 Acked Write 6 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Write 5 Sent Done Done false
2722 Acked Read 4 Sent Done Done false
2723 Acked Write 9 Sent Done Done false
2724 Acked Read 4 Sent Done Done false
2725 Acked Flush 0 Sent Done Done false
2726 Acked Read 5 Sent Done Done false
2727 Acked Write 9 Sent Done Done false
2728 Acked Write 8 Sent Done Done false
2729 Acked Flush 0 Sent Done Done false
2730 Acked Write 2 Sent Done Done false
2731 Acked Write 5 Sent Done Done false
2732 Acked Read 5 Sent Done Done false
2733 Acked Write 4 Sent Done Done false
2734 Acked Read 9 Sent Done Done false
2735 Acked Write 6 Sent Done Done false
2736 Acked Write 6 Sent Done Done false
2737 Acked Read 3 Sent Done Done false
2738 Acked Write 2 Sent Done Done false
2739 Acked Read 2 Sent Done Done false
2740 Acked Read 1 Sent Done Done false
2741 Acked Flush 0 Sent Done Done false
2742 Acked Write 3 Sent Done Done false
2743 Acked Read 5 Sent Done Done false
2744 Acked Flush 0 Sent Done Done false
2745 Acked Write 4 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Write 5 Sent Done Done false
2748 Acked Write 7 Sent Done Done false
2749 Acked Read 3 Sent Done Done false
2750 Acked Write 4 Sent Done Done false
2751 Acked Read 3 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 6 Sent Done Done false
2754 Acked Read 7 Sent Done Done false
2755 Acked Write 7 Sent Done Done false
2756 Acked Write 6 Sent Done Done false
2757 Acked Write 4 Sent Done Done false
2758 Acked Read 6 Sent Done Done false
2759 Acked Read 6 Sent Done Done false
2760 Acked Write 6 Sent Done Done false
2761 Acked Write 3 Sent Done Done false
2762 Acked Read 3 Sent Done Done false
2763 Acked Read 9 Sent Done Done false
2764 Acked Read 9 Sent Done Done false
2765 Acked Write 9 Sent Done Done false
2766 Acked Read 1 Sent Done Done false
2767 Acked Write 9 Sent Done Done false
2768 Acked Write 9 Sent Done Done false
2769 Acked Flush 0 Sent Done Done false
2770 Acked Write 6 Sent Done Done false
2771 Acked Read 7 Sent Done Done false
2772 Acked Flush 0 Sent Done Done false
2773 Acked Write 1 Sent Done Done false
2774 Acked Write 1 Sent Done Done false
2775 Acked Write 2 Sent Done Done false
2776 Acked Write 10 Sent Done Done false
2777 Acked Read 6 Sent Done Done false
2778 Acked Write 8 Sent Done Done false
2779 Acked Read 8 Sent Done Done false
2780 Acked Read 7 Sent Done Done false
2781 Acked Read 9 Sent Done Done false
2782 Acked Write 9 Sent Done Done false
2783 Acked Read 8 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Write 9 Sent Done Done false
2786 Acked Read 1 Sent Done Done false
2787 Acked Read 4 Sent Done Done false
2788 Acked Read 3 Sent Done Done false
2789 Acked Read 7 Sent Done Done false
2790 Acked Write 10 Sent Done Done false
2791 Acked Write 3 Sent Done Done false
2792 Acked Write 2 Sent Done Done false
2793 Acked Write 6 Sent Done Done false
2794 Acked Read 6 Sent Done Done false
2795 Acked Read 10 Sent Done Done false
2796 Acked Read 7 Sent Done Done false
2797 Acked Read 9 Sent Done Done false
2798 Acked Read 8 Sent Done Done false
2799 Acked Write 6 Sent Done Done false
2800 Acked Write 3 Sent Done Done false
2801 Acked Read 8 Sent Done Done false
2802 Acked Write 8 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Write 9 Sent Done Done false
2805 Acked Write 3 Sent Done Done false
2806 Acked Read 5 Sent Done Done false
2807 Acked Read 2 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 2807 2806 2805 2804 2803
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:37.869205635Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:38.49562525Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:39.497448926Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:40.499578819Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2507 Acked Read 3 Sent Done Done false
2508 Acked Read 4 Sent Done Done false
2509 Acked Write 6 Sent Done Done false
2510 Acked Read 7 Sent Done Done false
2511 Acked Flush 0 Sent Done Done false
2512 Acked Write 6 Sent Done Done false
2513 Acked Read 5 Sent Done Done false
2514 Acked Write 8 Sent Done Done false
2515 Acked Write 9 Sent Done Done false
2516 Acked Write 8 Sent Done Done false
2517 Acked Flush 0 Sent Done Done false
2518 Acked Write 2 Sent Done Done false
2519 Acked Write 8 Sent Done Done false
2520 Acked Write 10 Sent Done Done false
2521 Acked Read 5 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Write 9 Sent Done Done false
2524 Acked Read 4 Sent Done Done false
2525 Acked Write 7 Sent Done Done false
2526 Acked Flush 0 Sent Done Done false
2527 Acked Write 3 Sent Done Done false
2528 Acked Read 9 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Write 3 Sent Done Done false
2531 Acked Flush 0 Sent Done Done false
2532 Acked Write 8 Sent Done Done false
2533 Acked Write 5 Sent Done Done false
2534 Acked Read 10 Sent Done Done false
2535 Acked Write 7 Sent Done Done false
2536 Acked Write 9 Sent Done Done false
2537 Acked Read 4 Sent Done Done false
2538 Acked Write 10 Sent Done Done false
2539 Acked Flush 0 Sent Done Done false
2540 Acked Write 3 Sent Done Done false
2541 Acked Write 5 Sent Done Done false
2542 Acked Read 10 Sent Done Done false
2543 Acked Read 7 Sent Done Done false
2544 Acked Write 10 Sent Done Done false
2545 Acked Read 2 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 1 Sent Done Done false
2548 Acked Write 4 Sent Done Done false
2549 Acked Write 3 Sent Done Done false
2550 Acked Read 6 Sent Done Done false
2551 Acked Read 8 Sent Done Done false
2552 Acked Write 5 Sent Done Done false
2553 Acked Write 7 Sent Done Done false
2554 Acked Read 3 Sent Done Done false
2555 Acked Read 3 Sent Done Done false
2556 Acked Write 4 Sent Done Done false
2557 Acked Write 4 Sent Done Done false
2558 Acked Read 2 Sent Done Done false
2559 Acked Write 2 Sent Done Done false
2560 Acked Write 1 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Write 1 Sent Done Done false
2563 Acked Read 4 Sent Done Done false
2564 Acked Write 6 Sent Done Done false
2565 Acked Read 3 Sent Done Done false
2566 Acked Write 9 Sent Done Done false
2567 Acked Write 3 Sent Done Done false
2568 Acked Write 9 Sent Done Done false
2569 Acked Read 4 Sent Done Done false
2570 Acked Write 6 Sent Done Done false
2571 Acked Read 3 Sent Done Done false
2572 Acked Write 7 Sent Done Done false
2573 Acked Read 6 Sent Done Done false
2574 Acked Write 5 Sent Done Done false
2575 Acked Read 1 Sent Done Done false
2576 Acked Write 3 Sent Done Done false
2577 Acked Write 5 Sent Done Done false
2578 Acked Read 2 Sent Done Done false
2579 Acked Write 5 Sent Done Done false
2580 Acked Write 4 Sent Done Done false
2581 Acked Write 2 Sent Done Done false
2582 Acked Read 1 Sent Done Done false
2583 Acked Read 7 Sent Done Done false
2584 Acked Write 3 Sent Done Done false
2585 Acked Read 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Flush 0 Sent Done Done false
2588 Acked Write 8 Sent Done Done false
2589 Acked Read 7 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Write 2 Sent Done Done false
2592 Acked Write 7 Sent Done Done false
2593 Acked Write 9 Sent Done Done false
2594 Acked Read 10 Sent Done Done false
2595 Acked Write 10 Sent Done Done false
2596 Acked Read 2 Sent Done Done false
2597 Acked Write 9 Sent Done Done false
2598 Acked Read 7 Sent Done Done false
2599 Acked Read 1 Sent Done Done false
2600 Acked Write 3 Sent Done Done false
2601 Acked Read 10 Sent Done Done false
2602 Acked Read 9 Sent Done Done false
2603 Acked Read 1 Sent Done Done false
2604 Acked Write 2 Sent Done Done false
2605 Acked Read 3 Sent Done Done false
2606 Acked Read 5 Sent Done Done false
2607 Acked Read 7 Sent Done Done false
2608 Acked Read 9 Sent Done Done false
2609 Acked Write 6 Sent Done Done false
2610 Acked Write 9 Sent Done Done false
2611 Acked Write 6 Sent Done Done false
2612 Acked Read 7 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Write 5 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Read 6 Sent Done Done false
2617 Acked Read 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Flush 0 Sent Done Done false
2620 Acked Read 4 Sent Done Done false
2621 Acked Read 7 Sent Done Done false
2622 Acked Read 5 Sent Done Done false
2623 Acked Read 10 Sent Done Done false
2624 Acked Write 8 Sent Done Done false
2625 Acked Read 3 Sent Done Done false
2626 Acked Read 5 Sent Done Done false
2627 Acked Read 3 Sent Done Done false
2628 Acked Read 5 Sent Done Done false
2629 Acked Write 1 Sent Done Done false
2630 Acked Write 4 Sent Done Done false
2631 Acked Read 5 Sent Done Done false
2632 Acked Read 8 Sent Done Done false
2633 Acked Write 10 Sent Done Done false
2634 Acked Write 1 Sent Done Done false
2635 Acked Read 1 Sent Done Done false
2636 Acked Write 9 Sent Done Done false
2637 Acked Read 8 Sent Done Done false
2638 Acked Write 3 Sent Done Done false
2639 Acked Write 7 Sent Done Done false
2640 Acked Read 2 Sent Done Done false
2641 Acked Read 1 Sent Done Done false
2642 Acked Write 2 Sent Done Done false
2643 Acked Read 2 Sent Done Done false
2644 Acked Write 6 Sent Done Done false
2645 Acked Read 2 Sent Done Done false
2646 Acked Read 10 Sent Done Done false
2647 Acked Read 9 Sent Done Done false
2648 Acked Write 9 Sent Done Done false
2649 Acked Read 3 Sent Done Done false
2650 Acked Read 4 Sent Done Done false
2651 Acked Write 5 Sent Done Done false
2652 Acked Write 8 Sent Done Done false
2653 Acked Flush 0 Sent Done Done false
2654 Acked Write 1 Sent Done Done false
2655 Acked Read 5 Sent Done Done false
2656 Acked Write 8 Sent Done Done false
2657 Acked Read 10 Sent Done Done false
2658 Acked Write 3 Sent Done Done false
2659 Acked Read 4 Sent Done Done false
2660 Acked Read 7 Sent Done Done false
2661 Acked Write 7 Sent Done Done false
2662 Acked Write 3 Sent Done Done false
2663 Acked Read 7 Sent Done Done false
2664 Acked Write 8 Sent Done Done false
2665 Acked Write 9 Sent Done Done false
2666 Acked Write 10 Sent Done Done false
2667 Acked Read 8 Sent Done Done false
2668 Acked Write 6 Sent Done Done false
2669 Acked Flush 0 Sent Done Done false
2670 Acked Read 10 Sent Done Done false
2671 Acked Write 5 Sent Done Done false
2672 Acked Write 2 Sent Done Done false
2673 Acked Read 5 Sent Done Done false
2674 Acked Write 3 Sent Done Done false
2675 Acked Read 5 Sent Done Done false
2676 Acked Read 2 Sent Done Done false
2677 Acked Read 10 Sent Done Done false
2678 Acked Write 1 Sent Done Done false
2679 Acked Read 8 Sent Done Done false
2680 Acked Flush 0 Sent Done Done false
2681 Acked Write 4 Sent Done Done false
2682 Acked Write 8 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 7 Sent Done Done false
2685 Acked Read 6 Sent Done Done false
2686 Acked Read 8 Sent Done Done false
2687 Acked Write 1 Sent Done Done false
2688 Acked Read 7 Sent Done Done false
2689 Acked Read 1 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 6 Sent Done Done false
2692 Acked Read 7 Sent Done Done false
2693 Acked Read 4 Sent Done Done false
2694 Acked Read 2 Sent Done Done false
2695 Acked Read 1 Sent Done Done false
2696 Acked Write 3 Sent Done Done false
2697 Acked Read 9 Sent Done Done false
2698 Acked Flush 0 Sent Done Done false
2699 Acked Write 6 Sent Done Done false
2700 Acked Write 7 Sent Done Done false
2701 Acked Write 1 Sent Done Done false
2702 Acked Read 4 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Read 3 Sent Done Done false
2705 Acked Read 8 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Write 2 Sent Done Done false
2708 Acked Read 4 Sent Done Done false
2709 Acked Write 2 Sent Done Done false
2710 Acked Read 4 Sent Done Done false
2711 Acked Write 1 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Read 3 Sent Done Done false
2714 Acked Write 2 Sent Done Done false
2715 Acked Write 9 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 6 Sent Done Done false
2718 Acked Write 6 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Write 5 Sent Done Done false
2722 Acked Read 4 Sent Done Done false
2723 Acked Write 9 Sent Done Done false
2724 Acked Read 4 Sent Done Done false
2725 Acked Flush 0 Sent Done Done false
2726 Acked Read 5 Sent Done Done false
2727 Acked Write 9 Sent Done Done false
2728 Acked Write 8 Sent Done Done false
2729 Acked Flush 0 Sent Done Done false
2730 Acked Write 2 Sent Done Done false
2731 Acked Write 5 Sent Done Done false
2732 Acked Read 5 Sent Done Done false
2733 Acked Write 4 Sent Done Done false
2734 Acked Read 9 Sent Done Done false
2735 Acked Write 6 Sent Done Done false
2736 Acked Write 6 Sent Done Done false
2737 Acked Read 3 Sent Done Done false
2738 Acked Write 2 Sent Done Done false
2739 Acked Read 2 Sent Done Done false
2740 Acked Read 1 Sent Done Done false
2741 Acked Flush 0 Sent Done Done false
2742 Acked Write 3 Sent Done Done false
2743 Acked Read 5 Sent Done Done false
2744 Acked Flush 0 Sent Done Done false
2745 Acked Write 4 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Write 5 Sent Done Done false
2748 Acked Write 7 Sent Done Done false
2749 Acked Read 3 Sent Done Done false
2750 Acked Write 4 Sent Done Done false
2751 Acked Read 3 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 6 Sent Done Done false
2754 Acked Read 7 Sent Done Done false
2755 Acked Write 7 Sent Done Done false
2756 Acked Write 6 Sent Done Done false
2757 Acked Write 4 Sent Done Done false
2758 Acked Read 6 Sent Done Done false
2759 Acked Read 6 Sent Done Done false
2760 Acked Write 6 Sent Done Done false
2761 Acked Write 3 Sent Done Done false
2762 Acked Read 3 Sent Done Done false
2763 Acked Read 9 Sent Done Done false
2764 Acked Read 9 Sent Done Done false
2765 Acked Write 9 Sent Done Done false
2766 Acked Read 1 Sent Done Done false
2767 Acked Write 9 Sent Done Done false
2768 Acked Write 9 Sent Done Done false
2769 Acked Flush 0 Sent Done Done false
2770 Acked Write 6 Sent Done Done false
2771 Acked Read 7 Sent Done Done false
2772 Acked Flush 0 Sent Done Done false
2773 Acked Write 1 Sent Done Done false
2774 Acked Write 1 Sent Done Done false
2775 Acked Write 2 Sent Done Done false
2776 Acked Write 10 Sent Done Done false
2777 Acked Read 6 Sent Done Done false
2778 Acked Write 8 Sent Done Done false
2779 Acked Read 8 Sent Done Done false
2780 Acked Read 7 Sent Done Done false
2781 Acked Read 9 Sent Done Done false
2782 Acked Write 9 Sent Done Done false
2783 Acked Read 8 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Write 9 Sent Done Done false
2786 Acked Read 1 Sent Done Done false
2787 Acked Read 4 Sent Done Done false
2788 Acked Read 3 Sent Done Done false
2789 Acked Read 7 Sent Done Done false
2790 Acked Write 10 Sent Done Done false
2791 Acked Write 3 Sent Done Done false
2792 Acked Write 2 Sent Done Done false
2793 Acked Write 6 Sent Done Done false
2794 Acked Read 6 Sent Done Done false
2795 Acked Read 10 Sent Done Done false
2796 Acked Read 7 Sent Done Done false
2797 Acked Read 9 Sent Done Done false
2798 Acked Read 8 Sent Done Done false
2799 Acked Write 6 Sent Done Done false
2800 Acked Write 3 Sent Done Done false
2801 Acked Read 8 Sent Done Done false
2802 Acked Write 8 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Write 9 Sent Done Done false
2805 Acked Write 3 Sent Done Done false
2806 Acked Read 5 Sent Done Done false
2807 Acked Read 2 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 2807 2806 2805 2804 2803
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:41.871799328Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:42.501842204Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:43.13518136Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:43.135230258Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:43.135251103Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:43.135270129Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:43.13539064Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:43.636550774Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:44.505074916Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1217)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2507 Acked Read 3 Sent Done Done false
2508 Acked Read 4 Sent Done Done false
2509 Acked Write 6 Sent Done Done false
2510 Acked Read 7 Sent Done Done false
2511 Acked Flush 0 Sent Done Done false
2512 Acked Write 6 Sent Done Done false
2513 Acked Read 5 Sent Done Done false
2514 Acked Write 8 Sent Done Done false
2515 Acked Write 9 Sent Done Done false
2516 Acked Write 8 Sent Done Done false
2517 Acked Flush 0 Sent Done Done false
2518 Acked Write 2 Sent Done Done false
2519 Acked Write 8 Sent Done Done false
2520 Acked Write 10 Sent Done Done false
2521 Acked Read 5 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Write 9 Sent Done Done false
2524 Acked Read 4 Sent Done Done false
2525 Acked Write 7 Sent Done Done false
2526 Acked Flush 0 Sent Done Done false
2527 Acked Write 3 Sent Done Done false
2528 Acked Read 9 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Write 3 Sent Done Done false
2531 Acked Flush 0 Sent Done Done false
2532 Acked Write 8 Sent Done Done false
2533 Acked Write 5 Sent Done Done false
2534 Acked Read 10 Sent Done Done false
2535 Acked Write 7 Sent Done Done false
2536 Acked Write 9 Sent Done Done false
2537 Acked Read 4 Sent Done Done false
2538 Acked Write 10 Sent Done Done false
2539 Acked Flush 0 Sent Done Done false
2540 Acked Write 3 Sent Done Done false
2541 Acked Write 5 Sent Done Done false
2542 Acked Read 10 Sent Done Done false
2543 Acked Read 7 Sent Done Done false
2544 Acked Write 10 Sent Done Done false
2545 Acked Read 2 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 1 Sent Done Done false
2548 Acked Write 4 Sent Done Done false
2549 Acked Write 3 Sent Done Done false
2550 Acked Read 6 Sent Done Done false
2551 Acked Read 8 Sent Done Done false
2552 Acked Write 5 Sent Done Done false
2553 Acked Write 7 Sent Done Done false
2554 Acked Read 3 Sent Done Done false
2555 Acked Read 3 Sent Done Done false
2556 Acked Write 4 Sent Done Done false
2557 Acked Write 4 Sent Done Done false
2558 Acked Read 2 Sent Done Done false
2559 Acked Write 2 Sent Done Done false
2560 Acked Write 1 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Write 1 Sent Done Done false
2563 Acked Read 4 Sent Done Done false
2564 Acked Write 6 Sent Done Done false
2565 Acked Read 3 Sent Done Done false
2566 Acked Write 9 Sent Done Done false
2567 Acked Write 3 Sent Done Done false
2568 Acked Write 9 Sent Done Done false
2569 Acked Read 4 Sent Done Done false
2570 Acked Write 6 Sent Done Done false
2571 Acked Read 3 Sent Done Done false
2572 Acked Write 7 Sent Done Done false
2573 Acked Read 6 Sent Done Done false
2574 Acked Write 5 Sent Done Done false
2575 Acked Read 1 Sent Done Done false
2576 Acked Write 3 Sent Done Done false
2577 Acked Write 5 Sent Done Done false
2578 Acked Read 2 Sent Done Done false
2579 Acked Write 5 Sent Done Done false
2580 Acked Write 4 Sent Done Done false
2581 Acked Write 2 Sent Done Done false
2582 Acked Read 1 Sent Done Done false
2583 Acked Read 7 Sent Done Done false
2584 Acked Write 3 Sent Done Done false
2585 Acked Read 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Flush 0 Sent Done Done false
2588 Acked Write 8 Sent Done Done false
2589 Acked Read 7 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Write 2 Sent Done Done false
2592 Acked Write 7 Sent Done Done false
2593 Acked Write 9 Sent Done Done false
2594 Acked Read 10 Sent Done Done false
2595 Acked Write 10 Sent Done Done false
2596 Acked Read 2 Sent Done Done false
2597 Acked Write 9 Sent Done Done false
2598 Acked Read 7 Sent Done Done false
2599 Acked Read 1 Sent Done Done false
2600 Acked Write 3 Sent Done Done false
2601 Acked Read 10 Sent Done Done false
2602 Acked Read 9 Sent Done Done false
2603 Acked Read 1 Sent Done Done false
2604 Acked Write 2 Sent Done Done false
2605 Acked Read 3 Sent Done Done false
2606 Acked Read 5 Sent Done Done false
2607 Acked Read 7 Sent Done Done false
2608 Acked Read 9 Sent Done Done false
2609 Acked Write 6 Sent Done Done false
2610 Acked Write 9 Sent Done Done false
2611 Acked Write 6 Sent Done Done false
2612 Acked Read 7 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Write 5 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Read 6 Sent Done Done false
2617 Acked Read 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Flush 0 Sent Done Done false
2620 Acked Read 4 Sent Done Done false
2621 Acked Read 7 Sent Done Done false
2622 Acked Read 5 Sent Done Done false
2623 Acked Read 10 Sent Done Done false
2624 Acked Write 8 Sent Done Done false
2625 Acked Read 3 Sent Done Done false
2626 Acked Read 5 Sent Done Done false
2627 Acked Read 3 Sent Done Done false
2628 Acked Read 5 Sent Done Done false
2629 Acked Write 1 Sent Done Done false
2630 Acked Write 4 Sent Done Done false
2631 Acked Read 5 Sent Done Done false
2632 Acked Read 8 Sent Done Done false
2633 Acked Write 10 Sent Done Done false
2634 Acked Write 1 Sent Done Done false
2635 Acked Read 1 Sent Done Done false
2636 Acked Write 9 Sent Done Done false
2637 Acked Read 8 Sent Done Done false
2638 Acked Write 3 Sent Done Done false
2639 Acked Write 7 Sent Done Done false
2640 Acked Read 2 Sent Done Done false
2641 Acked Read 1 Sent Done Done false
2642 Acked Write 2 Sent Done Done false
2643 Acked Read 2 Sent Done Done false
2644 Acked Write 6 Sent Done Done false
2645 Acked Read 2 Sent Done Done false
2646 Acked Read 10 Sent Done Done false
2647 Acked Read 9 Sent Done Done false
2648 Acked Write 9 Sent Done Done false
2649 Acked Read 3 Sent Done Done false
2650 Acked Read 4 Sent Done Done false
2651 Acked Write 5 Sent Done Done false
2652 Acked Write 8 Sent Done Done false
2653 Acked Flush 0 Sent Done Done false
2654 Acked Write 1 Sent Done Done false
2655 Acked Read 5 Sent Done Done false
2656 Acked Write 8 Sent Done Done false
2657 Acked Read 10 Sent Done Done false
2658 Acked Write 3 Sent Done Done false
2659 Acked Read 4 Sent Done Done false
2660 Acked Read 7 Sent Done Done false
2661 Acked Write 7 Sent Done Done false
2662 Acked Write 3 Sent Done Done false
2663 Acked Read 7 Sent Done Done false
2664 Acked Write 8 Sent Done Done false
2665 Acked Write 9 Sent Done Done false
2666 Acked Write 10 Sent Done Done false
2667 Acked Read 8 Sent Done Done false
2668 Acked Write 6 Sent Done Done false
2669 Acked Flush 0 Sent Done Done false
2670 Acked Read 10 Sent Done Done false
2671 Acked Write 5 Sent Done Done false
2672 Acked Write 2 Sent Done Done false
2673 Acked Read 5 Sent Done Done false
2674 Acked Write 3 Sent Done Done false
2675 Acked Read 5 Sent Done Done false
2676 Acked Read 2 Sent Done Done false
2677 Acked Read 10 Sent Done Done false
2678 Acked Write 1 Sent Done Done false
2679 Acked Read 8 Sent Done Done false
2680 Acked Flush 0 Sent Done Done false
2681 Acked Write 4 Sent Done Done false
2682 Acked Write 8 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 7 Sent Done Done false
2685 Acked Read 6 Sent Done Done false
2686 Acked Read 8 Sent Done Done false
2687 Acked Write 1 Sent Done Done false
2688 Acked Read 7 Sent Done Done false
2689 Acked Read 1 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 6 Sent Done Done false
2692 Acked Read 7 Sent Done Done false
2693 Acked Read 4 Sent Done Done false
2694 Acked Read 2 Sent Done Done false
2695 Acked Read 1 Sent Done Done false
2696 Acked Write 3 Sent Done Done false
2697 Acked Read 9 Sent Done Done false
2698 Acked Flush 0 Sent Done Done false
2699 Acked Write 6 Sent Done Done false
2700 Acked Write 7 Sent Done Done false
2701 Acked Write 1 Sent Done Done false
2702 Acked Read 4 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Read 3 Sent Done Done false
2705 Acked Read 8 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Write 2 Sent Done Done false
2708 Acked Read 4 Sent Done Done false
2709 Acked Write 2 Sent Done Done false
2710 Acked Read 4 Sent Done Done false
2711 Acked Write 1 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Read 3 Sent Done Done false
2714 Acked Write 2 Sent Done Done false
2715 Acked Write 9 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 6 Sent Done Done false
2718 Acked Write 6 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Write 5 Sent Done Done false
2722 Acked Read 4 Sent Done Done false
2723 Acked Write 9 Sent Done Done false
2724 Acked Read 4 Sent Done Done false
2725 Acked Flush 0 Sent Done Done false
2726 Acked Read 5 Sent Done Done false
2727 Acked Write 9 Sent Done Done false
2728 Acked Write 8 Sent Done Done false
2729 Acked Flush 0 Sent Done Done false
2730 Acked Write 2 Sent Done Done false
2731 Acked Write 5 Sent Done Done false
2732 Acked Read 5 Sent Done Done false
2733 Acked Write 4 Sent Done Done false
2734 Acked Read 9 Sent Done Done false
2735 Acked Write 6 Sent Done Done false
2736 Acked Write 6 Sent Done Done false
2737 Acked Read 3 Sent Done Done false
2738 Acked Write 2 Sent Done Done false
2739 Acked Read 2 Sent Done Done false
2740 Acked Read 1 Sent Done Done false
2741 Acked Flush 0 Sent Done Done false
2742 Acked Write 3 Sent Done Done false
2743 Acked Read 5 Sent Done Done false
2744 Acked Flush 0 Sent Done Done false
2745 Acked Write 4 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Write 5 Sent Done Done false
2748 Acked Write 7 Sent Done Done false
2749 Acked Read 3 Sent Done Done false
2750 Acked Write 4 Sent Done Done false
2751 Acked Read 3 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 6 Sent Done Done false
2754 Acked Read 7 Sent Done Done false
2755 Acked Write 7 Sent Done Done false
2756 Acked Write 6 Sent Done Done false
2757 Acked Write 4 Sent Done Done false
2758 Acked Read 6 Sent Done Done false
2759 Acked Read 6 Sent Done Done false
2760 Acked Write 6 Sent Done Done false
2761 Acked Write 3 Sent Done Done false
2762 Acked Read 3 Sent Done Done false
2763 Acked Read 9 Sent Done Done false
2764 Acked Read 9 Sent Done Done false
2765 Acked Write 9 Sent Done Done false
2766 Acked Read 1 Sent Done Done false
2767 Acked Write 9 Sent Done Done false
2768 Acked Write 9 Sent Done Done false
2769 Acked Flush 0 Sent Done Done false
2770 Acked Write 6 Sent Done Done false
2771 Acked Read 7 Sent Done Done false
2772 Acked Flush 0 Sent Done Done false
2773 Acked Write 1 Sent Done Done false
2774 Acked Write 1 Sent Done Done false
2775 Acked Write 2 Sent Done Done false
2776 Acked Write 10 Sent Done Done false
2777 Acked Read 6 Sent Done Done false
2778 Acked Write 8 Sent Done Done false
2779 Acked Read 8 Sent Done Done false
2780 Acked Read 7 Sent Done Done false
2781 Acked Read 9 Sent Done Done false
2782 Acked Write 9 Sent Done Done false
2783 Acked Read 8 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Write 9 Sent Done Done false
2786 Acked Read 1 Sent Done Done false
2787 Acked Read 4 Sent Done Done false
2788 Acked Read 3 Sent Done Done false
2789 Acked Read 7 Sent Done Done false
2790 Acked Write 10 Sent Done Done false
2791 Acked Write 3 Sent Done Done false
2792 Acked Write 2 Sent Done Done false
2793 Acked Write 6 Sent Done Done false
2794 Acked Read 6 Sent Done Done false
2795 Acked Read 10 Sent Done Done false
2796 Acked Read 7 Sent Done Done false
2797 Acked Read 9 Sent Done Done false
2798 Acked Read 8 Sent Done Done false
2799 Acked Write 6 Sent Done Done false
2800 Acked Write 3 Sent Done Done false
2801 Acked Read 8 Sent Done Done false
2802 Acked Write 8 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Write 9 Sent Done Done false
2805 Acked Write 3 Sent Done Done false
2806 Acked Read 5 Sent Done Done false
2807 Acked Read 2 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 2807 2806 2805 2804 2803
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:45.87372261Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:46.508584771Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:47.509657577Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:48.511189384Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2507 Acked Read 3 Sent Done Done false
2508 Acked Read 4 Sent Done Done false
2509 Acked Write 6 Sent Done Done false
2510 Acked Read 7 Sent Done Done false
2511 Acked Flush 0 Sent Done Done false
2512 Acked Write 6 Sent Done Done false
2513 Acked Read 5 Sent Done Done false
2514 Acked Write 8 Sent Done Done false
2515 Acked Write 9 Sent Done Done false
2516 Acked Write 8 Sent Done Done false
2517 Acked Flush 0 Sent Done Done false
2518 Acked Write 2 Sent Done Done false
2519 Acked Write 8 Sent Done Done false
2520 Acked Write 10 Sent Done Done false
2521 Acked Read 5 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Write 9 Sent Done Done false
2524 Acked Read 4 Sent Done Done false
2525 Acked Write 7 Sent Done Done false
2526 Acked Flush 0 Sent Done Done false
2527 Acked Write 3 Sent Done Done false
2528 Acked Read 9 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Write 3 Sent Done Done false
2531 Acked Flush 0 Sent Done Done false
2532 Acked Write 8 Sent Done Done false
2533 Acked Write 5 Sent Done Done false
2534 Acked Read 10 Sent Done Done false
2535 Acked Write 7 Sent Done Done false
2536 Acked Write 9 Sent Done Done false
2537 Acked Read 4 Sent Done Done false
2538 Acked Write 10 Sent Done Done false
2539 Acked Flush 0 Sent Done Done false
2540 Acked Write 3 Sent Done Done false
2541 Acked Write 5 Sent Done Done false
2542 Acked Read 10 Sent Done Done false
2543 Acked Read 7 Sent Done Done false
2544 Acked Write 10 Sent Done Done false
2545 Acked Read 2 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 1 Sent Done Done false
2548 Acked Write 4 Sent Done Done false
2549 Acked Write 3 Sent Done Done false
2550 Acked Read 6 Sent Done Done false
2551 Acked Read 8 Sent Done Done false
2552 Acked Write 5 Sent Done Done false
2553 Acked Write 7 Sent Done Done false
2554 Acked Read 3 Sent Done Done false
2555 Acked Read 3 Sent Done Done false
2556 Acked Write 4 Sent Done Done false
2557 Acked Write 4 Sent Done Done false
2558 Acked Read 2 Sent Done Done false
2559 Acked Write 2 Sent Done Done false
2560 Acked Write 1 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Write 1 Sent Done Done false
2563 Acked Read 4 Sent Done Done false
2564 Acked Write 6 Sent Done Done false
2565 Acked Read 3 Sent Done Done false
2566 Acked Write 9 Sent Done Done false
2567 Acked Write 3 Sent Done Done false
2568 Acked Write 9 Sent Done Done false
2569 Acked Read 4 Sent Done Done false
2570 Acked Write 6 Sent Done Done false
2571 Acked Read 3 Sent Done Done false
2572 Acked Write 7 Sent Done Done false
2573 Acked Read 6 Sent Done Done false
2574 Acked Write 5 Sent Done Done false
2575 Acked Read 1 Sent Done Done false
2576 Acked Write 3 Sent Done Done false
2577 Acked Write 5 Sent Done Done false
2578 Acked Read 2 Sent Done Done false
2579 Acked Write 5 Sent Done Done false
2580 Acked Write 4 Sent Done Done false
2581 Acked Write 2 Sent Done Done false
2582 Acked Read 1 Sent Done Done false
2583 Acked Read 7 Sent Done Done false
2584 Acked Write 3 Sent Done Done false
2585 Acked Read 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Flush 0 Sent Done Done false
2588 Acked Write 8 Sent Done Done false
2589 Acked Read 7 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Write 2 Sent Done Done false
2592 Acked Write 7 Sent Done Done false
2593 Acked Write 9 Sent Done Done false
2594 Acked Read 10 Sent Done Done false
2595 Acked Write 10 Sent Done Done false
2596 Acked Read 2 Sent Done Done false
2597 Acked Write 9 Sent Done Done false
2598 Acked Read 7 Sent Done Done false
2599 Acked Read 1 Sent Done Done false
2600 Acked Write 3 Sent Done Done false
2601 Acked Read 10 Sent Done Done false
2602 Acked Read 9 Sent Done Done false
2603 Acked Read 1 Sent Done Done false
2604 Acked Write 2 Sent Done Done false
2605 Acked Read 3 Sent Done Done false
2606 Acked Read 5 Sent Done Done false
2607 Acked Read 7 Sent Done Done false
2608 Acked Read 9 Sent Done Done false
2609 Acked Write 6 Sent Done Done false
2610 Acked Write 9 Sent Done Done false
2611 Acked Write 6 Sent Done Done false
2612 Acked Read 7 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Write 5 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Read 6 Sent Done Done false
2617 Acked Read 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Flush 0 Sent Done Done false
2620 Acked Read 4 Sent Done Done false
2621 Acked Read 7 Sent Done Done false
2622 Acked Read 5 Sent Done Done false
2623 Acked Read 10 Sent Done Done false
2624 Acked Write 8 Sent Done Done false
2625 Acked Read 3 Sent Done Done false
2626 Acked Read 5 Sent Done Done false
2627 Acked Read 3 Sent Done Done false
2628 Acked Read 5 Sent Done Done false
2629 Acked Write 1 Sent Done Done false
2630 Acked Write 4 Sent Done Done false
2631 Acked Read 5 Sent Done Done false
2632 Acked Read 8 Sent Done Done false
2633 Acked Write 10 Sent Done Done false
2634 Acked Write 1 Sent Done Done false
2635 Acked Read 1 Sent Done Done false
2636 Acked Write 9 Sent Done Done false
2637 Acked Read 8 Sent Done Done false
2638 Acked Write 3 Sent Done Done false
2639 Acked Write 7 Sent Done Done false
2640 Acked Read 2 Sent Done Done false
2641 Acked Read 1 Sent Done Done false
2642 Acked Write 2 Sent Done Done false
2643 Acked Read 2 Sent Done Done false
2644 Acked Write 6 Sent Done Done false
2645 Acked Read 2 Sent Done Done false
2646 Acked Read 10 Sent Done Done false
2647 Acked Read 9 Sent Done Done false
2648 Acked Write 9 Sent Done Done false
2649 Acked Read 3 Sent Done Done false
2650 Acked Read 4 Sent Done Done false
2651 Acked Write 5 Sent Done Done false
2652 Acked Write 8 Sent Done Done false
2653 Acked Flush 0 Sent Done Done false
2654 Acked Write 1 Sent Done Done false
2655 Acked Read 5 Sent Done Done false
2656 Acked Write 8 Sent Done Done false
2657 Acked Read 10 Sent Done Done false
2658 Acked Write 3 Sent Done Done false
2659 Acked Read 4 Sent Done Done false
2660 Acked Read 7 Sent Done Done false
2661 Acked Write 7 Sent Done Done false
2662 Acked Write 3 Sent Done Done false
2663 Acked Read 7 Sent Done Done false
2664 Acked Write 8 Sent Done Done false
2665 Acked Write 9 Sent Done Done false
2666 Acked Write 10 Sent Done Done false
2667 Acked Read 8 Sent Done Done false
2668 Acked Write 6 Sent Done Done false
2669 Acked Flush 0 Sent Done Done false
2670 Acked Read 10 Sent Done Done false
2671 Acked Write 5 Sent Done Done false
2672 Acked Write 2 Sent Done Done false
2673 Acked Read 5 Sent Done Done false
2674 Acked Write 3 Sent Done Done false
2675 Acked Read 5 Sent Done Done false
2676 Acked Read 2 Sent Done Done false
2677 Acked Read 10 Sent Done Done false
2678 Acked Write 1 Sent Done Done false
2679 Acked Read 8 Sent Done Done false
2680 Acked Flush 0 Sent Done Done false
2681 Acked Write 4 Sent Done Done false
2682 Acked Write 8 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 7 Sent Done Done false
2685 Acked Read 6 Sent Done Done false
2686 Acked Read 8 Sent Done Done false
2687 Acked Write 1 Sent Done Done false
2688 Acked Read 7 Sent Done Done false
2689 Acked Read 1 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 6 Sent Done Done false
2692 Acked Read 7 Sent Done Done false
2693 Acked Read 4 Sent Done Done false
2694 Acked Read 2 Sent Done Done false
2695 Acked Read 1 Sent Done Done false
2696 Acked Write 3 Sent Done Done false
2697 Acked Read 9 Sent Done Done false
2698 Acked Flush 0 Sent Done Done false
2699 Acked Write 6 Sent Done Done false
2700 Acked Write 7 Sent Done Done false
2701 Acked Write 1 Sent Done Done false
2702 Acked Read 4 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Read 3 Sent Done Done false
2705 Acked Read 8 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Write 2 Sent Done Done false
2708 Acked Read 4 Sent Done Done false
2709 Acked Write 2 Sent Done Done false
2710 Acked Read 4 Sent Done Done false
2711 Acked Write 1 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Read 3 Sent Done Done false
2714 Acked Write 2 Sent Done Done false
2715 Acked Write 9 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 6 Sent Done Done false
2718 Acked Write 6 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Write 5 Sent Done Done false
2722 Acked Read 4 Sent Done Done false
2723 Acked Write 9 Sent Done Done false
2724 Acked Read 4 Sent Done Done false
2725 Acked Flush 0 Sent Done Done false
2726 Acked Read 5 Sent Done Done false
2727 Acked Write 9 Sent Done Done false
2728 Acked Write 8 Sent Done Done false
2729 Acked Flush 0 Sent Done Done false
2730 Acked Write 2 Sent Done Done false
2731 Acked Write 5 Sent Done Done false
2732 Acked Read 5 Sent Done Done false
2733 Acked Write 4 Sent Done Done false
2734 Acked Read 9 Sent Done Done false
2735 Acked Write 6 Sent Done Done false
2736 Acked Write 6 Sent Done Done false
2737 Acked Read 3 Sent Done Done false
2738 Acked Write 2 Sent Done Done false
2739 Acked Read 2 Sent Done Done false
2740 Acked Read 1 Sent Done Done false
2741 Acked Flush 0 Sent Done Done false
2742 Acked Write 3 Sent Done Done false
2743 Acked Read 5 Sent Done Done false
2744 Acked Flush 0 Sent Done Done false
2745 Acked Write 4 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Write 5 Sent Done Done false
2748 Acked Write 7 Sent Done Done false
2749 Acked Read 3 Sent Done Done false
2750 Acked Write 4 Sent Done Done false
2751 Acked Read 3 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 6 Sent Done Done false
2754 Acked Read 7 Sent Done Done false
2755 Acked Write 7 Sent Done Done false
2756 Acked Write 6 Sent Done Done false
2757 Acked Write 4 Sent Done Done false
2758 Acked Read 6 Sent Done Done false
2759 Acked Read 6 Sent Done Done false
2760 Acked Write 6 Sent Done Done false
2761 Acked Write 3 Sent Done Done false
2762 Acked Read 3 Sent Done Done false
2763 Acked Read 9 Sent Done Done false
2764 Acked Read 9 Sent Done Done false
2765 Acked Write 9 Sent Done Done false
2766 Acked Read 1 Sent Done Done false
2767 Acked Write 9 Sent Done Done false
2768 Acked Write 9 Sent Done Done false
2769 Acked Flush 0 Sent Done Done false
2770 Acked Write 6 Sent Done Done false
2771 Acked Read 7 Sent Done Done false
2772 Acked Flush 0 Sent Done Done false
2773 Acked Write 1 Sent Done Done false
2774 Acked Write 1 Sent Done Done false
2775 Acked Write 2 Sent Done Done false
2776 Acked Write 10 Sent Done Done false
2777 Acked Read 6 Sent Done Done false
2778 Acked Write 8 Sent Done Done false
2779 Acked Read 8 Sent Done Done false
2780 Acked Read 7 Sent Done Done false
2781 Acked Read 9 Sent Done Done false
2782 Acked Write 9 Sent Done Done false
2783 Acked Read 8 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Write 9 Sent Done Done false
2786 Acked Read 1 Sent Done Done false
2787 Acked Read 4 Sent Done Done false
2788 Acked Read 3 Sent Done Done false
2789 Acked Read 7 Sent Done Done false
2790 Acked Write 10 Sent Done Done false
2791 Acked Write 3 Sent Done Done false
2792 Acked Write 2 Sent Done Done false
2793 Acked Write 6 Sent Done Done false
2794 Acked Read 6 Sent Done Done false
2795 Acked Read 10 Sent Done Done false
2796 Acked Read 7 Sent Done Done false
2797 Acked Read 9 Sent Done Done false
2798 Acked Read 8 Sent Done Done false
2799 Acked Write 6 Sent Done Done false
2800 Acked Write 3 Sent Done Done false
2801 Acked Read 8 Sent Done Done false
2802 Acked Write 8 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Write 9 Sent Done Done false
2805 Acked Write 3 Sent Done Done false
2806 Acked Read 5 Sent Done Done false
2807 Acked Read 2 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 2807 2806 2805 2804 2803
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:49.875801195Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:50.514408359Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:51.516474347Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:52.518472511Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.136312272Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46410)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.136421306Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.13736291Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.137916738Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.137942242Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(2506))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.137956438Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.137969275Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(2506))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.137982112Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(2506))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:53.138274922Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:35
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2773 Acked Write 1 Done Done Done true
2774 Acked Write 1 Done Done Done true
2775 Acked Write 2 Done Done Done true
2776 Acked Write 10 Done Done Done true
2777 Acked Read 6 Done Done Done true
2778 Acked Write 8 Done Done Done true
2779 Acked Read 8 Done Done Done true
2780 Acked Read 7 Done Done Done true
2781 Acked Read 9 Done Done Done true
2782 Acked Write 9 Done Done Done true
2783 Acked Read 8 Done Done Done true
2784 Acked Read 8 Done Done Done true
2785 Acked Write 9 Done Done Done true
2786 Acked Read 1 Done Done Done true
2787 Acked Read 4 Done Done Done true
2788 Acked Read 3 Done Done Done true
2789 Acked Read 7 Done Done Done true
2790 Acked Write 10 Done Done Done true
2791 Acked Write 3 Done Done Done true
2792 Acked Write 2 Done Done Done true
2793 Acked Write 6 Done Done Done true
2794 Acked Read 6 Done Done Done true
2795 Acked Read 10 Done Done Done true
2796 Acked Read 7 Done Done Done true
2797 Acked Read 9 Done Done Done true
2798 Acked Read 8 Done Done Done true
2799 Acked Write 6 Done Done Done true
2800 Acked Write 3 Done Done Done true
2801 Acked Read 8 Done Done Done true
2802 Acked Write 8 Done Done Done true
2803 Acked Read 4 Done Done Done true
2804 Acked Write 9 Done Done Done true
2805 Acked Write 3 Done Done Done true
2806 Acked Read 5 Done Done Done true
2807 Acked Read 2 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 35 35 35 105
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2772)) Some(JobId(2772)) Some(JobId(2772))
Downstairs last five completed: 2772 2771 2770 2769 2768
Upstairs last five completed: 2807 2806 2805 2804 2803
CLIENT: Up:0 ds:35 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2808 2807 2806 2805 2804
Upstairs last five completed: 2808 2807 2806 2805 2804
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:02:57.378[8810] Got stop action so:false kr:false
INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 80e0812b-96f7-41dc-bae2-bdf88a89df44, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:57.379364248Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:57.37943906Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:02:57.379456456Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:02:57.379 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:57.379518811Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
118, response_code: 200, uri: /state/cid/0, method: GET, req_id: 61bab362-d83e-436e-893a-7e8e80ddabf8, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:03:01.382 INFO request completed, latency_us: 345, response_code: 200, uri: /state/cid/0, method: GET, req_id: 083cdd87-dfe7-46a1-97e1-1139f43fcf38, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 4211 len 28672 data: 3 3 3 2 2 2 2
002/300 Write block 1709 len 24576 data: 3 3 3 3 3 3
003/300 Write block 2430 len 4096 data: 2
004/300 Read block 4934 len 20480
005/300 Write block 2411 len 8192 data: 2 2
006/300 Read block 1664 len 4096
007/300 Read block 3183 len 24576
008/300 Read block 1345 len 8192
009/300 Read block 3309 len 32768
010/300 Read block 4698 len 24576
011/300 Read block 2474 len 16384
012/300 Write block 1774 len 8192 data: 2 2
013/300 Read block 4613 len 12288
014/300 Read block 3185 len 32768
015/300 Read block 4482 len 24576
016/300 Write block 4733 len 8192 data: 3 3
017/300 Flush
018/300 Read block 3353 len 20480
019/300 Read block 1412 len 12288
020/300 Read block 2598 len 12288
021/300 Read block 3211 len 16384
022/300 Read block 1451 len 24576
023/300 Read block 3719 len 4096
024/300 Read block 963 len 36864
025/300 Write block 333 len 36864 data: 2 2 2 3 4 4 3 3 4
026/300 Read block 2830 len 8192
027/300 Read block 194 len 8192
028/300 Write block 91 len 32768 data: 2 2 2 2 2 2 2 2
029/300 Read block 2664 len 12288
030/300 Read block 1584 len 28672
031/300 Write block 398 len 16384 data: 2 2 2 2
032/300 Write block 3903 len 28672 data: 3 3 3 3 4 3 3
033/300 Read block 4582 len 40960
034/300 Read block 2791 len 16384
035/300 Write block 508 len 16384 data: 2 3 2 2
036/300 Write block 2114 len 20480 data: 3 3 3 4 4
037/300 Read block 985 len 36864
038/300 Read block 2450 len 36864
039/300 Write block 92 len 16384 data: 3 3 3 3
040/300 Read block 4218 len 36864
041/300 Read block 2306 len 28672
042/300 Read block 549 len 20480
043/300 Write block 2442 len 32768 data: 2 2 2 3 3 3 2 2
044/300 Flush
045/300 Write block 3107 len 24576 data: 3 4 4 4 4 4
046/300 Read block 4732 len 12288
047/300 Write block 772 len 4096 data: 2
048/300 Read block 2356 len 20480
049/300 Write block 2061 len 20480 data: 2 2 2 3 3
050/300 Read block 125 len 40960
051/300 Write block 3436 len 12288 data: 4 4 4
052/300 Flush
053/300 Read block 4098 len 20480
054/300 Read block 2351 len 24576
055/300 Read block 4985 len 32768
056/300 Write block 4203 len 8192 data: 2 3
057/300 Read block 3175 len 8192
058/300 Flush
059/300 Flush
060/300 Write block 1440 len 24576 data: 3 4 4 4 4 4
061/300 Read block 794 len 24576
062/300 Read block 609 len 20480
063/300 Read block 1652 len 36864
064/300 Write block 796 len 16384 data: 4 4 4 3
065/300 Read block 4094 len 24576
066/300 Write block 3249 len 16384 data: 3 3 2 2
067/300 Flush
068/300 Read block 4830 len 36864
069/300 Write block 4903 len 8192 data: 4 4
070/300 Read block 4848 len 12288
071/300 Read block 1190 len 36864
072/300 Read block 1083 len 28672
073/300 Read block 2856 len 36864
074/300 Read block 880 len 40960
075/300 Read block 3022 len 4096
076/300 Write block 4095 len 40960 data: 3 2 2 2 2 2 3 3 3 3
077/300 Read block 3870 len 28672
078/300 Read block 864 len 12288
079/300 Write block 2452 len 36864 data: 2 2 3 4 3 3 3 2 2
080/300 Write block 1485 len 36864 data: 2 2 2 2 3 3 3 3 2
081/300 Write block 2740 len 28672 data: 3 3 3 3 3 3 3
082/300 Write block 1491 len 8192 data: 4 4
083/300 Read block 1318 len 8192
084/300 Flush
085/300 Read block 4069 len 4096
086/300 Write block 4357 len 12288 data: 3 3 3
087/300 Write block 195 len 24576 data: 4 4 3 3 3 3
088/300 Write block 25 len 8192 data: 4 3
089/300 Write block 3096 len 20480 data: 2 2 3 3 4
090/300 Read block 4956 len 32768
091/300 Read block 4235 len 36864
092/300 Read block 4767 len 4096
093/300 Read block 3576 len 36864
094/300 Write block 3986 len 8192 data: 2 2
095/300 Write block 2468 len 28672 data: 2 2 3 4 4 3 3
096/300 Write block 2304 len 12288 data: 2 2 3
097/300 Write block 4961 len 8192 data: 2 2
098/300 Read block 2372 len 12288
099/300 Write block 4832 len 28672 data: 4 2 2 3 3 4 4
100/300 Write block 897 len 8192 data: 3 3
101/300 Read block 4843 len 24576
102/300 Write block 4307 len 4096 data: 2
103/300 Read block 3665 len 32768
104/300 Read block 1236 len 40960
105/300 Write block 94 len 8192 data: 4 4
106/300 Write block 3875 len 36864 data: 5 5 4 4 3 3 2 3 4
107/300 Read block 3233 len 24576
108/300 Read block 3471 len 40960
109/300 Write block 3293 len 40960 data: 3 3 3 4 6 5 4 3 3 4
110/300 Write block 1032 len 36864 data: 2 2 2 2 2 3 3 3 4
111/300 Write block 4510 len 8192 data: 2 2
112/300 Read block 693 len 32768
113/300 Flush
114/300 Write block 2668 len 20480 data: 3 3 4 3 3
115/300 Read block 766 len 28672
116/300 Write block 2128 len 4096 data: 3
117/300 Write block 4430 len 16384 data: 3 3 2 2
118/300 Flush
119/300 Write block 827 len 4096 data: 3
120/300 Read block 152 len 24576
121/300 Read block 1256 len 32768
122/300 Read block 3173 len 28672
123/300 Read block 2696 len 12288
124/300 Write block 1164 len 28672 data: 3 3 3 3 3 3 2
125/300 Read block 1815 len 36864
126/300 Read block 2943 len 12288
127/300 Write block 870 len 8192 data: 4 3
128/300 Read block 4787 len 12288
129/300 Write block 2263 len 32768 data: 2 2 2 2 2 2 2 2
130/300 Write block 2509 len 20480 data: 2 3 4 5 5
131/300 Write block 697 len 36864 data: 2 2 2 3 3 3 3 3 2
132/300 Write block 235 len 40960 data: 3 4 6 6 4 3 3 3 2 2
133/300 Read block 1413 len 36864
134/300 Read block 2296 len 20480
135/300 Write block 1624 len 32768 data: 4 4 4 4 4 4 4 3
136/300 Write block 453 len 32768 data: 6 6 5 5 3 3 3 3
137/300 Read block 1225 len 16384
138/300 Read block 1859 len 20480
139/300 Read block 3809 len 16384
140/300 Read block 186 len 16384
141/300 Read block 2814 len 24576
142/300 Read block 1967 len 36864
143/300 Write block 2683 len 28672 data: 3 3 3 2 3 3 4
144/300 Read block 636 len 16384
145/300 Read block 527 len 32768
146/300 Write block 2524 len 28672 data: 5 5 3 3 3 3 2
147/300 Read block 2194 len 16384
148/300 Write block 3776 len 36864 data: 2 2 2 2 3 4 4 4 3
149/300 Read block 4108 len 28672
150/300 Write block 2291 len 12288 data: 2 2 2
151/300 Read block 2959 len 12288
152/300 Read block 1828 len 20480
153/300 Read block 2091 len 20480
154/300 Write block 4798 len 20480 data: 3 2 2 2 2
155/300 Read block 4892 len 32768
156/300 Write block 2965 len 12288 data: 4 4 4
157/300 Read block 3737 len 16384
158/300 Read block 4295 len 20480
159/300 Write block 4989 len 8192 data: 2 2
160/300 Read block 2424 len 16384
161/300 Write block 2783 len 8192 data: 6 5
162/300 Read block 2131 len 24576
163/300 Write block 3734 len 20480 data: 3 3 3 3 3
164/300 Write block 486 len 4096 data: 2
165/300 Read block 1983 len 12288
166/300 Read block 4259 len 12288
167/300 Read block 4857 len 32768
168/300 Write block 316 len 40960 data: 2 2 2 3 3 3 3 3 2 2
169/300 Read block 2815 len 8192
170/300 Read block 3397 len 12288
171/300 Read block 4127 len 20480
172/300 Write block 4569 len 32768 data: 2 2 2 2 2 2 2 2
173/300 Read block 597 len 28672
174/300 Write block 4384 len 32768 data: 3 3 4 3 3 3 4 4
175/300 Write block 1363 len 24576 data: 3 3 3 2 2 2
176/300 Write block 495 len 28672 data: 3 3 3 3 3 3 3
177/300 Write block 3585 len 36864 data: 2 2 2 2 3 3 3 3 3
178/300 Read block 1762 len 12288
179/300 Write block 1488 len 36864 data: 3 4 4 5 5 3 2 2 2
180/300 Write block 478 len 20480 data: 5 2 2 2 2
181/300 Read block 862 len 32768
182/300 Read block 2163 len 40960
183/300 Read block 3799 len 36864
184/300 Read block 3090 len 36864
185/300 Read block 2741 len 16384
186/300 Read block 2482 len 20480
187/300 Read block 2374 len 8192
188/300 Flush
189/300 Read block 2842 len 8192
190/300 Write block 4879 len 12288 data: 3 3 4
191/300 Flush
192/300 Write block 937 len 32768 data: 2 3 3 3 3 3 3 3
193/300 Write block 4696 len 16384 data: 3 3 2 3
194/300 Read block 4441 len 32768
195/300 Read block 3387 len 28672
196/300 Read block 3178 len 20480
197/300 Write block 1665 len 32768 data: 3 3 3 4 4 3 3 3
198/300 Flush
199/300 Read block 4135 len 24576
200/300 Write block 2093 len 36864 data: 3 3 3 3 3 3 2 2 2
201/300 Write block 611 len 28672 data: 2 2 2 2 2 2 2
202/300 Write block 2473 len 16384 data: 4 4 3 3
203/300 Flush
204/300 Write block 3718 len 24576 data: 4 4 4 4 4 4
205/300 Read block 3611 len 24576
206/300 Read block 2673 len 4096
207/300 Read block 4540 len 20480
208/300 Write block 2152 len 36864 data: 3 2 2 2 2 2 2 2 2
209/300 Read block 842 len 16384
210/300 Write block 1715 len 4096 data: 3
211/300 Read block 2358 len 16384
212/300 Read block 2883 len 40960
213/300 Read block 454 len 36864
214/300 Write block 2684 len 16384 data: 4 4 3 4
215/300 Flush
216/300 Write block 4566 len 8192 data: 2 2
217/300 Read block 931 len 40960
218/300 Write block 2107 len 8192 data: 3 4
219/300 Read block 262 len 4096
220/300 Write block 927 len 28672 data: 3 3 3 4 4 4 3
221/300 Read block 1439 len 4096
222/300 Read block 1473 len 28672
223/300 Read block 1878 len 8192
224/300 Read block 4374 len 36864
225/300 Flush
226/300 Read block 4480 len 4096
227/300 Read block 2062 len 40960
228/300 Write block 2759 len 20480 data: 3 3 3 3 3
229/300 Write block 739 len 8192 data: 2 2
230/300 Write block 1733 len 4096 data: 3
231/300 Write block 434 len 8192 data: 4 4
232/300 Write block 4158 len 16384 data: 2 3 3 3
233/300 Write block 4126 len 32768 data: 3 3 3 3 3 3 2 2
234/300 Read block 4187 len 4096
235/300 Write block 3583 len 4096 data: 2
236/300 Write block 4953 len 40960 data: 3 3 2 2 2 2 2 2 3 3
237/300 Read block 3667 len 28672
238/300 Read block 3786 len 28672
239/300 Read block 3902 len 28672
240/300 Write block 1318 len 4096 data: 2
241/300 Read block 3465 len 28672
242/300 Write block 1177 len 24576 data: 5 6 7 6 5 3
243/300 Write block 1525 len 20480 data: 3 3 4 4 4
244/300 Write block 2174 len 8192 data: 2 2
245/300 Read block 370 len 20480
246/300 Read block 691 len 40960
247/300 Read block 4134 len 24576
248/300 Write block 1271 len 24576 data: 3 3 3 3 2 2
249/300 Read block 581 len 32768
250/300 Write block 1703 len 24576 data: 4 3 2 3 3 3
251/300 Flush
252/300 Read block 813 len 20480
253/300 Write block 1305 len 4096 data: 2
254/300 Read block 3316 len 28672
255/300 Flush
256/300 Write block 3322 len 4096 data: 2
257/300 Read block 2238 len 8192
258/300 Read block 4248 len 12288
259/300 Write block 2946 len 40960 data: 2 3 3 3 3 3 3 3 3 3
260/300 Read block 2461 len 8192
261/300 Read block 805 len 40960
262/300 Read block 2395 len 8192
263/300 Read block 4170 len 28672
264/300 Read block 157 len 20480
265/300 Read block 3362 len 16384
266/300 Write block 3126 len 36864 data: 2 2 2 2 2 2 2 2 2
267/300 Read block 116 len 40960
268/300 Read block 4513 len 20480
269/300 Write block 2813 len 28672 data: 3 3 3 3 3 3 3
270/300 Flush
271/300 Flush
272/300 Write block 2488 len 16384 data: 3 3 2 2
273/300 Read block 2690 len 12288
274/300 Write block 1124 len 32768 data: 2 3 3 3 2 2 2 2
275/300 Write block 769 len 8192 data: 2 2
276/300 Flush
277/300 Read block 1980 len 12288
278/300 Read block 3020 len 28672
279/300 Write block 1466 len 36864 data: 3 3 3 3 2 2 2 3 3
280/300 Write block 2416 len 16384 data: 2 2 2 2
281/300 Write block 4830 len 4096 data: 5
282/300 Read block 1546 len 28672
283/300 Flush
284/300 Write block 442 len 40960 data: 5 6 6 6 5 6 6 7 7 7
285/300 Flush
286/300 Read block 2538 len 40960
287/300 Read block 2139 len 40960
288/300 Read block 4518 len 28672
289/300 Read block 4653 len 16384
290/300 Flush
291/300 Write block 1885 len 16384 data: 2 2 2 2
292/300 Write block 1207 len 20480 data: 2 2 2 2 2
293/300 Read block 2022 len 36864
294/300 Flush
295/300 Write block 4978 len 32768 data: 3 3 3 3 4 4 5 5
296/300 Read block 3687 len 20480
297/300 Write block 2483 len 36864 data: 2 3 3 3 3 4 4 3 3
298/300 Flush
299/300 Write block 3961 len 4096 data: 3
300/300 Write block 3465 len 12288 data: 2 2 2
301/300 Write block 4479 len 36864 data: 3 2 2 2 2 2 2 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:03:01.626 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/0, method: POST, req_id: bcaa173f-edde-488f-9501-adc3d0502b87, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2809 Acked Write 7 Sent Done Done false
2810 Acked Write 6 Sent Done Done false
2811 Acked Write 1 Sent Done Done false
2812 Acked Read 5 Sent Done Done false
2813 Acked Write 2 Sent Done Done false
2814 Acked Read 1 Sent Done Done false
2815 Acked Read 6 Sent Done Done false
2816 Acked Read 2 Sent Done Done false
2817 Acked Read 8 Sent Done Done false
2818 Acked Read 6 Sent Done Done false
2819 Acked Read 4 Sent Done Done false
2820 Acked Write 2 Sent Done Done false
2821 Acked Read 3 Sent Done Done false
2822 Acked Read 8 Sent Done Done false
2823 Acked Read 6 Sent Done Done false
2824 Acked Write 2 Sent Done Done false
2825 Acked Flush 0 Sent Done Done false
2826 Acked Read 5 Sent Done Done false
2827 Acked Read 3 Sent Done Done false
2828 Acked Read 3 Sent Done Done false
2829 Acked Read 4 Sent Done Done false
2830 Acked Read 6 Sent Done Done false
2831 Acked Read 1 Sent Done Done false
2832 Acked Read 9 Sent Done Done false
2833 Acked Write 9 Sent Done Done false
2834 Acked Read 2 Sent Done Done false
2835 Acked Read 2 Sent Done Done false
2836 Acked Write 8 Sent Done Done false
2837 Acked Read 3 Sent Done Done false
2838 Acked Read 7 Sent Done Done false
2839 Acked Write 4 Sent Done Done false
2840 Acked Write 7 Sent Done Done false
2841 Acked Read 10 Sent Done Done false
2842 Acked Read 4 Sent Done Done false
2843 Acked Write 4 Sent Done Done false
2844 Acked Write 5 Sent Done Done false
2845 Acked Read 9 Sent Done Done false
2846 Acked Read 9 Sent Done Done false
2847 Acked Write 4 Sent Done Done false
2848 Acked Read 9 Sent Done Done false
2849 Acked Read 7 Sent Done Done false
2850 Acked Read 5 Sent Done Done false
2851 Acked Write 8 Sent Done Done false
2852 Acked Flush 0 Sent Done Done false
2853 Acked Write 6 Sent Done Done false
2854 Acked Read 3 Sent Done Done false
2855 Acked Write 1 Sent Done Done false
2856 Acked Read 5 Sent Done Done false
2857 Acked Write 5 Sent Done Done false
2858 Acked Read 10 Sent Done Done false
2859 Acked Write 3 Sent Done Done false
2860 Acked Flush 0 Sent Done Done false
2861 Acked Read 5 Sent Done Done false
2862 Acked Read 6 Sent Done Done false
2863 Acked Read 8 Sent Done Done false
2864 Acked Write 2 Sent Done Done false
2865 Acked Read 2 Sent Done Done false
2866 Acked Flush 0 Sent Done Done false
2867 Acked Flush 0 Sent Done Done false
2868 Acked Write 6 Sent Done Done false
2869 Acked Read 6 Sent Done Done false
2870 Acked Read 5 Sent Done Done false
2871 Acked Read 9 Sent Done Done false
2872 Acked Write 4 Sent Done Done false
2873 Acked Read 6 Sent Done Done false
2874 Acked Write 4 Sent Done Done false
2875 Acked Flush 0 Sent Done Done false
2876 Acked Read 9 Sent Done Done false
2877 Acked Write 2 Sent Done Done false
2878 Acked Read 3 Sent Done Done false
2879 Acked Read 9 Sent Done Done false
2880 Acked Read 7 Sent Done Done false
2881 Acked Read 9 Sent Done Done false
2882 Acked Read 10 Sent Done Done false
2883 Acked Read 1 Sent Done Done false
2884 Acked Write 10 Sent Done Done false
2885 Acked Read 7 Sent Done Done false
2886 Acked Read 3 Sent Done Done false
2887 Acked Write 9 Sent Done Done false
2888 Acked Write 9 Sent Done Done false
2889 Acked Write 7 Sent Done Done false
2890 Acked Write 2 Sent Done Done false
2891 Acked Read 2 Sent Done Done false
2892 Acked Flush 0 Sent Done Done false
2893 Acked Read 1 Sent Done Done false
2894 Acked Write 3 Sent Done Done false
2895 Acked Write 6 Sent Done Done false
2896 Acked Write 2 Sent Done Done false
2897 Acked Write 5 Sent Done Done false
2898 Acked Read 8 Sent Done Done false
2899 Acked Read 9 Sent Done Done false
2900 Acked Read 1 Sent Done Done false
2901 Acked Read 9 Sent Done Done false
2902 Acked Write 2 Sent Done Done false
2903 Acked Write 7 Sent Done Done false
2904 Acked Write 3 Sent Done Done false
2905 Acked Write 2 Sent Done Done false
2906 Acked Read 3 Sent Done Done false
2907 Acked Write 7 Sent Done Done false
2908 Acked Write 2 Sent Done Done false
2909 Acked Read 6 Sent Done Done false
2910 Acked Write 1 Sent Done Done false
2911 Acked Read 8 Sent Done Done false
2912 Acked Read 10 Sent Done Done false
2913 Acked Write 2 Sent Done Done false
2914 Acked Write 9 Sent Done Done false
2915 Acked Read 6 Sent Done Done false
2916 Acked Read 10 Sent Done Done false
2917 Acked Write 10 Sent Done Done false
2918 Acked Write 9 Sent Done Done false
2919 Acked Write 2 Sent Done Done false
2920 Acked Read 8 Sent Done Done false
2921 Acked Flush 0 Sent Done Done false
2922 Acked Write 5 Sent Done Done false
2923 Acked Read 7 Sent Done Done false
2924 Acked Write 1 Sent Done Done false
2925 Acked Write 4 Sent Done Done false
2926 Acked Flush 0 Sent Done Done false
2927 Acked Write 1 Sent Done Done false
2928 Acked Read 6 Sent Done Done false
2929 Acked Read 8 Sent Done Done false
2930 Acked Read 7 Sent Done Done false
2931 Acked Read 3 Sent Done Done false
2932 Acked Write 7 Sent Done Done false
2933 Acked Read 9 Sent Done Done false
2934 Acked Read 3 Sent Done Done false
2935 Acked Write 2 Sent Done Done false
2936 Acked Read 3 Sent Done Done false
2937 Acked Write 8 Sent Done Done false
2938 Acked Write 5 Sent Done Done false
2939 Acked Write 9 Sent Done Done false
2940 Acked Write 10 Sent Done Done false
2941 Acked Read 9 Sent Done Done false
2942 Acked Read 5 Sent Done Done false
2943 Acked Write 8 Sent Done Done false
2944 Acked Write 8 Sent Done Done false
2945 Acked Read 4 Sent Done Done false
2946 Acked Read 5 Sent Done Done false
2947 Acked Read 4 Sent Done Done false
2948 Acked Read 4 Sent Done Done false
2949 Acked Read 6 Sent Done Done false
2950 Acked Read 9 Sent Done Done false
2951 Acked Write 7 Sent Done Done false
2952 Acked Read 4 Sent Done Done false
2953 Acked Read 8 Sent Done Done false
2954 Acked Write 7 Sent Done Done false
2955 Acked Read 4 Sent Done Done false
2956 Acked Write 9 Sent Done Done false
2957 Acked Read 7 Sent Done Done false
2958 Acked Write 3 Sent Done Done false
2959 Acked Read 3 Sent Done Done false
2960 Acked Read 5 Sent Done Done false
2961 Acked Read 5 Sent Done Done false
2962 Acked Write 5 Sent Done Done false
2963 Acked Read 8 Sent Done Done false
2964 Acked Write 3 Sent Done Done false
2965 Acked Read 4 Sent Done Done false
2966 Acked Read 5 Sent Done Done false
2967 Acked Write 2 Sent Done Done false
2968 Acked Read 4 Sent Done Done false
2969 Acked Write 2 Sent Done Done false
2970 Acked Read 6 Sent Done Done false
2971 Acked Write 5 Sent Done Done false
2972 Acked Write 1 Sent Done Done false
2973 Acked Read 3 Sent Done Done false
2974 Acked Read 3 Sent Done Done false
2975 Acked Read 8 Sent Done Done false
2976 Acked Write 10 Sent Done Done false
2977 Acked Read 2 Sent Done Done false
2978 Acked Read 3 Sent Done Done false
2979 Acked Read 5 Sent Done Done false
2980 Acked Write 8 Sent Done Done false
2981 Acked Read 7 Sent Done Done false
2982 Acked Write 8 Sent Done Done false
2983 Acked Write 6 Sent Done Done false
2984 Acked Write 7 Sent Done Done false
2985 Acked Write 9 Sent Done Done false
2986 Acked Read 3 Sent Done Done false
2987 Acked Write 9 Sent Done Done false
2988 Acked Write 5 Sent Done Done false
2989 Acked Read 8 Sent Done Done false
2990 Acked Read 10 Sent Done Done false
2991 Acked Read 9 Sent Done Done false
2992 Acked Read 9 Sent Done Done false
2993 Acked Read 4 Sent Done Done false
2994 Acked Read 5 Sent Done Done false
2995 Acked Read 2 Sent Done Done false
2996 Acked Flush 0 Sent Done Done false
2997 Acked Read 2 Sent Done Done false
2998 Acked Write 3 Sent Done Done false
2999 Acked Flush 0 Sent Done Done false
3000 Acked Write 8 Sent Done Done false
3001 Acked Write 4 Sent Done Done false
3002 Acked Read 8 Sent Done Done false
3003 Acked Read 7 Sent Done Done false
3004 Acked Read 5 Sent Done Done false
3005 Acked Write 8 Sent Done Done false
3006 Acked Flush 0 Sent Done Done false
3007 Acked Read 6 Sent Done Done false
3008 Acked Write 9 Sent Done Done false
3009 Acked Write 7 Sent Done Done false
3010 Acked Write 4 Sent Done Done false
3011 Acked Flush 0 Sent Done Done false
3012 Acked Write 6 Sent Done Done false
3013 Acked Read 6 Sent Done Done false
3014 Acked Read 1 Sent Done Done false
3015 Acked Read 5 Sent Done Done false
3016 Acked Write 9 Sent Done Done false
3017 Acked Read 4 Sent Done Done false
3018 Acked Write 1 Sent Done Done false
3019 Acked Read 4 Sent Done Done false
3020 Acked Read 10 Sent Done Done false
3021 Acked Read 9 Sent Done Done false
3022 Acked Write 4 Sent Done Done false
3023 Acked Flush 0 Sent Done Done false
3024 Acked Write 2 Sent Done Done false
3025 Acked Read 10 Sent Done Done false
3026 Acked Write 2 Sent Done Done false
3027 Acked Read 1 Sent Done Done false
3028 Acked Write 7 Sent Done Done false
3029 Acked Read 1 Sent Done Done false
3030 Acked Read 7 Sent Done Done false
3031 Acked Read 2 Sent Done Done false
3032 Acked Read 9 Sent Done Done false
3033 Acked Flush 0 Sent Done Done false
3034 Acked Read 1 Sent Done Done false
3035 Acked Read 10 Sent Done Done false
3036 Acked Write 5 Sent Done Done false
3037 Acked Write 2 Sent Done Done false
3038 Acked Write 1 Sent Done Done false
3039 Acked Write 2 Sent Done Done false
3040 Acked Write 4 Sent Done Done false
3041 Acked Write 8 Sent Done Done false
3042 Acked Read 1 Sent Done Done false
3043 Acked Write 1 Sent Done Done false
3044 Acked Write 10 Sent Done Done false
3045 Acked Read 7 Sent Done Done false
3046 Acked Read 7 Sent Done Done false
3047 Acked Read 7 Sent Done Done false
3048 Acked Write 1 Sent Done Done false
3049 Acked Read 7 Sent Done Done false
3050 Acked Write 6 Sent Done Done false
3051 Acked Write 5 Sent Done Done false
3052 Acked Write 2 Sent Done Done false
3053 Acked Read 5 Sent Done Done false
3054 Acked Read 10 Sent Done Done false
3055 Acked Read 6 Sent Done Done false
3056 Acked Write 6 Sent Done Done false
3057 Acked Read 8 Sent Done Done false
3058 Acked Write 6 Sent Done Done false
3059 Acked Flush 0 Sent Done Done false
3060 Acked Read 5 Sent Done Done false
3061 Acked Write 1 Sent Done Done false
3062 Acked Read 7 Sent Done Done false
3063 Acked Flush 0 Sent Done Done false
3064 Acked Write 1 Sent Done Done false
3065 Acked Read 2 Sent Done Done false
3066 Acked Read 3 Sent Done Done false
3067 Acked Write 10 Sent Done Done false
3068 Acked Read 2 Sent Done Done false
3069 Acked Read 10 Sent Done Done false
3070 Acked Read 2 Sent Done Done false
3071 Acked Read 7 Sent Done Done false
3072 Acked Read 5 Sent Done Done false
3073 Acked Read 4 Sent Done Done false
3074 Acked Write 9 Sent Done Done false
3075 Acked Read 10 Sent Done Done false
3076 Acked Read 5 Sent Done Done false
3077 Acked Write 7 Sent Done Done false
3078 Acked Flush 0 Sent Done Done false
3079 Acked Flush 0 Sent Done Done false
3080 Acked Write 4 Sent Done Done false
3081 Acked Read 3 Sent Done Done false
3082 Acked Write 8 Sent Done Done false
3083 Acked Write 2 Sent Done Done false
3084 Acked Flush 0 Sent Done Done false
3085 Acked Read 3 Sent Done Done false
3086 Acked Read 7 Sent Done Done false
3087 Acked Write 9 Sent Done Done false
3088 Acked Write 4 Sent Done Done false
3089 Acked Write 1 Sent Done Done false
3090 Acked Read 7 Sent Done Done false
3091 Acked Flush 0 Sent Done Done false
3092 Acked Write 10 Sent Done Done false
3093 Acked Flush 0 Sent Done Done false
3094 Acked Read 10 Sent Done Done false
3095 Acked Read 10 Sent Done Done false
3096 Acked Read 7 Sent Done Done false
3097 Acked Read 4 Sent Done Done false
3098 Acked Flush 0 Sent Done Done false
3099 Acked Write 4 Sent Done Done false
3100 Acked Write 5 Sent Done Done false
3101 Acked Read 9 Sent Done Done false
3102 Acked Flush 0 Sent Done Done false
3103 Acked Write 8 Sent Done Done false
3104 Acked Read 5 Sent Done Done false
3105 Acked Write 9 Sent Done Done false
3106 Acked Flush 0 Sent Done Done false
3107 Acked Write 1 Sent Done Done false
3108 Acked Write 3 Sent Done Done false
3109 Acked Write 9 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 1 302
Done 0 301 300 601
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2808)) Some(JobId(3106)) Some(JobId(3106))
Downstairs last five completed: 2808 2807 2806 2805 2804
Upstairs last five completed: 3109 3108 3107 3106 3105
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:02.532590173Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218)
[8810][0] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:03.534236711Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:04.535992493Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:05.538171195Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2809 Acked Write 7 Sent Done Done false
2810 Acked Write 6 Sent Done Done false
2811 Acked Write 1 Sent Done Done false
2812 Acked Read 5 Sent Done Done false
2813 Acked Write 2 Sent Done Done false
2814 Acked Read 1 Sent Done Done false
2815 Acked Read 6 Sent Done Done false
2816 Acked Read 2 Sent Done Done false
2817 Acked Read 8 Sent Done Done false
2818 Acked Read 6 Sent Done Done false
2819 Acked Read 4 Sent Done Done false
2820 Acked Write 2 Sent Done Done false
2821 Acked Read 3 Sent Done Done false
2822 Acked Read 8 Sent Done Done false
2823 Acked Read 6 Sent Done Done false
2824 Acked Write 2 Sent Done Done false
2825 Acked Flush 0 Sent Done Done false
2826 Acked Read 5 Sent Done Done false
2827 Acked Read 3 Sent Done Done false
2828 Acked Read 3 Sent Done Done false
2829 Acked Read 4 Sent Done Done false
2830 Acked Read 6 Sent Done Done false
2831 Acked Read 1 Sent Done Done false
2832 Acked Read 9 Sent Done Done false
2833 Acked Write 9 Sent Done Done false
2834 Acked Read 2 Sent Done Done false
2835 Acked Read 2 Sent Done Done false
2836 Acked Write 8 Sent Done Done false
2837 Acked Read 3 Sent Done Done false
2838 Acked Read 7 Sent Done Done false
2839 Acked Write 4 Sent Done Done false
2840 Acked Write 7 Sent Done Done false
2841 Acked Read 10 Sent Done Done false
2842 Acked Read 4 Sent Done Done false
2843 Acked Write 4 Sent Done Done false
2844 Acked Write 5 Sent Done Done false
2845 Acked Read 9 Sent Done Done false
2846 Acked Read 9 Sent Done Done false
2847 Acked Write 4 Sent Done Done false
2848 Acked Read 9 Sent Done Done false
2849 Acked Read 7 Sent Done Done false
2850 Acked Read 5 Sent Done Done false
2851 Acked Write 8 Sent Done Done false
2852 Acked Flush 0 Sent Done Done false
2853 Acked Write 6 Sent Done Done false
2854 Acked Read 3 Sent Done Done false
2855 Acked Write 1 Sent Done Done false
2856 Acked Read 5 Sent Done Done false
2857 Acked Write 5 Sent Done Done false
2858 Acked Read 10 Sent Done Done false
2859 Acked Write 3 Sent Done Done false
2860 Acked Flush 0 Sent Done Done false
2861 Acked Read 5 Sent Done Done false
2862 Acked Read 6 Sent Done Done false
2863 Acked Read 8 Sent Done Done false
2864 Acked Write 2 Sent Done Done false
2865 Acked Read 2 Sent Done Done false
2866 Acked Flush 0 Sent Done Done false
2867 Acked Flush 0 Sent Done Done false
2868 Acked Write 6 Sent Done Done false
2869 Acked Read 6 Sent Done Done false
2870 Acked Read 5 Sent Done Done false
2871 Acked Read 9 Sent Done Done false
2872 Acked Write 4 Sent Done Done false
2873 Acked Read 6 Sent Done Done false
2874 Acked Write 4 Sent Done Done false
2875 Acked Flush 0 Sent Done Done false
2876 Acked Read 9 Sent Done Done false
2877 Acked Write 2 Sent Done Done false
2878 Acked Read 3 Sent Done Done false
2879 Acked Read 9 Sent Done Done false
2880 Acked Read 7 Sent Done Done false
2881 Acked Read 9 Sent Done Done false
2882 Acked Read 10 Sent Done Done false
2883 Acked Read 1 Sent Done Done false
2884 Acked Write 10 Sent Done Done false
2885 Acked Read 7 Sent Done Done false
2886 Acked Read 3 Sent Done Done false
2887 Acked Write 9 Sent Done Done false
2888 Acked Write 9 Sent Done Done false
2889 Acked Write 7 Sent Done Done false
2890 Acked Write 2 Sent Done Done false
2891 Acked Read 2 Sent Done Done false
2892 Acked Flush 0 Sent Done Done false
2893 Acked Read 1 Sent Done Done false
2894 Acked Write 3 Sent Done Done false
2895 Acked Write 6 Sent Done Done false
2896 Acked Write 2 Sent Done Done false
2897 Acked Write 5 Sent Done Done false
2898 Acked Read 8 Sent Done Done false
2899 Acked Read 9 Sent Done Done false
2900 Acked Read 1 Sent Done Done false
2901 Acked Read 9 Sent Done Done false
2902 Acked Write 2 Sent Done Done false
2903 Acked Write 7 Sent Done Done false
2904 Acked Write 3 Sent Done Done false
2905 Acked Write 2 Sent Done Done false
2906 Acked Read 3 Sent Done Done false
2907 Acked Write 7 Sent Done Done false
2908 Acked Write 2 Sent Done Done false
2909 Acked Read 6 Sent Done Done false
2910 Acked Write 1 Sent Done Done false
2911 Acked Read 8 Sent Done Done false
2912 Acked Read 10 Sent Done Done false
2913 Acked Write 2 Sent Done Done false
2914 Acked Write 9 Sent Done Done false
2915 Acked Read 6 Sent Done Done false
2916 Acked Read 10 Sent Done Done false
2917 Acked Write 10 Sent Done Done false
2918 Acked Write 9 Sent Done Done false
2919 Acked Write 2 Sent Done Done false
2920 Acked Read 8 Sent Done Done false
2921 Acked Flush 0 Sent Done Done false
2922 Acked Write 5 Sent Done Done false
2923 Acked Read 7 Sent Done Done false
2924 Acked Write 1 Sent Done Done false
2925 Acked Write 4 Sent Done Done false
2926 Acked Flush 0 Sent Done Done false
2927 Acked Write 1 Sent Done Done false
2928 Acked Read 6 Sent Done Done false
2929 Acked Read 8 Sent Done Done false
2930 Acked Read 7 Sent Done Done false
2931 Acked Read 3 Sent Done Done false
2932 Acked Write 7 Sent Done Done false
2933 Acked Read 9 Sent Done Done false
2934 Acked Read 3 Sent Done Done false
2935 Acked Write 2 Sent Done Done false
2936 Acked Read 3 Sent Done Done false
2937 Acked Write 8 Sent Done Done false
2938 Acked Write 5 Sent Done Done false
2939 Acked Write 9 Sent Done Done false
2940 Acked Write 10 Sent Done Done false
2941 Acked Read 9 Sent Done Done false
2942 Acked Read 5 Sent Done Done false
2943 Acked Write 8 Sent Done Done false
2944 Acked Write 8 Sent Done Done false
2945 Acked Read 4 Sent Done Done false
2946 Acked Read 5 Sent Done Done false
2947 Acked Read 4 Sent Done Done false
2948 Acked Read 4 Sent Done Done false
2949 Acked Read 6 Sent Done Done false
2950 Acked Read 9 Sent Done Done false
2951 Acked Write 7 Sent Done Done false
2952 Acked Read 4 Sent Done Done false
2953 Acked Read 8 Sent Done Done false
2954 Acked Write 7 Sent Done Done false
2955 Acked Read 4 Sent Done Done false
2956 Acked Write 9 Sent Done Done false
2957 Acked Read 7 Sent Done Done false
2958 Acked Write 3 Sent Done Done false
2959 Acked Read 3 Sent Done Done false
2960 Acked Read 5 Sent Done Done false
2961 Acked Read 5 Sent Done Done false
2962 Acked Write 5 Sent Done Done false
2963 Acked Read 8 Sent Done Done false
2964 Acked Write 3 Sent Done Done false
2965 Acked Read 4 Sent Done Done false
2966 Acked Read 5 Sent Done Done false
2967 Acked Write 2 Sent Done Done false
2968 Acked Read 4 Sent Done Done false
2969 Acked Write 2 Sent Done Done false
2970 Acked Read 6 Sent Done Done false
2971 Acked Write 5 Sent Done Done false
2972 Acked Write 1 Sent Done Done false
2973 Acked Read 3 Sent Done Done false
2974 Acked Read 3 Sent Done Done false
2975 Acked Read 8 Sent Done Done false
2976 Acked Write 10 Sent Done Done false
2977 Acked Read 2 Sent Done Done false
2978 Acked Read 3 Sent Done Done false
2979 Acked Read 5 Sent Done Done false
2980 Acked Write 8 Sent Done Done false
2981 Acked Read 7 Sent Done Done false
2982 Acked Write 8 Sent Done Done false
2983 Acked Write 6 Sent Done Done false
2984 Acked Write 7 Sent Done Done false
2985 Acked Write 9 Sent Done Done false
2986 Acked Read 3 Sent Done Done false
2987 Acked Write 9 Sent Done Done false
2988 Acked Write 5 Sent Done Done false
2989 Acked Read 8 Sent Done Done false
2990 Acked Read 10 Sent Done Done false
2991 Acked Read 9 Sent Done Done false
2992 Acked Read 9 Sent Done Done false
2993 Acked Read 4 Sent Done Done false
2994 Acked Read 5 Sent Done Done false
2995 Acked Read 2 Sent Done Done false
2996 Acked Flush 0 Sent Done Done false
2997 Acked Read 2 Sent Done Done false
2998 Acked Write 3 Sent Done Done false
2999 Acked Flush 0 Sent Done Done false
3000 Acked Write 8 Sent Done Done false
3001 Acked Write 4 Sent Done Done false
3002 Acked Read 8 Sent Done Done false
3003 Acked Read 7 Sent Done Done false
3004 Acked Read 5 Sent Done Done false
3005 Acked Write 8 Sent Done Done false
3006 Acked Flush 0 Sent Done Done false
3007 Acked Read 6 Sent Done Done false
3008 Acked Write 9 Sent Done Done false
3009 Acked Write 7 Sent Done Done false
3010 Acked Write 4 Sent Done Done false
3011 Acked Flush 0 Sent Done Done false
3012 Acked Write 6 Sent Done Done false
3013 Acked Read 6 Sent Done Done false
3014 Acked Read 1 Sent Done Done false
3015 Acked Read 5 Sent Done Done false
3016 Acked Write 9 Sent Done Done false
3017 Acked Read 4 Sent Done Done false
3018 Acked Write 1 Sent Done Done false
3019 Acked Read 4 Sent Done Done false
3020 Acked Read 10 Sent Done Done false
3021 Acked Read 9 Sent Done Done false
3022 Acked Write 4 Sent Done Done false
3023 Acked Flush 0 Sent Done Done false
3024 Acked Write 2 Sent Done Done false
3025 Acked Read 10 Sent Done Done false
3026 Acked Write 2 Sent Done Done false
3027 Acked Read 1 Sent Done Done false
3028 Acked Write 7 Sent Done Done false
3029 Acked Read 1 Sent Done Done false
3030 Acked Read 7 Sent Done Done false
3031 Acked Read 2 Sent Done Done false
3032 Acked Read 9 Sent Done Done false
3033 Acked Flush 0 Sent Done Done false
3034 Acked Read 1 Sent Done Done false
3035 Acked Read 10 Sent Done Done false
3036 Acked Write 5 Sent Done Done false
3037 Acked Write 2 Sent Done Done false
3038 Acked Write 1 Sent Done Done false
3039 Acked Write 2 Sent Done Done false
3040 Acked Write 4 Sent Done Done false
3041 Acked Write 8 Sent Done Done false
3042 Acked Read 1 Sent Done Done false
3043 Acked Write 1 Sent Done Done false
3044 Acked Write 10 Sent Done Done false
3045 Acked Read 7 Sent Done Done false
3046 Acked Read 7 Sent Done Done false
3047 Acked Read 7 Sent Done Done false
3048 Acked Write 1 Sent Done Done false
3049 Acked Read 7 Sent Done Done false
3050 Acked Write 6 Sent Done Done false
3051 Acked Write 5 Sent Done Done false
3052 Acked Write 2 Sent Done Done false
3053 Acked Read 5 Sent Done Done false
3054 Acked Read 10 Sent Done Done false
3055 Acked Read 6 Sent Done Done false
3056 Acked Write 6 Sent Done Done false
3057 Acked Read 8 Sent Done Done false
3058 Acked Write 6 Sent Done Done false
3059 Acked Flush 0 Sent Done Done false
3060 Acked Read 5 Sent Done Done false
3061 Acked Write 1 Sent Done Done false
3062 Acked Read 7 Sent Done Done false
3063 Acked Flush 0 Sent Done Done false
3064 Acked Write 1 Sent Done Done false
3065 Acked Read 2 Sent Done Done false
3066 Acked Read 3 Sent Done Done false
3067 Acked Write 10 Sent Done Done false
3068 Acked Read 2 Sent Done Done false
3069 Acked Read 10 Sent Done Done false
3070 Acked Read 2 Sent Done Done false
3071 Acked Read 7 Sent Done Done false
3072 Acked Read 5 Sent Done Done false
3073 Acked Read 4 Sent Done Done false
3074 Acked Write 9 Sent Done Done false
3075 Acked Read 10 Sent Done Done false
3076 Acked Read 5 Sent Done Done false
3077 Acked Write 7 Sent Done Done false
3078 Acked Flush 0 Sent Done Done false
3079 Acked Flush 0 Sent Done Done false
3080 Acked Write 4 Sent Done Done false
3081 Acked Read 3 Sent Done Done false
3082 Acked Write 8 Sent Done Done false
3083 Acked Write 2 Sent Done Done false
3084 Acked Flush 0 Sent Done Done false
3085 Acked Read 3 Sent Done Done false
3086 Acked Read 7 Sent Done Done false
3087 Acked Write 9 Sent Done Done false
3088 Acked Write 4 Sent Done Done false
3089 Acked Write 1 Sent Done Done false
3090 Acked Read 7 Sent Done Done false
3091 Acked Flush 0 Sent Done Done false
3092 Acked Write 10 Sent Done Done false
3093 Acked Flush 0 Sent Done Done false
3094 Acked Read 10 Sent Done Done false
3095 Acked Read 10 Sent Done Done false
3096 Acked Read 7 Sent Done Done false
3097 Acked Read 4 Sent Done Done false
3098 Acked Flush 0 Sent Done Done false
3099 Acked Write 4 Sent Done Done false
3100 Acked Write 5 Sent Done Done false
3101 Acked Read 9 Sent Done Done false
3102 Acked Flush 0 Sent Done Done false
3103 Acked Write 8 Sent Done Done false
3104 Acked Read 5 Sent Done Done false
3105 Acked Write 9 Sent Done Done false
3106 Acked Flush 0 Sent Done Done false
3107 Acked Write 1 Sent Done Done false
3108 Acked Write 3 Sent Done Done false
3109 Acked Write 9 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(2808)) Some(JobId(3106)) Some(JobId(3106))
Downstairs last five completed: 2808 2807 2806 2805 2804
Upstairs last five completed: 3109 3108 3107 3106 3105
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:06.540367152Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.380417388Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61029)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.380616171Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.381666679Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.382164619Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.382193552Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(2808))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.382212528Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.382228814Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(2808))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.38224498Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(2808))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:07.382439683Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3110 3109 3108 3107 3106
Upstairs last five completed: 3110 3109 3108 3107 3106
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:03:09.632 INFO request completed, latency_us: 364, response_code: 204, uri:[8820] Got stop action so:false kr:false
/stop/cid/1, method: POST, req_id: bd6f3783-e092-4cbe-b044-c18a3eb082b7, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:03:09.632 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/1, method: GET, req_id: f8d365fd-74e8-4553-bc4a-de55e447c6b9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:09.632806448Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:09.632907523Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:09.632935727Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:09.63300586Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:03:13.635 INFO request completed, latency_us: 286, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2156f9fb-29d2-4206-bd74-c7a9e6df4e4f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 4063 len 28672
003/300 Write block 3453 len 28672 data: 3 3 3 3 3 3 3
004/300 Read block 1330 len 20480
005/300 Read block 3262 len 16384
006/300 Write block 466 len 12288 data: 2 2 3
007/300 Write block 4875 len 20480 data: 2 2 2 3 4
008/300 Read block 599 len 24576
009/300 Write block 744 len 32768 data: 2 2 2 2 2 3 3 3
010/300 Read block 4107 len 12288
011/300 Write block 2075 len 20480 data: 2 2 2 2 2
012/300 Read block 3622 len 20480
013/300 Read block 149 len 24576
014/300 Write block 2028 len 8192 data: 3 3
015/300 Read block 1931 len 36864
016/300 Read block 2427 len 24576
017/300 Read block 1134 len 40960
018/300 Read block 238 len 16384
019/300 Read block 651 len 8192
020/300 Read block 1725 len 40960
021/300 Write block 2326 len 4096 data: 3
022/300 Read block 1255 len 36864
023/300 Write block 1043 len 32768 data: 4 4 4 3 2 2 3 3
024/300 Read block 3450 len 20480
025/300 Write block 2389 len 24576 data: 2 2 2 3 3 2
026/300 Read block 4805 len 16384
027/300 Read block 4283 len 4096
028/300 Read block 3764 len 32768
029/300 Read block 4548 len 36864
030/300 Read block 3108 len 40960
031/300 Write block 1683 len 16384 data: 3 3 3 3
032/300 Write block 4262 len 20480 data: 3 3 2 2 2
033/300 Write block 4621 len 16384 data: 3 3 4 4
034/300 Flush
035/300 Write block 4659 len 40960 data: 2 2 2 2 2 2 2 2 2 2
036/300 Write block 3112 len 12288 data: 5 4 5
037/300 Write block 3532 len 20480 data: 3 3 3 2 2
038/300 Write block 1941 len 28672 data: 3 3 3 3 2 3 3
039/300 Read block 3931 len 20480
040/300 Write block 4567 len 8192 data: 3 2
041/300 Write block 1400 len 4096 data: 4
042/300 Write block 894 len 36864 data: 3 2 2 4 4 3 3 3 4
043/300 Write block 4339 len 40960 data: 4 3 2 2 2 2 2 2 2 2
044/300 Write block 4162 len 28672 data: 3 3 3 4 4 4 4
045/300 Read block 1826 len 40960
046/300 Write block 1371 len 16384 data: 2 2 3 3
047/300 Write block 2480 len 32768 data: 2 2 2 3 4 4 4 4
048/300 Read block 94 len 28672
049/300 Read block 4905 len 16384
050/300 Read block 1329 len 24576
051/300 Write block 1721 len 28672 data: 2 2 3 3 3 3 4
052/300 Write block 1310 len 16384 data: 2 2 2 2
053/300 Read block 4090 len 12288
054/300 Read block 2308 len 20480
055/300 Write block 2674 len 28672 data: 3 3 3 2 3 3 3
056/300 Write block 4390 len 28672 data: 5 5 3 3 3 3 3
057/300 Read block 1386 len 12288
058/300 Write block 1138 len 32768 data: 2 2 2 2 3 3 3 3
059/300 Read block 3984 len 20480
060/300 Flush
061/300 Read block 2180 len 8192
062/300 Write block 4225 len 28672 data: 2 3 3 3 3 3 3
063/300 Flush
064/300 Read block 356 len 16384
065/300 Read block 2923 len 40960
066/300 Read block 3558 len 28672
067/300 Read block 3356 len 12288
068/300 Write block 11 len 40960 data: 3 3 2 2 2 2 2 2 2 2
069/300 Read block 1812 len 8192
070/300 Write block 3707 len 24576 data: 2 2 2 2 2 2
071/300 Read block 4896 len 20480
072/300 Flush
073/300 Write block 4374 len 32768 data: 2 3 3 3 3 3 3 3
074/300 Read block 1046 len 40960
075/300 Write block 42 len 32768 data: 4 2 2 2 2 2 2 2
076/300 Write block 523 len 16384 data: 3 3 2 3
077/300 Write block 3810 len 4096 data: 2
078/300 Read block 2037 len 20480
079/300 Write block 3850 len 28672 data: 3 3 4 4 4 3 3
080/300 Read block 1663 len 16384
081/300 Read block 4242 len 12288
082/300 Read block 4269 len 36864
083/300 Read block 2482 len 8192
084/300 Read block 3182 len 16384
085/300 Read block 3455 len 16384
086/300 Flush
087/300 Write block 4229 len 12288 data: 4 4 4
088/300 Flush
089/300 Read block 3291 len 28672
090/300 Read block 527 len 40960
091/300 Read block 2804 len 28672
092/300 Read block 1003 len 24576
093/300 Write block 1633 len 4096 data: 4
094/300 Read block 3655 len 12288
095/300 Read block 3348 len 24576
096/300 Read block 1707 len 28672
097/300 Write block 1019 len 12288 data: 3 3 3
098/300 Read block 406 len 36864
099/300 Read block 4819 len 16384
100/300 Read block 2864 len 24576
101/300 Write block 4944 len 4096 data: 2
102/300 Read block 112 len 8192
103/300 Write block 4589 len 16384 data: 2 3 3 3
104/300 Write block 3489 len 12288 data: 3 2 2
105/300 Read block 2536 len 16384
106/300 Read block 3647 len 40960
107/300 Read block 1723 len 36864
108/300 Read block 2135 len 32768
109/300 Read block 4350 len 40960
110/300 Write block 1483 len 24576 data: 2 2 3 3 3 4
111/300 Read block 178 len 16384
112/300 Write block 2096 len 28672 data: 4 4 4 3 3 3 2
113/300 Read block 1451 len 32768
114/300 Read block 4709 len 12288
115/300 Read block 726 len 4096
116/300 Flush
117/300 Write block 1820 len 12288 data: 2 3 3
118/300 Write block 4402 len 16384 data: 2 2 2 4
119/300 Write block 4116 len 20480 data: 3 3 2 2 2
120/300 Read block 2755 len 12288
121/300 Read block 4630 len 20480
122/300 Write block 2986 len 28672 data: 3 3 3 3 4 4 4
123/300 Write block 2559 len 28672 data: 2 2 2 2 3 3 2
124/300 Read block 1579 len 24576
125/300 Read block 2108 len 32768
126/300 Read block 1134 len 12288
127/300 Read block 1464 len 12288
128/300 Write block 716 len 20480 data: 3 3 3 2 2
129/300 Write block 278 len 12288 data: 5 4 4
130/300 Write block 3417 len 12288 data: 4 3 2
131/300 Write block 4517 len 32768 data: 2 3 3 3 3 2 2 3
132/300 Read block 2360 len 4096
133/300 Read block 943 len 32768
134/300 Read block 3695 len 8192
135/300 Write block 3006 len 24576 data: 2 3 3 3 3 3
136/300 Read block 1982 len 20480
137/300 Read block 917 len 20480
138/300 Read block 1865 len 16384
139/300 Read block 4265 len 36864
140/300 Write block 444 len 32768 data: 7 7 6 7 7 8 8 8
141/300 Read block 3725 len 28672
142/300 Write block 4941 len 32768 data: 2 2 2 3 2 2 2 2
143/300 Read block 3296 len 24576
144/300 Flush
145/300 Read block 412 len 40960
146/300 Read block 4339 len 8192
147/300 Read block 823 len 16384
148/300 Read block 2286 len 28672
149/300 Write block 2662 len 36864 data: 3 3 4 4 4 4 4 4 5
150/300 Write block 1270 len 20480 data: 3 4 4 4 4
151/300 Read block 3340 len 40960
152/300 Write block 4643 len 16384 data: 2 2 2 2
153/300 Read block 3806 len 16384
154/300 Read block 849 len 16384
155/300 Read block 1210 len 8192
156/300 Read block 4980 len 28672
157/300 Flush
158/300 Read block 1961 len 24576
159/300 Write block 4585 len 16384 data: 2 2 2 2
160/300 Read block 3248 len 16384
161/300 Write block 1227 len 16384 data: 2 2 2 2
162/300 Read block 2041 len 8192
163/300 Read block 3227 len 24576
164/300 Read block 1305 len 4096
165/300 Read block 4433 len 32768
166/300 Read block 2243 len 24576
167/300 Write block 126 len 20480 data: 3 4 4 4 4
168/300 Read block 943 len 32768
169/300 Read block 3090 len 24576
170/300 Read block 2821 len 40960
171/300 Read block 4480 len 32768
172/300 Flush
173/300 Read block 3193 len 32768
174/300 Write block 2700 len 40960 data: 2 3 3 3 3 4 4 4 4 4
175/300 Read block 2353 len 28672
176/300 Read block 3823 len 24576
177/300 Write block 1205 len 4096 data: 3
178/300 Read block 4278 len 36864
179/300 Write block 1229 len 12288 data: 3 3 2
180/300 Read block 2277 len 16384
181/300 Write block 4595 len 16384 data: 4 3 3 3
182/300 Write block 4521 len 32768 data: 4 3 3 4 3 3 3 4
183/300 Write block 3809 len 4096 data: 2
184/300 Write block 3685 len 32768 data: 3 3 3 4 2 2 2 2
185/300 Write block 1308 len 24576 data: 3 3 3 3 3 3
186/300 Read block 3049 len 28672
187/300 Read block 2394 len 8192
188/300 Write block 2730 len 32768 data: 4 3 3 3 2 2 2 2
189/300 Write block 2611 len 4096 data: 3
190/300 Read block 2631 len 4096
191/300 Write block 4652 len 16384 data: 4 4 4 4
192/300 Write block 3679 len 20480 data: 2 2 3 3 3
193/300 Write block 4228 len 40960 data: 4 5 5 5 2 2 2 2 2 2
194/300 Flush
195/300 Write block 2400 len 4096 data: 2
196/300 Read block 3305 len 24576
197/300 Read block 3782 len 12288
198/300 Read block 1325 len 32768
199/300 Read block 1356 len 20480
200/300 Write block 1593 len 20480 data: 2 2 2 2 2
201/300 Read block 4178 len 12288
202/300 Read block 1045 len 28672
203/300 Read block 1490 len 36864
204/300 Read block 2627 len 16384
205/300 Write block 4332 len 24576 data: 2 3 3 3 4 4
206/300 Flush
207/300 Write block 701 len 40960 data: 4 4 4 4 3 3 3 3 4 4
208/300 Read block 3811 len 16384
209/300 Write block 4617 len 16384 data: 2 2 2 2
210/300 Write block 3782 len 8192 data: 5 5
211/300 Write block 469 len 16384 data: 4 5 3 4
212/300 Write block 3925 len 4096 data: 3
213/300 Flush
214/300 Read block 3743 len 8192
215/300 Flush
216/300 Flush
217/300 Write block 2713 len 32768 data: 5 4 4 4 4 3 2 3
218/300 Write block 3405 len 24576 data: 2 2 2 2 2 3
219/300 Write block 359 len 4096 data: 3
220/300 Read block 1108 len 32768
221/300 Read block 3409 len 8192
222/300 Read block 3256 len 32768
223/300 Read block 4696 len 16384
224/300 Flush
225/300 Flush
226/300 Flush
227/300 Write block 520 len 24576 data: 3 4 4 4 4 3
228/300 Read block 4764 len 36864
229/300 Read block 1940 len 20480
230/300 Write block 2051 len 12288 data: 3 3 3
231/300 Write block 3940 len 20480 data: 6 5 4 3 3
232/300 Flush
233/300 Read block 1278 len 40960
234/300 Flush
235/300 Write block 292 len 4096 data: 3
236/300 Write block 2164 len 28672 data: 2 2 2 2 3 2 2
237/300 Read block 4607 len 36864
238/300 Read block 948 len 16384
239/300 Write block 2786 len 20480 data: 4 4 3 4 3
240/300 Read block 4387 len 8192
241/300 Read block 49 len 36864
242/300 Read block 1277 len 32768
243/300 Read block 4853 len 20480
244/300 Read block 3347 len 36864
245/300 Read block 3717 len 4096
246/300 Read block 322 len 32768
247/300 Write block 1877 len 20480 data: 3 3 3 3 3
248/300 Write block 3276 len 16384 data: 2 2 2 2
249/300 Write block 1027 len 20480 data: 3 2 2 2 2
250/300 Flush
251/300 Write block 3303 len 24576 data: 3 3 3 4 4 4
252/300 Read block 897 len 8192
253/300 Write block 1289 len 28672 data: 2 2 3 3 3 3 2
254/300 Read block 4910 len 36864
255/300 Read block 2978 len 36864
256/300 Write block 1735 len 36864 data: 2 2 2 2 3 3 3 2 2
257/300 Read block 3813 len 40960
258/300 Read block 4038 len 12288
259/300 Write block 529 len 4096 data: 3
260/300 Read block 4706 len 28672
261/300 Read block 1859 len 24576
262/300 Write block 3305 len 36864 data: 4 5 5 5 4 3 3 4 4
263/300 Flush
264/300 Write block 1381 len 28672 data: 2 2 3 3 3 3 3
265/300 Read block 861 len 28672
266/300 Write block 1566 len 4096 data: 2
267/300 Flush
268/300 Write block 2702 len 20480 data: 4 4 4 5 5
269/300 Write block 1080 len 4096 data: 5
270/300 Read block 409 len 36864
271/300 Flush
272/300 Flush
273/300 Flush
274/300 Read block 1189 len 20480
275/300 Read block 3086 len 4096
276/300 Write block 3320 len 20480 data: 2 2 3 3 2
277/300 Write block 2882 len 32768 data: 2 2 2 3 3 3 3 3
278/300 Flush
279/300 Read block 1292 len 12288
280/300 Flush
281/300 Read block 3212 len 4096
282/300 Read block 2036 len 16384
283/300 Write block 3034 len 36864 data: 3 2 2 2 2 2 2 2 2
284/300 Read block 758 len 16384
285/300 Write block 1707 len 32768 data: 4 4 4 4 4 4 4 4
286/300 Write block 162 len 12288 data: 4 3 3
287/300 Write block 997 len 24576 data: 2 2 2 3 4 4
288/300 Read block 4746 len 4096
289/300 Write block 4514 len 16384 data: 2 2 2 3
290/300 Flush
291/300 Write block 3201 len 36864 data: 2 2 2 2 3 3 3 2 2
292/300 Write block 3274 len 8192 data: 2 2
293/300 Read block 2993 len 24576
294/300 Read block 3021 len 32768
295/300 Write block 419 len 4096 data: 5
296/300 Read block 1659 len 36864
297/300 Write block 768 len 40960 data: 2 3 3 2 3 2 2 2 2 2
298/300 Read block 1148 len 12288
299/300 Flush
300/300 Flush
301/300 Read block 3405 len 16384
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:03:13.877 INFO request completed, latency_us: 181, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6fa29bd4-578f-4fbc-a27e-a87dd001b953, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3111 Acked Flush 0 Done Sent Done false
3112 Acked Read 7 Done Sent Done false
3113 Acked Write 7 Done Sent Done false
3114 Acked Read 5 Done Sent Done false
3115 Acked Read 4 Done Sent Done false
3116 Acked Write 3 Done Sent Done false
3117 Acked Write 5 Done Sent Done false
3118 Acked Read 6 Done Sent Done false
3119 Acked Write 8 Done Sent Done false
3120 Acked Read 3 Done Sent Done false
3121 Acked Write 5 Done Sent Done false
3122 Acked Read 5 Done Sent Done false
3123 Acked Read 6 Done Sent Done false
3124 Acked Write 2 Done Sent Done false
3125 Acked Read 9 Done Sent Done false
3126 Acked Read 6 Done Sent Done false
3127 Acked Read 10 Done Sent Done false
3128 Acked Read 4 Done Sent Done false
3129 Acked Read 2 Done Sent Done false
3130 Acked Read 10 Done Sent Done false
3131 Acked Write 1 Done Sent Done false
3132 Acked Read 9 Done Sent Done false
3133 Acked Write 8 Done Sent Done false
3134 Acked Read 5 Done Sent Done false
3135 Acked Write 6 Done Sent Done false
3136 Acked Read 4 Done Sent Done false
3137 Acked Read 1 Done Sent Done false
3138 Acked Read 8 Done Sent Done false
3139 Acked Read 9 Done Sent Done false
3140 Acked Read 10 Done Sent Done false
3141 Acked Write 4 Done Sent Done false
3142 Acked Write 5 Done Sent Done false
3143 Acked Write 4 Done Sent Done false
3144 Acked Flush 0 Done Sent Done false
3145 Acked Write 10 Done Sent Done false
3146 Acked Write 3 Done Sent Done false
3147 Acked Write 5 Done Sent Done false
3148 Acked Write 7 Done Sent Done false
3149 Acked Read 5 Done Sent Done false
3150 Acked Write 2 Done Sent Done false
3151 Acked Write 1 Done Sent Done false
3152 Acked Write 9 Done Sent Done false
3153 Acked Write 10 Done Sent Done false
3154 Acked Write 7 Done Sent Done false
3155 Acked Read 10 Done Sent Done false
3156 Acked Write 4 Done Sent Done false
3157 Acked Write 8 Done Sent Done false
3158 Acked Read 7 Done Sent Done false
3159 Acked Read 4 Done Sent Done false
3160 Acked Read 6 Done Sent Done false
3161 Acked Write 7 Done Sent Done false
3162 Acked Write 4 Done Sent Done false
3163 Acked Read 3 Done Sent Done false
3164 Acked Read 5 Done Sent Done false
3165 Acked Write 7 Done Sent Done false
3166 Acked Write 7 Done Sent Done false
3167 Acked Read 3 Done Sent Done false
3168 Acked Write 8 Done Sent Done false
3169 Acked Read 5 Done Sent Done false
3170 Acked Flush 0 Done Sent Done false
3171 Acked Read 2 Done Sent Done false
3172 Acked Write 7 Done Sent Done false
3173 Acked Flush 0 Done Sent Done false
3174 Acked Read 4 Done Sent Done false
3175 Acked Read 10 Done Sent Done false
3176 Acked Read 7 Done Sent Done false
3177 Acked Read 3 Done Sent Done false
3178 Acked Write 10 Done Sent Done false
3179 Acked Read 2 Done Sent Done false
3180 Acked Write 6 Done Sent Done false
3181 Acked Read 5 Done Sent Done false
3182 Acked Flush 0 Done Sent Done false
3183 Acked Write 8 Done Sent Done false
3184 Acked Read 10 Done Sent Done false
3185 Acked Write 8 Done Sent Done false
3186 Acked Write 4 Done Sent Done false
3187 Acked Write 1 Done Sent Done false
3188 Acked Read 5 Done Sent Done false
3189 Acked Write 7 Done Sent Done false
3190 Acked Read 4 Done Sent Done false
3191 Acked Read 3 Done Sent Done false
3192 Acked Read 9 Done Sent Done false
3193 Acked Read 2 Done Sent Done false
3194 Acked Read 4 Done Sent Done false
3195 Acked Read 4 Done Sent Done false
3196 Acked Flush 0 Done Sent Done false
3197 Acked Write 3 Done Sent Done false
3198 Acked Flush 0 Done Sent Done false
3199 Acked Read 7 Done Sent Done false
3200 Acked Read 10 Done Sent Done false
3201 Acked Read 7 Done Sent Done false
3202 Acked Read 6 Done Sent Done false
3203 Acked Write 1 Done Sent Done false
3204 Acked Read 3 Done Sent Done false
3205 Acked Read 6 Done Sent Done false
3206 Acked Read 7 Done Sent Done false
3207 Acked Write 3 Done Sent Done false
3208 Acked Read 9 Done Sent Done false
3209 Acked Read 4 Done Sent Done false
3210 Acked Read 6 Done Sent Done false
3211 Acked Write 1 Done Sent Done false
3212 Acked Read 2 Done Sent Done false
3213 Acked Write 4 Done Sent Done false
3214 Acked Write 3 Done Sent Done false
3215 Acked Read 4 Done Sent Done false
3216 Acked Read 10 Done Sent Done false
3217 Acked Read 9 Done Sent Done false
3218 Acked Read 8 Done Sent Done false
3219 Acked Read 10 Done Sent Done false
3220 Acked Write 6 Done Sent Done false
3221 Acked Read 4 Done Sent Done false
3222 Acked Write 7 Done Sent Done false
3223 Acked Read 8 Done Sent Done false
3224 Acked Read 3 Done Sent Done false
3225 Acked Read 1 Done Sent Done false
3226 Acked Flush 0 Done Sent Done false
3227 Acked Write 3 Done Sent Done false
3228 Acked Write 4 Done Sent Done false
3229 Acked Write 5 Done Sent Done false
3230 Acked Read 3 Done Sent Done false
3231 Acked Read 5 Done Sent Done false
3232 Acked Write 7 Done Sent Done false
3233 Acked Write 7 Done Sent Done false
3234 Acked Read 6 Done Sent Done false
3235 Acked Read 8 Done Sent Done false
3236 Acked Read 3 Done Sent Done false
3237 Acked Read 3 Done Sent Done false
3238 Acked Write 5 Done Sent Done false
3239 Acked Write 3 Done Sent Done false
3240 Acked Write 3 Done Sent Done false
3241 Acked Write 8 Done Sent Done false
3242 Acked Read 1 Done Sent Done false
3243 Acked Read 8 Done Sent Done false
3244 Acked Read 2 Done Sent Done false
3245 Acked Write 6 Done Sent Done false
3246 Acked Read 5 Done Sent Done false
3247 Acked Read 5 Done Sent Done false
3248 Acked Read 4 Done Sent Done false
3249 Acked Read 9 Done Sent Done false
3250 Acked Write 8 Done Sent Done false
3251 Acked Read 7 Done Sent Done false
3252 Acked Write 8 Done Sent Done false
3253 Acked Read 6 Done Sent Done false
3254 Acked Flush 0 Done Sent Done false
3255 Acked Read 10 Done Sent Done false
3256 Acked Read 2 Done Sent Done false
3257 Acked Read 4 Done Sent Done false
3258 Acked Read 7 Done Sent Done false
3259 Acked Write 9 Done Sent Done false
3260 Acked Write 5 Done Sent Done false
3261 Acked Read 10 Done Sent Done false
3262 Acked Write 4 Done Sent Done false
3263 Acked Read 4 Done Sent Done false
3264 Acked Read 4 Done Sent Done false
3265 Acked Read 2 Done Sent Done false
3266 Acked Read 7 Done Sent Done false
3267 Acked Flush 0 Done Sent Done false
3268 Acked Read 6 Done Sent Done false
3269 Acked Write 4 Done Sent Done false
3270 Acked Read 4 Done Sent Done false
3271 Acked Write 4 Done Sent Done false
3272 Acked Read 2 Done Sent Done false
3273 Acked Read 6 Done Sent Done false
3274 Acked Read 1 Done Sent Done false
3275 Acked Read 8 Done Sent Done false
3276 Acked Read 6 Done Sent Done false
3277 Acked Write 5 Done Sent Done false
3278 Acked Read 8 Done Sent Done false
3279 Acked Read 6 Done Sent Done false
3280 Acked Read 10 Done Sent Done false
3281 Acked Read 8 Done Sent Done false
3282 Acked Flush 0 Done Sent Done false
3283 Acked Read 8 Done Sent Done false
3284 Acked Write 10 Done Sent Done false
3285 Acked Read 7 Done Sent Done false
3286 Acked Read 6 Done Sent Done false
3287 Acked Write 1 Done Sent Done false
3288 Acked Read 9 Done Sent Done false
3289 Acked Write 3 Done Sent Done false
3290 Acked Read 4 Done Sent Done false
3291 Acked Write 4 Done Sent Done false
3292 Acked Write 8 Done Sent Done false
3293 Acked Write 1 Done Sent Done false
3294 Acked Write 8 Done Sent Done false
3295 Acked Write 6 Done Sent Done false
3296 Acked Read 7 Done Sent Done false
3297 Acked Read 2 Done Sent Done false
3298 Acked Write 8 Done Sent Done false
3299 Acked Write 1 Done Sent Done false
3300 Acked Read 1 Done Sent Done false
3301 Acked Write 4 Done Sent Done false
3302 Acked Write 5 Done Sent Done false
3303 Acked Write 10 Done Sent Done false
3304 Acked Flush 0 Done Sent Done false
3305 Acked Write 1 Done Sent Done false
3306 Acked Read 6 Done Sent Done false
3307 Acked Read 3 Done Sent Done false
3308 Acked Read 8 Done Sent Done false
3309 Acked Read 5 Done Sent Done false
3310 Acked Write 5 Done Sent Done false
3311 Acked Read 3 Done Sent Done false
3312 Acked Read 7 Done Sent Done false
3313 Acked Read 9 Done Sent Done false
3314 Acked Read 4 Done Sent Done false
3315 Acked Write 6 Done Sent Done false
3316 Acked Flush 0 Done Sent Done false
3317 Acked Write 10 Done Sent Done false
3318 Acked Read 4 Done Sent Done false
3319 Acked Write 4 Done Sent Done false
3320 Acked Write 2 Done Sent Done false
3321 Acked Write 4 Done Sent Done false
3322 Acked Write 1 Done Sent Done false
3323 Acked Flush 0 Done Sent Done false
3324 Acked Read 2 Done Sent Done false
3325 Acked Flush 0 Done Sent Done false
3326 Acked Flush 0 Done Sent Done false
3327 Acked Write 8 Done Sent Done false
3328 Acked Write 6 Done Sent Done false
3329 Acked Write 1 Done Sent Done false
3330 Acked Read 8 Done Sent Done false
3331 Acked Read 2 Done Sent Done false
3332 Acked Read 8 Done Sent Done false
3333 Acked Read 4 Done Sent Done false
3334 Acked Flush 0 Done Sent Done false
3335 Acked Flush 0 Done Sent Done false
3336 Acked Flush 0 Done Sent Done false
3337 Acked Write 6 Done Sent Done false
3338 Acked Read 9 Done Sent Done false
3339 Acked Read 5 Done Sent Done false
3340 Acked Write 3 Done Sent Done false
3341 Acked Write 5 Done Sent Done false
3342 Acked Flush 0 Done Sent Done false
3343 Acked Read 10 Done Sent Done false
3344 Acked Flush 0 Done Sent Done false
3345 Acked Write 1 Done Sent Done false
3346 Acked Write 7 Done Sent Done false
3347 Acked Read 9 Done Sent Done false
3348 Acked Read 4 Done Sent Done false
3349 Acked Write 5 Done Sent Done false
3350 Acked Read 2 Done Sent Done false
3351 Acked Read 9 Done Sent Done false
3352 Acked Read 8 Done Sent Done false
3353 Acked Read 5 Done Sent Done false
3354 Acked Read 9 Done Sent Done false
3355 Acked Read 1 Done Sent Done false
3356 Acked Read 8 Done Sent Done false
3357 Acked Write 5 Done Sent Done false
3358 Acked Write 4 Done Sent Done false
3359 Acked Write 5 Done Sent Done false
3360 Acked Flush 0 Done Sent Done false
3361 Acked Write 6 Done Sent Done false
3362 Acked Read 2 Done Sent Done false
3363 Acked Write 7 Done Sent Done false
3364 Acked Read 9 Done Sent Done false
3365 Acked Read 9 Done Sent Done false
3366 Acked Write 9 Done Sent Done false
3367 Acked Read 10 Done Sent Done false
3368 Acked Read 3 Done Sent Done false
3369 Acked Write 1 Done Sent Done false
3370 Acked Read 7 Done Sent Done false
3371 Acked Read 6 Done Sent Done false
3372 Acked Write 9 Done Sent Done false
3373 Acked Flush 0 Done Sent Done false
3374 Acked Write 7 Done Sent Done false
3375 Acked Read 7 Done Sent Done false
3376 Acked Write 1 Done Sent Done false
3377 Acked Flush 0 Done Sent Done false
3378 Acked Write 5 Done Sent Done false
3379 Acked Write 1 Done Sent Done false
3380 Acked Read 9 Done Sent Done false
3381 Acked Flush 0 Done Sent Done false
3382 Acked Flush 0 Done Sent Done false
3383 Acked Flush 0 Done Sent Done false
3384 Acked Read 5 Done Sent Done false
3385 Acked Read 1 Done Sent Done false
3386 Acked Write 5 Done Sent Done false
3387 Acked Write 8 Done Sent Done false
3388 Acked Flush 0 Done Sent Done false
3389 Acked Read 3 Done Sent Done false
3390 Acked Flush 0 Done Sent Done false
3391 Acked Read 1 Done Sent Done false
3392 Acked Read 4 Done Sent Done false
3393 Acked Write 9 Done Sent Done false
3394 Acked Read 4 Done Sent Done false
3395 Acked Write 8 Done Sent Done false
3396 Acked Write 3 Done Sent Done false
3397 Acked Write 6 Done Sent Done false
3398 Acked Read 1 Done Sent Done false
3399 Acked Write 4 Done Sent Done false
3400 Acked Flush 0 Done Sent Done false
3401 Acked Write 9 Done Sent Done false
3402 Acked Write 2 Done Sent Done false
3403 Acked Read 6 Done Sent Done false
3404 Acked Read 8 Done Sent Done false
3405 Acked Write 1 Done Sent Done false
3406 Acked Read 9 Done Sent Done false
3407 Acked Write 10 Done Sent Done false
3408 Acked Read 3 Done Sent Done false
3409 Acked Flush 0 Done Sent Done false
3410 Acked Flush 0 Done Sent Done false
3411 Acked Read 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3410)) Some(JobId(3110)) Some(JobId(3410))
Downstairs last five completed: 3110 3109 3108 3107 3106
Upstairs last five completed: 3411 3410 3409 3408 3407
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:14.55659332Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:15.557770141Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:16.559161692Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3111 Acked Flush 0 Done Sent Done false
3112 Acked Read 7 Done Sent Done false
3113 Acked Write 7 Done Sent Done false
3114 Acked Read 5 Done Sent Done false
3115 Acked Read 4 Done Sent Done false
3116 Acked Write 3 Done Sent Done false
3117 Acked Write 5 Done Sent Done false
3118 Acked Read 6 Done Sent Done false
3119 Acked Write 8 Done Sent Done false
3120 Acked Read 3 Done Sent Done false
3121 Acked Write 5 Done Sent Done false
3122 Acked Read 5 Done Sent Done false
3123 Acked Read 6 Done Sent Done false
3124 Acked Write 2 Done Sent Done false
3125 Acked Read 9 Done Sent Done false
3126 Acked Read 6 Done Sent Done false
3127 Acked Read 10 Done Sent Done false
3128 Acked Read 4 Done Sent Done false
3129 Acked Read 2 Done Sent Done false
3130 Acked Read 10 Done Sent Done false
3131 Acked Write 1 Done Sent Done false
3132 Acked Read 9 Done Sent Done false
3133 Acked Write 8 Done Sent Done false
3134 Acked Read 5 Done Sent Done false
3135 Acked Write 6 Done Sent Done false
3136 Acked Read 4 Done Sent Done false
3137 Acked Read 1 Done Sent Done false
3138 Acked Read 8 Done Sent Done false
3139 Acked Read 9 Done Sent Done false
3140 Acked Read 10 Done Sent Done false
3141 Acked Write 4 Done Sent Done false
3142 Acked Write 5 Done Sent Done false
3143 Acked Write 4 Done Sent Done false
3144 Acked Flush 0 Done Sent Done false
3145 Acked Write 10 Done Sent Done false
3146 Acked Write 3 Done Sent Done false
3147 Acked Write 5 Done Sent Done false
3148 Acked Write 7 Done Sent Done false
3149 Acked Read 5 Done Sent Done false
3150 Acked Write 2 Done Sent Done false
3151 Acked Write 1 Done Sent Done false
3152 Acked Write 9 Done Sent Done false
3153 Acked Write 10 Done Sent Done false
3154 Acked Write 7 Done Sent Done false
3155 Acked Read 10 Done Sent Done false
3156 Acked Write 4 Done Sent Done false
3157 Acked Write 8 Done Sent Done false
3158 Acked Read 7 Done Sent Done false
3159 Acked Read 4 Done Sent Done false
3160 Acked Read 6 Done Sent Done false
3161 Acked Write 7 Done Sent Done false
3162 Acked Write 4 Done Sent Done false
3163 Acked Read 3 Done Sent Done false
3164 Acked Read 5 Done Sent Done false
3165 Acked Write 7 Done Sent Done false
3166 Acked Write 7 Done Sent Done false
3167 Acked Read 3 Done Sent Done false
3168 Acked Write 8 Done Sent Done false
3169 Acked Read 5 Done Sent Done false
3170 Acked Flush 0 Done Sent Done false
3171 Acked Read 2 Done Sent Done false
3172 Acked Write 7 Done Sent Done false
3173 Acked Flush 0 Done Sent Done false
3174 Acked Read 4 Done Sent Done false
3175 Acked Read 10 Done Sent Done false
3176 Acked Read 7 Done Sent Done false
3177 Acked Read 3 Done Sent Done false
3178 Acked Write 10 Done Sent Done false
3179 Acked Read 2 Done Sent Done false
3180 Acked Write 6 Done Sent Done false
3181 Acked Read 5 Done Sent Done false
3182 Acked Flush 0 Done Sent Done false
3183 Acked Write 8 Done Sent Done false
3184 Acked Read 10 Done Sent Done false
3185 Acked Write 8 Done Sent Done false
3186 Acked Write 4 Done Sent Done false
3187 Acked Write 1 Done Sent Done false
3188 Acked Read 5 Done Sent Done false
3189 Acked Write 7 Done Sent Done false
3190 Acked Read 4 Done Sent Done false
3191 Acked Read 3 Done Sent Done false
3192 Acked Read 9 Done Sent Done false
3193 Acked Read 2 Done Sent Done false
3194 Acked Read 4 Done Sent Done false
3195 Acked Read 4 Done Sent Done false
3196 Acked Flush 0 Done Sent Done false
3197 Acked Write 3 Done Sent Done false
3198 Acked Flush 0 Done Sent Done false
3199 Acked Read 7 Done Sent Done false
3200 Acked Read 10 Done Sent Done false
3201 Acked Read 7 Done Sent Done false
3202 Acked Read 6 Done Sent Done false
3203 Acked Write 1 Done Sent Done false
3204 Acked Read 3 Done Sent Done false
3205 Acked Read 6 Done Sent Done false
3206 Acked Read 7 Done Sent Done false
3207 Acked Write 3 Done Sent Done false
3208 Acked Read 9 Done Sent Done false
3209 Acked Read 4 Done Sent Done false
3210 Acked Read 6 Done Sent Done false
3211 Acked Write 1 Done Sent Done false
3212 Acked Read 2 Done Sent Done false
3213 Acked Write 4 Done Sent Done false
3214 Acked Write 3 Done Sent Done false
3215 Acked Read 4 Done Sent Done false
3216 Acked Read 10 Done Sent Done false
3217 Acked Read 9 Done Sent Done false
3218 Acked Read 8 Done Sent Done false
3219 Acked Read 10 Done Sent Done false
3220 Acked Write 6 Done Sent Done false
3221 Acked Read 4 Done Sent Done false
3222 Acked Write 7 Done Sent Done false
3223 Acked Read 8 Done Sent Done false
3224 Acked Read 3 Done Sent Done false
3225 Acked Read 1 Done Sent Done false
3226 Acked Flush 0 Done Sent Done false
3227 Acked Write 3 Done Sent Done false
3228 Acked Write 4 Done Sent Done false
3229 Acked Write 5 Done Sent Done false
3230 Acked Read 3 Done Sent Done false
3231 Acked Read 5 Done Sent Done false
3232 Acked Write 7 Done Sent Done false
3233 Acked Write 7 Done Sent Done false
3234 Acked Read 6 Done Sent Done false
3235 Acked Read 8 Done Sent Done false
3236 Acked Read 3 Done Sent Done false
3237 Acked Read 3 Done Sent Done false
3238 Acked Write 5 Done Sent Done false
3239 Acked Write 3 Done Sent Done false
3240 Acked Write 3 Done Sent Done false
3241 Acked Write 8 Done Sent Done false
3242 Acked Read 1 Done Sent Done false
3243 Acked Read 8 Done Sent Done false
3244 Acked Read 2 Done Sent Done false
3245 Acked Write 6 Done Sent Done false
3246 Acked Read 5 Done Sent Done false
3247 Acked Read 5 Done Sent Done false
3248 Acked Read 4 Done Sent Done false
3249 Acked Read 9 Done Sent Done false
3250 Acked Write 8 Done Sent Done false
3251 Acked Read 7 Done Sent Done false
3252 Acked Write 8 Done Sent Done false
3253 Acked Read 6 Done Sent Done false
3254 Acked Flush 0 Done Sent Done false
3255 Acked Read 10 Done Sent Done false
3256 Acked Read 2 Done Sent Done false
3257 Acked Read 4 Done Sent Done false
3258 Acked Read 7 Done Sent Done false
3259 Acked Write 9 Done Sent Done false
3260 Acked Write 5 Done Sent Done false
3261 Acked Read 10 Done Sent Done false
3262 Acked Write 4 Done Sent Done false
3263 Acked Read 4 Done Sent Done false
3264 Acked Read 4 Done Sent Done false
3265 Acked Read 2 Done Sent Done false
3266 Acked Read 7 Done Sent Done false
3267 Acked Flush 0 Done Sent Done false
3268 Acked Read 6 Done Sent Done false
3269 Acked Write 4 Done Sent Done false
3270 Acked Read 4 Done Sent Done false
3271 Acked Write 4 Done Sent Done false
3272 Acked Read 2 Done Sent Done false
3273 Acked Read 6 Done Sent Done false
3274 Acked Read 1 Done Sent Done false
3275 Acked Read 8 Done Sent Done false
3276 Acked Read 6 Done Sent Done false
3277 Acked Write 5 Done Sent Done false
3278 Acked Read 8 Done Sent Done false
3279 Acked Read 6 Done Sent Done false
3280 Acked Read 10 Done Sent Done false
3281 Acked Read 8 Done Sent Done false
3282 Acked Flush 0 Done Sent Done false
3283 Acked Read 8 Done Sent Done false
3284 Acked Write 10 Done Sent Done false
3285 Acked Read 7 Done Sent Done false
3286 Acked Read 6 Done Sent Done false
3287 Acked Write 1 Done Sent Done false
3288 Acked Read 9 Done Sent Done false
3289 Acked Write 3 Done Sent Done false
3290 Acked Read 4 Done Sent Done false
3291 Acked Write 4 Done Sent Done false
3292 Acked Write 8 Done Sent Done false
3293 Acked Write 1 Done Sent Done false
3294 Acked Write 8 Done Sent Done false
3295 Acked Write 6 Done Sent Done false
3296 Acked Read 7 Done Sent Done false
3297 Acked Read 2 Done Sent Done false
3298 Acked Write 8 Done Sent Done false
3299 Acked Write 1 Done Sent Done false
3300 Acked Read 1 Done Sent Done false
3301 Acked Write 4 Done Sent Done false
3302 Acked Write 5 Done Sent Done false
3303 Acked Write 10 Done Sent Done false
3304 Acked Flush 0 Done Sent Done false
3305 Acked Write 1 Done Sent Done false
3306 Acked Read 6 Done Sent Done false
3307 Acked Read 3 Done Sent Done false
3308 Acked Read 8 Done Sent Done false
3309 Acked Read 5 Done Sent Done false
3310 Acked Write 5 Done Sent Done false
3311 Acked Read 3 Done Sent Done false
3312 Acked Read 7 Done Sent Done false
3313 Acked Read 9 Done Sent Done false
3314 Acked Read 4 Done Sent Done false
3315 Acked Write 6 Done Sent Done false
3316 Acked Flush 0 Done Sent Done false
3317 Acked Write 10 Done Sent Done false
3318 Acked Read 4 Done Sent Done false
3319 Acked Write 4 Done Sent Done false
3320 Acked Write 2 Done Sent Done false
3321 Acked Write 4 Done Sent Done false
3322 Acked Write 1 Done Sent Done false
3323 Acked Flush 0 Done Sent Done false
3324 Acked Read 2 Done Sent Done false
3325 Acked Flush 0 Done Sent Done false
3326 Acked Flush 0 Done Sent Done false
3327 Acked Write 8 Done Sent Done false
3328 Acked Write 6 Done Sent Done false
3329 Acked Write 1 Done Sent Done false
3330 Acked Read 8 Done Sent Done false
3331 Acked Read 2 Done Sent Done false
3332 Acked Read 8 Done Sent Done false
3333 Acked Read 4 Done Sent Done false
3334 Acked Flush 0 Done Sent Done false
3335 Acked Flush 0 Done Sent Done false
3336 Acked Flush 0 Done Sent Done false
3337 Acked Write 6 Done Sent Done false
3338 Acked Read 9 Done Sent Done false
3339 Acked Read 5 Done Sent Done false
3340 Acked Write 3 Done Sent Done false
3341 Acked Write 5 Done Sent Done false
3342 Acked Flush 0 Done Sent Done false
3343 Acked Read 10 Done Sent Done false
3344 Acked Flush 0 Done Sent Done false
3345 Acked Write 1 Done Sent Done false
3346 Acked Write 7 Done Sent Done false
3347 Acked Read 9 Done Sent Done false
3348 Acked Read 4 Done Sent Done false
3349 Acked Write 5 Done Sent Done false
3350 Acked Read 2 Done Sent Done false
3351 Acked Read 9 Done Sent Done false
3352 Acked Read 8 Done Sent Done false
3353 Acked Read 5 Done Sent Done false
3354 Acked Read 9 Done Sent Done false
3355 Acked Read 1 Done Sent Done false
3356 Acked Read 8 Done Sent Done false
3357 Acked Write 5 Done Sent Done false
3358 Acked Write 4 Done Sent Done false
3359 Acked Write 5 Done Sent Done false
3360 Acked Flush 0 Done Sent Done false
3361 Acked Write 6 Done Sent Done false
3362 Acked Read 2 Done Sent Done false
3363 Acked Write 7 Done Sent Done false
3364 Acked Read 9 Done Sent Done false
3365 Acked Read 9 Done Sent Done false
3366 Acked Write 9 Done Sent Done false
3367 Acked Read 10 Done Sent Done false
3368 Acked Read 3 Done Sent Done false
3369 Acked Write 1 Done Sent Done false
3370 Acked Read 7 Done Sent Done false
3371 Acked Read 6 Done Sent Done false
3372 Acked Write 9 Done Sent Done false
3373 Acked Flush 0 Done Sent Done false
3374 Acked Write 7 Done Sent Done false
3375 Acked Read 7 Done Sent Done false
3376 Acked Write 1 Done Sent Done false
3377 Acked Flush 0 Done Sent Done false
3378 Acked Write 5 Done Sent Done false
3379 Acked Write 1 Done Sent Done false
3380 Acked Read 9 Done Sent Done false
3381 Acked Flush 0 Done Sent Done false
3382 Acked Flush 0 Done Sent Done false
3383 Acked Flush 0 Done Sent Done false
3384 Acked Read 5 Done Sent Done false
3385 Acked Read 1 Done Sent Done false
3386 Acked Write 5 Done Sent Done false
3387 Acked Write 8 Done Sent Done false
3388 Acked Flush 0 Done Sent Done false
3389 Acked Read 3 Done Sent Done false
3390 Acked Flush 0 Done Sent Done false
3391 Acked Read 1 Done Sent Done false
3392 Acked Read 4 Done Sent Done false
3393 Acked Write 9 Done Sent Done false
3394 Acked Read 4 Done Sent Done false
3395 Acked Write 8 Done Sent Done false
3396 Acked Write 3 Done Sent Done false
3397 Acked Write 6 Done Sent Done false
3398 Acked Read 1 Done Sent Done false
3399 Acked Write 4 Done Sent Done false
3400 Acked Flush 0 Done Sent Done false
3401 Acked Write 9 Done Sent Done false
3402 Acked Write 2 Done Sent Done false
3403 Acked Read 6 Done Sent Done false
3404 Acked Read 8 Done Sent Done false
3405 Acked Write 1 Done Sent Done false
3406 Acked Read 9 Done Sent Done false
3407 Acked Write 10 Done Sent Done false
3408 Acked Read 3 Done Sent Done false
3409 Acked Flush 0 Done Sent Done false
3410 Acked Flush 0 Done Sent Done false
3411 Acked Read 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3410)) Some(JobId(3110)) Some(JobId(3410))
Downstairs last five completed: 3110 3109 3108 3107 3106
Upstairs last five completed: 3411 3410 3409 3408 3407
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:18.563179816Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:19.565284664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.634548835Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52870)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.634753696Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.635743958Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636277811Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636297586Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(3110))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636311313Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636324239Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(3110))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636337566Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(3110))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:19.636536109Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3412 3411 3410 3409 3408
Upstairs last five completed: 3412 3411 3410 3409 3408
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 28 06:03:21.882 INFO request completed, latency_us: 327, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7ccec0b5-38d9-4767-bd99-c6be8b5f76f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:03:21.883 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/2, method: GET, req_id: 32b6a838-f234-4d60-9762-afb6912cd3be, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:21.883040418Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:21.883135226Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:21.883162589Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:21.883250068Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:03:25.884 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: d1360034-b88b-4543-96b2-04a589fdf7c6, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 2672 len 16384 data: 4 3 4 4
002/300 Write block 475 len 8192 data: 4 4
003/300 Flush
004/300 Write block 833 len 20480 data: 3 3 3 2 2
005/300 Read block 4623 len 8192
006/300 Read block 3077 len 16384
007/300 Read block 2965 len 32768
008/300 Write block 2866 len 36864 data: 4 4 4 4 4 4 4 3 3
009/300 Read block 1420 len 36864
010/300 Read block 4666 len 24576
011/300 Flush
012/300 Write block 249 len 24576 data: 2 2 3 3 3 2
013/300 Read block 3272 len 36864
014/300 Read block 261 len 12288
015/300 Write block 1831 len 40960 data: 2 2 2 2 2 2 2 3 4 4
016/300 Read block 457 len 24576
017/300 Read block 684 len 20480
018/300 Write block 3597 len 4096 data: 3
019/300 Write block 3220 len 40960 data: 3 3 3 3 3 3 3 2 2 2
020/300 Read block 1110 len 8192
021/300 Read block 3814 len 4096
022/300 Read block 900 len 16384
023/300 Read block 4655 len 20480
024/300 Flush
025/300 Write block 4815 len 40960 data: 2 2 2 2 2 2 2 2 2 2
026/300 Write block 3218 len 16384 data: 3 3 4 4
027/300 Write block 3061 len 40960 data: 2 2 2 2 2 2 3 2 2 2
028/300 Read block 4837 len 16384
029/300 Read block 1060 len 36864
030/300 Write block 219 len 24576 data: 4 3 4 3 3 3
031/300 Read block 815 len 36864
032/300 Write block 4692 len 8192 data: 2 2
033/300 Read block 72 len 24576
034/300 Read block 118 len 8192
035/300 Read block 717 len 28672
036/300 Write block 1901 len 12288 data: 3 2 2
037/300 Write block 3720 len 36864 data: 5 5 5 5 5 3 2 2 2
038/300 Write block 1440 len 28672 data: 4 5 5 5 5 5 4
039/300 Write block 1322 len 32768 data: 3 3 3 3 3 3 3 4
040/300 Read block 560 len 16384
041/300 Read block 795 len 8192
042/300 Read block 3225 len 8192
043/300 Write block 4887 len 40960 data: 5 6 6 5 5 3 3 3 3 3
044/300 Read block 3822 len 24576
045/300 Write block 2214 len 36864 data: 6 6 6 6 6 4 2 2 2
046/300 Read block 3548 len 28672
047/300 Read block 3651 len 40960
048/300 Read block 3848 len 4096
049/300 Write block 3463 len 20480 data: 2 2 3 3 3
050/300 Read block 1862 len 20480
051/300 Write block 1485 len 24576 data: 4 4 4 5 5 5
052/300 Write block 4212 len 24576 data: 4 4 3 3 3 3
053/300 Read block 1050 len 8192
054/300 Read block 1334 len 32768
055/300 Read block 1769 len 8192
056/300 Flush
057/300 Write block 1420 len 32768 data: 2 2 2 2 2 2 2 2
058/300 Write block 2272 len 36864 data: 2 4 4 4 4 4 5 5 5
059/300 Flush
060/300 Flush
061/300 Write block 3488 len 24576 data: 3 4 3 3 2 2
062/300 Write block 1616 len 36864 data: 4 4 3 3 4 4 2 2 5
063/300 Read block 3637 len 36864
064/300 Read block 1457 len 4096
065/300 Read block 2804 len 16384
066/300 Read block 494 len 12288
067/300 Read block 3467 len 40960
068/300 Read block 4244 len 24576
069/300 Read block 4458 len 20480
070/300 Write block 2950 len 36864 data: 4 4 4 4 4 4 3 4 4
071/300 Read block 1827 len 24576
072/300 Read block 4521 len 32768
073/300 Read block 4175 len 8192
074/300 Write block 4456 len 40960 data: 2 2 2 2 2 3 3 3 3 3
075/300 Read block 2190 len 24576
076/300 Flush
077/300 Read block 275 len 4096
078/300 Write block 3704 len 24576 data: 2 2 2 3 3 3
079/300 Write block 3862 len 12288 data: 2 2 3
080/300 Flush
081/300 Flush
082/300 Read block 4395 len 16384
083/300 Write block 1810 len 24576 data: 2 2 2 2 2 2
084/300 Read block 1174 len 28672
085/300 Read block 2416 len 20480
086/300 Write block 2570 len 16384 data: 4 3 3 3
087/300 Write block 372 len 40960 data: 2 3 3 3 3 4 5 5 6 5
088/300 Read block 116 len 28672
089/300 Write block 4442 len 28672 data: 3 3 3 3 2 2 2
090/300 Write block 2990 len 40960 data: 5 5 5 2 2 2 2 2 2 2
091/300 Read block 3751 len 16384
092/300 Read block 4248 len 12288
093/300 Write block 3049 len 8192 data: 2 2
094/300 Read block 3863 len 4096
095/300 Read block 2327 len 36864
096/300 Read block 1632 len 12288
097/300 Write block 4337 len 40960 data: 5 4 5 4 3 3 3 3 3 3
098/300 Write block 1264 len 4096 data: 5
099/300 Read block 1565 len 4096
100/300 Flush
101/300 Read block 149 len 4096
102/300 Write block 2440 len 36864 data: 3 2 3 3 3 4 4 4 3
103/300 Read block 4442 len 32768
104/300 Read block 2678 len 12288
105/300 Read block 3665 len 20480
106/300 Read block 1065 len 4096
107/300 Flush
108/300 Read block 2575 len 24576
109/300 Read block 2838 len 12288
110/300 Write block 3252 len 28672 data: 3 2 2 2 2 2 2
111/300 Read block 3239 len 36864
112/300 Read block 1303 len 24576
113/300 Read block 4581 len 40960
114/300 Read block 3268 len 20480
115/300 Write block 4600 len 32768 data: 2 2 2 2 2 2 2 2
116/300 Read block 2734 len 24576
117/300 Read block 1974 len 8192
118/300 Flush
119/300 Write block 1169 len 40960 data: 4 3 2 3 3 4 4 5 6 7
120/300 Read block 2198 len 32768
121/300 Read block 3813 len 40960
122/300 Write block 3056 len 4096 data: 3
123/300 Write block 2348 len 8192 data: 2 2
124/300 Read block 2457 len 28672
125/300 Read block 574 len 12288
126/300 Write block 4807 len 8192 data: 2 3
127/300 Write block 242 len 40960 data: 4 3 3 3 3 2 2 3 3 4
128/300 Read block 980 len 12288
129/300 Flush
130/300 Write block 1899 len 20480 data: 3 3 4 3 3
131/300 Read block 4401 len 40960
132/300 Write block 315 len 36864 data: 2 3 3 3 4 4 4 4 4
133/300 Write block 1017 len 40960 data: 3 3 4 4 4 3 2 2 3 3
134/300 Write block 4043 len 20480 data: 3 3 3 3 2
135/300 Read block 1667 len 32768
136/300 Write block 3881 len 12288 data: 3 4 5
137/300 Write block 2790 len 24576 data: 4 3 3 2 3 3
138/300 Read block 2728 len 12288
139/300 Read block 1435 len 12288
140/300 Read block 4309 len 40960
141/300 Write block 4085 len 28672 data: 2 2 2 2 2 2 2
142/300 Read block 4588 len 36864
143/300 Flush
144/300 Write block 3435 len 40960 data: 4 5 5 5 3 2 3 3 3 3
145/300 Read block 4978 len 8192
146/300 Write block 4644 len 8192 data: 3 3
147/300 Write block 4346 len 40960 data: 4 3 3 2 2 2 3 3 3 3
148/300 Write block 4081 len 4096 data: 3
149/300 Write block 2918 len 40960 data: 3 2 2 2 2 3 3 3 3 2
150/300 Read block 302 len 36864
151/300 Write block 4724 len 36864 data: 2 2 2 2 2 3 3 3 3
152/300 Flush
153/300 Read block 2821 len 40960
154/300 Write block 1239 len 16384 data: 2 2 2 2
155/300 Read block 2038 len 24576
156/300 Read block 1102 len 28672
157/300 Write block 3272 len 20480 data: 3 2 3 3 3
158/300 Write block 1297 len 8192 data: 2 2
159/300 Write block 1445 len 4096 data: 6
160/300 Write block 2056 len 32768 data: 2 2 2 2 2 3 3 3
161/300 Write block 364 len 24576 data: 3 3 3 2 2 2
162/300 Write block 3826 len 28672 data: 2 2 2 2 2 2 2
163/300 Read block 3934 len 24576
164/300 Read block 3879 len 24576
165/300 Write block 1437 len 24576 data: 2 2 2 5 6 6
166/300 Read block 1002 len 24576
167/300 Write block 4699 len 4096 data: 4
168/300 Read block 4350 len 20480
169/300 Write block 2200 len 12288 data: 3 3 4
170/300 Read block 1867 len 16384
171/300 Read block 3359 len 8192
172/300 Read block 557 len 40960
173/300 Read block 3203 len 12288
174/300 Write block 1255 len 12288 data: 3 3 3
175/300 Write block 908 len 32768 data: 3 2 3 4 4 4 3 3
176/300 Read block 123 len 24576
177/300 Read block 2183 len 12288
178/300 Write block 4154 len 28672 data: 2 2 2 2 3 4 4
179/300 Write block 75 len 28672 data: 4 4 4 4 4 3 4
180/300 Flush
181/300 Read block 4112 len 16384
182/300 Write block 2298 len 4096 data: 2
183/300 Write block 689 len 16384 data: 2 3 3 3
184/300 Read block 1385 len 40960
185/300 Write block 481 len 28672 data: 3 3 2 2 2 3 2
186/300 Write block 3522 len 36864 data: 4 4 5 5 5 6 6 5 3
187/300 Flush
188/300 Write block 435 len 12288 data: 5 5 4
189/300 Read block 424 len 12288
190/300 Write block 3334 len 24576 data: 2 3 4 4 4 4
191/300 Read block 2578 len 40960
192/300 Write block 1018 len 40960 data: 4 5 5 5 4 3 3 4 4 4
193/300 Read block 3109 len 32768
194/300 Write block 2591 len 12288 data: 3 2 2
195/300 Write block 3407 len 8192 data: 3 3
196/300 Read block 3570 len 32768
197/300 Read block 3859 len 12288
198/300 Flush
199/300 Write block 2131 len 20480 data: 3 3 3 3 2
200/300 Read block 1467 len 4096
201/300 Write block 1969 len 36864 data: 3 3 3 3 3 3 3 2 2
202/300 Write block 350 len 28672 data: 4 3 3 3 2 2 2
203/300 Read block 3647 len 24576
204/300 Read block 4963 len 4096
205/300 Read block 4838 len 8192
206/300 Write block 4642 len 32768 data: 2 3 4 4 3 5 4 3
207/300 Read block 3430 len 40960
208/300 Write block 702 len 24576 data: 5 5 5 4 4 4
209/300 Read block 2529 len 4096
210/300 Flush
211/300 Write block 2841 len 4096 data: 2
212/300 Read block 4824 len 8192
213/300 Read block 90 len 4096
214/300 Write block 4520 len 32768 data: 4 5 4 4 5 4 4 4
215/300 Read block 525 len 20480
216/300 Write block 2176 len 16384 data: 2 2 3 5
217/300 Write block 4524 len 32768 data: 6 5 5 5 5 5 5 3
218/300 Read block 2244 len 4096
219/300 Read block 4498 len 36864
220/300 Flush
221/300 Read block 255 len 20480
222/300 Read block 3575 len 28672
223/300 Read block 4043 len 4096
224/300 Write block 3903 len 12288 data: 4 4 4
225/300 Write block 2570 len 20480 data: 5 4 4 4 2
226/300 Write block 1326 len 24576 data: 4 4 4 5 4 4
227/300 Read block 4236 len 28672
228/300 Read block 2722 len 32768
229/300 Write block 404 len 20480 data: 3 3 3 3 3
230/300 Write block 3756 len 32768 data: 2 2 2 2 3 3 3 2
231/300 Read block 2700 len 24576
232/300 Read block 2986 len 28672
233/300 Read block 1521 len 40960
234/300 Write block 4111 len 36864 data: 2 3 3 3 3 4 4 3 3
235/300 Write block 557 len 36864 data: 2 2 2 2 2 2 2 2 2
236/300 Read block 3050 len 16384
237/300 Write block 3370 len 8192 data: 2 2
238/300 Read block 4198 len 12288
239/300 Read block 4894 len 36864
240/300 Read block 2180 len 12288
241/300 Read block 4691 len 4096
242/300 Write block 1296 len 40960 data: 2 3 3 2 2 2 2 2 2 3
243/300 Read block 162 len 8192
244/300 Write block 1608 len 28672 data: 2 2 2 2 2 4 4
245/300 Read block 384 len 12288
246/300 Read block 3933 len 4096
247/300 Write block 792 len 20480 data: 3 3 3 4 5
248/300 Write block 3887 len 20480 data: 3 3 2 2 2
249/300 Write block 4506 len 40960 data: 4 3 2 2 3 3 2 2 3 3
250/300 Read block 1101 len 12288
251/300 Flush
252/300 Read block 4940 len 4096
253/300 Read block 2239 len 32768
254/300 Write block 3114 len 24576 data: 6 5 5 5 4 4
255/300 Read block 4155 len 8192
256/300 Write block 3733 len 12288 data: 3 4 4
257/300 Write block 2638 len 24576 data: 2 2 2 2 2 2
258/300 Write block 3286 len 20480 data: 2 2 2 2 2
259/300 Read block 2405 len 8192
260/300 Write block 685 len 8192 data: 3 2
261/300 Write block 3556 len 16384 data: 2 2 2 2
262/300 Read block 547 len 32768
263/300 Write block 3816 len 16384 data: 2 2 2 2
264/300 Flush
265/300 Read block 3021 len 24576
266/300 Write block 2809 len 36864 data: 2 2 3 3 4 4 4 4 4
267/300 Write block 410 len 40960 data: 3 2 2 3 4 4 5 5 5 6
268/300 Write block 4736 len 28672 data: 3 3 3 3 3 3 3
269/300 Write block 574 len 40960 data: 3 3 3 4 4 4 5 4 4 4
270/300 Flush
271/300 Read block 1666 len 20480
272/300 Write block 1170 len 8192 data: 4 3
273/300 Write block 3645 len 8192 data: 2 2
274/300 Write block 3293 len 16384 data: 4 4 4 5
275/300 Read block 1161 len 12288
276/300 Read block 2916 len 24576
277/300 Write block 10 len 4096 data: 3
278/300 Flush
279/300 Read block 1255 len 16384
280/300 Write block 1098 len 8192 data: 2 2
281/300 Read block 179 len 4096
282/300 Flush
283/300 Read block 4368 len 24576
284/300 Flush
285/300 Read block 2495 len 36864
286/300 Write block 4667 len 20480 data: 3 3 2 2 2
287/300 Write block 2146 len 32768 data: 3 3 3 3 3 3 4 3
288/300 Flush
289/300 Read block 2888 len 16384
290/300 Read block 3214 len 24576
291/300 Write block 3861 len 36864 data: 3 3 3 4 3 3 3 3 4
292/300 Write block 4867 len 28672 data: 2 2 2 2 2 2 2
293/300 Write block 674 len 12288 data: 4 4 3
294/300 Read block 4522 len 4096
295/300 Write block 2432 len 4096 data: 2
296/300 Read block 3728 len 20480
297/300 Read block 3012 len 20480
298/300 Read block 532 len 4096
299/300 Read block 4339 len 36864
300/300 Read block 237 len 12288
301/300 Read block 4238 len 32768
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:03:26.124 INFO request completed, latency_us: 176, response_code: 204, uri: /start/cid/2, method: POST, req_id: e488d2cf-7d79-41ef-9ae3-6f4eefadae31, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3413 Acked Write 4 Done Done Sent false
3414 Acked Write 2 Done Done Sent false
3415 Acked Flush 0 Done Done Sent false
3416 Acked Write 5 Done Done Sent false
3417 Acked Read 2 Done Done Sent false
3418 Acked Read 4 Done Done Sent false
3419 Acked Read 8 Done Done Sent false
3420 Acked Write 9 Done Done Sent false
3421 Acked Read 9 Done Done Sent false
3422 Acked Read 6 Done Done Sent false
3423 Acked Flush 0 Done Done Sent false
3424 Acked Write 6 Done Done Sent false
3425 Acked Read 9 Done Done Sent false
3426 Acked Read 3 Done Done Sent false
3427 Acked Write 10 Done Done Sent false
3428 Acked Read 6 Done Done Sent false
3429 Acked Read 5 Done Done Sent false
3430 Acked Write 1 Done Done Sent false
3431 Acked Write 10 Done Done Sent false
3432 Acked Read 2 Done Done Sent false
3433 Acked Read 1 Done Done Sent false
3434 Acked Read 4 Done Done Sent false
3435 Acked Read 5 Done Done Sent false
3436 Acked Flush 0 Done Done Sent false
3437 Acked Write 10 Done Done Sent false
3438 Acked Write 4 Done Done Sent false
3439 Acked Write 10 Done Done Sent false
3440 Acked Read 4 Done Done Sent false
3441 Acked Read 9 Done Done Sent false
3442 Acked Write 6 Done Done Sent false
3443 Acked Read 9 Done Done Sent false
3444 Acked Write 2 Done Done Sent false
3445 Acked Read 6 Done Done Sent false
3446 Acked Read 2 Done Done Sent false
3447 Acked Read 7 Done Done Sent false
3448 Acked Write 3 Done Done Sent false
3449 Acked Write 9 Done Done Sent false
3450 Acked Write 7 Done Done Sent false
3451 Acked Write 8 Done Done Sent false
3452 Acked Read 4 Done Done Sent false
3453 Acked Read 2 Done Done Sent false
3454 Acked Read 2 Done Done Sent false
3455 Acked Write 10 Done Done Sent false
3456 Acked Read 6 Done Done Sent false
3457 Acked Write 9 Done Done Sent false
3458 Acked Read 7 Done Done Sent false
3459 Acked Read 10 Done Done Sent false
3460 Acked Read 1 Done Done Sent false
3461 Acked Write 5 Done Done Sent false
3462 Acked Read 5 Done Done Sent false
3463 Acked Write 6 Done Done Sent false
3464 Acked Write 6 Done Done Sent false
3465 Acked Read 2 Done Done Sent false
3466 Acked Read 8 Done Done Sent false
3467 Acked Read 2 Done Done Sent false
3468 Acked Flush 0 Done Done Sent false
3469 Acked Write 8 Done Done Sent false
3470 Acked Write 9 Done Done Sent false
3471 Acked Flush 0 Done Done Sent false
3472 Acked Flush 0 Done Done Sent false
3473 Acked Write 6 Done Done Sent false
3474 Acked Write 9 Done Done Sent false
3475 Acked Read 9 Done Done Sent false
3476 Acked Read 1 Done Done Sent false
3477 Acked Read 4 Done Done Sent false
3478 Acked Read 3 Done Done Sent false
3479 Acked Read 10 Done Done Sent false
3480 Acked Read 6 Done Done Sent false
3481 Acked Read 5 Done Done Sent false
3482 Acked Write 9 Done Done Sent false
3483 Acked Read 6 Done Done Sent false
3484 Acked Read 8 Done Done Sent false
3485 Acked Read 2 Done Done Sent false
3486 Acked Write 10 Done Done Sent false
3487 Acked Read 6 Done Done Sent false
3488 Acked Flush 0 Done Done Sent false
3489 Acked Read 1 Done Done Sent false
3490 Acked Write 6 Done Done Sent false
3491 Acked Write 3 Done Done Sent false
3492 Acked Flush 0 Done Done Sent false
3493 Acked Flush 0 Done Done Sent false
3494 Acked Read 4 Done Done Sent false
3495 Acked Write 6 Done Done Sent false
3496 Acked Read 7 Done Done Sent false
3497 Acked Read 5 Done Done Sent false
3498 Acked Write 4 Done Done Sent false
3499 Acked Write 10 Done Done Sent false
3500 Acked Read 7 Done Done Sent false
3501 Acked Write 7 Done Done Sent false
3502 Acked Write 10 Done Done Sent false
3503 Acked Read 4 Done Done Sent false
3504 Acked Read 3 Done Done Sent false
3505 Acked Write 2 Done Done Sent false
3506 Acked Read 1 Done Done Sent false
3507 Acked Read 9 Done Done Sent false
3508 Acked Read 3 Done Done Sent false
3509 Acked Write 10 Done Done Sent false
3510 Acked Write 1 Done Done Sent false
3511 Acked Read 1 Done Done Sent false
3512 Acked Flush 0 Done Done Sent false
3513 Acked Read 1 Done Done Sent false
3514 Acked Write 9 Done Done Sent false
3515 Acked Read 8 Done Done Sent false
3516 Acked Read 3 Done Done Sent false
3517 Acked Read 5 Done Done Sent false
3518 Acked Read 1 Done Done Sent false
3519 Acked Flush 0 Done Done Sent false
3520 Acked Read 6 Done Done Sent false
3521 Acked Read 3 Done Done Sent false
3522 Acked Write 7 Done Done Sent false
3523 Acked Read 9 Done Done Sent false
3524 Acked Read 6 Done Done Sent false
3525 Acked Read 10 Done Done Sent false
3526 Acked Read 5 Done Done Sent false
3527 Acked Write 8 Done Done Sent false
3528 Acked Read 6 Done Done Sent false
3529 Acked Read 2 Done Done Sent false
3530 Acked Flush 0 Done Done Sent false
3531 Acked Write 10 Done Done Sent false
3532 Acked Read 8 Done Done Sent false
3533 Acked Read 10 Done Done Sent false
3534 Acked Write 1 Done Done Sent false
3535 Acked Write 2 Done Done Sent false
3536 Acked Read 7 Done Done Sent false
3537 Acked Read 3 Done Done Sent false
3538 Acked Write 2 Done Done Sent false
3539 Acked Write 10 Done Done Sent false
3540 Acked Read 3 Done Done Sent false
3541 Acked Flush 0 Done Done Sent false
3542 Acked Write 5 Done Done Sent false
3543 Acked Read 10 Done Done Sent false
3544 Acked Write 9 Done Done Sent false
3545 Acked Write 10 Done Done Sent false
3546 Acked Write 5 Done Done Sent false
3547 Acked Read 8 Done Done Sent false
3548 Acked Write 3 Done Done Sent false
3549 Acked Write 6 Done Done Sent false
3550 Acked Read 3 Done Done Sent false
3551 Acked Read 3 Done Done Sent false
3552 Acked Read 10 Done Done Sent false
3553 Acked Write 7 Done Done Sent false
3554 Acked Read 9 Done Done Sent false
3555 Acked Flush 0 Done Done Sent false
3556 Acked Write 10 Done Done Sent false
3557 Acked Read 2 Done Done Sent false
3558 Acked Write 2 Done Done Sent false
3559 Acked Write 10 Done Done Sent false
3560 Acked Write 1 Done Done Sent false
3561 Acked Write 10 Done Done Sent false
3562 Acked Read 9 Done Done Sent false
3563 Acked Write 9 Done Done Sent false
3564 Acked Flush 0 Done Done Sent false
3565 Acked Read 10 Done Done Sent false
3566 Acked Write 4 Done Done Sent false
3567 Acked Read 6 Done Done Sent false
3568 Acked Read 7 Done Done Sent false
3569 Acked Write 5 Done Done Sent false
3570 Acked Write 2 Done Done Sent false
3571 Acked Write 1 Done Done Sent false
3572 Acked Write 8 Done Done Sent false
3573 Acked Write 6 Done Done Sent false
3574 Acked Write 7 Done Done Sent false
3575 Acked Read 6 Done Done Sent false
3576 Acked Read 6 Done Done Sent false
3577 Acked Write 6 Done Done Sent false
3578 Acked Read 6 Done Done Sent false
3579 Acked Write 1 Done Done Sent false
3580 Acked Read 5 Done Done Sent false
3581 Acked Write 3 Done Done Sent false
3582 Acked Read 4 Done Done Sent false
3583 Acked Read 2 Done Done Sent false
3584 Acked Read 10 Done Done Sent false
3585 Acked Read 3 Done Done Sent false
3586 Acked Write 3 Done Done Sent false
3587 Acked Write 8 Done Done Sent false
3588 Acked Read 6 Done Done Sent false
3589 Acked Read 3 Done Done Sent false
3590 Acked Write 7 Done Done Sent false
3591 Acked Write 7 Done Done Sent false
3592 Acked Flush 0 Done Done Sent false
3593 Acked Read 4 Done Done Sent false
3594 Acked Write 1 Done Done Sent false
3595 Acked Write 4 Done Done Sent false
3596 Acked Read 10 Done Done Sent false
3597 Acked Write 7 Done Done Sent false
3598 Acked Write 9 Done Done Sent false
3599 Acked Flush 0 Done Done Sent false
3600 Acked Write 3 Done Done Sent false
3601 Acked Read 3 Done Done Sent false
3602 Acked Write 6 Done Done Sent false
3603 Acked Read 10 Done Done Sent false
3604 Acked Write 10 Done Done Sent false
3605 Acked Read 8 Done Done Sent false
3606 Acked Write 3 Done Done Sent false
3607 Acked Write 2 Done Done Sent false
3608 Acked Read 8 Done Done Sent false
3609 Acked Read 3 Done Done Sent false
3610 Acked Flush 0 Done Done Sent false
3611 Acked Write 5 Done Done Sent false
3612 Acked Read 1 Done Done Sent false
3613 Acked Write 9 Done Done Sent false
3614 Acked Write 7 Done Done Sent false
3615 Acked Read 6 Done Done Sent false
3616 Acked Read 1 Done Done Sent false
3617 Acked Read 2 Done Done Sent false
3618 Acked Write 8 Done Done Sent false
3619 Acked Read 10 Done Done Sent false
3620 Acked Write 6 Done Done Sent false
3621 Acked Read 1 Done Done Sent false
3622 Acked Flush 0 Done Done Sent false
3623 Acked Write 1 Done Done Sent false
3624 Acked Read 2 Done Done Sent false
3625 Acked Read 1 Done Done Sent false
3626 Acked Write 8 Done Done Sent false
3627 Acked Read 5 Done Done Sent false
3628 Acked Write 4 Done Done Sent false
3629 Acked Write 8 Done Done Sent false
3630 Acked Read 1 Done Done Sent false
3631 Acked Read 9 Done Done Sent false
3632 Acked Flush 0 Done Done Sent false
3633 Acked Read 5 Done Done Sent false
3634 Acked Read 7 Done Done Sent false
3635 Acked Read 1 Done Done Sent false
3636 Acked Write 3 Done Done Sent false
3637 Acked Write 5 Done Done Sent false
3638 Acked Write 6 Done Done Sent false
3639 Acked Read 7 Done Done Sent false
3640 Acked Read 8 Done Done Sent false
3641 Acked Write 5 Done Done Sent false
3642 Acked Write 8 Done Done Sent false
3643 Acked Read 6 Done Done Sent false
3644 Acked Read 7 Done Done Sent false
3645 Acked Read 10 Done Done Sent false
3646 Acked Write 9 Done Done Sent false
3647 Acked Write 9 Done Done Sent false
3648 Acked Read 4 Done Done Sent false
3649 Acked Write 2 Done Done Sent false
3650 Acked Read 3 Done Done Sent false
3651 Acked Read 9 Done Done Sent false
3652 Acked Read 3 Done Done Sent false
3653 Acked Read 1 Done Done Sent false
3654 Acked Write 10 Done Done Sent false
3655 Acked Read 2 Done Done Sent false
3656 Acked Write 7 Done Done Sent false
3657 Acked Read 3 Done Done Sent false
3658 Acked Read 1 Done Done Sent false
3659 Acked Write 5 Done Done Sent false
3660 Acked Write 5 Done Done Sent false
3661 Acked Write 10 Done Done Sent false
3662 Acked Read 3 Done Done Sent false
3663 Acked Flush 0 Done Done Sent false
3664 Acked Read 1 Done Done Sent false
3665 Acked Read 8 Done Done Sent false
3666 Acked Write 6 Done Done Sent false
3667 Acked Read 2 Done Done Sent false
3668 Acked Write 3 Done Done Sent false
3669 Acked Write 6 Done Done Sent false
3670 Acked Write 5 Done Done Sent false
3671 Acked Read 2 Done Done Sent false
3672 Acked Write 2 Done Done Sent false
3673 Acked Write 4 Done Done Sent false
3674 Acked Read 8 Done Done Sent false
3675 Acked Write 4 Done Done Sent false
3676 Acked Flush 0 Done Done Sent false
3677 Acked Read 6 Done Done Sent false
3678 Acked Write 9 Done Done Sent false
3679 Acked Write 10 Done Done Sent false
3680 Acked Write 7 Done Done Sent false
3681 Acked Write 10 Done Done Sent false
3682 Acked Flush 0 Done Done Sent false
3683 Acked Read 5 Done Done Sent false
3684 Acked Write 2 Done Done Sent false
3685 Acked Write 2 Done Done Sent false
3686 Acked Write 4 Done Done Sent false
3687 Acked Read 3 Done Done Sent false
3688 Acked Read 6 Done Done Sent false
3689 Acked Write 1 Done Done Sent false
3690 Acked Flush 0 Done Done Sent false
3691 Acked Read 4 Done Done Sent false
3692 Acked Write 2 Done Done Sent false
3693 Acked Read 1 Done Done Sent false
3694 Acked Flush 0 Done Done Sent false
3695 Acked Read 6 Done Done Sent false
3696 Acked Flush 0 Done Done Sent false
3697 Acked Read 9 Done Done Sent false
3698 Acked Write 5 Done Done Sent false
3699 Acked Write 8 Done Done Sent false
3700 Acked Flush 0 Done Done Sent false
3701 Acked Read 4 Done Done Sent false
3702 Acked Read 6 Done Done Sent false
3703 Acked Write 9 Done Done Sent false
3704 Acked Write 7 Done Done Sent false
3705 Acked Write 3 Done Done Sent false
3706 Acked Read 1 Done Done Sent false
3707 Acked Write 1 Done Done Sent false
3708 Acked Read 5 Done Done Sent false
3709 Acked Read 5 Done Done Sent false
3710 Acked Read 1 Done Done Sent false
3711 Acked Read 9 Done Done Sent false
3712 Acked Read 3 Done Done Sent false
3713 Acked Read 8 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3700)) Some(JobId(3700)) Some(JobId(3412))
Downstairs last five completed: 3412 3411 3410 3409 3408
Upstairs last five completed: 3713 3712 3711 3710 3709
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:26.626191492Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:27.887098437Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:28.581310635Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:29.583233097Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3413 Acked Write 4 Done Done Sent false
3414 Acked Write 2 Done Done Sent false
3415 Acked Flush 0 Done Done Sent false
3416 Acked Write 5 Done Done Sent false
3417 Acked Read 2 Done Done Sent false
3418 Acked Read 4 Done Done Sent false
3419 Acked Read 8 Done Done Sent false
3420 Acked Write 9 Done Done Sent false
3421 Acked Read 9 Done Done Sent false
3422 Acked Read 6 Done Done Sent false
3423 Acked Flush 0 Done Done Sent false
3424 Acked Write 6 Done Done Sent false
3425 Acked Read 9 Done Done Sent false
3426 Acked Read 3 Done Done Sent false
3427 Acked Write 10 Done Done Sent false
3428 Acked Read 6 Done Done Sent false
3429 Acked Read 5 Done Done Sent false
3430 Acked Write 1 Done Done Sent false
3431 Acked Write 10 Done Done Sent false
3432 Acked Read 2 Done Done Sent false
3433 Acked Read 1 Done Done Sent false
3434 Acked Read 4 Done Done Sent false
3435 Acked Read 5 Done Done Sent false
3436 Acked Flush 0 Done Done Sent false
3437 Acked Write 10 Done Done Sent false
3438 Acked Write 4 Done Done Sent false
3439 Acked Write 10 Done Done Sent false
3440 Acked Read 4 Done Done Sent false
3441 Acked Read 9 Done Done Sent false
3442 Acked Write 6 Done Done Sent false
3443 Acked Read 9 Done Done Sent false
3444 Acked Write 2 Done Done Sent false
3445 Acked Read 6 Done Done Sent false
3446 Acked Read 2 Done Done Sent false
3447 Acked Read 7 Done Done Sent false
3448 Acked Write 3 Done Done Sent false
3449 Acked Write 9 Done Done Sent false
3450 Acked Write 7 Done Done Sent false
3451 Acked Write 8 Done Done Sent false
3452 Acked Read 4 Done Done Sent false
3453 Acked Read 2 Done Done Sent false
3454 Acked Read 2 Done Done Sent false
3455 Acked Write 10 Done Done Sent false
3456 Acked Read 6 Done Done Sent false
3457 Acked Write 9 Done Done Sent false
3458 Acked Read 7 Done Done Sent false
3459 Acked Read 10 Done Done Sent false
3460 Acked Read 1 Done Done Sent false
3461 Acked Write 5 Done Done Sent false
3462 Acked Read 5 Done Done Sent false
3463 Acked Write 6 Done Done Sent false
3464 Acked Write 6 Done Done Sent false
3465 Acked Read 2 Done Done Sent false
3466 Acked Read 8 Done Done Sent false
3467 Acked Read 2 Done Done Sent false
3468 Acked Flush 0 Done Done Sent false
3469 Acked Write 8 Done Done Sent false
3470 Acked Write 9 Done Done Sent false
3471 Acked Flush 0 Done Done Sent false
3472 Acked Flush 0 Done Done Sent false
3473 Acked Write 6 Done Done Sent false
3474 Acked Write 9 Done Done Sent false
3475 Acked Read 9 Done Done Sent false
3476 Acked Read 1 Done Done Sent false
3477 Acked Read 4 Done Done Sent false
3478 Acked Read 3 Done Done Sent false
3479 Acked Read 10 Done Done Sent false
3480 Acked Read 6 Done Done Sent false
3481 Acked Read 5 Done Done Sent false
3482 Acked Write 9 Done Done Sent false
3483 Acked Read 6 Done Done Sent false
3484 Acked Read 8 Done Done Sent false
3485 Acked Read 2 Done Done Sent false
3486 Acked Write 10 Done Done Sent false
3487 Acked Read 6 Done Done Sent false
3488 Acked Flush 0 Done Done Sent false
3489 Acked Read 1 Done Done Sent false
3490 Acked Write 6 Done Done Sent false
3491 Acked Write 3 Done Done Sent false
3492 Acked Flush 0 Done Done Sent false
3493 Acked Flush 0 Done Done Sent false
3494 Acked Read 4 Done Done Sent false
3495 Acked Write 6 Done Done Sent false
3496 Acked Read 7 Done Done Sent false
3497 Acked Read 5 Done Done Sent false
3498 Acked Write 4 Done Done Sent false
3499 Acked Write 10 Done Done Sent false
3500 Acked Read 7 Done Done Sent false
3501 Acked Write 7 Done Done Sent false
3502 Acked Write 10 Done Done Sent false
3503 Acked Read 4 Done Done Sent false
3504 Acked Read 3 Done Done Sent false
3505 Acked Write 2 Done Done Sent false
3506 Acked Read 1 Done Done Sent false
3507 Acked Read 9 Done Done Sent false
3508 Acked Read 3 Done Done Sent false
3509 Acked Write 10 Done Done Sent false
3510 Acked Write 1 Done Done Sent false
3511 Acked Read 1 Done Done Sent false
3512 Acked Flush 0 Done Done Sent false
3513 Acked Read 1 Done Done Sent false
3514 Acked Write 9 Done Done Sent false
3515 Acked Read 8 Done Done Sent false
3516 Acked Read 3 Done Done Sent false
3517 Acked Read 5 Done Done Sent false
3518 Acked Read 1 Done Done Sent false
3519 Acked Flush 0 Done Done Sent false
3520 Acked Read 6 Done Done Sent false
3521 Acked Read 3 Done Done Sent false
3522 Acked Write 7 Done Done Sent false
3523 Acked Read 9 Done Done Sent false
3524 Acked Read 6 Done Done Sent false
3525 Acked Read 10 Done Done Sent false
3526 Acked Read 5 Done Done Sent false
3527 Acked Write 8 Done Done Sent false
3528 Acked Read 6 Done Done Sent false
3529 Acked Read 2 Done Done Sent false
3530 Acked Flush 0 Done Done Sent false
3531 Acked Write 10 Done Done Sent false
3532 Acked Read 8 Done Done Sent false
3533 Acked Read 10 Done Done Sent false
3534 Acked Write 1 Done Done Sent false
3535 Acked Write 2 Done Done Sent false
3536 Acked Read 7 Done Done Sent false
3537 Acked Read 3 Done Done Sent false
3538 Acked Write 2 Done Done Sent false
3539 Acked Write 10 Done Done Sent false
3540 Acked Read 3 Done Done Sent false
3541 Acked Flush 0 Done Done Sent false
3542 Acked Write 5 Done Done Sent false
3543 Acked Read 10 Done Done Sent false
3544 Acked Write 9 Done Done Sent false
3545 Acked Write 10 Done Done Sent false
3546 Acked Write 5 Done Done Sent false
3547 Acked Read 8 Done Done Sent false
3548 Acked Write 3 Done Done Sent false
3549 Acked Write 6 Done Done Sent false
3550 Acked Read 3 Done Done Sent false
3551 Acked Read 3 Done Done Sent false
3552 Acked Read 10 Done Done Sent false
3553 Acked Write 7 Done Done Sent false
3554 Acked Read 9 Done Done Sent false
3555 Acked Flush 0 Done Done Sent false
3556 Acked Write 10 Done Done Sent false
3557 Acked Read 2 Done Done Sent false
3558 Acked Write 2 Done Done Sent false
3559 Acked Write 10 Done Done Sent false
3560 Acked Write 1 Done Done Sent false
3561 Acked Write 10 Done Done Sent false
3562 Acked Read 9 Done Done Sent false
3563 Acked Write 9 Done Done Sent false
3564 Acked Flush 0 Done Done Sent false
3565 Acked Read 10 Done Done Sent false
3566 Acked Write 4 Done Done Sent false
3567 Acked Read 6 Done Done Sent false
3568 Acked Read 7 Done Done Sent false
3569 Acked Write 5 Done Done Sent false
3570 Acked Write 2 Done Done Sent false
3571 Acked Write 1 Done Done Sent false
3572 Acked Write 8 Done Done Sent false
3573 Acked Write 6 Done Done Sent false
3574 Acked Write 7 Done Done Sent false
3575 Acked Read 6 Done Done Sent false
3576 Acked Read 6 Done Done Sent false
3577 Acked Write 6 Done Done Sent false
3578 Acked Read 6 Done Done Sent false
3579 Acked Write 1 Done Done Sent false
3580 Acked Read 5 Done Done Sent false
3581 Acked Write 3 Done Done Sent false
3582 Acked Read 4 Done Done Sent false
3583 Acked Read 2 Done Done Sent false
3584 Acked Read 10 Done Done Sent false
3585 Acked Read 3 Done Done Sent false
3586 Acked Write 3 Done Done Sent false
3587 Acked Write 8 Done Done Sent false
3588 Acked Read 6 Done Done Sent false
3589 Acked Read 3 Done Done Sent false
3590 Acked Write 7 Done Done Sent false
3591 Acked Write 7 Done Done Sent false
3592 Acked Flush 0 Done Done Sent false
3593 Acked Read 4 Done Done Sent false
3594 Acked Write 1 Done Done Sent false
3595 Acked Write 4 Done Done Sent false
3596 Acked Read 10 Done Done Sent false
3597 Acked Write 7 Done Done Sent false
3598 Acked Write 9 Done Done Sent false
3599 Acked Flush 0 Done Done Sent false
3600 Acked Write 3 Done Done Sent false
3601 Acked Read 3 Done Done Sent false
3602 Acked Write 6 Done Done Sent false
3603 Acked Read 10 Done Done Sent false
3604 Acked Write 10 Done Done Sent false
3605 Acked Read 8 Done Done Sent false
3606 Acked Write 3 Done Done Sent false
3607 Acked Write 2 Done Done Sent false
3608 Acked Read 8 Done Done Sent false
3609 Acked Read 3 Done Done Sent false
3610 Acked Flush 0 Done Done Sent false
3611 Acked Write 5 Done Done Sent false
3612 Acked Read 1 Done Done Sent false
3613 Acked Write 9 Done Done Sent false
3614 Acked Write 7 Done Done Sent false
3615 Acked Read 6 Done Done Sent false
3616 Acked Read 1 Done Done Sent false
3617 Acked Read 2 Done Done Sent false
3618 Acked Write 8 Done Done Sent false
3619 Acked Read 10 Done Done Sent false
3620 Acked Write 6 Done Done Sent false
3621 Acked Read 1 Done Done Sent false
3622 Acked Flush 0 Done Done Sent false
3623 Acked Write 1 Done Done Sent false
3624 Acked Read 2 Done Done Sent false
3625 Acked Read 1 Done Done Sent false
3626 Acked Write 8 Done Done Sent false
3627 Acked Read 5 Done Done Sent false
3628 Acked Write 4 Done Done Sent false
3629 Acked Write 8 Done Done Sent false
3630 Acked Read 1 Done Done Sent false
3631 Acked Read 9 Done Done Sent false
3632 Acked Flush 0 Done Done Sent false
3633 Acked Read 5 Done Done Sent false
3634 Acked Read 7 Done Done Sent false
3635 Acked Read 1 Done Done Sent false
3636 Acked Write 3 Done Done Sent false
3637 Acked Write 5 Done Done Sent false
3638 Acked Write 6 Done Done Sent false
3639 Acked Read 7 Done Done Sent false
3640 Acked Read 8 Done Done Sent false
3641 Acked Write 5 Done Done Sent false
3642 Acked Write 8 Done Done Sent false
3643 Acked Read 6 Done Done Sent false
3644 Acked Read 7 Done Done Sent false
3645 Acked Read 10 Done Done Sent false
3646 Acked Write 9 Done Done Sent false
3647 Acked Write 9 Done Done Sent false
3648 Acked Read 4 Done Done Sent false
3649 Acked Write 2 Done Done Sent false
3650 Acked Read 3 Done Done Sent false
3651 Acked Read 9 Done Done Sent false
3652 Acked Read 3 Done Done Sent false
3653 Acked Read 1 Done Done Sent false
3654 Acked Write 10 Done Done Sent false
3655 Acked Read 2 Done Done Sent false
3656 Acked Write 7 Done Done Sent false
3657 Acked Read 3 Done Done Sent false
3658 Acked Read 1 Done Done Sent false
3659 Acked Write 5 Done Done Sent false
3660 Acked Write 5 Done Done Sent false
3661 Acked Write 10 Done Done Sent false
3662 Acked Read 3 Done Done Sent false
3663 Acked Flush 0 Done Done Sent false
3664 Acked Read 1 Done Done Sent false
3665 Acked Read 8 Done Done Sent false
3666 Acked Write 6 Done Done Sent false
3667 Acked Read 2 Done Done Sent false
3668 Acked Write 3 Done Done Sent false
3669 Acked Write 6 Done Done Sent false
3670 Acked Write 5 Done Done Sent false
3671 Acked Read 2 Done Done Sent false
3672 Acked Write 2 Done Done Sent false
3673 Acked Write 4 Done Done Sent false
3674 Acked Read 8 Done Done Sent false
3675 Acked Write 4 Done Done Sent false
3676 Acked Flush 0 Done Done Sent false
3677 Acked Read 6 Done Done Sent false
3678 Acked Write 9 Done Done Sent false
3679 Acked Write 10 Done Done Sent false
3680 Acked Write 7 Done Done Sent false
3681 Acked Write 10 Done Done Sent false
3682 Acked Flush 0 Done Done Sent false
3683 Acked Read 5 Done Done Sent false
3684 Acked Write 2 Done Done Sent false
3685 Acked Write 2 Done Done Sent false
3686 Acked Write 4 Done Done Sent false
3687 Acked Read 3 Done Done Sent false
3688 Acked Read 6 Done Done Sent false
3689 Acked Write 1 Done Done Sent false
3690 Acked Flush 0 Done Done Sent false
3691 Acked Read 4 Done Done Sent false
3692 Acked Write 2 Done Done Sent false
3693 Acked Read 1 Done Done Sent false
3694 Acked Flush 0 Done Done Sent false
3695 Acked Read 6 Done Done Sent false
3696 Acked Flush 0 Done Done Sent false
3697 Acked Read 9 Done Done Sent false
3698 Acked Write 5 Done Done Sent false
3699 Acked Write 8 Done Done Sent false
3700 Acked Flush 0 Done Done Sent false
3701 Acked Read 4 Done Done Sent false
3702 Acked Read 6 Done Done Sent false
3703 Acked Write 9 Done Done Sent false
3704 Acked Write 7 Done Done Sent false
3705 Acked Write 3 Done Done Sent false
3706 Acked Read 1 Done Done Sent false
3707 Acked Write 1 Done Done Sent false
3708 Acked Read 5 Done Done Sent false
3709 Acked Read 5 Done Done Sent false
3710 Acked Read 1 Done Done Sent false
3711 Acked Read 9 Done Done Sent false
3712 Acked Read 3 Done Done Sent false
3713 Acked Read 8 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3700)) Some(JobId(3700)) Some(JobId(3412))
Downstairs last five completed: 3412 3411 3410 3409 3408
Upstairs last five completed: 3713 3712 3711 3710 3709
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:30.628036253Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:31.586366492Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.884668672Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60177)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.884898546Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886014219Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886565167Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.88659472Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(3412))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886613046Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886629462Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(3412))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886646378Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(3412))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:31.886837452Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 3714 3713 3712 3711 3710
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:03:34.129 INFO request completed[8810] Got stop action so:false kr:false
, latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3c412538-928e-47a0-ba49-f00443a84843, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:34.130250592Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:34.130316696Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:34.130333982Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:03:34.130 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/0, method: GET, req_id: ea5085f7-6aa0-4a18-87e8-de36a10033c8, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:34.130443956Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:03:38.132 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/0, method: GET, req_id: c71b93af-d91a-48dc-ac5b-5474876b3182, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 106 len 28672
002/300 Read block 2646 len 8192
003/300 Write block 3163 len 32768 data: 2 2 3 3 3 3 3 3
004/300 Write block 3249 len 16384 data: 4 4 3 4
005/300 Read block 22 len 32768
006/300 Flush
007/300 Read block 2074 len 36864
008/300 Read block 3140 len 16384
009/300 Read block 1048 len 24576
010/300 Read block 174 len 20480
011/300 Read block 1211 len 36864
012/300 Write block 3157 len 20480 data: 3 4 4 4 3
013/300 Read block 560 len 40960
014/300 Read block 3800 len 28672
015/300 Read block 969 len 40960
016/300 Write block 238 len 16384 data: 7 5 4 4
017/300 Write block 2846 len 32768 data: 3 3 3 3 3 2 3 3
018/300 Read block 3334 len 8192
019/300 Write block 2139 len 8192 data: 2 2
020/300 Write block 3403 len 4096 data: 2
021/300 Flush
022/300 Read block 2933 len 40960
023/300 Read block 3543 len 4096
024/300 Read block 4359 len 24576
025/300 Read block 982 len 24576
026/300 Write block 1978 len 28672 data: 2 2 2 2 2 3 3
027/300 Read block 4375 len 40960
028/300 Write block 4341 len 36864 data: 4 4 4 4 4 5 4 4 3
029/300 Read block 3357 len 32768
030/300 Write block 975 len 24576 data: 2 2 2 3 3 3
031/300 Write block 214 len 24576 data: 3 3 3 3 4 5
032/300 Flush
033/300 Read block 1092 len 24576
034/300 Read block 4490 len 24576
035/300 Read block 3613 len 32768
036/300 Read block 518 len 4096
037/300 Write block 1766 len 8192 data: 3 2
038/300 Flush
039/300 Read block 4199 len 4096
040/300 Read block 3230 len 16384
041/300 Read block 3737 len 40960
042/300 Write block 4271 len 36864 data: 5 5 4 4 3 3 3 3 3
043/300 Write block 3478 len 40960 data: 3 4 4 3 3 3 3 3 3 3
044/300 Read block 510 len 28672
045/300 Write block 4225 len 40960 data: 3 4 4 5 6 6 6 3 3 3
046/300 Write block 748 len 12288 data: 3 4 4
047/300 Read block 2556 len 12288
048/300 Read block 294 len 4096
049/300 Read block 3438 len 36864
050/300 Read block 4249 len 16384
051/300 Write block 1171 len 16384 data: 4 4 4 5
052/300 Write block 4318 len 16384 data: 3 4 4 4
053/300 Read block 958 len 8192
054/300 Write block 657 len 40960 data: 2 2 2 2 2 2 2 2 2 2
055/300 Read block 3855 len 20480
056/300 Read block 1808 len 32768
057/300 Read block 4444 len 28672
058/300 Write block 4762 len 28672 data: 3 3 4 3 3 3 3
059/300 Flush
060/300 Write block 1872 len 28672 data: 3 3 3 3 3 4 4
061/300 Read block 139 len 32768
062/300 Read block 1124 len 12288
063/300 Write block 4383 len 40960 data: 3 4 4 5 4 4 4 6 6 4
064/300 Read block 4915 len 36864
065/300 Write block 3134 len 20480 data: 3 2 2 2 2
066/300 Read block 4641 len 40960
067/300 Read block 3938 len 12288
068/300 Read block 3715 len 20480
069/300 Write block 2331 len 20480 data: 2 2 2 2 2
070/300 Write block 4204 len 12288 data: 4 3 3
071/300 Read block 1878 len 4096
072/300 Read block 4831 len 40960
073/300 Write block 3888 len 24576 data: 4 3 3 3 2 2
074/300 Read block 4907 len 16384
075/300 Read block 3865 len 40960
076/300 Write block 2851 len 36864 data: 3 4 4 3 3 3 3 3 2
077/300 Write block 4632 len 32768 data: 5 5 4 3 3 3 3 3
078/300 Flush
079/300 Read block 2768 len 12288
080/300 Write block 436 len 28672 data: 6 5 5 5 5 4 6
081/300 Read block 2993 len 12288
082/300 Write block 1544 len 32768 data: 2 2 3 3 3 3 3 3
083/300 Flush
084/300 Write block 355 len 8192 data: 3 3
085/300 Write block 2475 len 24576 data: 4 4 3 3 3 3
086/300 Write block 3914 len 28672 data: 3 3 3 2 2 2 2
087/300 Read block 4546 len 12288
088/300 Read block 3044 len 4096
089/300 Read block 839 len 36864
090/300 Write block 658 len 32768 data: 3 3 3 3 3 3 3 3
091/300 Write block 4212 len 24576 data: 5 5 4 4 4 4
092/300 Read block 3970 len 20480
093/300 Write block 3829 len 4096 data: 3
094/300 Read block 1275 len 32768
095/300 Read block 1408 len 16384
096/300 Read block 2383 len 32768
097/300 Write block 2431 len 24576 data: 2 3 2 2 2 2
098/300 Write block 1707 len 16384 data: 5 5 5 5
099/300 Flush
100/300 Write block 4862 len 28672 data: 2 2 2 2 2 3 3
101/300 Write block 3807 len 16384 data: 2 2 3 3
102/300 Write block 3525 len 16384 data: 6 6 7 7
103/300 Write block 1641 len 8192 data: 2 2
104/300 Read block 825 len 20480
105/300 Read block 2181 len 4096
106/300 Write block 316 len 12288 data: 4 4 4
107/300 Write block 2071 len 16384 data: 3 3 2 2
108/300 Write block 414 len 4096 data: 5
109/300 Write block 3716 len 12288 data: 2 3 5
110/300 Write block 4508 len 32768 data: 3 3 4 4 3 3 4 4
111/300 Read block 3223 len 24576
112/300 Flush
113/300 Flush
114/300 Read block 4733 len 36864
115/300 Read block 4418 len 36864
116/300 Write block 1954 len 24576 data: 3 3 3 3 3 3
117/300 Read block 2058 len 32768
118/300 Write block 3536 len 28672 data: 3 2 2 2 2 2 2
119/300 Write block 2228 len 16384 data: 4 4 4 5
120/300 Read block 1940 len 24576
121/300 Read block 4270 len 12288
122/300 Write block 4754 len 40960 data: 2 2 3 3 3 3 3 3 4 4
123/300 Write block 3123 len 20480 data: 3 3 3 3 3
124/300 Flush
125/300 Write block 2523 len 8192 data: 5 6
126/300 Read block 691 len 20480
127/300 Read block 4174 len 4096
128/300 Read block 2103 len 12288
129/300 Read block 3839 len 8192
130/300 Write block 382 len 20480 data: 5 5 4 4 4
131/300 Write block 3354 len 16384 data: 3 3 3 2
132/300 Read block 1476 len 16384
133/300 Read block 3625 len 24576
134/300 Write block 1136 len 20480 data: 2 2 3 3 3
135/300 Write block 2320 len 4096 data: 4
136/300 Read block 3681 len 20480
137/300 Write block 1681 len 28672 data: 2 3 4 4 4 4 3
138/300 Read block 710 len 24576
139/300 Write block 2712 len 28672 data: 4 6 5 5 5 5 4
140/300 Write block 1741 len 40960 data: 4 3 3 2 2 2 3 3 3 3
141/300 Flush
142/300 Write block 4780 len 12288 data: 5 5 4
143/300 Read block 471 len 24576
144/300 Read block 919 len 40960
145/300 Flush
146/300 Write block 1408 len 8192 data: 3 4
147/300 Write block 2164 len 36864 data: 3 3 3 3 4 3 3 2 2
148/300 Write block 4496 len 4096 data: 5
149/300 Read block 3647 len 12288
150/300 Write block 1423 len 4096 data: 3
151/300 Write block 1031 len 4096 data: 3
152/300 Read block 4939 len 32768
153/300 Read block 4431 len 16384
154/300 Read block 2300 len 32768
155/300 Read block 4293 len 8192
156/300 Read block 4847 len 36864
157/300 Read block 721 len 8192
158/300 Write block 4823 len 40960 data: 3 3 3 3 3 4 4 6 6 5
159/300 Read block 2535 len 20480
160/300 Read block 437 len 4096
161/300 Read block 4663 len 4096
162/300 Read block 1644 len 8192
163/300 Read block 1993 len 24576
164/300 Write block 1692 len 40960 data: 4 3 4 4 5 5 4 4 5 5
165/300 Read block 4133 len 4096
166/300 Read block 1134 len 32768
167/300 Write block 4871 len 12288 data: 3 3 3
168/300 Write block 408 len 4096 data: 4
169/300 Read block 3085 len 28672
170/300 Read block 1422 len 4096
171/300 Read block 453 len 12288
172/300 Write block 3058 len 28672 data: 2 2 2 3 3 3 3
173/300 Read block 4714 len 20480
174/300 Read block 1784 len 4096
175/300 Read block 4718 len 20480
176/300 Write block 1747 len 36864 data: 4 4 4 4 2 2 2 2 2
177/300 Flush
178/300 Flush
179/300 Read block 148 len 24576
180/300 Write block 1578 len 32768 data: 4 4 4 5 5 5 5 5
181/300 Write block 672 len 28672 data: 5 4 5 5 4 3 4
182/300 Write block 4223 len 28672 data: 2 2 4 5 5 6 7
183/300 Write block 1906 len 12288 data: 2 2 2
184/300 Write block 4531 len 24576 data: 4 2 2 2 3 3
185/300 Write block 2234 len 24576 data: 6 4 5 4 4 3
186/300 Write block 1273 len 36864 data: 5 5 3 3 2 3 3 4 4
187/300 Read block 4785 len 32768
188/300 Read block 4832 len 36864
189/300 Read block 1511 len 32768
190/300 Write block 1966 len 8192 data: 2 2
191/300 Read block 4531 len 16384
192/300 Read block 4708 len 32768
193/300 Read block 2422 len 20480
194/300 Read block 433 len 12288
195/300 Write block 789 len 36864 data: 2 2 3 4 4 4 5 6 5
196/300 Read block 3619 len 24576
197/300 Write block 926 len 36864 data: 4 4 4 4 5 5 5 4 3
198/300 Read block 3981 len 28672
199/300 Read block 4454 len 12288
200/300 Read block 3603 len 4096
201/300 Write block 1054 len 32768 data: 4 3 3 2 2 2 2 2
202/300 Read block 2912 len 12288
203/300 Read block 528 len 4096
204/300 Read block 688 len 32768
205/300 Write block 4342 len 4096 data: 5
206/300 Flush
207/300 Read block 314 len 12288
208/300 Read block 852 len 24576
209/300 Write block 3061 len 40960 data: 4 4 4 4 3 3 4 3 3 3
210/300 Write block 4365 len 4096 data: 3
211/300 Read block 2539 len 28672
212/300 Write block 4108 len 40960 data: 2 2 3 3 4 4 4 4 5 5
213/300 Flush
214/300 Read block 4122 len 40960
215/300 Read block 2500 len 4096
216/300 Read block 3593 len 16384
217/300 Write block 4954 len 20480 data: 4 3 3 3 3
218/300 Read block 771 len 36864
219/300 Flush
220/300 Read block 4000 len 20480
221/300 Read block 2556 len 36864
222/300 Write block 657 len 36864 data: 3 4 4 4 4 4 4 4 4
223/300 Read block 4084 len 8192
224/300 Write block 4889 len 40960 data: 7 6 6 4 4 4 4 4 4 5
225/300 Write block 3699 len 16384 data: 2 2 2 2
226/300 Read block 1976 len 12288
227/300 Write block 3301 len 20480 data: 4 5 4 4 5
228/300 Read block 3260 len 12288
229/300 Write block 4503 len 20480 data: 4 4 4 5 4
230/300 Write block 3536 len 28672 data: 4 3 3 3 3 3 3
231/300 Read block 4597 len 12288
232/300 Write block 2335 len 36864 data: 3 3 3 3 3 3 3 3 3
233/300 Read block 4379 len 4096
234/300 Flush
235/300 Read block 438 len 32768
236/300 Write block 2231 len 8192 data: 6 4
237/300 Flush
238/300 Read block 3158 len 20480
239/300 Flush
240/300 Write block 209 len 24576 data: 2 2 2 2 2 4
241/300 Read block 769 len 4096
242/300 Write block 4020 len 36864 data: 3 3 3 3 3 3 2 2 2
243/300 Write block 2641 len 12288 data: 3 3 3
244/300 Write block 4405 len 12288 data: 5 4 6
245/300 Read block 2097 len 16384
246/300 Read block 2572 len 12288
247/300 Write block 4029 len 8192 data: 2 2
248/300 Read block 3464 len 4096
249/300 Read block 963 len 12288
250/300 Read block 2448 len 40960
251/300 Write block 3844 len 40960 data: 2 2 2 2 3 3 4 4 5 5
252/300 Write block 2933 len 8192 data: 3 3
253/300 Read block 1752 len 36864
254/300 Write block 2685 len 20480 data: 5 4 5 4 5
255/300 Write block 1575 len 24576 data: 3 3 3 5 5 5
256/300 Write block 2371 len 28672 data: 3 2 2 3 4 5 4
257/300 Read block 2744 len 28672
258/300 Read block 1621 len 28672
259/300 Read block 1214 len 8192
260/300 Read block 744 len 36864
261/300 Read block 2015 len 4096
262/300 Read block 703 len 12288
263/300 Read block 4336 len 8192
264/300 Flush
265/300 Flush
266/300 Read block 3672 len 40960
267/300 Write block 4876 len 16384 data: 3 3 4 5
268/300 Read block 1286 len 24576
269/300 Read block 3495 len 20480
270/300 Read block 1714 len 16384
271/300 Write block 3778 len 12288 data: 3 3 4
272/300 Read block 3332 len 24576
273/300 Read block 746 len 36864
274/300 Flush
275/300 Flush
276/300 Read block 3146 len 20480
277/300 Write block 1131 len 8192 data: 3 2
278/300 Write block 2225 len 12288 data: 3 3 4
279/300 Read block 4975 len 8192
280/300 Write block 602 len 12288 data: 3 3 2
281/300 Read block 1682 len 16384
282/300 Read block 2056 len 32768
283/300 Write block 3838 len 20480 data: 2 2 2 2 2
284/300 Write block 4447 len 4096 data: 3
285/300 Write block 2861 len 16384 data: 3 3 4 3
286/300 Read block 160 len 36864
287/300 Read block 3798 len 24576
288/300 Read block 4908 len 32768
289/300 Write block 838 len 36864 data: 2 2 2 3 3 3 5 5 4
290/300 Flush
291/300 Write block 1828 len 12288 data: 3 3 2
292/300 Write block 2637 len 36864 data: 2 3 3 3 4 4 4 2 2
293/300 Read block 1948 len 4096
294/300 Write block 1253 len 40960 data: 2 3 4 4 4 3 2 3 5 5
295/300 Read block 3921 len 4096
296/300 Read block 2281 len 8192
297/300 Read block 97 len 40960
298/300 Read block 180 len 16384
299/300 Write block 670 len 16384 data: 5 5 6 5
300/300 Read block 2756 len 4096
301/300 Write block 1741 len 28672 data: 5 4 4 3 3 3 5
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:03:38.370 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/0, method: POST, req_id: a56164b5-3466-4eb6-b457-ee314e15cc0a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3715 Acked Read 7 Sent Done Done false
3716 Acked Read 2 Sent Done Done false
3717 Acked Write 8 Sent Done Done false
3718 Acked Write 4 Sent Done Done false
3719 Acked Read 8 Sent Done Done false
3720 Acked Flush 0 Sent Done Done false
3721 Acked Read 9 Sent Done Done false
3722 Acked Read 4 Sent Done Done false
3723 Acked Read 6 Sent Done Done false
3724 Acked Read 5 Sent Done Done false
3725 Acked Read 9 Sent Done Done false
3726 Acked Write 5 Sent Done Done false
3727 Acked Read 10 Sent Done Done false
3728 Acked Read 7 Sent Done Done false
3729 Acked Read 10 Sent Done Done false
3730 Acked Write 4 Sent Done Done false
3731 Acked Write 8 Sent Done Done false
3732 Acked Read 2 Sent Done Done false
3733 Acked Write 2 Sent Done Done false
3734 Acked Write 1 Sent Done Done false
3735 Acked Flush 0 Sent Done Done false
3736 Acked Read 10 Sent Done Done false
3737 Acked Read 1 Sent Done Done false
3738 Acked Read 6 Sent Done Done false
3739 Acked Read 6 Sent Done Done false
3740 Acked Write 7 Sent Done Done false
3741 Acked Read 10 Sent Done Done false
3742 Acked Write 9 Sent Done Done false
3743 Acked Read 8 Sent Done Done false
3744 Acked Write 6 Sent Done Done false
3745 Acked Write 6 Sent Done Done false
3746 Acked Flush 0 Sent Done Done false
3747 Acked Read 6 Sent Done Done false
3748 Acked Read 6 Sent Done Done false
3749 Acked Read 8 Sent Done Done false
3750 Acked Read 1 Sent Done Done false
3751 Acked Write 2 Sent Done Done false
3752 Acked Flush 0 Sent Done Done false
3753 Acked Read 1 Sent Done Done false
3754 Acked Read 4 Sent Done Done false
3755 Acked Read 10 Sent Done Done false
3756 Acked Write 9 Sent Done Done false
3757 Acked Write 10 Sent Done Done false
3758 Acked Read 7 Sent Done Done false
3759 Acked Write 10 Sent Done Done false
3760 Acked Write 3 Sent Done Done false
3761 Acked Read 3 Sent Done Done false
3762 Acked Read 1 Sent Done Done false
3763 Acked Read 9 Sent Done Done false
3764 Acked Read 4 Sent Done Done false
3765 Acked Write 4 Sent Done Done false
3766 Acked Write 4 Sent Done Done false
3767 Acked Read 2 Sent Done Done false
3768 Acked Write 10 Sent Done Done false
3769 Acked Read 5 Sent Done Done false
3770 Acked Read 8 Sent Done Done false
3771 Acked Read 7 Sent Done Done false
3772 Acked Write 7 Sent Done Done false
3773 Acked Flush 0 Sent Done Done false
3774 Acked Write 7 Sent Done Done false
3775 Acked Read 8 Sent Done Done false
3776 Acked Read 3 Sent Done Done false
3777 Acked Write 10 Sent Done Done false
3778 Acked Read 9 Sent Done Done false
3779 Acked Write 5 Sent Done Done false
3780 Acked Read 10 Sent Done Done false
3781 Acked Read 3 Sent Done Done false
3782 Acked Read 5 Sent Done Done false
3783 Acked Write 5 Sent Done Done false
3784 Acked Write 3 Sent Done Done false
3785 Acked Read 1 Sent Done Done false
3786 Acked Read 10 Sent Done Done false
3787 Acked Write 6 Sent Done Done false
3788 Acked Read 4 Sent Done Done false
3789 Acked Read 10 Sent Done Done false
3790 Acked Write 9 Sent Done Done false
3791 Acked Write 8 Sent Done Done false
3792 Acked Flush 0 Sent Done Done false
3793 Acked Read 3 Sent Done Done false
3794 Acked Write 7 Sent Done Done false
3795 Acked Read 3 Sent Done Done false
3796 Acked Write 8 Sent Done Done false
3797 Acked Flush 0 Sent Done Done false
3798 Acked Write 2 Sent Done Done false
3799 Acked Write 6 Sent Done Done false
3800 Acked Write 7 Sent Done Done false
3801 Acked Read 3 Sent Done Done false
3802 Acked Read 1 Sent Done Done false
3803 Acked Read 9 Sent Done Done false
3804 Acked Write 8 Sent Done Done false
3805 Acked Write 6 Sent Done Done false
3806 Acked Read 5 Sent Done Done false
3807 Acked Write 1 Sent Done Done false
3808 Acked Read 8 Sent Done Done false
3809 Acked Read 4 Sent Done Done false
3810 Acked Read 8 Sent Done Done false
3811 Acked Write 6 Sent Done Done false
3812 Acked Write 4 Sent Done Done false
3813 Acked Flush 0 Sent Done Done false
3814 Acked Write 7 Sent Done Done false
3815 Acked Write 4 Sent Done Done false
3816 Acked Write 4 Sent Done Done false
3817 Acked Write 2 Sent Done Done false
3818 Acked Read 5 Sent Done Done false
3819 Acked Read 1 Sent Done Done false
3820 Acked Write 3 Sent Done Done false
3821 Acked Write 4 Sent Done Done false
3822 Acked Write 1 Sent Done Done false
3823 Acked Write 3 Sent Done Done false
3824 Acked Write 8 Sent Done Done false
3825 Acked Read 6 Sent Done Done false
3826 Acked Flush 0 Sent Done Done false
3827 Acked Flush 0 Sent Done Done false
3828 Acked Read 9 Sent Done Done false
3829 Acked Read 9 Sent Done Done false
3830 Acked Write 6 Sent Done Done false
3831 Acked Read 8 Sent Done Done false
3832 Acked Write 7 Sent Done Done false
3833 Acked Write 4 Sent Done Done false
3834 Acked Read 6 Sent Done Done false
3835 Acked Read 3 Sent Done Done false
3836 Acked Write 10 Sent Done Done false
3837 Acked Write 5 Sent Done Done false
3838 Acked Flush 0 Sent Done Done false
3839 Acked Write 2 Sent Done Done false
3840 Acked Read 5 Sent Done Done false
3841 Acked Read 1 Sent Done Done false
3842 Acked Read 3 Sent Done Done false
3843 Acked Read 2 Sent Done Done false
3844 Acked Write 5 Sent Done Done false
3845 Acked Write 4 Sent Done Done false
3846 Acked Read 4 Sent Done Done false
3847 Acked Read 6 Sent Done Done false
3848 Acked Write 5 Sent Done Done false
3849 Acked Write 1 Sent Done Done false
3850 Acked Read 5 Sent Done Done false
3851 Acked Write 7 Sent Done Done false
3852 Acked Read 6 Sent Done Done false
3853 Acked Write 7 Sent Done Done false
3854 Acked Write 10 Sent Done Done false
3855 Acked Flush 0 Sent Done Done false
3856 Acked Write 3 Sent Done Done false
3857 Acked Read 6 Sent Done Done false
3858 Acked Read 10 Sent Done Done false
3859 Acked Flush 0 Sent Done Done false
3860 Acked Write 2 Sent Done Done false
3861 Acked Write 9 Sent Done Done false
3862 Acked Write 1 Sent Done Done false
3863 Acked Read 3 Sent Done Done false
3864 Acked Write 1 Sent Done Done false
3865 Acked Write 1 Sent Done Done false
3866 Acked Read 8 Sent Done Done false
3867 Acked Read 4 Sent Done Done false
3868 Acked Read 8 Sent Done Done false
3869 Acked Read 2 Sent Done Done false
3870 Acked Read 9 Sent Done Done false
3871 Acked Read 2 Sent Done Done false
3872 Acked Write 10 Sent Done Done false
3873 Acked Read 5 Sent Done Done false
3874 Acked Read 1 Sent Done Done false
3875 Acked Read 1 Sent Done Done false
3876 Acked Read 2 Sent Done Done false
3877 Acked Read 6 Sent Done Done false
3878 Acked Write 10 Sent Done Done false
3879 Acked Read 1 Sent Done Done false
3880 Acked Read 8 Sent Done Done false
3881 Acked Write 3 Sent Done Done false
3882 Acked Write 1 Sent Done Done false
3883 Acked Read 7 Sent Done Done false
3884 Acked Read 1 Sent Done Done false
3885 Acked Read 3 Sent Done Done false
3886 Acked Write 7 Sent Done Done false
3887 Acked Read 5 Sent Done Done false
3888 Acked Read 1 Sent Done Done false
3889 Acked Read 5 Sent Done Done false
3890 Acked Write 9 Sent Done Done false
3891 Acked Flush 0 Sent Done Done false
3892 Acked Flush 0 Sent Done Done false
3893 Acked Read 6 Sent Done Done false
3894 Acked Write 8 Sent Done Done false
3895 Acked Write 7 Sent Done Done false
3896 Acked Write 7 Sent Done Done false
3897 Acked Write 3 Sent Done Done false
3898 Acked Write 6 Sent Done Done false
3899 Acked Write 6 Sent Done Done false
3900 Acked Write 9 Sent Done Done false
3901 Acked Read 8 Sent Done Done false
3902 Acked Read 9 Sent Done Done false
3903 Acked Read 8 Sent Done Done false
3904 Acked Write 2 Sent Done Done false
3905 Acked Read 4 Sent Done Done false
3906 Acked Read 8 Sent Done Done false
3907 Acked Read 5 Sent Done Done false
3908 Acked Read 3 Sent Done Done false
3909 Acked Write 9 Sent Done Done false
3910 Acked Read 6 Sent Done Done false
3911 Acked Write 9 Sent Done Done false
3912 Acked Read 7 Sent Done Done false
3913 Acked Read 3 Sent Done Done false
3914 Acked Read 1 Sent Done Done false
3915 Acked Write 8 Sent Done Done false
3916 Acked Read 3 Sent Done Done false
3917 Acked Read 1 Sent Done Done false
3918 Acked Read 8 Sent Done Done false
3919 Acked Write 1 Sent Done Done false
3920 Acked Flush 0 Sent Done Done false
3921 Acked Read 3 Sent Done Done false
3922 Acked Read 6 Sent Done Done false
3923 Acked Write 10 Sent Done Done false
3924 Acked Write 1 Sent Done Done false
3925 Acked Read 7 Sent Done Done false
3926 Acked Write 10 Sent Done Done false
3927 Acked Flush 0 Sent Done Done false
3928 Acked Read 10 Sent Done Done false
3929 Acked Read 1 Sent Done Done false
3930 Acked Read 4 Sent Done Done false
3931 Acked Write 5 Sent Done Done false
3932 Acked Read 9 Sent Done Done false
3933 Acked Flush 0 Sent Done Done false
3934 Acked Read 5 Sent Done Done false
3935 Acked Read 9 Sent Done Done false
3936 Acked Write 9 Sent Done Done false
3937 Acked Read 2 Sent Done Done false
3938 Acked Write 10 Sent Done Done false
3939 Acked Write 4 Sent Done Done false
3940 Acked Read 3 Sent Done Done false
3941 Acked Write 5 Sent Done Done false
3942 Acked Read 3 Sent Done Done false
3943 Acked Write 5 Sent Done Done false
3944 Acked Write 7 Sent Done Done false
3945 Acked Read 3 Sent Done Done false
3946 Acked Write 9 Sent Done Done false
3947 Acked Read 1 Sent Done Done false
3948 Acked Flush 0 Sent Done Done false
3949 Acked Read 8 Sent Done Done false
3950 Acked Write 2 Sent Done Done false
3951 Acked Flush 0 Sent Done Done false
3952 Acked Read 5 Sent Done Done false
3953 Acked Flush 0 Sent Done Done false
3954 Acked Write 6 Sent Done Done false
3955 Acked Read 1 Sent Done Done false
3956 Acked Write 9 Sent Done Done false
3957 Acked Write 3 Sent Done Done false
3958 Acked Write 3 Sent Done Done false
3959 Acked Read 4 Sent Done Done false
3960 Acked Read 3 Sent Done Done false
3961 Acked Write 2 Sent Done Done false
3962 Acked Read 1 Sent Done Done false
3963 Acked Read 3 Sent Done Done false
3964 Acked Read 10 Sent Done Done false
3965 Acked Write 10 Sent Done Done false
3966 Acked Write 2 Sent Done Done false
3967 Acked Read 9 Sent Done Done false
3968 Acked Write 5 Sent Done Done false
3969 Acked Write 6 Sent Done Done false
3970 Acked Write 7 Sent Done Done false
3971 Acked Read 7 Sent Done Done false
3972 Acked Read 7 Sent Done Done false
3973 Acked Read 2 Sent Done Done false
3974 Acked Read 9 Sent Done Done false
3975 Acked Read 1 Sent Done Done false
3976 Acked Read 3 Sent Done Done false
3977 Acked Read 2 Sent Done Done false
3978 Acked Flush 0 Sent Done Done false
3979 Acked Flush 0 Sent Done Done false
3980 Acked Read 10 Sent Done Done false
3981 Acked Write 4 Sent Done Done false
3982 Acked Read 6 Sent Done Done false
3983 Acked Read 5 Sent Done Done false
3984 Acked Read 4 Sent Done Done false
3985 Acked Write 3 Sent Done Done false
3986 Acked Read 6 Sent Done Done false
3987 Acked Read 9 Sent Done Done false
3988 Acked Flush 0 Sent Done Done false
3989 Acked Flush 0 Sent Done Done false
3990 Acked Read 5 Sent Done Done false
3991 Acked Write 2 Sent Done Done false
3992 Acked Write 3 Sent Done Done false
3993 Acked Read 2 Sent Done Done false
3994 Acked Write 3 Sent Done Done false
3995 Acked Read 4 Sent Done Done false
3996 Acked Read 8 Sent Done Done false
3997 Acked Write 5 Sent Done Done false
3998 Acked Write 1 Sent Done Done false
3999 Acked Write 4 Sent Done Done false
4000 Acked Read 9 Sent Done Done false
4001 Acked Read 6 Sent Done Done false
4002 Acked Read 8 Sent Done Done false
4003 Acked Write 9 Sent Done Done false
4004 Acked Flush 0 Sent Done Done false
4005 Acked Write 3 Sent Done Done false
4006 Acked Write 9 Sent Done Done false
4007 Acked Read 1 Sent Done Done false
4008 Acked Write 10 Sent Done Done false
4009 Acked Read 1 Sent Done Done false
4010 Acked Read 2 Sent Done Done false
4011 Acked Read 10 Sent Done Done false
4012 Acked Read 4 Sent Done Done false
4013 Acked Write 4 Sent Done Done false
4014 Acked Read 1 Sent Done Done false
4015 Acked Write 7 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3714)) Some(JobId(4004)) Some(JobId(4004))
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 4015 4014 4013 4012 4011
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:38.873231475Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:39.599326471Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:40.600609187Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:41.601560942Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3715 Acked Read 7 Sent Done Done false
3716 Acked Read 2 Sent Done Done false
3717 Acked Write 8 Sent Done Done false
3718 Acked Write 4 Sent Done Done false
3719 Acked Read 8 Sent Done Done false
3720 Acked Flush 0 Sent Done Done false
3721 Acked Read 9 Sent Done Done false
3722 Acked Read 4 Sent Done Done false
3723 Acked Read 6 Sent Done Done false
3724 Acked Read 5 Sent Done Done false
3725 Acked Read 9 Sent Done Done false
3726 Acked Write 5 Sent Done Done false
3727 Acked Read 10 Sent Done Done false
3728 Acked Read 7 Sent Done Done false
3729 Acked Read 10 Sent Done Done false
3730 Acked Write 4 Sent Done Done false
3731 Acked Write 8 Sent Done Done false
3732 Acked Read 2 Sent Done Done false
3733 Acked Write 2 Sent Done Done false
3734 Acked Write 1 Sent Done Done false
3735 Acked Flush 0 Sent Done Done false
3736 Acked Read 10 Sent Done Done false
3737 Acked Read 1 Sent Done Done false
3738 Acked Read 6 Sent Done Done false
3739 Acked Read 6 Sent Done Done false
3740 Acked Write 7 Sent Done Done false
3741 Acked Read 10 Sent Done Done false
3742 Acked Write 9 Sent Done Done false
3743 Acked Read 8 Sent Done Done false
3744 Acked Write 6 Sent Done Done false
3745 Acked Write 6 Sent Done Done false
3746 Acked Flush 0 Sent Done Done false
3747 Acked Read 6 Sent Done Done false
3748 Acked Read 6 Sent Done Done false
3749 Acked Read 8 Sent Done Done false
3750 Acked Read 1 Sent Done Done false
3751 Acked Write 2 Sent Done Done false
3752 Acked Flush 0 Sent Done Done false
3753 Acked Read 1 Sent Done Done false
3754 Acked Read 4 Sent Done Done false
3755 Acked Read 10 Sent Done Done false
3756 Acked Write 9 Sent Done Done false
3757 Acked Write 10 Sent Done Done false
3758 Acked Read 7 Sent Done Done false
3759 Acked Write 10 Sent Done Done false
3760 Acked Write 3 Sent Done Done false
3761 Acked Read 3 Sent Done Done false
3762 Acked Read 1 Sent Done Done false
3763 Acked Read 9 Sent Done Done false
3764 Acked Read 4 Sent Done Done false
3765 Acked Write 4 Sent Done Done false
3766 Acked Write 4 Sent Done Done false
3767 Acked Read 2 Sent Done Done false
3768 Acked Write 10 Sent Done Done false
3769 Acked Read 5 Sent Done Done false
3770 Acked Read 8 Sent Done Done false
3771 Acked Read 7 Sent Done Done false
3772 Acked Write 7 Sent Done Done false
3773 Acked Flush 0 Sent Done Done false
3774 Acked Write 7 Sent Done Done false
3775 Acked Read 8 Sent Done Done false
3776 Acked Read 3 Sent Done Done false
3777 Acked Write 10 Sent Done Done false
3778 Acked Read 9 Sent Done Done false
3779 Acked Write 5 Sent Done Done false
3780 Acked Read 10 Sent Done Done false
3781 Acked Read 3 Sent Done Done false
3782 Acked Read 5 Sent Done Done false
3783 Acked Write 5 Sent Done Done false
3784 Acked Write 3 Sent Done Done false
3785 Acked Read 1 Sent Done Done false
3786 Acked Read 10 Sent Done Done false
3787 Acked Write 6 Sent Done Done false
3788 Acked Read 4 Sent Done Done false
3789 Acked Read 10 Sent Done Done false
3790 Acked Write 9 Sent Done Done false
3791 Acked Write 8 Sent Done Done false
3792 Acked Flush 0 Sent Done Done false
3793 Acked Read 3 Sent Done Done false
3794 Acked Write 7 Sent Done Done false
3795 Acked Read 3 Sent Done Done false
3796 Acked Write 8 Sent Done Done false
3797 Acked Flush 0 Sent Done Done false
3798 Acked Write 2 Sent Done Done false
3799 Acked Write 6 Sent Done Done false
3800 Acked Write 7 Sent Done Done false
3801 Acked Read 3 Sent Done Done false
3802 Acked Read 1 Sent Done Done false
3803 Acked Read 9 Sent Done Done false
3804 Acked Write 8 Sent Done Done false
3805 Acked Write 6 Sent Done Done false
3806 Acked Read 5 Sent Done Done false
3807 Acked Write 1 Sent Done Done false
3808 Acked Read 8 Sent Done Done false
3809 Acked Read 4 Sent Done Done false
3810 Acked Read 8 Sent Done Done false
3811 Acked Write 6 Sent Done Done false
3812 Acked Write 4 Sent Done Done false
3813 Acked Flush 0 Sent Done Done false
3814 Acked Write 7 Sent Done Done false
3815 Acked Write 4 Sent Done Done false
3816 Acked Write 4 Sent Done Done false
3817 Acked Write 2 Sent Done Done false
3818 Acked Read 5 Sent Done Done false
3819 Acked Read 1 Sent Done Done false
3820 Acked Write 3 Sent Done Done false
3821 Acked Write 4 Sent Done Done false
3822 Acked Write 1 Sent Done Done false
3823 Acked Write 3 Sent Done Done false
3824 Acked Write 8 Sent Done Done false
3825 Acked Read 6 Sent Done Done false
3826 Acked Flush 0 Sent Done Done false
3827 Acked Flush 0 Sent Done Done false
3828 Acked Read 9 Sent Done Done false
3829 Acked Read 9 Sent Done Done false
3830 Acked Write 6 Sent Done Done false
3831 Acked Read 8 Sent Done Done false
3832 Acked Write 7 Sent Done Done false
3833 Acked Write 4 Sent Done Done false
3834 Acked Read 6 Sent Done Done false
3835 Acked Read 3 Sent Done Done false
3836 Acked Write 10 Sent Done Done false
3837 Acked Write 5 Sent Done Done false
3838 Acked Flush 0 Sent Done Done false
3839 Acked Write 2 Sent Done Done false
3840 Acked Read 5 Sent Done Done false
3841 Acked Read 1 Sent Done Done false
3842 Acked Read 3 Sent Done Done false
3843 Acked Read 2 Sent Done Done false
3844 Acked Write 5 Sent Done Done false
3845 Acked Write 4 Sent Done Done false
3846 Acked Read 4 Sent Done Done false
3847 Acked Read 6 Sent Done Done false
3848 Acked Write 5 Sent Done Done false
3849 Acked Write 1 Sent Done Done false
3850 Acked Read 5 Sent Done Done false
3851 Acked Write 7 Sent Done Done false
3852 Acked Read 6 Sent Done Done false
3853 Acked Write 7 Sent Done Done false
3854 Acked Write 10 Sent Done Done false
3855 Acked Flush 0 Sent Done Done false
3856 Acked Write 3 Sent Done Done false
3857 Acked Read 6 Sent Done Done false
3858 Acked Read 10 Sent Done Done false
3859 Acked Flush 0 Sent Done Done false
3860 Acked Write 2 Sent Done Done false
3861 Acked Write 9 Sent Done Done false
3862 Acked Write 1 Sent Done Done false
3863 Acked Read 3 Sent Done Done false
3864 Acked Write 1 Sent Done Done false
3865 Acked Write 1 Sent Done Done false
3866 Acked Read 8 Sent Done Done false
3867 Acked Read 4 Sent Done Done false
3868 Acked Read 8 Sent Done Done false
3869 Acked Read 2 Sent Done Done false
3870 Acked Read 9 Sent Done Done false
3871 Acked Read 2 Sent Done Done false
3872 Acked Write 10 Sent Done Done false
3873 Acked Read 5 Sent Done Done false
3874 Acked Read 1 Sent Done Done false
3875 Acked Read 1 Sent Done Done false
3876 Acked Read 2 Sent Done Done false
3877 Acked Read 6 Sent Done Done false
3878 Acked Write 10 Sent Done Done false
3879 Acked Read 1 Sent Done Done false
3880 Acked Read 8 Sent Done Done false
3881 Acked Write 3 Sent Done Done false
3882 Acked Write 1 Sent Done Done false
3883 Acked Read 7 Sent Done Done false
3884 Acked Read 1 Sent Done Done false
3885 Acked Read 3 Sent Done Done false
3886 Acked Write 7 Sent Done Done false
3887 Acked Read 5 Sent Done Done false
3888 Acked Read 1 Sent Done Done false
3889 Acked Read 5 Sent Done Done false
3890 Acked Write 9 Sent Done Done false
3891 Acked Flush 0 Sent Done Done false
3892 Acked Flush 0 Sent Done Done false
3893 Acked Read 6 Sent Done Done false
3894 Acked Write 8 Sent Done Done false
3895 Acked Write 7 Sent Done Done false
3896 Acked Write 7 Sent Done Done false
3897 Acked Write 3 Sent Done Done false
3898 Acked Write 6 Sent Done Done false
3899 Acked Write 6 Sent Done Done false
3900 Acked Write 9 Sent Done Done false
3901 Acked Read 8 Sent Done Done false
3902 Acked Read 9 Sent Done Done false
3903 Acked Read 8 Sent Done Done false
3904 Acked Write 2 Sent Done Done false
3905 Acked Read 4 Sent Done Done false
3906 Acked Read 8 Sent Done Done false
3907 Acked Read 5 Sent Done Done false
3908 Acked Read 3 Sent Done Done false
3909 Acked Write 9 Sent Done Done false
3910 Acked Read 6 Sent Done Done false
3911 Acked Write 9 Sent Done Done false
3912 Acked Read 7 Sent Done Done false
3913 Acked Read 3 Sent Done Done false
3914 Acked Read 1 Sent Done Done false
3915 Acked Write 8 Sent Done Done false
3916 Acked Read 3 Sent Done Done false
3917 Acked Read 1 Sent Done Done false
3918 Acked Read 8 Sent Done Done false
3919 Acked Write 1 Sent Done Done false
3920 Acked Flush 0 Sent Done Done false
3921 Acked Read 3 Sent Done Done false
3922 Acked Read 6 Sent Done Done false
3923 Acked Write 10 Sent Done Done false
3924 Acked Write 1 Sent Done Done false
3925 Acked Read 7 Sent Done Done false
3926 Acked Write 10 Sent Done Done false
3927 Acked Flush 0 Sent Done Done false
3928 Acked Read 10 Sent Done Done false
3929 Acked Read 1 Sent Done Done false
3930 Acked Read 4 Sent Done Done false
3931 Acked Write 5 Sent Done Done false
3932 Acked Read 9 Sent Done Done false
3933 Acked Flush 0 Sent Done Done false
3934 Acked Read 5 Sent Done Done false
3935 Acked Read 9 Sent Done Done false
3936 Acked Write 9 Sent Done Done false
3937 Acked Read 2 Sent Done Done false
3938 Acked Write 10 Sent Done Done false
3939 Acked Write 4 Sent Done Done false
3940 Acked Read 3 Sent Done Done false
3941 Acked Write 5 Sent Done Done false
3942 Acked Read 3 Sent Done Done false
3943 Acked Write 5 Sent Done Done false
3944 Acked Write 7 Sent Done Done false
3945 Acked Read 3 Sent Done Done false
3946 Acked Write 9 Sent Done Done false
3947 Acked Read 1 Sent Done Done false
3948 Acked Flush 0 Sent Done Done false
3949 Acked Read 8 Sent Done Done false
3950 Acked Write 2 Sent Done Done false
3951 Acked Flush 0 Sent Done Done false
3952 Acked Read 5 Sent Done Done false
3953 Acked Flush 0 Sent Done Done false
3954 Acked Write 6 Sent Done Done false
3955 Acked Read 1 Sent Done Done false
3956 Acked Write 9 Sent Done Done false
3957 Acked Write 3 Sent Done Done false
3958 Acked Write 3 Sent Done Done false
3959 Acked Read 4 Sent Done Done false
3960 Acked Read 3 Sent Done Done false
3961 Acked Write 2 Sent Done Done false
3962 Acked Read 1 Sent Done Done false
3963 Acked Read 3 Sent Done Done false
3964 Acked Read 10 Sent Done Done false
3965 Acked Write 10 Sent Done Done false
3966 Acked Write 2 Sent Done Done false
3967 Acked Read 9 Sent Done Done false
3968 Acked Write 5 Sent Done Done false
3969 Acked Write 6 Sent Done Done false
3970 Acked Write 7 Sent Done Done false
3971 Acked Read 7 Sent Done Done false
3972 Acked Read 7 Sent Done Done false
3973 Acked Read 2 Sent Done Done false
3974 Acked Read 9 Sent Done Done false
3975 Acked Read 1 Sent Done Done false
3976 Acked Read 3 Sent Done Done false
3977 Acked Read 2 Sent Done Done false
3978 Acked Flush 0 Sent Done Done false
3979 Acked Flush 0 Sent Done Done false
3980 Acked Read 10 Sent Done Done false
3981 Acked Write 4 Sent Done Done false
3982 Acked Read 6 Sent Done Done false
3983 Acked Read 5 Sent Done Done false
3984 Acked Read 4 Sent Done Done false
3985 Acked Write 3 Sent Done Done false
3986 Acked Read 6 Sent Done Done false
3987 Acked Read 9 Sent Done Done false
3988 Acked Flush 0 Sent Done Done false
3989 Acked Flush 0 Sent Done Done false
3990 Acked Read 5 Sent Done Done false
3991 Acked Write 2 Sent Done Done false
3992 Acked Write 3 Sent Done Done false
3993 Acked Read 2 Sent Done Done false
3994 Acked Write 3 Sent Done Done false
3995 Acked Read 4 Sent Done Done false
3996 Acked Read 8 Sent Done Done false
3997 Acked Write 5 Sent Done Done false
3998 Acked Write 1 Sent Done Done false
3999 Acked Write 4 Sent Done Done false
4000 Acked Read 9 Sent Done Done false
4001 Acked Read 6 Sent Done Done false
4002 Acked Read 8 Sent Done Done false
4003 Acked Write 9 Sent Done Done false
4004 Acked Flush 0 Sent Done Done false
4005 Acked Write 3 Sent Done Done false
4006 Acked Write 9 Sent Done Done false
4007 Acked Read 1 Sent Done Done false
4008 Acked Write 10 Sent Done Done false
4009 Acked Read 1 Sent Done Done false
4010 Acked Read 2 Sent Done Done false
4011 Acked Read 10 Sent Done Done false
4012 Acked Read 4 Sent Done Done false
4013 Acked Write 4 Sent Done Done false
4014 Acked Read 1 Sent Done Done false
4015 Acked Write 7 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(3714)) Some(JobId(4004)) Some(JobId(4004))
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 4015 4014 4013 4012 4011
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:42.876229769Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:43.605229799Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.131463554Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40982)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.131672124Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.132924963Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133447648Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133470263Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(3714))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133484849Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133498296Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(3714))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133511633Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(3714))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:44.133748226Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4016 4015 4014 4013 4012
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:03:46.378 INFO request completed[8820] Got stop action so:false kr:false
, latency_us: 658, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 57ed5b77-071d-43bf-9849-8223cee10e6f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:46.379471515Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:46.379560893Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:03:46.379 INFO request completed, latency_us: 137, response_code{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:46.379584088Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
: 200, uri: /state/cid/1, method: GET, req_id: 1c250499-10a9-46c9-85fb-45f1177886cf, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:46.379686053Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:03:50.382 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6681ed0f-09ae-4776-9463-ca1faa5e9eb5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 3543 len 40960
003/300 Read block 3062 len 40960
004/300 Read block 1554 len 20480
005/300 Write block 3546 len 24576 data: 2 2 2 2 2 2
006/300 Write block 2511 len 8192 data: 5 6
007/300 Read block 2634 len 36864
008/300 Write block 502 len 32768 data: 3 3 3 3 3 3 3 4
009/300 Write block 4574 len 16384 data: 3 3 3 2
010/300 Flush
011/300 Write block 3822 len 4096 data: 2
012/300 Write block 4994 len 24576 data: 2 2 2 2 2 2
013/300 Flush
014/300 Read block 2308 len 32768
015/300 Read block 3618 len 32768
016/300 Read block 103 len 12288
017/300 Write block 75 len 4096 data: 5
018/300 Read block 3761 len 8192
019/300 Write block 4206 len 4096 data: 4
020/300 Write block 3833 len 12288 data: 2 2 2
021/300 Read block 707 len 16384
022/300 Read block 3218 len 20480
023/300 Write block 1493 len 16384 data: 4 3 3 3
024/300 Read block 3321 len 40960
025/300 Write block 4684 len 16384 data: 2 2 2 2
026/300 Write block 2140 len 24576 data: 3 2 2 2 2 2
027/300 Write block 444 len 8192 data: 8 8
028/300 Flush
029/300 Write block 1681 len 16384 data: 3 4 5 5
030/300 Read block 1177 len 28672
031/300 Read block 1919 len 32768
032/300 Write block 3748 len 32768 data: 3 3 3 4 3 3 2 2
033/300 Write block 641 len 36864 data: 3 3 3 3 3 3 4 4 3
034/300 Write block 2200 len 40960 data: 4 4 5 3 3 2 2 2 2 4
035/300 Read block 4721 len 32768
036/300 Read block 3932 len 24576
037/300 Write block 3450 len 40960 data: 2 2 3 4 4 4 4 4 4 4
038/300 Write block 150 len 8192 data: 4 4
039/300 Write block 2058 len 16384 data: 3 3 3 4
040/300 Read block 1926 len 36864
041/300 Flush
042/300 Read block 2628 len 36864
043/300 Write block 4923 len 24576 data: 2 2 3 4 3 2
044/300 Read block 4419 len 16384
045/300 Read block 2021 len 8192
046/300 Read block 2728 len 16384
047/300 Write block 4965 len 4096 data: 3
048/300 Write block 1184 len 16384 data: 3 3 4 3
049/300 Read block 3802 len 20480
050/300 Flush
051/300 Read block 1430 len 20480
052/300 Read block 1115 len 12288
053/300 Read block 681 len 20480
054/300 Write block 4197 len 12288 data: 3 3 3
055/300 Write block 1380 len 32768 data: 2 3 3 4 4 4 4 4
056/300 Write block 928 len 40960 data: 5 5 6 6 6 5 4 3 2 3
057/300 Write block 4473 len 36864 data: 2 2 3 3 3 3 4 3 3
058/300 Write block 3860 len 36864 data: 3 4 4 4 5 4 4 4 4
059/300 Read block 334 len 36864
060/300 Read block 3814 len 16384
061/300 Write block 1173 len 12288 data: 5 6 5
062/300 Write block 3970 len 24576 data: 3 3 3 3 4 4
063/300 Read block 1547 len 32768
064/300 Flush
065/300 Read block 2863 len 40960
066/300 Flush
067/300 Read block 2702 len 40960
068/300 Read block 3603 len 4096
069/300 Write block 1791 len 8192 data: 3 3
070/300 Read block 715 len 32768
071/300 Read block 4647 len 4096
072/300 Write block 18 len 32768 data: 3 3 3 3 3 3 3 5
073/300 Flush
074/300 Write block 2357 len 32768 data: 3 3 3 3 3 4 5 6
075/300 Write block 4354 len 28672 data: 4 4 3 4 4 4 2
076/300 Read block 702 len 8192
077/300 Write block 1312 len 8192 data: 4 4
078/300 Write block 2264 len 20480 data: 3 3 3 3 3
079/300 Write block 1155 len 24576 data: 3 3 3 2 2 3
080/300 Flush
081/300 Write block 1077 len 24576 data: 3 4 4 6 4 3
082/300 Flush
083/300 Write block 3371 len 16384 data: 3 2 2 2
084/300 Read block 2518 len 24576
085/300 Read block 1176 len 36864
086/300 Write block 2087 len 20480 data: 3 3 3 3 3
087/300 Write block 4627 len 20480 data: 4 4 3 4 5
088/300 Write block 4971 len 36864 data: 3 3 3 3 2 2 2 4 4
089/300 Read block 2126 len 32768
090/300 Flush
091/300 Write block 2232 len 24576 data: 5 4 7 5 6 5
092/300 Read block 2250 len 16384
093/300 Read block 1335 len 28672
094/300 Read block 1258 len 40960
095/300 Read block 3321 len 28672
096/300 Flush
097/300 Read block 4852 len 36864
098/300 Read block 4175 len 8192
099/300 Write block 3047 len 20480 data: 2 2 3 3 2
100/300 Read block 52 len 28672
101/300 Write block 3603 len 20480 data: 2 2 3 3 3
102/300 Flush
103/300 Read block 4032 len 28672
104/300 Flush
105/300 Write block 3607 len 24576 data: 4 3 3 3 3 3
106/300 Read block 2934 len 8192
107/300 Read block 722 len 36864
108/300 Read block 4388 len 16384
109/300 Write block 3621 len 8192 data: 2 2
110/300 Read block 1371 len 40960
111/300 Read block 513 len 20480
112/300 Write block 3692 len 8192 data: 3 2
113/300 Read block 1656 len 24576
114/300 Write block 4724 len 16384 data: 3 3 3 3
115/300 Write block 3794 len 12288 data: 3 3 3
116/300 Read block 441 len 16384
117/300 Write block 4143 len 36864 data: 4 3 4 4 4 4 4 4 3
118/300 Write block 2814 len 32768 data: 5 5 5 5 4 4 3 2
119/300 Write block 2424 len 32768 data: 3 3 3 3 2 2 3 3
120/300 Read block 2992 len 28672
121/300 Read block 4587 len 8192
122/300 Read block 3603 len 40960
123/300 Read block 1 len 12288
124/300 Read block 4155 len 12288
125/300 Write block 2975 len 36864 data: 2 2 2 2 2 2 3 3 4
126/300 Write block 3840 len 24576 data: 3 3 3 2 3 3
127/300 Read block 4119 len 36864
128/300 Read block 129 len 40960
129/300 Write block 4572 len 16384 data: 3 3 4 4
130/300 Read block 3038 len 24576
131/300 Read block 847 len 16384
132/300 Read block 210 len 16384
133/300 Write block 2936 len 20480 data: 2 3 4 4 4
134/300 Write block 4572 len 16384 data: 4 4 5 5
135/300 Write block 872 len 12288 data: 3 3 3
136/300 Write block 1983 len 36864 data: 4 4 3 3 3 4 4 3 3
137/300 Flush
138/300 Read block 3417 len 12288
139/300 Read block 4481 len 16384
140/300 Read block 128 len 16384
141/300 Read block 1959 len 24576
142/300 Read block 2716 len 36864
143/300 Flush
144/300 Read block 3398 len 20480
145/300 Read block 3808 len 24576
146/300 Write block 743 len 32768 data: 2 3 3 3 3 4 5 5
147/300 Read block 483 len 8192
148/300 Read block 28 len 16384
149/300 Write block 607 len 8192 data: 2 2
150/300 Write block 2214 len 28672 data: 7 7 7 7 7 5 3
151/300 Write block 395 len 28672 data: 2 2 2 3 3 3 3
152/300 Write block 2477 len 36864 data: 4 4 4 4 3 3 4 5 5
153/300 Flush
154/300 Read block 383 len 32768
155/300 Read block 3488 len 24576
156/300 Read block 1603 len 36864
157/300 Write block 3301 len 36864 data: 5 6 5 5 6 6 6 6 5
158/300 Read block 4578 len 40960
159/300 Read block 953 len 32768
160/300 Write block 1663 len 16384 data: 3 3 4 4
161/300 Write block 2416 len 36864 data: 3 3 3 3 3 3 3 3 4
162/300 Read block 1613 len 4096
163/300 Write block 4744 len 20480 data: 3 3 2 2 2
164/300 Read block 639 len 28672
165/300 Write block 4481 len 40960 data: 4 3 3 3 3 3 3 2 2 2
166/300 Write block 2006 len 40960 data: 2 2 2 2 2 2 2 2 2 2
167/300 Write block 3343 len 24576 data: 4 4 4 4 3 2
168/300 Write block 4895 len 36864 data: 5 5 5 6 5 5 4 4 5
169/300 Write block 2976 len 28672 data: 3 3 3 3 3 4 4
170/300 Write block 3066 len 32768 data: 4 5 4 4 4 3 3 4
171/300 Read block 2045 len 36864
172/300 Read block 1886 len 36864
173/300 Write block 2176 len 16384 data: 3 3 4 6
174/300 Read block 4243 len 4096
175/300 Write block 1046 len 40960 data: 4 3 3 4 4 3 4 4 5 4
176/300 Write block 1025 len 16384 data: 5 5 5 3
177/300 Write block 2233 len 4096 data: 5
178/300 Read block 2655 len 12288
179/300 Write block 1145 len 36864 data: 4 3 2 3 3 3 3 3 3
180/300 Write block 3030 len 40960 data: 4 4 4 4 4 3 3 3 3 3
181/300 Write block 4029 len 16384 data: 3 3 2 2
182/300 Write block 4627 len 12288 data: 5 5 4
183/300 Read block 2602 len 12288
184/300 Flush
185/300 Write block 4400 len 24576 data: 2 2 3 3 3 6
186/300 Write block 2712 len 12288 data: 5 7 6
187/300 Write block 2759 len 8192 data: 4 4
188/300 Flush
189/300 Write block 546 len 32768 data: 2 3 3 3 3 4 5 5
190/300 Write block 4235 len 12288 data: 3 3 3
191/300 Read block 1269 len 8192
192/300 Read block 2712 len 32768
193/300 Write block 2086 len 4096 data: 3
194/300 Read block 3413 len 36864
195/300 Read block 2717 len 40960
196/300 Write block 2841 len 28672 data: 3 3 3 3 3 4 4
197/300 Write block 2025 len 40960 data: 2 2 2 4 4 3 3 3 3 2
198/300 Read block 1901 len 12288
199/300 Read block 289 len 36864
200/300 Read block 1163 len 12288
201/300 Read block 2397 len 20480
202/300 Write block 2194 len 28672 data: 3 4 4 4 3 3 5
203/300 Read block 2448 len 36864
204/300 Flush
205/300 Write block 1955 len 4096 data: 4
206/300 Read block 71 len 20480
207/300 Write block 1153 len 28672 data: 4 3 4 4 4 3 3
208/300 Write block 1825 len 20480 data: 3 3 3 4 4
209/300 Read block 4096 len 8192
210/300 Read block 4768 len 20480
211/300 Read block 3887 len 24576
212/300 Read block 1806 len 16384
213/300 Write block 2437 len 24576 data: 2 2 3 4 3 4
214/300 Write block 2245 len 12288 data: 2 2 2
215/300 Read block 1023 len 28672
216/300 Read block 1254 len 24576
217/300 Read block 3610 len 32768
218/300 Read block 1199 len 12288
219/300 Write block 3182 len 32768 data: 2 2 2 2 2 2 2 2
220/300 Write block 2672 len 40960 data: 5 4 5 5 4 3 4 4 4 3
221/300 Write block 3368 len 4096 data: 3
222/300 Read block 82 len 28672
223/300 Write block 2835 len 40960 data: 2 2 2 2 2 2 4 4 4 4
224/300 Read block 4031 len 24576
225/300 Write block 4647 len 32768 data: 6 5 4 3 3 5 5 5
226/300 Read block 3139 len 40960
227/300 Write block 1249 len 40960 data: 3 3 2 2 3 4 5 5 5 4
228/300 Write block 2275 len 28672 data: 5 5 5 6 6 6 5
229/300 Write block 3338 len 24576 data: 5 5 4 5 4 5
230/300 Flush
231/300 Read block 4927 len 4096
232/300 Flush
233/300 Write block 4428 len 16384 data: 4 4 4 4
234/300 Write block 4526 len 8192 data: 6 6
235/300 Read block 2516 len 32768
236/300 Flush
237/300 Write block 2773 len 36864 data: 2 2 2 3 3 3 4 4 4
238/300 Write block 4169 len 24576 data: 2 2 2 2 2 2
239/300 Read block 3602 len 36864
240/300 Read block 2095 len 28672
241/300 Write block 495 len 8192 data: 4 4
242/300 Write block 1487 len 40960 data: 5 6 6 6 6 6 5 4 4 4
243/300 Read block 1281 len 28672
244/300 Flush
245/300 Write block 119 len 24576 data: 2 2 2 3 3 3
246/300 Read block 945 len 32768
247/300 Read block 4500 len 16384
248/300 Read block 501 len 12288
249/300 Write block 4031 len 32768 data: 3 3 2 2 2 2 2 2
250/300 Read block 312 len 12288
251/300 Write block 505 len 8192 data: 4 4
252/300 Read block 1781 len 40960
253/300 Write block 3502 len 16384 data: 3 3 2 2
254/300 Write block 2663 len 32768 data: 4 5 5 5 5 5 5 6
255/300 Read block 4528 len 16384
256/300 Flush
257/300 Write block 676 len 4096 data: 5
258/300 Read block 1819 len 16384
259/300 Read block 1252 len 8192
260/300 Read block 3154 len 8192
261/300 Read block 2040 len 4096
262/300 Read block 803 len 20480
263/300 Write block 2188 len 4096 data: 3
264/300 Flush
265/300 Read block 3537 len 4096
266/300 Write block 1009 len 36864 data: 2 2 2 2 2 2 2 3 4
267/300 Flush
268/300 Write block 4174 len 12288 data: 3 2 2
269/300 Write block 4637 len 24576 data: 4 4 4 3 2 3
270/300 Read block 4661 len 32768
271/300 Write block 3344 len 16384 data: 5 5 5 4
272/300 Write block 3956 len 32768 data: 3 3 3 3 3 4 3 2
273/300 Read block 2350 len 12288
274/300 Write block 2883 len 28672 data: 3 3 4 4 4 4 4
275/300 Read block 3973 len 8192
276/300 Write block 3535 len 16384 data: 3 5 4 4
277/300 Read block 3746 len 16384
278/300 Read block 444 len 4096
279/300 Write block 3154 len 28672 data: 2 2 3 4 5 5 5
280/300 Read block 2245 len 32768
281/300 Read block 1770 len 36864
282/300 Write block 3320 len 24576 data: 3 3 4 4 3 2
283/300 Read block 4940 len 20480
284/300 Write block 3692 len 4096 data: 4
285/300 Write block 936 len 40960 data: 3 4 4 4 4 4 4 4 4 3
286/300 Read block 1233 len 8192
287/300 Write block 47 len 28672 data: 3 3 3 2 3 3 4
288/300 Read block 957 len 28672
289/300 Read block 963 len 4096
290/300 Read block 851 len 12288
291/300 Read block 306 len 12288
292/300 Read block 888 len 36864
293/300 Write block 2137 len 20480 data: 3 2 3 4 3
294/300 Read block 1710 len 8192
295/300 Flush
296/300 Write block 4925 len 16384 data: 4 5 4 3
297/300 Write block 256 len 4096 data: 2
298/300 Read block 875 len 36864
299/300 Write block 1874 len 36864 data: 4 4 4 5 5 4 4 4 3
300/300 Read block 1387 len 24576
301/300 Write block 1493 len 12288 data: 6 5 5
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:03:50.619 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/1, method: POST, req_id: 396e01ad-cb0a-4acc-9ddd-bd4e43d494fb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Flush 0 Done Sent Done false
4018 Acked Read 10 Done Sent Done false
4019 Acked Read 10 Done Sent Done false
4020 Acked Read 5 Done Sent Done false
4021 Acked Write 6 Done Sent Done false
4022 Acked Write 2 Done Sent Done false
4023 Acked Read 9 Done Sent Done false
4024 Acked Write 8 Done Sent Done false
4025 Acked Write 4 Done Sent Done false
4026 Acked Flush 0 Done Sent Done false
4027 Acked Write 1 Done Sent Done false
4028 Acked Write 6 Done Sent Done false
4029 Acked Flush 0 Done Sent Done false
4030 Acked Read 8 Done Sent Done false
4031 Acked Read 8 Done Sent Done false
4032 Acked Read 3 Done Sent Done false
4033 Acked Write 1 Done Sent Done false
4034 Acked Read 2 Done Sent Done false
4035 Acked Write 1 Done Sent Done false
4036 Acked Write 3 Done Sent Done false
4037 Acked Read 4 Done Sent Done false
4038 Acked Read 5 Done Sent Done false
4039 Acked Write 4 Done Sent Done false
4040 Acked Read 10 Done Sent Done false
4041 Acked Write 4 Done Sent Done false
4042 Acked Write 6 Done Sent Done false
4043 Acked Write 2 Done Sent Done false
4044 Acked Flush 0 Done Sent Done false
4045 Acked Write 4 Done Sent Done false
4046 Acked Read 7 Done Sent Done false
4047 Acked Read 8 Done Sent Done false
4048 Acked Write 8 Done Sent Done false
4049 Acked Write 9 Done Sent Done false
4050 Acked Write 10 Done Sent Done false
4051 Acked Read 8 Done Sent Done false
4052 Acked Read 6 Done Sent Done false
4053 Acked Write 10 Done Sent Done false
4054 Acked Write 2 Done Sent Done false
4055 Acked Write 4 Done Sent Done false
4056 Acked Read 9 Done Sent Done false
4057 Acked Flush 0 Done Sent Done false
4058 Acked Read 9 Done Sent Done false
4059 Acked Write 6 Done Sent Done false
4060 Acked Read 4 Done Sent Done false
4061 Acked Read 2 Done Sent Done false
4062 Acked Read 4 Done Sent Done false
4063 Acked Write 1 Done Sent Done false
4064 Acked Write 4 Done Sent Done false
4065 Acked Read 5 Done Sent Done false
4066 Acked Flush 0 Done Sent Done false
4067 Acked Read 5 Done Sent Done false
4068 Acked Read 3 Done Sent Done false
4069 Acked Read 5 Done Sent Done false
4070 Acked Write 3 Done Sent Done false
4071 Acked Write 8 Done Sent Done false
4072 Acked Write 10 Done Sent Done false
4073 Acked Write 9 Done Sent Done false
4074 Acked Write 9 Done Sent Done false
4075 Acked Read 9 Done Sent Done false
4076 Acked Read 4 Done Sent Done false
4077 Acked Write 3 Done Sent Done false
4078 Acked Write 6 Done Sent Done false
4079 Acked Read 8 Done Sent Done false
4080 Acked Flush 0 Done Sent Done false
4081 Acked Read 10 Done Sent Done false
4082 Acked Flush 0 Done Sent Done false
4083 Acked Read 10 Done Sent Done false
4084 Acked Read 1 Done Sent Done false
4085 Acked Write 2 Done Sent Done false
4086 Acked Read 8 Done Sent Done false
4087 Acked Read 1 Done Sent Done false
4088 Acked Write 8 Done Sent Done false
4089 Acked Flush 0 Done Sent Done false
4090 Acked Write 8 Done Sent Done false
4091 Acked Write 7 Done Sent Done false
4092 Acked Read 2 Done Sent Done false
4093 Acked Write 2 Done Sent Done false
4094 Acked Write 5 Done Sent Done false
4095 Acked Write 6 Done Sent Done false
4096 Acked Flush 0 Done Sent Done false
4097 Acked Write 6 Done Sent Done false
4098 Acked Flush 0 Done Sent Done false
4099 Acked Write 4 Done Sent Done false
4100 Acked Read 6 Done Sent Done false
4101 Acked Read 9 Done Sent Done false
4102 Acked Write 5 Done Sent Done false
4103 Acked Write 5 Done Sent Done false
4104 Acked Write 9 Done Sent Done false
4105 Acked Read 8 Done Sent Done false
4106 Acked Flush 0 Done Sent Done false
4107 Acked Write 6 Done Sent Done false
4108 Acked Read 4 Done Sent Done false
4109 Acked Read 7 Done Sent Done false
4110 Acked Read 10 Done Sent Done false
4111 Acked Read 7 Done Sent Done false
4112 Acked Flush 0 Done Sent Done false
4113 Acked Read 9 Done Sent Done false
4114 Acked Read 2 Done Sent Done false
4115 Acked Write 5 Done Sent Done false
4116 Acked Read 7 Done Sent Done false
4117 Acked Write 5 Done Sent Done false
4118 Acked Flush 0 Done Sent Done false
4119 Acked Read 7 Done Sent Done false
4120 Acked Flush 0 Done Sent Done false
4121 Acked Write 6 Done Sent Done false
4122 Acked Read 2 Done Sent Done false
4123 Acked Read 9 Done Sent Done false
4124 Acked Read 4 Done Sent Done false
4125 Acked Write 2 Done Sent Done false
4126 Acked Read 10 Done Sent Done false
4127 Acked Read 5 Done Sent Done false
4128 Acked Write 2 Done Sent Done false
4129 Acked Read 6 Done Sent Done false
4130 Acked Write 4 Done Sent Done false
4131 Acked Write 3 Done Sent Done false
4132 Acked Read 4 Done Sent Done false
4133 Acked Write 9 Done Sent Done false
4134 Acked Write 8 Done Sent Done false
4135 Acked Write 8 Done Sent Done false
4136 Acked Read 7 Done Sent Done false
4137 Acked Read 2 Done Sent Done false
4138 Acked Read 10 Done Sent Done false
4139 Acked Read 3 Done Sent Done false
4140 Acked Read 3 Done Sent Done false
4141 Acked Write 9 Done Sent Done false
4142 Acked Write 6 Done Sent Done false
4143 Acked Read 9 Done Sent Done false
4144 Acked Read 10 Done Sent Done false
4145 Acked Write 4 Done Sent Done false
4146 Acked Read 6 Done Sent Done false
4147 Acked Read 4 Done Sent Done false
4148 Acked Read 4 Done Sent Done false
4149 Acked Write 5 Done Sent Done false
4150 Acked Write 4 Done Sent Done false
4151 Acked Write 3 Done Sent Done false
4152 Acked Write 9 Done Sent Done false
4153 Acked Flush 0 Done Sent Done false
4154 Acked Read 3 Done Sent Done false
4155 Acked Read 4 Done Sent Done false
4156 Acked Read 4 Done Sent Done false
4157 Acked Read 6 Done Sent Done false
4158 Acked Read 9 Done Sent Done false
4159 Acked Flush 0 Done Sent Done false
4160 Acked Read 5 Done Sent Done false
4161 Acked Read 6 Done Sent Done false
4162 Acked Write 8 Done Sent Done false
4163 Acked Read 2 Done Sent Done false
4164 Acked Read 4 Done Sent Done false
4165 Acked Write 2 Done Sent Done false
4166 Acked Write 7 Done Sent Done false
4167 Acked Write 7 Done Sent Done false
4168 Acked Write 9 Done Sent Done false
4169 Acked Flush 0 Done Sent Done false
4170 Acked Read 8 Done Sent Done false
4171 Acked Read 6 Done Sent Done false
4172 Acked Read 9 Done Sent Done false
4173 Acked Write 9 Done Sent Done false
4174 Acked Read 10 Done Sent Done false
4175 Acked Read 8 Done Sent Done false
4176 Acked Write 4 Done Sent Done false
4177 Acked Write 9 Done Sent Done false
4178 Acked Read 1 Done Sent Done false
4179 Acked Write 5 Done Sent Done false
4180 Acked Read 7 Done Sent Done false
4181 Acked Write 10 Done Sent Done false
4182 Acked Write 10 Done Sent Done false
4183 Acked Write 6 Done Sent Done false
4184 Acked Write 9 Done Sent Done false
4185 Acked Write 7 Done Sent Done false
4186 Acked Write 8 Done Sent Done false
4187 Acked Read 9 Done Sent Done false
4188 Acked Read 9 Done Sent Done false
4189 Acked Write 4 Done Sent Done false
4190 Acked Read 1 Done Sent Done false
4191 Acked Write 10 Done Sent Done false
4192 Acked Write 4 Done Sent Done false
4193 Acked Write 1 Done Sent Done false
4194 Acked Read 3 Done Sent Done false
4195 Acked Write 9 Done Sent Done false
4196 Acked Write 10 Done Sent Done false
4197 Acked Write 4 Done Sent Done false
4198 Acked Write 3 Done Sent Done false
4199 Acked Read 3 Done Sent Done false
4200 Acked Flush 0 Done Sent Done false
4201 Acked Write 6 Done Sent Done false
4202 Acked Write 3 Done Sent Done false
4203 Acked Write 2 Done Sent Done false
4204 Acked Flush 0 Done Sent Done false
4205 Acked Write 8 Done Sent Done false
4206 Acked Write 3 Done Sent Done false
4207 Acked Read 2 Done Sent Done false
4208 Acked Read 8 Done Sent Done false
4209 Acked Write 1 Done Sent Done false
4210 Acked Read 9 Done Sent Done false
4211 Acked Read 10 Done Sent Done false
4212 Acked Write 7 Done Sent Done false
4213 Acked Write 10 Done Sent Done false
4214 Acked Read 3 Done Sent Done false
4215 Acked Read 9 Done Sent Done false
4216 Acked Read 3 Done Sent Done false
4217 Acked Read 5 Done Sent Done false
4218 Acked Write 7 Done Sent Done false
4219 Acked Read 9 Done Sent Done false
4220 Acked Flush 0 Done Sent Done false
4221 Acked Write 1 Done Sent Done false
4222 Acked Read 5 Done Sent Done false
4223 Acked Write 7 Done Sent Done false
4224 Acked Write 5 Done Sent Done false
4225 Acked Read 2 Done Sent Done false
4226 Acked Read 5 Done Sent Done false
4227 Acked Read 6 Done Sent Done false
4228 Acked Read 4 Done Sent Done false
4229 Acked Write 6 Done Sent Done false
4230 Acked Write 3 Done Sent Done false
4231 Acked Read 7 Done Sent Done false
4232 Acked Read 6 Done Sent Done false
4233 Acked Read 8 Done Sent Done false
4234 Acked Read 3 Done Sent Done false
4235 Acked Write 8 Done Sent Done false
4236 Acked Write 10 Done Sent Done false
4237 Acked Write 1 Done Sent Done false
4238 Acked Read 7 Done Sent Done false
4239 Acked Write 10 Done Sent Done false
4240 Acked Read 6 Done Sent Done false
4241 Acked Write 8 Done Sent Done false
4242 Acked Read 10 Done Sent Done false
4243 Acked Write 10 Done Sent Done false
4244 Acked Write 7 Done Sent Done false
4245 Acked Write 6 Done Sent Done false
4246 Acked Flush 0 Done Sent Done false
4247 Acked Read 1 Done Sent Done false
4248 Acked Flush 0 Done Sent Done false
4249 Acked Write 4 Done Sent Done false
4250 Acked Write 2 Done Sent Done false
4251 Acked Read 8 Done Sent Done false
4252 Acked Flush 0 Done Sent Done false
4253 Acked Write 9 Done Sent Done false
4254 Acked Write 6 Done Sent Done false
4255 Acked Read 9 Done Sent Done false
4256 Acked Read 7 Done Sent Done false
4257 Acked Write 2 Done Sent Done false
4258 Acked Write 10 Done Sent Done false
4259 Acked Read 7 Done Sent Done false
4260 Acked Flush 0 Done Sent Done false
4261 Acked Write 6 Done Sent Done false
4262 Acked Read 8 Done Sent Done false
4263 Acked Read 4 Done Sent Done false
4264 Acked Read 3 Done Sent Done false
4265 Acked Write 8 Done Sent Done false
4266 Acked Read 3 Done Sent Done false
4267 Acked Write 2 Done Sent Done false
4268 Acked Read 10 Done Sent Done false
4269 Acked Write 4 Done Sent Done false
4270 Acked Write 8 Done Sent Done false
4271 Acked Read 4 Done Sent Done false
4272 Acked Flush 0 Done Sent Done false
4273 Acked Write 1 Done Sent Done false
4274 Acked Read 4 Done Sent Done false
4275 Acked Read 2 Done Sent Done false
4276 Acked Read 2 Done Sent Done false
4277 Acked Read 1 Done Sent Done false
4278 Acked Read 5 Done Sent Done false
4279 Acked Write 1 Done Sent Done false
4280 Acked Flush 0 Done Sent Done false
4281 Acked Read 1 Done Sent Done false
4282 Acked Write 9 Done Sent Done false
4283 Acked Flush 0 Done Sent Done false
4284 Acked Write 3 Done Sent Done false
4285 Acked Write 6 Done Sent Done false
4286 Acked Read 8 Done Sent Done false
4287 Acked Write 4 Done Sent Done false
4288 Acked Write 8 Done Sent Done false
4289 Acked Read 3 Done Sent Done false
4290 Acked Write 7 Done Sent Done false
4291 Acked Read 2 Done Sent Done false
4292 Acked Write 4 Done Sent Done false
4293 Acked Read 4 Done Sent Done false
4294 Acked Read 1 Done Sent Done false
4295 Acked Write 7 Done Sent Done false
4296 Acked Read 8 Done Sent Done false
4297 Acked Read 9 Done Sent Done false
4298 Acked Write 6 Done Sent Done false
4299 Acked Read 5 Done Sent Done false
4300 Acked Write 1 Done Sent Done false
4301 Acked Write 10 Done Sent Done false
4302 Acked Read 2 Done Sent Done false
4303 Acked Write 7 Done Sent Done false
4304 Acked Read 7 Done Sent Done false
4305 Acked Read 1 Done Sent Done false
4306 Acked Read 3 Done Sent Done false
4307 Acked Read 3 Done Sent Done false
4308 Acked Read 9 Done Sent Done false
4309 Acked Write 5 Done Sent Done false
4310 Acked Read 2 Done Sent Done false
4311 Acked Flush 0 Done Sent Done false
4312 Acked Write 4 Done Sent Done false
4313 Acked Write 1 Done Sent Done false
4314 Acked Read 9 Done Sent Done false
4315 Acked Write 9 Done Sent Done false
4316 Acked Read 6 Done Sent Done false
4317 Acked Write 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:51.618301233Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:52.620050586Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:53.622295882Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Flush 0 Done Sent Done false
4018 Acked Read 10 Done Sent Done false
4019 Acked Read 10 Done Sent Done false
4020 Acked Read 5 Done Sent Done false
4021 Acked Write 6 Done Sent Done false
4022 Acked Write 2 Done Sent Done false
4023 Acked Read 9 Done Sent Done false
4024 Acked Write 8 Done Sent Done false
4025 Acked Write 4 Done Sent Done false
4026 Acked Flush 0 Done Sent Done false
4027 Acked Write 1 Done Sent Done false
4028 Acked Write 6 Done Sent Done false
4029 Acked Flush 0 Done Sent Done false
4030 Acked Read 8 Done Sent Done false
4031 Acked Read 8 Done Sent Done false
4032 Acked Read 3 Done Sent Done false
4033 Acked Write 1 Done Sent Done false
4034 Acked Read 2 Done Sent Done false
4035 Acked Write 1 Done Sent Done false
4036 Acked Write 3 Done Sent Done false
4037 Acked Read 4 Done Sent Done false
4038 Acked Read 5 Done Sent Done false
4039 Acked Write 4 Done Sent Done false
4040 Acked Read 10 Done Sent Done false
4041 Acked Write 4 Done Sent Done false
4042 Acked Write 6 Done Sent Done false
4043 Acked Write 2 Done Sent Done false
4044 Acked Flush 0 Done Sent Done false
4045 Acked Write 4 Done Sent Done false
4046 Acked Read 7 Done Sent Done false
4047 Acked Read 8 Done Sent Done false
4048 Acked Write 8 Done Sent Done false
4049 Acked Write 9 Done Sent Done false
4050 Acked Write 10 Done Sent Done false
4051 Acked Read 8 Done Sent Done false
4052 Acked Read 6 Done Sent Done false
4053 Acked Write 10 Done Sent Done false
4054 Acked Write 2 Done Sent Done false
4055 Acked Write 4 Done Sent Done false
4056 Acked Read 9 Done Sent Done false
4057 Acked Flush 0 Done Sent Done false
4058 Acked Read 9 Done Sent Done false
4059 Acked Write 6 Done Sent Done false
4060 Acked Read 4 Done Sent Done false
4061 Acked Read 2 Done Sent Done false
4062 Acked Read 4 Done Sent Done false
4063 Acked Write 1 Done Sent Done false
4064 Acked Write 4 Done Sent Done false
4065 Acked Read 5 Done Sent Done false
4066 Acked Flush 0 Done Sent Done false
4067 Acked Read 5 Done Sent Done false
4068 Acked Read 3 Done Sent Done false
4069 Acked Read 5 Done Sent Done false
4070 Acked Write 3 Done Sent Done false
4071 Acked Write 8 Done Sent Done false
4072 Acked Write 10 Done Sent Done false
4073 Acked Write 9 Done Sent Done false
4074 Acked Write 9 Done Sent Done false
4075 Acked Read 9 Done Sent Done false
4076 Acked Read 4 Done Sent Done false
4077 Acked Write 3 Done Sent Done false
4078 Acked Write 6 Done Sent Done false
4079 Acked Read 8 Done Sent Done false
4080 Acked Flush 0 Done Sent Done false
4081 Acked Read 10 Done Sent Done false
4082 Acked Flush 0 Done Sent Done false
4083 Acked Read 10 Done Sent Done false
4084 Acked Read 1 Done Sent Done false
4085 Acked Write 2 Done Sent Done false
4086 Acked Read 8 Done Sent Done false
4087 Acked Read 1 Done Sent Done false
4088 Acked Write 8 Done Sent Done false
4089 Acked Flush 0 Done Sent Done false
4090 Acked Write 8 Done Sent Done false
4091 Acked Write 7 Done Sent Done false
4092 Acked Read 2 Done Sent Done false
4093 Acked Write 2 Done Sent Done false
4094 Acked Write 5 Done Sent Done false
4095 Acked Write 6 Done Sent Done false
4096 Acked Flush 0 Done Sent Done false
4097 Acked Write 6 Done Sent Done false
4098 Acked Flush 0 Done Sent Done false
4099 Acked Write 4 Done Sent Done false
4100 Acked Read 6 Done Sent Done false
4101 Acked Read 9 Done Sent Done false
4102 Acked Write 5 Done Sent Done false
4103 Acked Write 5 Done Sent Done false
4104 Acked Write 9 Done Sent Done false
4105 Acked Read 8 Done Sent Done false
4106 Acked Flush 0 Done Sent Done false
4107 Acked Write 6 Done Sent Done false
4108 Acked Read 4 Done Sent Done false
4109 Acked Read 7 Done Sent Done false
4110 Acked Read 10 Done Sent Done false
4111 Acked Read 7 Done Sent Done false
4112 Acked Flush 0 Done Sent Done false
4113 Acked Read 9 Done Sent Done false
4114 Acked Read 2 Done Sent Done false
4115 Acked Write 5 Done Sent Done false
4116 Acked Read 7 Done Sent Done false
4117 Acked Write 5 Done Sent Done false
4118 Acked Flush 0 Done Sent Done false
4119 Acked Read 7 Done Sent Done false
4120 Acked Flush 0 Done Sent Done false
4121 Acked Write 6 Done Sent Done false
4122 Acked Read 2 Done Sent Done false
4123 Acked Read 9 Done Sent Done false
4124 Acked Read 4 Done Sent Done false
4125 Acked Write 2 Done Sent Done false
4126 Acked Read 10 Done Sent Done false
4127 Acked Read 5 Done Sent Done false
4128 Acked Write 2 Done Sent Done false
4129 Acked Read 6 Done Sent Done false
4130 Acked Write 4 Done Sent Done false
4131 Acked Write 3 Done Sent Done false
4132 Acked Read 4 Done Sent Done false
4133 Acked Write 9 Done Sent Done false
4134 Acked Write 8 Done Sent Done false
4135 Acked Write 8 Done Sent Done false
4136 Acked Read 7 Done Sent Done false
4137 Acked Read 2 Done Sent Done false
4138 Acked Read 10 Done Sent Done false
4139 Acked Read 3 Done Sent Done false
4140 Acked Read 3 Done Sent Done false
4141 Acked Write 9 Done Sent Done false
4142 Acked Write 6 Done Sent Done false
4143 Acked Read 9 Done Sent Done false
4144 Acked Read 10 Done Sent Done false
4145 Acked Write 4 Done Sent Done false
4146 Acked Read 6 Done Sent Done false
4147 Acked Read 4 Done Sent Done false
4148 Acked Read 4 Done Sent Done false
4149 Acked Write 5 Done Sent Done false
4150 Acked Write 4 Done Sent Done false
4151 Acked Write 3 Done Sent Done false
4152 Acked Write 9 Done Sent Done false
4153 Acked Flush 0 Done Sent Done false
4154 Acked Read 3 Done Sent Done false
4155 Acked Read 4 Done Sent Done false
4156 Acked Read 4 Done Sent Done false
4157 Acked Read 6 Done Sent Done false
4158 Acked Read 9 Done Sent Done false
4159 Acked Flush 0 Done Sent Done false
4160 Acked Read 5 Done Sent Done false
4161 Acked Read 6 Done Sent Done false
4162 Acked Write 8 Done Sent Done false
4163 Acked Read 2 Done Sent Done false
4164 Acked Read 4 Done Sent Done false
4165 Acked Write 2 Done Sent Done false
4166 Acked Write 7 Done Sent Done false
4167 Acked Write 7 Done Sent Done false
4168 Acked Write 9 Done Sent Done false
4169 Acked Flush 0 Done Sent Done false
4170 Acked Read 8 Done Sent Done false
4171 Acked Read 6 Done Sent Done false
4172 Acked Read 9 Done Sent Done false
4173 Acked Write 9 Done Sent Done false
4174 Acked Read 10 Done Sent Done false
4175 Acked Read 8 Done Sent Done false
4176 Acked Write 4 Done Sent Done false
4177 Acked Write 9 Done Sent Done false
4178 Acked Read 1 Done Sent Done false
4179 Acked Write 5 Done Sent Done false
4180 Acked Read 7 Done Sent Done false
4181 Acked Write 10 Done Sent Done false
4182 Acked Write 10 Done Sent Done false
4183 Acked Write 6 Done Sent Done false
4184 Acked Write 9 Done Sent Done false
4185 Acked Write 7 Done Sent Done false
4186 Acked Write 8 Done Sent Done false
4187 Acked Read 9 Done Sent Done false
4188 Acked Read 9 Done Sent Done false
4189 Acked Write 4 Done Sent Done false
4190 Acked Read 1 Done Sent Done false
4191 Acked Write 10 Done Sent Done false
4192 Acked Write 4 Done Sent Done false
4193 Acked Write 1 Done Sent Done false
4194 Acked Read 3 Done Sent Done false
4195 Acked Write 9 Done Sent Done false
4196 Acked Write 10 Done Sent Done false
4197 Acked Write 4 Done Sent Done false
4198 Acked Write 3 Done Sent Done false
4199 Acked Read 3 Done Sent Done false
4200 Acked Flush 0 Done Sent Done false
4201 Acked Write 6 Done Sent Done false
4202 Acked Write 3 Done Sent Done false
4203 Acked Write 2 Done Sent Done false
4204 Acked Flush 0 Done Sent Done false
4205 Acked Write 8 Done Sent Done false
4206 Acked Write 3 Done Sent Done false
4207 Acked Read 2 Done Sent Done false
4208 Acked Read 8 Done Sent Done false
4209 Acked Write 1 Done Sent Done false
4210 Acked Read 9 Done Sent Done false
4211 Acked Read 10 Done Sent Done false
4212 Acked Write 7 Done Sent Done false
4213 Acked Write 10 Done Sent Done false
4214 Acked Read 3 Done Sent Done false
4215 Acked Read 9 Done Sent Done false
4216 Acked Read 3 Done Sent Done false
4217 Acked Read 5 Done Sent Done false
4218 Acked Write 7 Done Sent Done false
4219 Acked Read 9 Done Sent Done false
4220 Acked Flush 0 Done Sent Done false
4221 Acked Write 1 Done Sent Done false
4222 Acked Read 5 Done Sent Done false
4223 Acked Write 7 Done Sent Done false
4224 Acked Write 5 Done Sent Done false
4225 Acked Read 2 Done Sent Done false
4226 Acked Read 5 Done Sent Done false
4227 Acked Read 6 Done Sent Done false
4228 Acked Read 4 Done Sent Done false
4229 Acked Write 6 Done Sent Done false
4230 Acked Write 3 Done Sent Done false
4231 Acked Read 7 Done Sent Done false
4232 Acked Read 6 Done Sent Done false
4233 Acked Read 8 Done Sent Done false
4234 Acked Read 3 Done Sent Done false
4235 Acked Write 8 Done Sent Done false
4236 Acked Write 10 Done Sent Done false
4237 Acked Write 1 Done Sent Done false
4238 Acked Read 7 Done Sent Done false
4239 Acked Write 10 Done Sent Done false
4240 Acked Read 6 Done Sent Done false
4241 Acked Write 8 Done Sent Done false
4242 Acked Read 10 Done Sent Done false
4243 Acked Write 10 Done Sent Done false
4244 Acked Write 7 Done Sent Done false
4245 Acked Write 6 Done Sent Done false
4246 Acked Flush 0 Done Sent Done false
4247 Acked Read 1 Done Sent Done false
4248 Acked Flush 0 Done Sent Done false
4249 Acked Write 4 Done Sent Done false
4250 Acked Write 2 Done Sent Done false
4251 Acked Read 8 Done Sent Done false
4252 Acked Flush 0 Done Sent Done false
4253 Acked Write 9 Done Sent Done false
4254 Acked Write 6 Done Sent Done false
4255 Acked Read 9 Done Sent Done false
4256 Acked Read 7 Done Sent Done false
4257 Acked Write 2 Done Sent Done false
4258 Acked Write 10 Done Sent Done false
4259 Acked Read 7 Done Sent Done false
4260 Acked Flush 0 Done Sent Done false
4261 Acked Write 6 Done Sent Done false
4262 Acked Read 8 Done Sent Done false
4263 Acked Read 4 Done Sent Done false
4264 Acked Read 3 Done Sent Done false
4265 Acked Write 8 Done Sent Done false
4266 Acked Read 3 Done Sent Done false
4267 Acked Write 2 Done Sent Done false
4268 Acked Read 10 Done Sent Done false
4269 Acked Write 4 Done Sent Done false
4270 Acked Write 8 Done Sent Done false
4271 Acked Read 4 Done Sent Done false
4272 Acked Flush 0 Done Sent Done false
4273 Acked Write 1 Done Sent Done false
4274 Acked Read 4 Done Sent Done false
4275 Acked Read 2 Done Sent Done false
4276 Acked Read 2 Done Sent Done false
4277 Acked Read 1 Done Sent Done false
4278 Acked Read 5 Done Sent Done false
4279 Acked Write 1 Done Sent Done false
4280 Acked Flush 0 Done Sent Done false
4281 Acked Read 1 Done Sent Done false
4282 Acked Write 9 Done Sent Done false
4283 Acked Flush 0 Done Sent Done false
4284 Acked Write 3 Done Sent Done false
4285 Acked Write 6 Done Sent Done false
4286 Acked Read 8 Done Sent Done false
4287 Acked Write 4 Done Sent Done false
4288 Acked Write 8 Done Sent Done false
4289 Acked Read 3 Done Sent Done false
4290 Acked Write 7 Done Sent Done false
4291 Acked Read 2 Done Sent Done false
4292 Acked Write 4 Done Sent Done false
4293 Acked Read 4 Done Sent Done false
4294 Acked Read 1 Done Sent Done false
4295 Acked Write 7 Done Sent Done false
4296 Acked Read 8 Done Sent Done false
4297 Acked Read 9 Done Sent Done false
4298 Acked Write 6 Done Sent Done false
4299 Acked Read 5 Done Sent Done false
4300 Acked Write 1 Done Sent Done false
4301 Acked Write 10 Done Sent Done false
4302 Acked Read 2 Done Sent Done false
4303 Acked Write 7 Done Sent Done false
4304 Acked Read 7 Done Sent Done false
4305 Acked Read 1 Done Sent Done false
4306 Acked Read 3 Done Sent Done false
4307 Acked Read 3 Done Sent Done false
4308 Acked Read 9 Done Sent Done false
4309 Acked Write 5 Done Sent Done false
4310 Acked Read 2 Done Sent Done false
4311 Acked Flush 0 Done Sent Done false
4312 Acked Write 4 Done Sent Done false
4313 Acked Write 1 Done Sent Done false
4314 Acked Read 9 Done Sent Done false
4315 Acked Write 9 Done Sent Done false
4316 Acked Read 6 Done Sent Done false
4317 Acked Write 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:55.123662949Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:55.625389177Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:56.381736313Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:56.381812815Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:56.381863453Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:56.381901773Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:03:56.382073562Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:56.883322784Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:57.627881766Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Flush 0 Done Sent Done false
4018 Acked Read 10 Done Sent Done false
4019 Acked Read 10 Done Sent Done false
4020 Acked Read 5 Done Sent Done false
4021 Acked Write 6 Done Sent Done false
4022 Acked Write 2 Done Sent Done false
4023 Acked Read 9 Done Sent Done false
4024 Acked Write 8 Done Sent Done false
4025 Acked Write 4 Done Sent Done false
4026 Acked Flush 0 Done Sent Done false
4027 Acked Write 1 Done Sent Done false
4028 Acked Write 6 Done Sent Done false
4029 Acked Flush 0 Done Sent Done false
4030 Acked Read 8 Done Sent Done false
4031 Acked Read 8 Done Sent Done false
4032 Acked Read 3 Done Sent Done false
4033 Acked Write 1 Done Sent Done false
4034 Acked Read 2 Done Sent Done false
4035 Acked Write 1 Done Sent Done false
4036 Acked Write 3 Done Sent Done false
4037 Acked Read 4 Done Sent Done false
4038 Acked Read 5 Done Sent Done false
4039 Acked Write 4 Done Sent Done false
4040 Acked Read 10 Done Sent Done false
4041 Acked Write 4 Done Sent Done false
4042 Acked Write 6 Done Sent Done false
4043 Acked Write 2 Done Sent Done false
4044 Acked Flush 0 Done Sent Done false
4045 Acked Write 4 Done Sent Done false
4046 Acked Read 7 Done Sent Done false
4047 Acked Read 8 Done Sent Done false
4048 Acked Write 8 Done Sent Done false
4049 Acked Write 9 Done Sent Done false
4050 Acked Write 10 Done Sent Done false
4051 Acked Read 8 Done Sent Done false
4052 Acked Read 6 Done Sent Done false
4053 Acked Write 10 Done Sent Done false
4054 Acked Write 2 Done Sent Done false
4055 Acked Write 4 Done Sent Done false
4056 Acked Read 9 Done Sent Done false
4057 Acked Flush 0 Done Sent Done false
4058 Acked Read 9 Done Sent Done false
4059 Acked Write 6 Done Sent Done false
4060 Acked Read 4 Done Sent Done false
4061 Acked Read 2 Done Sent Done false
4062 Acked Read 4 Done Sent Done false
4063 Acked Write 1 Done Sent Done false
4064 Acked Write 4 Done Sent Done false
4065 Acked Read 5 Done Sent Done false
4066 Acked Flush 0 Done Sent Done false
4067 Acked Read 5 Done Sent Done false
4068 Acked Read 3 Done Sent Done false
4069 Acked Read 5 Done Sent Done false
4070 Acked Write 3 Done Sent Done false
4071 Acked Write 8 Done Sent Done false
4072 Acked Write 10 Done Sent Done false
4073 Acked Write 9 Done Sent Done false
4074 Acked Write 9 Done Sent Done false
4075 Acked Read 9 Done Sent Done false
4076 Acked Read 4 Done Sent Done false
4077 Acked Write 3 Done Sent Done false
4078 Acked Write 6 Done Sent Done false
4079 Acked Read 8 Done Sent Done false
4080 Acked Flush 0 Done Sent Done false
4081 Acked Read 10 Done Sent Done false
4082 Acked Flush 0 Done Sent Done false
4083 Acked Read 10 Done Sent Done false
4084 Acked Read 1 Done Sent Done false
4085 Acked Write 2 Done Sent Done false
4086 Acked Read 8 Done Sent Done false
4087 Acked Read 1 Done Sent Done false
4088 Acked Write 8 Done Sent Done false
4089 Acked Flush 0 Done Sent Done false
4090 Acked Write 8 Done Sent Done false
4091 Acked Write 7 Done Sent Done false
4092 Acked Read 2 Done Sent Done false
4093 Acked Write 2 Done Sent Done false
4094 Acked Write 5 Done Sent Done false
4095 Acked Write 6 Done Sent Done false
4096 Acked Flush 0 Done Sent Done false
4097 Acked Write 6 Done Sent Done false
4098 Acked Flush 0 Done Sent Done false
4099 Acked Write 4 Done Sent Done false
4100 Acked Read 6 Done Sent Done false
4101 Acked Read 9 Done Sent Done false
4102 Acked Write 5 Done Sent Done false
4103 Acked Write 5 Done Sent Done false
4104 Acked Write 9 Done Sent Done false
4105 Acked Read 8 Done Sent Done false
4106 Acked Flush 0 Done Sent Done false
4107 Acked Write 6 Done Sent Done false
4108 Acked Read 4 Done Sent Done false
4109 Acked Read 7 Done Sent Done false
4110 Acked Read 10 Done Sent Done false
4111 Acked Read 7 Done Sent Done false
4112 Acked Flush 0 Done Sent Done false
4113 Acked Read 9 Done Sent Done false
4114 Acked Read 2 Done Sent Done false
4115 Acked Write 5 Done Sent Done false
4116 Acked Read 7 Done Sent Done false
4117 Acked Write 5 Done Sent Done false
4118 Acked Flush 0 Done Sent Done false
4119 Acked Read 7 Done Sent Done false
4120 Acked Flush 0 Done Sent Done false
4121 Acked Write 6 Done Sent Done false
4122 Acked Read 2 Done Sent Done false
4123 Acked Read 9 Done Sent Done false
4124 Acked Read 4 Done Sent Done false
4125 Acked Write 2 Done Sent Done false
4126 Acked Read 10 Done Sent Done false
4127 Acked Read 5 Done Sent Done false
4128 Acked Write 2 Done Sent Done false
4129 Acked Read 6 Done Sent Done false
4130 Acked Write 4 Done Sent Done false
4131 Acked Write 3 Done Sent Done false
4132 Acked Read 4 Done Sent Done false
4133 Acked Write 9 Done Sent Done false
4134 Acked Write 8 Done Sent Done false
4135 Acked Write 8 Done Sent Done false
4136 Acked Read 7 Done Sent Done false
4137 Acked Read 2 Done Sent Done false
4138 Acked Read 10 Done Sent Done false
4139 Acked Read 3 Done Sent Done false
4140 Acked Read 3 Done Sent Done false
4141 Acked Write 9 Done Sent Done false
4142 Acked Write 6 Done Sent Done false
4143 Acked Read 9 Done Sent Done false
4144 Acked Read 10 Done Sent Done false
4145 Acked Write 4 Done Sent Done false
4146 Acked Read 6 Done Sent Done false
4147 Acked Read 4 Done Sent Done false
4148 Acked Read 4 Done Sent Done false
4149 Acked Write 5 Done Sent Done false
4150 Acked Write 4 Done Sent Done false
4151 Acked Write 3 Done Sent Done false
4152 Acked Write 9 Done Sent Done false
4153 Acked Flush 0 Done Sent Done false
4154 Acked Read 3 Done Sent Done false
4155 Acked Read 4 Done Sent Done false
4156 Acked Read 4 Done Sent Done false
4157 Acked Read 6 Done Sent Done false
4158 Acked Read 9 Done Sent Done false
4159 Acked Flush 0 Done Sent Done false
4160 Acked Read 5 Done Sent Done false
4161 Acked Read 6 Done Sent Done false
4162 Acked Write 8 Done Sent Done false
4163 Acked Read 2 Done Sent Done false
4164 Acked Read 4 Done Sent Done false
4165 Acked Write 2 Done Sent Done false
4166 Acked Write 7 Done Sent Done false
4167 Acked Write 7 Done Sent Done false
4168 Acked Write 9 Done Sent Done false
4169 Acked Flush 0 Done Sent Done false
4170 Acked Read 8 Done Sent Done false
4171 Acked Read 6 Done Sent Done false
4172 Acked Read 9 Done Sent Done false
4173 Acked Write 9 Done Sent Done false
4174 Acked Read 10 Done Sent Done false
4175 Acked Read 8 Done Sent Done false
4176 Acked Write 4 Done Sent Done false
4177 Acked Write 9 Done Sent Done false
4178 Acked Read 1 Done Sent Done false
4179 Acked Write 5 Done Sent Done false
4180 Acked Read 7 Done Sent Done false
4181 Acked Write 10 Done Sent Done false
4182 Acked Write 10 Done Sent Done false
4183 Acked Write 6 Done Sent Done false
4184 Acked Write 9 Done Sent Done false
4185 Acked Write 7 Done Sent Done false
4186 Acked Write 8 Done Sent Done false
4187 Acked Read 9 Done Sent Done false
4188 Acked Read 9 Done Sent Done false
4189 Acked Write 4 Done Sent Done false
4190 Acked Read 1 Done Sent Done false
4191 Acked Write 10 Done Sent Done false
4192 Acked Write 4 Done Sent Done false
4193 Acked Write 1 Done Sent Done false
4194 Acked Read 3 Done Sent Done false
4195 Acked Write 9 Done Sent Done false
4196 Acked Write 10 Done Sent Done false
4197 Acked Write 4 Done Sent Done false
4198 Acked Write 3 Done Sent Done false
4199 Acked Read 3 Done Sent Done false
4200 Acked Flush 0 Done Sent Done false
4201 Acked Write 6 Done Sent Done false
4202 Acked Write 3 Done Sent Done false
4203 Acked Write 2 Done Sent Done false
4204 Acked Flush 0 Done Sent Done false
4205 Acked Write 8 Done Sent Done false
4206 Acked Write 3 Done Sent Done false
4207 Acked Read 2 Done Sent Done false
4208 Acked Read 8 Done Sent Done false
4209 Acked Write 1 Done Sent Done false
4210 Acked Read 9 Done Sent Done false
4211 Acked Read 10 Done Sent Done false
4212 Acked Write 7 Done Sent Done false
4213 Acked Write 10 Done Sent Done false
4214 Acked Read 3 Done Sent Done false
4215 Acked Read 9 Done Sent Done false
4216 Acked Read 3 Done Sent Done false
4217 Acked Read 5 Done Sent Done false
4218 Acked Write 7 Done Sent Done false
4219 Acked Read 9 Done Sent Done false
4220 Acked Flush 0 Done Sent Done false
4221 Acked Write 1 Done Sent Done false
4222 Acked Read 5 Done Sent Done false
4223 Acked Write 7 Done Sent Done false
4224 Acked Write 5 Done Sent Done false
4225 Acked Read 2 Done Sent Done false
4226 Acked Read 5 Done Sent Done false
4227 Acked Read 6 Done Sent Done false
4228 Acked Read 4 Done Sent Done false
4229 Acked Write 6 Done Sent Done false
4230 Acked Write 3 Done Sent Done false
4231 Acked Read 7 Done Sent Done false
4232 Acked Read 6 Done Sent Done false
4233 Acked Read 8 Done Sent Done false
4234 Acked Read 3 Done Sent Done false
4235 Acked Write 8 Done Sent Done false
4236 Acked Write 10 Done Sent Done false
4237 Acked Write 1 Done Sent Done false
4238 Acked Read 7 Done Sent Done false
4239 Acked Write 10 Done Sent Done false
4240 Acked Read 6 Done Sent Done false
4241 Acked Write 8 Done Sent Done false
4242 Acked Read 10 Done Sent Done false
4243 Acked Write 10 Done Sent Done false
4244 Acked Write 7 Done Sent Done false
4245 Acked Write 6 Done Sent Done false
4246 Acked Flush 0 Done Sent Done false
4247 Acked Read 1 Done Sent Done false
4248 Acked Flush 0 Done Sent Done false
4249 Acked Write 4 Done Sent Done false
4250 Acked Write 2 Done Sent Done false
4251 Acked Read 8 Done Sent Done false
4252 Acked Flush 0 Done Sent Done false
4253 Acked Write 9 Done Sent Done false
4254 Acked Write 6 Done Sent Done false
4255 Acked Read 9 Done Sent Done false
4256 Acked Read 7 Done Sent Done false
4257 Acked Write 2 Done Sent Done false
4258 Acked Write 10 Done Sent Done false
4259 Acked Read 7 Done Sent Done false
4260 Acked Flush 0 Done Sent Done false
4261 Acked Write 6 Done Sent Done false
4262 Acked Read 8 Done Sent Done false
4263 Acked Read 4 Done Sent Done false
4264 Acked Read 3 Done Sent Done false
4265 Acked Write 8 Done Sent Done false
4266 Acked Read 3 Done Sent Done false
4267 Acked Write 2 Done Sent Done false
4268 Acked Read 10 Done Sent Done false
4269 Acked Write 4 Done Sent Done false
4270 Acked Write 8 Done Sent Done false
4271 Acked Read 4 Done Sent Done false
4272 Acked Flush 0 Done Sent Done false
4273 Acked Write 1 Done Sent Done false
4274 Acked Read 4 Done Sent Done false
4275 Acked Read 2 Done Sent Done false
4276 Acked Read 2 Done Sent Done false
4277 Acked Read 1 Done Sent Done false
4278 Acked Read 5 Done Sent Done false
4279 Acked Write 1 Done Sent Done false
4280 Acked Flush 0 Done Sent Done false
4281 Acked Read 1 Done Sent Done false
4282 Acked Write 9 Done Sent Done false
4283 Acked Flush 0 Done Sent Done false
4284 Acked Write 3 Done Sent Done false
4285 Acked Write 6 Done Sent Done false
4286 Acked Read 8 Done Sent Done false
4287 Acked Write 4 Done Sent Done false
4288 Acked Write 8 Done Sent Done false
4289 Acked Read 3 Done Sent Done false
4290 Acked Write 7 Done Sent Done false
4291 Acked Read 2 Done Sent Done false
4292 Acked Write 4 Done Sent Done false
4293 Acked Read 4 Done Sent Done false
4294 Acked Read 1 Done Sent Done false
4295 Acked Write 7 Done Sent Done false
4296 Acked Read 8 Done Sent Done false
4297 Acked Read 9 Done Sent Done false
4298 Acked Write 6 Done Sent Done false
4299 Acked Read 5 Done Sent Done false
4300 Acked Write 1 Done Sent Done false
4301 Acked Write 10 Done Sent Done false
4302 Acked Read 2 Done Sent Done false
4303 Acked Write 7 Done Sent Done false
4304 Acked Read 7 Done Sent Done false
4305 Acked Read 1 Done Sent Done false
4306 Acked Read 3 Done Sent Done false
4307 Acked Read 3 Done Sent Done false
4308 Acked Read 9 Done Sent Done false
4309 Acked Write 5 Done Sent Done false
4310 Acked Read 2 Done Sent Done false
4311 Acked Flush 0 Done Sent Done false
4312 Acked Write 4 Done Sent Done false
4313 Acked Write 1 Done Sent Done false
4314 Acked Read 9 Done Sent Done false
4315 Acked Write 9 Done Sent Done false
4316 Acked Read 6 Done Sent Done false
4317 Acked Write 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:59.125663332Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:59.637361411Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:00.633771854Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:01.633875683Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Flush 0 Done Sent Done false
4018 Acked Read 10 Done Sent Done false
4019 Acked Read 10 Done Sent Done false
4020 Acked Read 5 Done Sent Done false
4021 Acked Write 6 Done Sent Done false
4022 Acked Write 2 Done Sent Done false
4023 Acked Read 9 Done Sent Done false
4024 Acked Write 8 Done Sent Done false
4025 Acked Write 4 Done Sent Done false
4026 Acked Flush 0 Done Sent Done false
4027 Acked Write 1 Done Sent Done false
4028 Acked Write 6 Done Sent Done false
4029 Acked Flush 0 Done Sent Done false
4030 Acked Read 8 Done Sent Done false
4031 Acked Read 8 Done Sent Done false
4032 Acked Read 3 Done Sent Done false
4033 Acked Write 1 Done Sent Done false
4034 Acked Read 2 Done Sent Done false
4035 Acked Write 1 Done Sent Done false
4036 Acked Write 3 Done Sent Done false
4037 Acked Read 4 Done Sent Done false
4038 Acked Read 5 Done Sent Done false
4039 Acked Write 4 Done Sent Done false
4040 Acked Read 10 Done Sent Done false
4041 Acked Write 4 Done Sent Done false
4042 Acked Write 6 Done Sent Done false
4043 Acked Write 2 Done Sent Done false
4044 Acked Flush 0 Done Sent Done false
4045 Acked Write 4 Done Sent Done false
4046 Acked Read 7 Done Sent Done false
4047 Acked Read 8 Done Sent Done false
4048 Acked Write 8 Done Sent Done false
4049 Acked Write 9 Done Sent Done false
4050 Acked Write 10 Done Sent Done false
4051 Acked Read 8 Done Sent Done false
4052 Acked Read 6 Done Sent Done false
4053 Acked Write 10 Done Sent Done false
4054 Acked Write 2 Done Sent Done false
4055 Acked Write 4 Done Sent Done false
4056 Acked Read 9 Done Sent Done false
4057 Acked Flush 0 Done Sent Done false
4058 Acked Read 9 Done Sent Done false
4059 Acked Write 6 Done Sent Done false
4060 Acked Read 4 Done Sent Done false
4061 Acked Read 2 Done Sent Done false
4062 Acked Read 4 Done Sent Done false
4063 Acked Write 1 Done Sent Done false
4064 Acked Write 4 Done Sent Done false
4065 Acked Read 5 Done Sent Done false
4066 Acked Flush 0 Done Sent Done false
4067 Acked Read 5 Done Sent Done false
4068 Acked Read 3 Done Sent Done false
4069 Acked Read 5 Done Sent Done false
4070 Acked Write 3 Done Sent Done false
4071 Acked Write 8 Done Sent Done false
4072 Acked Write 10 Done Sent Done false
4073 Acked Write 9 Done Sent Done false
4074 Acked Write 9 Done Sent Done false
4075 Acked Read 9 Done Sent Done false
4076 Acked Read 4 Done Sent Done false
4077 Acked Write 3 Done Sent Done false
4078 Acked Write 6 Done Sent Done false
4079 Acked Read 8 Done Sent Done false
4080 Acked Flush 0 Done Sent Done false
4081 Acked Read 10 Done Sent Done false
4082 Acked Flush 0 Done Sent Done false
4083 Acked Read 10 Done Sent Done false
4084 Acked Read 1 Done Sent Done false
4085 Acked Write 2 Done Sent Done false
4086 Acked Read 8 Done Sent Done false
4087 Acked Read 1 Done Sent Done false
4088 Acked Write 8 Done Sent Done false
4089 Acked Flush 0 Done Sent Done false
4090 Acked Write 8 Done Sent Done false
4091 Acked Write 7 Done Sent Done false
4092 Acked Read 2 Done Sent Done false
4093 Acked Write 2 Done Sent Done false
4094 Acked Write 5 Done Sent Done false
4095 Acked Write 6 Done Sent Done false
4096 Acked Flush 0 Done Sent Done false
4097 Acked Write 6 Done Sent Done false
4098 Acked Flush 0 Done Sent Done false
4099 Acked Write 4 Done Sent Done false
4100 Acked Read 6 Done Sent Done false
4101 Acked Read 9 Done Sent Done false
4102 Acked Write 5 Done Sent Done false
4103 Acked Write 5 Done Sent Done false
4104 Acked Write 9 Done Sent Done false
4105 Acked Read 8 Done Sent Done false
4106 Acked Flush 0 Done Sent Done false
4107 Acked Write 6 Done Sent Done false
4108 Acked Read 4 Done Sent Done false
4109 Acked Read 7 Done Sent Done false
4110 Acked Read 10 Done Sent Done false
4111 Acked Read 7 Done Sent Done false
4112 Acked Flush 0 Done Sent Done false
4113 Acked Read 9 Done Sent Done false
4114 Acked Read 2 Done Sent Done false
4115 Acked Write 5 Done Sent Done false
4116 Acked Read 7 Done Sent Done false
4117 Acked Write 5 Done Sent Done false
4118 Acked Flush 0 Done Sent Done false
4119 Acked Read 7 Done Sent Done false
4120 Acked Flush 0 Done Sent Done false
4121 Acked Write 6 Done Sent Done false
4122 Acked Read 2 Done Sent Done false
4123 Acked Read 9 Done Sent Done false
4124 Acked Read 4 Done Sent Done false
4125 Acked Write 2 Done Sent Done false
4126 Acked Read 10 Done Sent Done false
4127 Acked Read 5 Done Sent Done false
4128 Acked Write 2 Done Sent Done false
4129 Acked Read 6 Done Sent Done false
4130 Acked Write 4 Done Sent Done false
4131 Acked Write 3 Done Sent Done false
4132 Acked Read 4 Done Sent Done false
4133 Acked Write 9 Done Sent Done false
4134 Acked Write 8 Done Sent Done false
4135 Acked Write 8 Done Sent Done false
4136 Acked Read 7 Done Sent Done false
4137 Acked Read 2 Done Sent Done false
4138 Acked Read 10 Done Sent Done false
4139 Acked Read 3 Done Sent Done false
4140 Acked Read 3 Done Sent Done false
4141 Acked Write 9 Done Sent Done false
4142 Acked Write 6 Done Sent Done false
4143 Acked Read 9 Done Sent Done false
4144 Acked Read 10 Done Sent Done false
4145 Acked Write 4 Done Sent Done false
4146 Acked Read 6 Done Sent Done false
4147 Acked Read 4 Done Sent Done false
4148 Acked Read 4 Done Sent Done false
4149 Acked Write 5 Done Sent Done false
4150 Acked Write 4 Done Sent Done false
4151 Acked Write 3 Done Sent Done false
4152 Acked Write 9 Done Sent Done false
4153 Acked Flush 0 Done Sent Done false
4154 Acked Read 3 Done Sent Done false
4155 Acked Read 4 Done Sent Done false
4156 Acked Read 4 Done Sent Done false
4157 Acked Read 6 Done Sent Done false
4158 Acked Read 9 Done Sent Done false
4159 Acked Flush 0 Done Sent Done false
4160 Acked Read 5 Done Sent Done false
4161 Acked Read 6 Done Sent Done false
4162 Acked Write 8 Done Sent Done false
4163 Acked Read 2 Done Sent Done false
4164 Acked Read 4 Done Sent Done false
4165 Acked Write 2 Done Sent Done false
4166 Acked Write 7 Done Sent Done false
4167 Acked Write 7 Done Sent Done false
4168 Acked Write 9 Done Sent Done false
4169 Acked Flush 0 Done Sent Done false
4170 Acked Read 8 Done Sent Done false
4171 Acked Read 6 Done Sent Done false
4172 Acked Read 9 Done Sent Done false
4173 Acked Write 9 Done Sent Done false
4174 Acked Read 10 Done Sent Done false
4175 Acked Read 8 Done Sent Done false
4176 Acked Write 4 Done Sent Done false
4177 Acked Write 9 Done Sent Done false
4178 Acked Read 1 Done Sent Done false
4179 Acked Write 5 Done Sent Done false
4180 Acked Read 7 Done Sent Done false
4181 Acked Write 10 Done Sent Done false
4182 Acked Write 10 Done Sent Done false
4183 Acked Write 6 Done Sent Done false
4184 Acked Write 9 Done Sent Done false
4185 Acked Write 7 Done Sent Done false
4186 Acked Write 8 Done Sent Done false
4187 Acked Read 9 Done Sent Done false
4188 Acked Read 9 Done Sent Done false
4189 Acked Write 4 Done Sent Done false
4190 Acked Read 1 Done Sent Done false
4191 Acked Write 10 Done Sent Done false
4192 Acked Write 4 Done Sent Done false
4193 Acked Write 1 Done Sent Done false
4194 Acked Read 3 Done Sent Done false
4195 Acked Write 9 Done Sent Done false
4196 Acked Write 10 Done Sent Done false
4197 Acked Write 4 Done Sent Done false
4198 Acked Write 3 Done Sent Done false
4199 Acked Read 3 Done Sent Done false
4200 Acked Flush 0 Done Sent Done false
4201 Acked Write 6 Done Sent Done false
4202 Acked Write 3 Done Sent Done false
4203 Acked Write 2 Done Sent Done false
4204 Acked Flush 0 Done Sent Done false
4205 Acked Write 8 Done Sent Done false
4206 Acked Write 3 Done Sent Done false
4207 Acked Read 2 Done Sent Done false
4208 Acked Read 8 Done Sent Done false
4209 Acked Write 1 Done Sent Done false
4210 Acked Read 9 Done Sent Done false
4211 Acked Read 10 Done Sent Done false
4212 Acked Write 7 Done Sent Done false
4213 Acked Write 10 Done Sent Done false
4214 Acked Read 3 Done Sent Done false
4215 Acked Read 9 Done Sent Done false
4216 Acked Read 3 Done Sent Done false
4217 Acked Read 5 Done Sent Done false
4218 Acked Write 7 Done Sent Done false
4219 Acked Read 9 Done Sent Done false
4220 Acked Flush 0 Done Sent Done false
4221 Acked Write 1 Done Sent Done false
4222 Acked Read 5 Done Sent Done false
4223 Acked Write 7 Done Sent Done false
4224 Acked Write 5 Done Sent Done false
4225 Acked Read 2 Done Sent Done false
4226 Acked Read 5 Done Sent Done false
4227 Acked Read 6 Done Sent Done false
4228 Acked Read 4 Done Sent Done false
4229 Acked Write 6 Done Sent Done false
4230 Acked Write 3 Done Sent Done false
4231 Acked Read 7 Done Sent Done false
4232 Acked Read 6 Done Sent Done false
4233 Acked Read 8 Done Sent Done false
4234 Acked Read 3 Done Sent Done false
4235 Acked Write 8 Done Sent Done false
4236 Acked Write 10 Done Sent Done false
4237 Acked Write 1 Done Sent Done false
4238 Acked Read 7 Done Sent Done false
4239 Acked Write 10 Done Sent Done false
4240 Acked Read 6 Done Sent Done false
4241 Acked Write 8 Done Sent Done false
4242 Acked Read 10 Done Sent Done false
4243 Acked Write 10 Done Sent Done false
4244 Acked Write 7 Done Sent Done false
4245 Acked Write 6 Done Sent Done false
4246 Acked Flush 0 Done Sent Done false
4247 Acked Read 1 Done Sent Done false
4248 Acked Flush 0 Done Sent Done false
4249 Acked Write 4 Done Sent Done false
4250 Acked Write 2 Done Sent Done false
4251 Acked Read 8 Done Sent Done false
4252 Acked Flush 0 Done Sent Done false
4253 Acked Write 9 Done Sent Done false
4254 Acked Write 6 Done Sent Done false
4255 Acked Read 9 Done Sent Done false
4256 Acked Read 7 Done Sent Done false
4257 Acked Write 2 Done Sent Done false
4258 Acked Write 10 Done Sent Done false
4259 Acked Read 7 Done Sent Done false
4260 Acked Flush 0 Done Sent Done false
4261 Acked Write 6 Done Sent Done false
4262 Acked Read 8 Done Sent Done false
4263 Acked Read 4 Done Sent Done false
4264 Acked Read 3 Done Sent Done false
4265 Acked Write 8 Done Sent Done false
4266 Acked Read 3 Done Sent Done false
4267 Acked Write 2 Done Sent Done false
4268 Acked Read 10 Done Sent Done false
4269 Acked Write 4 Done Sent Done false
4270 Acked Write 8 Done Sent Done false
4271 Acked Read 4 Done Sent Done false
4272 Acked Flush 0 Done Sent Done false
4273 Acked Write 1 Done Sent Done false
4274 Acked Read 4 Done Sent Done false
4275 Acked Read 2 Done Sent Done false
4276 Acked Read 2 Done Sent Done false
4277 Acked Read 1 Done Sent Done false
4278 Acked Read 5 Done Sent Done false
4279 Acked Write 1 Done Sent Done false
4280 Acked Flush 0 Done Sent Done false
4281 Acked Read 1 Done Sent Done false
4282 Acked Write 9 Done Sent Done false
4283 Acked Flush 0 Done Sent Done false
4284 Acked Write 3 Done Sent Done false
4285 Acked Write 6 Done Sent Done false
4286 Acked Read 8 Done Sent Done false
4287 Acked Write 4 Done Sent Done false
4288 Acked Write 8 Done Sent Done false
4289 Acked Read 3 Done Sent Done false
4290 Acked Write 7 Done Sent Done false
4291 Acked Read 2 Done Sent Done false
4292 Acked Write 4 Done Sent Done false
4293 Acked Read 4 Done Sent Done false
4294 Acked Read 1 Done Sent Done false
4295 Acked Write 7 Done Sent Done false
4296 Acked Read 8 Done Sent Done false
4297 Acked Read 9 Done Sent Done false
4298 Acked Write 6 Done Sent Done false
4299 Acked Read 5 Done Sent Done false
4300 Acked Write 1 Done Sent Done false
4301 Acked Write 10 Done Sent Done false
4302 Acked Read 2 Done Sent Done false
4303 Acked Write 7 Done Sent Done false
4304 Acked Read 7 Done Sent Done false
4305 Acked Read 1 Done Sent Done false
4306 Acked Read 3 Done Sent Done false
4307 Acked Read 3 Done Sent Done false
4308 Acked Read 9 Done Sent Done false
4309 Acked Write 5 Done Sent Done false
4310 Acked Read 2 Done Sent Done false
4311 Acked Flush 0 Done Sent Done false
4312 Acked Write 4 Done Sent Done false
4313 Acked Write 1 Done Sent Done false
4314 Acked Read 9 Done Sent Done false
4315 Acked Write 9 Done Sent Done false
4316 Acked Read 6 Done Sent Done false
4317 Acked Write 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:03.126888992Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:03.638023065Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1222)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:04.63968539Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:05.640461677Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.383150297Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56219)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.383299401Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.384200246Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.384718391Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.384739186Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(4016))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.384753323Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.38476639Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(4016))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.384780236Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(4016))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:06.385026308Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4312 Acked Write 4 Done Done Done true
4313 Acked Write 1 Done Done Done true
4314 Acked Read 9 Done Done Done true
4315 Acked Write 9 Done Done Done true
4316 Acked Read 6 Done Done Done true
4317 Acked Write 3 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4311)) Some(JobId(4311)) Some(JobId(4311))
Downstairs last five completed: 4311 4310 4309 4308 4307
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:6 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4318 4317 4316 4315 4314
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 28 06:04:10.628 INFO request completed, latency_us: 319, response_code: 204, uri: /stop/cid/2, method: POST, req_id: bf66f5b6-de9a-45f4-8a5d-1c1ae13f5b98, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:10.629472648Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:04:10.629{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:10.629526365Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
INFO request completed, latency_us: 110, response_code: 200, uri: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:10.629553348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
/state/cid/2, method: GET, req_id: 4a253fc8-e9f5-4d8d-b4dd-e9a2e3cb517e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:10.629618513Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:04:14.631 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 58297db9-d4e8-4498-81fb-1d89f50af94a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 3548 len 28672 data: 3 3 3 3 2 2 2
002/300 Write block 113 len 12288 data: 2 2 3
003/300 Read block 3582 len 8192
004/300 Flush
005/300 Read block 2009 len 8192
006/300 Write block 3850 len 12288 data: 5 5 6
007/300 Write block 382 len 8192 data: 6 6
008/300 Read block 2819 len 40960
009/300 Read block 3979 len 32768
010/300 Read block 4224 len 8192
011/300 Write block 704 len 32768 data: 6 5 5 5 4 5 5 4
012/300 Flush
013/300 Read block 1091 len 12288
014/300 Read block 2365 len 20480
015/300 Read block 652 len 16384
016/300 Read block 2068 len 20480
017/300 Read block 3314 len 24576
018/300 Write block 2769 len 16384 data: 2 2 2 2
019/300 Write block 2002 len 24576 data: 3 3 2 2 3 3
020/300 Read block 4924 len 16384
021/300 Write block 2311 len 32768 data: 3 3 3 3 2 3 4 5
022/300 Read block 1728 len 16384
023/300 Read block 2924 len 4096
024/300 Read block 3983 len 16384
025/300 Write block 2298 len 16384 data: 3 2 2 2
026/300 Flush
027/300 Write block 854 len 16384 data: 3 3 2 3
028/300 Write block 287 len 36864 data: 3 3 3 3 3 4 3 3 4
029/300 Read block 207 len 4096
030/300 Read block 1732 len 28672
031/300 Read block 624 len 12288
032/300 Flush
033/300 Read block 1858 len 4096
034/300 Read block 9 len 40960
035/300 Read block 391 len 36864
036/300 Flush
037/300 Read block 4511 len 12288
038/300 Read block 2259 len 8192
039/300 Read block 1485 len 40960
040/300 Flush
041/300 Read block 2826 len 32768
042/300 Write block 3015 len 12288 data: 3 2 2
043/300 Write block 873 len 4096 data: 4
044/300 Write block 2214 len 28672 data: 8 8 8 8 8 6 4
045/300 Write block 54 len 32768 data: 4 4 4 4 4 3 3 2
046/300 Write block 949 len 32768 data: 3 3 3 2 2 2 2 2
047/300 Write block 70 len 20480 data: 2 2 2 2 2
048/300 Write block 1988 len 8192 data: 5 5
049/300 Write block 498 len 28672 data: 4 4 4 4 4 4 4
050/300 Read block 4559 len 16384
051/300 Read block 58 len 24576
052/300 Read block 116 len 28672
053/300 Read block 3314 len 36864
054/300 Read block 1581 len 4096
055/300 Write block 816 len 12288 data: 3 3 3
056/300 Write block 2086 len 24576 data: 4 4 4 4 4 4
057/300 Write block 95 len 4096 data: 5
058/300 Read block 148 len 20480
059/300 Write block 4371 len 4096 data: 3
060/300 Read block 1462 len 4096
061/300 Read block 3874 len 40960
062/300 Flush
063/300 Read block 2807 len 32768
064/300 Read block 4673 len 8192
065/300 Read block 1352 len 24576
066/300 Write block 325 len 8192 data: 3 2
067/300 Read block 2550 len 24576
068/300 Read block 2704 len 32768
069/300 Read block 3345 len 16384
070/300 Read block 108 len 12288
071/300 Write block 3141 len 16384 data: 2 2 2 2
072/300 Flush
073/300 Flush
074/300 Write block 4267 len 16384 data: 3 3 3 3
075/300 Read block 1500 len 40960
076/300 Read block 2390 len 36864
077/300 Write block 1992 len 12288 data: 3 2 2
078/300 Write block 4062 len 4096 data: 3
079/300 Write block 61 len 24576 data: 3 3 3 3 2 2
080/300 Write block 3106 len 36864 data: 3 4 5 5 5 5 6 5 7
081/300 Flush
082/300 Read block 3390 len 20480
083/300 Read block 4413 len 4096
084/300 Write block 3378 len 8192 data: 2 2
085/300 Read block 40 len 8192
086/300 Read block 1707 len 12288
087/300 Read block 3974 len 4096
088/300 Write block 984 len 32768 data: 3 3 3 3 3 3 3 4
089/300 Flush
090/300 Write block 4298 len 4096 data: 3
091/300 Write block 159 len 12288 data: 2 3 4
092/300 Write block 3376 len 16384 data: 3 2 3 3
093/300 Read block 3224 len 36864
094/300 Flush
095/300 Read block 4813 len 16384
096/300 Read block 617 len 36864
097/300 Write block 3307 len 40960 data: 7 7 6 4 4 5 5 3 3 3
098/300 Read block 2794 len 32768
099/300 Write block 3156 len 8192 data: 4 5
100/300 Read block 3985 len 32768
101/300 Flush
102/300 Write block 2491 len 40960 data: 4 2 2 2 2 2 2 2 2 2
103/300 Read block 3393 len 24576
104/300 Write block 2302 len 16384 data: 2 2 3 3
105/300 Write block 4755 len 24576 data: 3 4 4 4 4 4
106/300 Read block 281 len 24576
107/300 Write block 2165 len 16384 data: 4 4 4 5
108/300 Write block 1844 len 32768 data: 3 3 3 2 3 3 3 4
109/300 Write block 553 len 4096 data: 6
110/300 Write block 2097 len 24576 data: 5 5 4 4 4 3
111/300 Read block 33 len 40960
112/300 Read block 1629 len 12288
113/300 Read block 1610 len 20480
114/300 Read block 2603 len 36864
115/300 Read block 1720 len 32768
116/300 Read block 4736 len 12288
117/300 Read block 1499 len 32768
118/300 Read block 848 len 20480
119/300 Flush
120/300 Write block 870 len 8192 data: 5 4
121/300 Write block 2290 len 16384 data: 2 3 3 3
122/300 Read block 3574 len 32768
123/300 Write block 3481 len 12288 data: 4 4 4
124/300 Read block 3291 len 12288
125/300 Write block 4066 len 8192 data: 3 3
126/300 Flush
127/300 Flush
128/300 Flush
129/300 Read block 1569 len 8192
130/300 Read block 1325 len 24576
131/300 Write block 3043 len 24576 data: 2 2 2 2 3 3
132/300 Write block 1820 len 8192 data: 3 4
133/300 Write block 4375 len 24576 data: 4 4 4 4 4 4
134/300 Write block 2913 len 20480 data: 2 2 2 2 3
135/300 Read block 1431 len 40960
136/300 Read block 4250 len 40960
137/300 Read block 2576 len 36864
138/300 Read block 1043 len 16384
139/300 Write block 1252 len 24576 data: 3 4 5 6 6 6
140/300 Flush
141/300 Write block 515 len 40960 data: 3 3 3 3 3 4 5 5 5 5
142/300 Flush
143/300 Write block 4436 len 28672 data: 3 3 3 2 2 3 4
144/300 Read block 1239 len 40960
145/300 Write block 2184 len 12288 data: 2 2 3
146/300 Write block 4306 len 28672 data: 2 3 2 2 2 2 2
147/300 Write block 3706 len 8192 data: 3 4
148/300 Write block 2553 len 24576 data: 2 2 2 2 2 2
149/300 Flush
150/300 Write block 1201 len 36864 data: 2 2 3 3 4 3 3 3 3
151/300 Read block 4016 len 40960
152/300 Write block 2373 len 4096 data: 3
153/300 Write block 1286 len 32768 data: 2 2 2 3 3 4 4 4
154/300 Read block 2378 len 4096
155/300 Read block 2116 len 36864
156/300 Write block 1752 len 36864 data: 3 3 3 3 2 2 3 3 3
157/300 Read block 1665 len 4096
158/300 Write block 1419 len 28672 data: 2 3 3 3 4 3 3
159/300 Read block 4042 len 32768
160/300 Write block 3557 len 20480 data: 3 3 3 2 2
161/300 Write block 1503 len 32768 data: 2 2 2 2 2 2 2 2
162/300 Write block 4828 len 20480 data: 5 5 7 7 6
163/300 Read block 3379 len 24576
164/300 Flush
165/300 Read block 3009 len 24576
166/300 Write block 1210 len 32768 data: 3 3 2 2 2 2 2 2
167/300 Read block 1171 len 36864
168/300 Read block 3358 len 28672
169/300 Write block 1802 len 4096 data: 2
170/300 Flush
171/300 Write block 4783 len 32768 data: 4 3 3 3 3 4 4 3
172/300 Write block 3568 len 16384 data: 2 2 2 2
173/300 Write block 2897 len 16384 data: 2 3 2 2
174/300 Flush
175/300 Read block 200 len 16384
176/300 Write block 207 len 24576 data: 2 2 3 3 3 3
177/300 Read block 2606 len 36864
178/300 Write block 3457 len 24576 data: 5 5 5 2 3 2
179/300 Read block 889 len 24576
180/300 Read block 4611 len 28672
181/300 Write block 654 len 36864 data: 3 3 2 4 5 5 5 5 5
182/300 Write block 183 len 36864 data: 2 2 2 2 2 3 3 3 4
183/300 Write block 2735 len 28672 data: 3 3 3 2 2 4 4
184/300 Write block 2032 len 32768 data: 4 4 3 2 2 2 2 2
185/300 Write block 2490 len 4096 data: 4
186/300 Read block 119 len 36864
187/300 Read block 3344 len 12288
188/300 Read block 4573 len 24576
189/300 Read block 3487 len 4096
190/300 Write block 2647 len 20480 data: 3 3 3 3 3
191/300 Read block 3456 len 36864
192/300 Write block 4996 len 12288 data: 3 3 3
193/300 Write block 2917 len 20480 data: 4 4 3 3 3
194/300 Write block 2221 len 32768 data: 3 3 3 3 4 4 5 5
195/300 Write block 2262 len 40960 data: 3 3 4 4 4 4 4 3 3 2
196/300 Read block 2346 len 32768
197/300 Read block 624 len 20480
198/300 Flush
199/300 Read block 4588 len 32768
200/300 Read block 3359 len 16384
201/300 Read block 1359 len 40960
202/300 Read block 3279 len 20480
203/300 Read block 3436 len 20480
204/300 Read block 3887 len 28672
205/300 Read block 2336 len 28672
206/300 Read block 3663 len 24576
207/300 Read block 393 len 36864
208/300 Write block 739 len 16384 data: 3 3 2 2
209/300 Read block 145 len 40960
210/300 Write block 4833 len 24576 data: 3 3 4 4 5 5
211/300 Flush
212/300 Flush
213/300 Read block 4635 len 20480
214/300 Read block 2688 len 32768
215/300 Write block 3143 len 12288 data: 3 3 2
216/300 Read block 262 len 24576
217/300 Write block 226 len 36864 data: 3 3 2 2 3 3 2 2 3
218/300 Flush
219/300 Read block 4725 len 36864
220/300 Read block 3036 len 24576
221/300 Read block 323 len 40960
222/300 Read block 1051 len 4096
223/300 Read block 336 len 8192
224/300 Write block 3850 len 28672 data: 6 6 7 6 5 4 4
225/300 Read block 4809 len 28672
226/300 Write block 811 len 32768 data: 3 4 4 3 2 4 4 4
227/300 Read block 4031 len 8192
228/300 Write block 474 len 12288 data: 4 5 5
229/300 Read block 1564 len 24576
230/300 Write block 1433 len 20480 data: 2 2 2 2 3
231/300 Write block 4096 len 32768 data: 3 3 3 3 3 4 4 4
232/300 Read block 4059 len 40960
233/300 Write block 2729 len 20480 data: 3 5 4 4 4
234/300 Write block 188 len 40960 data: 4 4 4 5 5 5 5 5 5 4
235/300 Flush
236/300 Write block 4224 len 32768 data: 3 5 6 6 7 8 7 7
237/300 Read block 75 len 20480
238/300 Flush
239/300 Read block 189 len 12288
240/300 Read block 1342 len 36864
241/300 Flush
242/300 Write block 1394 len 12288 data: 2 2 2
243/300 Read block 2493 len 28672
244/300 Flush
245/300 Read block 1684 len 24576
246/300 Write block 4127 len 28672 data: 4 4 4 4 4 3 3
247/300 Write block 63 len 8192 data: 4 4
248/300 Read block 4559 len 12288
249/300 Read block 4892 len 8192
250/300 Read block 2359 len 8192
251/300 Read block 4862 len 8192
252/300 Read block 4441 len 32768
253/300 Read block 3345 len 12288
254/300 Write block 1477 len 40960 data: 2 2 2 2 2 2 3 3 5 5
255/300 Read block 971 len 28672
256/300 Read block 1584 len 20480
257/300 Flush
258/300 Write block 2276 len 40960 data: 6 6 7 7 7 6 4 5 3 3
259/300 Read block 1862 len 40960
260/300 Read block 920 len 4096
261/300 Read block 4131 len 24576
262/300 Write block 2020 len 4096 data: 3
263/300 Write block 4950 len 40960 data: 2 3 3 4 5 4 4 4 4 3
264/300 Read block 3517 len 24576
265/300 Read block 863 len 36864
266/300 Read block 78 len 36864
267/300 Flush
268/300 Write block 2806 len 24576 data: 4 4 3 3 3 4
269/300 Write block 2121 len 8192 data: 4 4
270/300 Read block 2080 len 12288
271/300 Write block 311 len 4096 data: 2
272/300 Read block 3034 len 40960
273/300 Read block 3423 len 28672
274/300 Read block 3241 len 28672
275/300 Read block 1865 len 20480
276/300 Read block 1601 len 40960
277/300 Write block 3347 len 20480 data: 5 3 2 2 3
278/300 Flush
279/300 Read block 1665 len 20480
280/300 Read block 4788 len 40960
281/300 Read block 3440 len 40960
282/300 Write block 1008 len 40960 data: 3 3 3 3 3 3 3 3 4 5
283/300 Write block 958 len 12288 data: 3 3 3
284/300 Write block 1283 len 4096 data: 4
285/300 Write block 2026 len 32768 data: 3 3 5 5 4 4 5 5
286/300 Read block 3571 len 8192
287/300 Read block 1274 len 20480
288/300 Flush
289/300 Read block 3874 len 12288
290/300 Write block 3159 len 8192 data: 6 6
291/300 Read block 560 len 4096
292/300 Write block 4285 len 40960 data: 3 3 3 2 3 4 4 4 3 3
293/300 Write block 3101 len 12288 data: 5 5 5
294/300 Read block 453 len 12288
295/300 Read block 281 len 40960
296/300 Write block 3792 len 32768 data: 3 2 4 4 4 3 3 3
297/300 Write block 4592 len 32768 data: 4 4 4 5 4 4 4 2
298/300 Read block 4816 len 4096
299/300 Flush
300/300 Read block 4916 len 36864
301/300 Write block 975 len 16384 data: 3 3 3 4
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:04:14.874 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1f7506d9-18fb-47f3-9ec1-e5758272bb42, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4319 Acked Write 7 Done Done Sent false
4320 Acked Write 3 Done Done Sent false
4321 Acked Read 2 Done Done Sent false
4322 Acked Flush 0 Done Done Sent false
4323 Acked Read 2 Done Done Sent false
4324 Acked Write 3 Done Done Sent false
4325 Acked Write 2 Done Done Sent false
4326 Acked Read 10 Done Done Sent false
4327 Acked Read 8 Done Done Sent false
4328 Acked Read 2 Done Done Sent false
4329 Acked Write 8 Done Done Sent false
4330 Acked Flush 0 Done Done Sent false
4331 Acked Read 3 Done Done Sent false
4332 Acked Read 5 Done Done Sent false
4333 Acked Read 4 Done Done Sent false
4334 Acked Read 5 Done Done Sent false
4335 Acked Read 6 Done Done Sent false
4336 Acked Write 4 Done Done Sent false
4337 Acked Write 6 Done Done Sent false
4338 Acked Read 4 Done Done Sent false
4339 Acked Write 8 Done Done Sent false
4340 Acked Read 4 Done Done Sent false
4341 Acked Read 1 Done Done Sent false
4342 Acked Read 4 Done Done Sent false
4343 Acked Write 4 Done Done Sent false
4344 Acked Flush 0 Done Done Sent false
4345 Acked Write 4 Done Done Sent false
4346 Acked Write 9 Done Done Sent false
4347 Acked Read 1 Done Done Sent false
4348 Acked Read 7 Done Done Sent false
4349 Acked Read 3 Done Done Sent false
4350 Acked Flush 0 Done Done Sent false
4351 Acked Read 1 Done Done Sent false
4352 Acked Read 10 Done Done Sent false
4353 Acked Read 9 Done Done Sent false
4354 Acked Flush 0 Done Done Sent false
4355 Acked Read 3 Done Done Sent false
4356 Acked Read 2 Done Done Sent false
4357 Acked Read 10 Done Done Sent false
4358 Acked Flush 0 Done Done Sent false
4359 Acked Read 8 Done Done Sent false
4360 Acked Write 3 Done Done Sent false
4361 Acked Write 1 Done Done Sent false
4362 Acked Write 7 Done Done Sent false
4363 Acked Write 8 Done Done Sent false
4364 Acked Write 8 Done Done Sent false
4365 Acked Write 5 Done Done Sent false
4366 Acked Write 2 Done Done Sent false
4367 Acked Write 7 Done Done Sent false
4368 Acked Read 4 Done Done Sent false
4369 Acked Read 6 Done Done Sent false
4370 Acked Read 7 Done Done Sent false
4371 Acked Read 9 Done Done Sent false
4372 Acked Read 1 Done Done Sent false
4373 Acked Write 3 Done Done Sent false
4374 Acked Write 6 Done Done Sent false
4375 Acked Write 1 Done Done Sent false
4376 Acked Read 5 Done Done Sent false
4377 Acked Write 1 Done Done Sent false
4378 Acked Read 1 Done Done Sent false
4379 Acked Read 10 Done Done Sent false
4380 Acked Flush 0 Done Done Sent false
4381 Acked Read 8 Done Done Sent false
4382 Acked Read 2 Done Done Sent false
4383 Acked Read 6 Done Done Sent false
4384 Acked Write 2 Done Done Sent false
4385 Acked Read 6 Done Done Sent false
4386 Acked Read 8 Done Done Sent false
4387 Acked Read 4 Done Done Sent false
4388 Acked Read 3 Done Done Sent false
4389 Acked Write 4 Done Done Sent false
4390 Acked Flush 0 Done Done Sent false
4391 Acked Flush 0 Done Done Sent false
4392 Acked Write 4 Done Done Sent false
4393 Acked Read 10 Done Done Sent false
4394 Acked Read 9 Done Done Sent false
4395 Acked Write 3 Done Done Sent false
4396 Acked Write 1 Done Done Sent false
4397 Acked Write 6 Done Done Sent false
4398 Acked Write 9 Done Done Sent false
4399 Acked Flush 0 Done Done Sent false
4400 Acked Read 5 Done Done Sent false
4401 Acked Read 1 Done Done Sent false
4402 Acked Write 2 Done Done Sent false
4403 Acked Read 2 Done Done Sent false
4404 Acked Read 3 Done Done Sent false
4405 Acked Read 1 Done Done Sent false
4406 Acked Write 8 Done Done Sent false
4407 Acked Flush 0 Done Done Sent false
4408 Acked Write 1 Done Done Sent false
4409 Acked Write 3 Done Done Sent false
4410 Acked Write 4 Done Done Sent false
4411 Acked Read 9 Done Done Sent false
4412 Acked Flush 0 Done Done Sent false
4413 Acked Read 4 Done Done Sent false
4414 Acked Read 9 Done Done Sent false
4415 Acked Write 10 Done Done Sent false
4416 Acked Read 8 Done Done Sent false
4417 Acked Write 2 Done Done Sent false
4418 Acked Read 8 Done Done Sent false
4419 Acked Flush 0 Done Done Sent false
4420 Acked Write 10 Done Done Sent false
4421 Acked Read 6 Done Done Sent false
4422 Acked Write 4 Done Done Sent false
4423 Acked Write 6 Done Done Sent false
4424 Acked Read 6 Done Done Sent false
4425 Acked Write 4 Done Done Sent false
4426 Acked Write 8 Done Done Sent false
4427 Acked Write 1 Done Done Sent false
4428 Acked Write 6 Done Done Sent false
4429 Acked Read 10 Done Done Sent false
4430 Acked Read 3 Done Done Sent false
4431 Acked Read 5 Done Done Sent false
4432 Acked Read 9 Done Done Sent false
4433 Acked Read 8 Done Done Sent false
4434 Acked Read 3 Done Done Sent false
4435 Acked Read 8 Done Done Sent false
4436 Acked Read 5 Done Done Sent false
4437 Acked Flush 0 Done Done Sent false
4438 Acked Write 2 Done Done Sent false
4439 Acked Write 4 Done Done Sent false
4440 Acked Read 8 Done Done Sent false
4441 Acked Write 3 Done Done Sent false
4442 Acked Read 3 Done Done Sent false
4443 Acked Write 2 Done Done Sent false
4444 Acked Flush 0 Done Done Sent false
4445 Acked Flush 0 Done Done Sent false
4446 Acked Flush 0 Done Done Sent false
4447 Acked Read 2 Done Done Sent false
4448 Acked Read 6 Done Done Sent false
4449 Acked Write 6 Done Done Sent false
4450 Acked Write 2 Done Done Sent false
4451 Acked Write 6 Done Done Sent false
4452 Acked Write 5 Done Done Sent false
4453 Acked Read 10 Done Done Sent false
4454 Acked Read 10 Done Done Sent false
4455 Acked Read 9 Done Done Sent false
4456 Acked Read 4 Done Done Sent false
4457 Acked Write 6 Done Done Sent false
4458 Acked Flush 0 Done Done Sent false
4459 Acked Write 10 Done Done Sent false
4460 Acked Flush 0 Done Done Sent false
4461 Acked Write 7 Done Done Sent false
4462 Acked Read 10 Done Done Sent false
4463 Acked Write 3 Done Done Sent false
4464 Acked Write 7 Done Done Sent false
4465 Acked Write 2 Done Done Sent false
4466 Acked Write 6 Done Done Sent false
4467 Acked Flush 0 Done Done Sent false
4468 Acked Write 9 Done Done Sent false
4469 Acked Read 10 Done Done Sent false
4470 Acked Write 1 Done Done Sent false
4471 Acked Write 8 Done Done Sent false
4472 Acked Read 1 Done Done Sent false
4473 Acked Read 9 Done Done Sent false
4474 Acked Write 9 Done Done Sent false
4475 Acked Read 1 Done Done Sent false
4476 Acked Write 7 Done Done Sent false
4477 Acked Read 8 Done Done Sent false
4478 Acked Write 5 Done Done Sent false
4479 Acked Write 8 Done Done Sent false
4480 Acked Write 5 Done Done Sent false
4481 Acked Read 6 Done Done Sent false
4482 Acked Flush 0 Done Done Sent false
4483 Acked Read 6 Done Done Sent false
4484 Acked Write 8 Done Done Sent false
4485 Acked Read 9 Done Done Sent false
4486 Acked Read 7 Done Done Sent false
4487 Acked Write 1 Done Done Sent false
4488 Acked Flush 0 Done Done Sent false
4489 Acked Write 8 Done Done Sent false
4490 Acked Write 4 Done Done Sent false
4491 Acked Write 4 Done Done Sent false
4492 Acked Flush 0 Done Done Sent false
4493 Acked Read 4 Done Done Sent false
4494 Acked Write 6 Done Done Sent false
4495 Acked Read 9 Done Done Sent false
4496 Acked Write 6 Done Done Sent false
4497 Acked Read 6 Done Done Sent false
4498 Acked Read 7 Done Done Sent false
4499 Acked Write 9 Done Done Sent false
4500 Acked Write 9 Done Done Sent false
4501 Acked Write 7 Done Done Sent false
4502 Acked Write 8 Done Done Sent false
4503 Acked Write 1 Done Done Sent false
4504 Acked Read 9 Done Done Sent false
4505 Acked Read 3 Done Done Sent false
4506 Acked Read 6 Done Done Sent false
4507 Acked Read 1 Done Done Sent false
4508 Acked Write 5 Done Done Sent false
4509 Acked Read 9 Done Done Sent false
4510 Acked Write 3 Done Done Sent false
4511 Acked Write 5 Done Done Sent false
4512 Acked Write 8 Done Done Sent false
4513 Acked Write 10 Done Done Sent false
4514 Acked Read 8 Done Done Sent false
4515 Acked Read 5 Done Done Sent false
4516 Acked Flush 0 Done Done Sent false
4517 Acked Read 8 Done Done Sent false
4518 Acked Read 4 Done Done Sent false
4519 Acked Read 10 Done Done Sent false
4520 Acked Read 5 Done Done Sent false
4521 Acked Read 5 Done Done Sent false
4522 Acked Read 7 Done Done Sent false
4523 Acked Read 7 Done Done Sent false
4524 Acked Read 6 Done Done Sent false
4525 Acked Read 9 Done Done Sent false
4526 Acked Write 4 Done Done Sent false
4527 Acked Read 10 Done Done Sent false
4528 Acked Write 6 Done Done Sent false
4529 Acked Flush 0 Done Done Sent false
4530 Acked Flush 0 Done Done Sent false
4531 Acked Read 5 Done Done Sent false
4532 Acked Read 8 Done Done Sent false
4533 Acked Write 3 Done Done Sent false
4534 Acked Read 6 Done Done Sent false
4535 Acked Write 9 Done Done Sent false
4536 Acked Flush 0 Done Done Sent false
4537 Acked Read 9 Done Done Sent false
4538 Acked Read 6 Done Done Sent false
4539 Acked Read 10 Done Done Sent false
4540 Acked Read 1 Done Done Sent false
4541 Acked Read 2 Done Done Sent false
4542 Acked Write 7 Done Done Sent false
4543 Acked Read 7 Done Done Sent false
4544 Acked Write 8 Done Done Sent false
4545 Acked Read 2 Done Done Sent false
4546 Acked Write 3 Done Done Sent false
4547 Acked Read 6 Done Done Sent false
4548 Acked Write 5 Done Done Sent false
4549 Acked Write 8 Done Done Sent false
4550 Acked Read 10 Done Done Sent false
4551 Acked Write 5 Done Done Sent false
4552 Acked Write 10 Done Done Sent false
4553 Acked Flush 0 Done Done Sent false
4554 Acked Write 8 Done Done Sent false
4555 Acked Read 5 Done Done Sent false
4556 Acked Flush 0 Done Done Sent false
4557 Acked Read 3 Done Done Sent false
4558 Acked Read 9 Done Done Sent false
4559 Acked Flush 0 Done Done Sent false
4560 Acked Write 3 Done Done Sent false
4561 Acked Read 7 Done Done Sent false
4562 Acked Flush 0 Done Done Sent false
4563 Acked Read 6 Done Done Sent false
4564 Acked Write 7 Done Done Sent false
4565 Acked Write 2 Done Done Sent false
4566 Acked Read 3 Done Done Sent false
4567 Acked Read 2 Done Done Sent false
4568 Acked Read 2 Done Done Sent false
4569 Acked Read 2 Done Done Sent false
4570 Acked Read 8 Done Done Sent false
4571 Acked Read 3 Done Done Sent false
4572 Acked Write 10 Done Done Sent false
4573 Acked Read 7 Done Done Sent false
4574 Acked Read 5 Done Done Sent false
4575 Acked Flush 0 Done Done Sent false
4576 Acked Write 10 Done Done Sent false
4577 Acked Read 10 Done Done Sent false
4578 Acked Read 1 Done Done Sent false
4579 Acked Read 6 Done Done Sent false
4580 Acked Write 1 Done Done Sent false
4581 Acked Write 10 Done Done Sent false
4582 Acked Read 6 Done Done Sent false
4583 Acked Read 9 Done Done Sent false
4584 Acked Read 9 Done Done Sent false
4585 Acked Flush 0 Done Done Sent false
4586 Acked Write 6 Done Done Sent false
4587 Acked Write 2 Done Done Sent false
4588 Acked Read 3 Done Done Sent false
4589 Acked Write 1 Done Done Sent false
4590 Acked Read 10 Done Done Sent false
4591 Acked Read 7 Done Done Sent false
4592 Acked Read 7 Done Done Sent false
4593 Acked Read 5 Done Done Sent false
4594 Acked Read 10 Done Done Sent false
4595 Acked Write 5 Done Done Sent false
4596 Acked Flush 0 Done Done Sent false
4597 Acked Read 5 Done Done Sent false
4598 Acked Read 10 Done Done Sent false
4599 Acked Read 10 Done Done Sent false
4600 Acked Write 10 Done Done Sent false
4601 Acked Write 3 Done Done Sent false
4602 Acked Write 1 Done Done Sent false
4603 Acked Write 8 Done Done Sent false
4604 Acked Read 2 Done Done Sent false
4605 Acked Read 5 Done Done Sent false
4606 Acked Flush 0 Done Done Sent false
4607 Acked Read 3 Done Done Sent false
4608 Acked Write 2 Done Done Sent false
4609 Acked Read 1 Done Done Sent false
4610 Acked Write 10 Done Done Sent false
4611 Acked Write 3 Done Done Sent false
4612 Acked Read 3 Done Done Sent false
4613 Acked Read 10 Done Done Sent false
4614 Acked Write 8 Done Done Sent false
4615 Acked Write 8 Done Done Sent false
4616 Acked Read 1 Done Done Sent false
4617 Acked Flush 0 Done Done Sent false
4618 Acked Read 9 Done Done Sent false
4619 Acked Write 4 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4617)) Some(JobId(4617)) Some(JobId(4318))
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4619 4618 4617 4616 4615
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:15.658883765Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:16.885970825Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:17.662093682Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:18.664093836Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4319 Acked Write 7 Done Done Sent false
4320 Acked Write 3 Done Done Sent false
4321 Acked Read 2 Done Done Sent false
4322 Acked Flush 0 Done Done Sent false
4323 Acked Read 2 Done Done Sent false
4324 Acked Write 3 Done Done Sent false
4325 Acked Write 2 Done Done Sent false
4326 Acked Read 10 Done Done Sent false
4327 Acked Read 8 Done Done Sent false
4328 Acked Read 2 Done Done Sent false
4329 Acked Write 8 Done Done Sent false
4330 Acked Flush 0 Done Done Sent false
4331 Acked Read 3 Done Done Sent false
4332 Acked Read 5 Done Done Sent false
4333 Acked Read 4 Done Done Sent false
4334 Acked Read 5 Done Done Sent false
4335 Acked Read 6 Done Done Sent false
4336 Acked Write 4 Done Done Sent false
4337 Acked Write 6 Done Done Sent false
4338 Acked Read 4 Done Done Sent false
4339 Acked Write 8 Done Done Sent false
4340 Acked Read 4 Done Done Sent false
4341 Acked Read 1 Done Done Sent false
4342 Acked Read 4 Done Done Sent false
4343 Acked Write 4 Done Done Sent false
4344 Acked Flush 0 Done Done Sent false
4345 Acked Write 4 Done Done Sent false
4346 Acked Write 9 Done Done Sent false
4347 Acked Read 1 Done Done Sent false
4348 Acked Read 7 Done Done Sent false
4349 Acked Read 3 Done Done Sent false
4350 Acked Flush 0 Done Done Sent false
4351 Acked Read 1 Done Done Sent false
4352 Acked Read 10 Done Done Sent false
4353 Acked Read 9 Done Done Sent false
4354 Acked Flush 0 Done Done Sent false
4355 Acked Read 3 Done Done Sent false
4356 Acked Read 2 Done Done Sent false
4357 Acked Read 10 Done Done Sent false
4358 Acked Flush 0 Done Done Sent false
4359 Acked Read 8 Done Done Sent false
4360 Acked Write 3 Done Done Sent false
4361 Acked Write 1 Done Done Sent false
4362 Acked Write 7 Done Done Sent false
4363 Acked Write 8 Done Done Sent false
4364 Acked Write 8 Done Done Sent false
4365 Acked Write 5 Done Done Sent false
4366 Acked Write 2 Done Done Sent false
4367 Acked Write 7 Done Done Sent false
4368 Acked Read 4 Done Done Sent false
4369 Acked Read 6 Done Done Sent false
4370 Acked Read 7 Done Done Sent false
4371 Acked Read 9 Done Done Sent false
4372 Acked Read 1 Done Done Sent false
4373 Acked Write 3 Done Done Sent false
4374 Acked Write 6 Done Done Sent false
4375 Acked Write 1 Done Done Sent false
4376 Acked Read 5 Done Done Sent false
4377 Acked Write 1 Done Done Sent false
4378 Acked Read 1 Done Done Sent false
4379 Acked Read 10 Done Done Sent false
4380 Acked Flush 0 Done Done Sent false
4381 Acked Read 8 Done Done Sent false
4382 Acked Read 2 Done Done Sent false
4383 Acked Read 6 Done Done Sent false
4384 Acked Write 2 Done Done Sent false
4385 Acked Read 6 Done Done Sent false
4386 Acked Read 8 Done Done Sent false
4387 Acked Read 4 Done Done Sent false
4388 Acked Read 3 Done Done Sent false
4389 Acked Write 4 Done Done Sent false
4390 Acked Flush 0 Done Done Sent false
4391 Acked Flush 0 Done Done Sent false
4392 Acked Write 4 Done Done Sent false
4393 Acked Read 10 Done Done Sent false
4394 Acked Read 9 Done Done Sent false
4395 Acked Write 3 Done Done Sent false
4396 Acked Write 1 Done Done Sent false
4397 Acked Write 6 Done Done Sent false
4398 Acked Write 9 Done Done Sent false
4399 Acked Flush 0 Done Done Sent false
4400 Acked Read 5 Done Done Sent false
4401 Acked Read 1 Done Done Sent false
4402 Acked Write 2 Done Done Sent false
4403 Acked Read 2 Done Done Sent false
4404 Acked Read 3 Done Done Sent false
4405 Acked Read 1 Done Done Sent false
4406 Acked Write 8 Done Done Sent false
4407 Acked Flush 0 Done Done Sent false
4408 Acked Write 1 Done Done Sent false
4409 Acked Write 3 Done Done Sent false
4410 Acked Write 4 Done Done Sent false
4411 Acked Read 9 Done Done Sent false
4412 Acked Flush 0 Done Done Sent false
4413 Acked Read 4 Done Done Sent false
4414 Acked Read 9 Done Done Sent false
4415 Acked Write 10 Done Done Sent false
4416 Acked Read 8 Done Done Sent false
4417 Acked Write 2 Done Done Sent false
4418 Acked Read 8 Done Done Sent false
4419 Acked Flush 0 Done Done Sent false
4420 Acked Write 10 Done Done Sent false
4421 Acked Read 6 Done Done Sent false
4422 Acked Write 4 Done Done Sent false
4423 Acked Write 6 Done Done Sent false
4424 Acked Read 6 Done Done Sent false
4425 Acked Write 4 Done Done Sent false
4426 Acked Write 8 Done Done Sent false
4427 Acked Write 1 Done Done Sent false
4428 Acked Write 6 Done Done Sent false
4429 Acked Read 10 Done Done Sent false
4430 Acked Read 3 Done Done Sent false
4431 Acked Read 5 Done Done Sent false
4432 Acked Read 9 Done Done Sent false
4433 Acked Read 8 Done Done Sent false
4434 Acked Read 3 Done Done Sent false
4435 Acked Read 8 Done Done Sent false
4436 Acked Read 5 Done Done Sent false
4437 Acked Flush 0 Done Done Sent false
4438 Acked Write 2 Done Done Sent false
4439 Acked Write 4 Done Done Sent false
4440 Acked Read 8 Done Done Sent false
4441 Acked Write 3 Done Done Sent false
4442 Acked Read 3 Done Done Sent false
4443 Acked Write 2 Done Done Sent false
4444 Acked Flush 0 Done Done Sent false
4445 Acked Flush 0 Done Done Sent false
4446 Acked Flush 0 Done Done Sent false
4447 Acked Read 2 Done Done Sent false
4448 Acked Read 6 Done Done Sent false
4449 Acked Write 6 Done Done Sent false
4450 Acked Write 2 Done Done Sent false
4451 Acked Write 6 Done Done Sent false
4452 Acked Write 5 Done Done Sent false
4453 Acked Read 10 Done Done Sent false
4454 Acked Read 10 Done Done Sent false
4455 Acked Read 9 Done Done Sent false
4456 Acked Read 4 Done Done Sent false
4457 Acked Write 6 Done Done Sent false
4458 Acked Flush 0 Done Done Sent false
4459 Acked Write 10 Done Done Sent false
4460 Acked Flush 0 Done Done Sent false
4461 Acked Write 7 Done Done Sent false
4462 Acked Read 10 Done Done Sent false
4463 Acked Write 3 Done Done Sent false
4464 Acked Write 7 Done Done Sent false
4465 Acked Write 2 Done Done Sent false
4466 Acked Write 6 Done Done Sent false
4467 Acked Flush 0 Done Done Sent false
4468 Acked Write 9 Done Done Sent false
4469 Acked Read 10 Done Done Sent false
4470 Acked Write 1 Done Done Sent false
4471 Acked Write 8 Done Done Sent false
4472 Acked Read 1 Done Done Sent false
4473 Acked Read 9 Done Done Sent false
4474 Acked Write 9 Done Done Sent false
4475 Acked Read 1 Done Done Sent false
4476 Acked Write 7 Done Done Sent false
4477 Acked Read 8 Done Done Sent false
4478 Acked Write 5 Done Done Sent false
4479 Acked Write 8 Done Done Sent false
4480 Acked Write 5 Done Done Sent false
4481 Acked Read 6 Done Done Sent false
4482 Acked Flush 0 Done Done Sent false
4483 Acked Read 6 Done Done Sent false
4484 Acked Write 8 Done Done Sent false
4485 Acked Read 9 Done Done Sent false
4486 Acked Read 7 Done Done Sent false
4487 Acked Write 1 Done Done Sent false
4488 Acked Flush 0 Done Done Sent false
4489 Acked Write 8 Done Done Sent false
4490 Acked Write 4 Done Done Sent false
4491 Acked Write 4 Done Done Sent false
4492 Acked Flush 0 Done Done Sent false
4493 Acked Read 4 Done Done Sent false
4494 Acked Write 6 Done Done Sent false
4495 Acked Read 9 Done Done Sent false
4496 Acked Write 6 Done Done Sent false
4497 Acked Read 6 Done Done Sent false
4498 Acked Read 7 Done Done Sent false
4499 Acked Write 9 Done Done Sent false
4500 Acked Write 9 Done Done Sent false
4501 Acked Write 7 Done Done Sent false
4502 Acked Write 8 Done Done Sent false
4503 Acked Write 1 Done Done Sent false
4504 Acked Read 9 Done Done Sent false
4505 Acked Read 3 Done Done Sent false
4506 Acked Read 6 Done Done Sent false
4507 Acked Read 1 Done Done Sent false
4508 Acked Write 5 Done Done Sent false
4509 Acked Read 9 Done Done Sent false
4510 Acked Write 3 Done Done Sent false
4511 Acked Write 5 Done Done Sent false
4512 Acked Write 8 Done Done Sent false
4513 Acked Write 10 Done Done Sent false
4514 Acked Read 8 Done Done Sent false
4515 Acked Read 5 Done Done Sent false
4516 Acked Flush 0 Done Done Sent false
4517 Acked Read 8 Done Done Sent false
4518 Acked Read 4 Done Done Sent false
4519 Acked Read 10 Done Done Sent false
4520 Acked Read 5 Done Done Sent false
4521 Acked Read 5 Done Done Sent false
4522 Acked Read 7 Done Done Sent false
4523 Acked Read 7 Done Done Sent false
4524 Acked Read 6 Done Done Sent false
4525 Acked Read 9 Done Done Sent false
4526 Acked Write 4 Done Done Sent false
4527 Acked Read 10 Done Done Sent false
4528 Acked Write 6 Done Done Sent false
4529 Acked Flush 0 Done Done Sent false
4530 Acked Flush 0 Done Done Sent false
4531 Acked Read 5 Done Done Sent false
4532 Acked Read 8 Done Done Sent false
4533 Acked Write 3 Done Done Sent false
4534 Acked Read 6 Done Done Sent false
4535 Acked Write 9 Done Done Sent false
4536 Acked Flush 0 Done Done Sent false
4537 Acked Read 9 Done Done Sent false
4538 Acked Read 6 Done Done Sent false
4539 Acked Read 10 Done Done Sent false
4540 Acked Read 1 Done Done Sent false
4541 Acked Read 2 Done Done Sent false
4542 Acked Write 7 Done Done Sent false
4543 Acked Read 7 Done Done Sent false
4544 Acked Write 8 Done Done Sent false
4545 Acked Read 2 Done Done Sent false
4546 Acked Write 3 Done Done Sent false
4547 Acked Read 6 Done Done Sent false
4548 Acked Write 5 Done Done Sent false
4549 Acked Write 8 Done Done Sent false
4550 Acked Read 10 Done Done Sent false
4551 Acked Write 5 Done Done Sent false
4552 Acked Write 10 Done Done Sent false
4553 Acked Flush 0 Done Done Sent false
4554 Acked Write 8 Done Done Sent false
4555 Acked Read 5 Done Done Sent false
4556 Acked Flush 0 Done Done Sent false
4557 Acked Read 3 Done Done Sent false
4558 Acked Read 9 Done Done Sent false
4559 Acked Flush 0 Done Done Sent false
4560 Acked Write 3 Done Done Sent false
4561 Acked Read 7 Done Done Sent false
4562 Acked Flush 0 Done Done Sent false
4563 Acked Read 6 Done Done Sent false
4564 Acked Write 7 Done Done Sent false
4565 Acked Write 2 Done Done Sent false
4566 Acked Read 3 Done Done Sent false
4567 Acked Read 2 Done Done Sent false
4568 Acked Read 2 Done Done Sent false
4569 Acked Read 2 Done Done Sent false
4570 Acked Read 8 Done Done Sent false
4571 Acked Read 3 Done Done Sent false
4572 Acked Write 10 Done Done Sent false
4573 Acked Read 7 Done Done Sent false
4574 Acked Read 5 Done Done Sent false
4575 Acked Flush 0 Done Done Sent false
4576 Acked Write 10 Done Done Sent false
4577 Acked Read 10 Done Done Sent false
4578 Acked Read 1 Done Done Sent false
4579 Acked Read 6 Done Done Sent false
4580 Acked Write 1 Done Done Sent false
4581 Acked Write 10 Done Done Sent false
4582 Acked Read 6 Done Done Sent false
4583 Acked Read 9 Done Done Sent false
4584 Acked Read 9 Done Done Sent false
4585 Acked Flush 0 Done Done Sent false
4586 Acked Write 6 Done Done Sent false
4587 Acked Write 2 Done Done Sent false
4588 Acked Read 3 Done Done Sent false
4589 Acked Write 1 Done Done Sent false
4590 Acked Read 10 Done Done Sent false
4591 Acked Read 7 Done Done Sent false
4592 Acked Read 7 Done Done Sent false
4593 Acked Read 5 Done Done Sent false
4594 Acked Read 10 Done Done Sent false
4595 Acked Write 5 Done Done Sent false
4596 Acked Flush 0 Done Done Sent false
4597 Acked Read 5 Done Done Sent false
4598 Acked Read 10 Done Done Sent false
4599 Acked Read 10 Done Done Sent false
4600 Acked Write 10 Done Done Sent false
4601 Acked Write 3 Done Done Sent false
4602 Acked Write 1 Done Done Sent false
4603 Acked Write 8 Done Done Sent false
4604 Acked Read 2 Done Done Sent false
4605 Acked Read 5 Done Done Sent false
4606 Acked Flush 0 Done Done Sent false
4607 Acked Read 3 Done Done Sent false
4608 Acked Write 2 Done Done Sent false
4609 Acked Read 1 Done Done Sent false
4610 Acked Write 10 Done Done Sent false
4611 Acked Write 3 Done Done Sent false
4612 Acked Read 3 Done Done Sent false
4613 Acked Read 10 Done Done Sent false
4614 Acked Write 8 Done Done Sent false
4615 Acked Write 8 Done Done Sent false
4616 Acked Read 1 Done Done Sent false
4617 Acked Flush 0 Done Done Sent false
4618 Acked Read 9 Done Done Sent false
4619 Acked Write 4 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4617)) Some(JobId(4617)) Some(JobId(4318))
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4619 4618 4617 4616 4615
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:19.666454003Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.630730868Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41280)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.630959453Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.631921003Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632444458Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632468812Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(4318))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632483438Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632496415Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(4318))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632509612Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(4318))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:20.632759562Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4620 4619 4618 4617 4616
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:04:22.880 INFO request completed, latency_us: 409, [8810] Got stop action so:false kr:false
response_code: 204, uri: /stop/cid/0, method: POST, req_id: 848f1a7b-d1dd-4d0a-85bd-9f8dd8651c02, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:22.880831966Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:22.88089706Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:22.880925193Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:04:22.880 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/0, method: GET, req_id: dd9c85eb-9905-4a27-aa4d-dd1b3e7fa0db, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:22.881020591Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:04:26.882 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 93353865-ce57-4863-81ee-7f723bcf5000, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 2615 len 16384 data: 2 2 2 2
002/300 Write block 4874 len 40960 data: 2 3 4 4 5 6 4 5 4 4
003/300 Write block 3752 len 4096 data: 4
004/300 Flush
005/300 Flush
006/300 Read block 2985 len 24576
007/300 Write block 242 len 16384 data: 5 4 4 4
008/300 Write block 1564 len 24576 data: 2 2 3 2 3 3
009/300 Write block 2960 len 24576 data: 3 4 5 4 4 5
010/300 Write block 3266 len 24576 data: 2 2 2 3 3 3
011/300 Write block 4596 len 8192 data: 5 5
012/300 Read block 2327 len 40960
013/300 Read block 1264 len 36864
014/300 Read block 3970 len 20480
015/300 Read block 3741 len 20480
016/300 Read block 3986 len 20480
017/300 Read block 1323 len 4096
018/300 Read block 1125 len 16384
019/300 Read block 2907 len 24576
020/300 Read block 4384 len 16384
021/300 Read block 3840 len 24576
022/300 Write block 4957 len 40960 data: 5 5 4 3 4 4 3 3 4 3
023/300 Read block 3772 len 4096
024/300 Write block 1712 len 28672 data: 5 5 5 4 2 2 2
025/300 Read block 1236 len 8192
026/300 Read block 2956 len 24576
027/300 Write block 303 len 4096 data: 3
028/300 Read block 3361 len 40960
029/300 Read block 2009 len 4096
030/300 Read block 3634 len 28672
031/300 Write block 792 len 12288 data: 5 5 5
032/300 Write block 1575 len 4096 data: 4
033/300 Read block 562 len 40960
034/300 Write block 3035 len 12288 data: 4 4 4
035/300 Flush
036/300 Flush
037/300 Read block 853 len 40960
038/300 Write block 1660 len 4096 data: 4
039/300 Write block 2124 len 20480 data: 5 4 4 3 4
040/300 Read block 297 len 24576
041/300 Read block 4940 len 8192
042/300 Read block 3165 len 12288
043/300 Write block 4968 len 20480 data: 3 3 3 4 4
044/300 Read block 1300 len 32768
045/300 Read block 565 len 16384
046/300 Write block 1843 len 20480 data: 3 4 4 4 3
047/300 Read block 4682 len 12288
048/300 Write block 3563 len 40960 data: 2 2 2 2 2 3 3 3 3 2
049/300 Write block 1368 len 4096 data: 3
050/300 Flush
051/300 Write block 4502 len 24576 data: 4 5 5 5 6 5
052/300 Write block 260 len 28672 data: 2 2 2 2 3 3 3
053/300 Flush
054/300 Read block 169 len 32768
055/300 Flush
056/300 Flush
057/300 Read block 3856 len 40960
058/300 Write block 757 len 8192 data: 3 3
059/300 Write block 2537 len 28672 data: 2 2 2 2 2 2 2
060/300 Write block 1116 len 20480 data: 2 2 2 2 2
061/300 Write block 3242 len 24576 data: 2 2 3 3 3 3
062/300 Read block 2246 len 16384
063/300 Read block 3288 len 16384
064/300 Read block 3528 len 16384
065/300 Read block 1468 len 32768
066/300 Write block 352 len 24576 data: 4 4 3 4 4 2
067/300 Write block 2824 len 20480 data: 3 3 3 2 2
068/300 Read block 2537 len 16384
069/300 Read block 473 len 8192
070/300 Read block 512 len 36864
071/300 Write block 2996 len 24576 data: 3 3 3 3 2 2
072/300 Write block 496 len 8192 data: 5 4
073/300 Read block 869 len 20480
074/300 Write block 1971 len 20480 data: 4 4 4 4 4
075/300 Flush
076/300 Read block 2161 len 16384
077/300 Write block 4625 len 40960 data: 4 4 6 6 5 5 6 6 6 5
078/300 Read block 4160 len 16384
079/300 Read block 2603 len 24576
080/300 Read block 3172 len 28672
081/300 Flush
082/300 Write block 4036 len 24576 data: 3 3 3 3 2 2
083/300 Write block 3197 len 40960 data: 3 3 3 2 3 3 3 3 4 4
084/300 Write block 90 len 28672 data: 2 3 4 4 5 6 3
085/300 Flush
086/300 Write block 1058 len 12288 data: 3 3 3
087/300 Write block 1069 len 40960 data: 3 3 2 2 2 2 2 2 4 5
088/300 Read block 1529 len 8192
089/300 Read block 2791 len 32768
090/300 Flush
091/300 Read block 4419 len 4096
092/300 Write block 1889 len 40960 data: 3 3 5 5 5 5 5 4 3 3
093/300 Read block 316 len 8192
094/300 Read block 2150 len 8192
095/300 Flush
096/300 Write block 2164 len 40960 data: 4 5 5 5 6 4 4 3 3 2
097/300 Read block 2696 len 4096
098/300 Write block 4153 len 32768 data: 2 3 3 3 3 4 5 5
099/300 Read block 497 len 40960
100/300 Flush
101/300 Write block 4482 len 40960 data: 4 4 4 4 4 4 3 3 3 2
102/300 Read block 404 len 12288
103/300 Flush
104/300 Read block 2735 len 36864
105/300 Read block 4022 len 4096
106/300 Read block 1437 len 40960
107/300 Write block 3473 len 20480 data: 3 3 4 3 3
108/300 Write block 1647 len 40960 data: 2 2 2 2 2 2 2 3 2 3
109/300 Write block 3728 len 16384 data: 3 2 2 3
110/300 Read block 84 len 8192
111/300 Write block 3869 len 24576 data: 5 4 3 3 5 5
112/300 Write block 1247 len 4096 data: 3
113/300 Write block 3348 len 12288 data: 4 3 3
114/300 Write block 327 len 4096 data: 2
115/300 Read block 3276 len 4096
116/300 Read block 3936 len 36864
117/300 Flush
118/300 Read block 3667 len 32768
119/300 Read block 4789 len 32768
120/300 Write block 3545 len 28672 data: 2 3 3 4 4 4 4
121/300 Write block 219 len 4096 data: 6
122/300 Write block 4427 len 24576 data: 4 5 5 5 5 3
123/300 Read block 9 len 16384
124/300 Read block 352 len 8192
125/300 Read block 787 len 28672
126/300 Write block 483 len 32768 data: 3 3 3 4 3 2 2 2
127/300 Read block 3714 len 32768
128/300 Write block 2803 len 32768 data: 3 3 4 5 5 4 4 4
129/300 Read block 4868 len 40960
130/300 Write block 3424 len 36864 data: 2 2 2 2 2 2 3 2 2
131/300 Write block 4728 len 36864 data: 3 4 4 4 4 4 4 3 4
132/300 Read block 224 len 28672
133/300 Write block 2000 len 24576 data: 4 3 4 4 3 3
134/300 Read block 1005 len 20480
135/300 Read block 447 len 36864
136/300 Write block 3480 len 40960 data: 5 5 5 5 4 4 4 4 4 5
137/300 Read block 3895 len 24576
138/300 Flush
139/300 Read block 3782 len 12288
140/300 Write block 798 len 40960 data: 5 4 4 3 2 2 2 3 3 3
141/300 Read block 2502 len 28672
142/300 Write block 4677 len 32768 data: 2 4 4 4 4 3 3 3
143/300 Read block 1591 len 28672
144/300 Read block 4015 len 4096
145/300 Write block 3625 len 4096 data: 3
146/300 Read block 1614 len 40960
147/300 Read block 1166 len 16384
148/300 Flush
149/300 Read block 3409 len 32768
150/300 Write block 3628 len 32768 data: 3 3 2 2 2 3 3 3
151/300 Read block 4889 len 28672
152/300 Read block 648 len 28672
153/300 Read block 2662 len 24576
154/300 Read block 1510 len 28672
155/300 Read block 489 len 28672
156/300 Write block 659 len 32768 data: 6 6 6 6 5 5 5 3
157/300 Write block 4077 len 20480 data: 3 3 4 3 4
158/300 Write block 2064 len 20480 data: 4 4 3 3 3
159/300 Read block 1616 len 32768
160/300 Read block 80 len 24576
161/300 Read block 3739 len 28672
162/300 Write block 3199 len 8192 data: 4 3
163/300 Read block 3208 len 16384
164/300 Write block 4377 len 36864 data: 5 5 5 5 4 2 4 5 5
165/300 Write block 1720 len 16384 data: 2 3 3 4
166/300 Read block 4533 len 16384
167/300 Write block 2283 len 16384 data: 6 4 4 3
168/300 Write block 3804 len 24576 data: 3 3 3 3 3 4
169/300 Read block 619 len 40960
170/300 Read block 2706 len 40960
171/300 Read block 4421 len 16384
172/300 Read block 4801 len 32768
173/300 Read block 2555 len 36864
174/300 Read block 2902 len 12288
175/300 Read block 1299 len 8192
176/300 Read block 687 len 36864
177/300 Write block 1843 len 4096 data: 4
178/300 Read block 4886 len 28672
179/300 Write block 1039 len 16384 data: 4 5 4 4
180/300 Read block 4518 len 24576
181/300 Read block 3889 len 12288
182/300 Write block 4711 len 32768 data: 2 2 2 2 2 2 3 3
183/300 Write block 4038 len 32768 data: 4 4 3 3 2 4 4 4
184/300 Write block 4838 len 12288 data: 6 4 4
185/300 Read block 1227 len 24576
186/300 Read block 2045 len 12288
187/300 Read block 2185 len 28672
188/300 Read block 1049 len 4096
189/300 Read block 4907 len 4096
190/300 Write block 3479 len 32768 data: 5 6 6 6 6 5 5 5
191/300 Read block 750 len 12288
192/300 Read block 1902 len 20480
193/300 Read block 2268 len 8192
194/300 Read block 2938 len 4096
195/300 Flush
196/300 Write block 2230 len 20480 data: 5 7 6 6 8
197/300 Read block 3960 len 8192
198/300 Write block 3717 len 40960 data: 4 6 5 6 6 6 6 6 4 3
199/300 Write block 3559 len 8192 data: 4 3
200/300 Write block 4349 len 40960 data: 4 3 3 4 4 5 5 4 5 5
201/300 Write block 4369 len 12288 data: 3 3 4
202/300 Write block 3538 len 4096 data: 5
203/300 Write block 4619 len 4096 data: 3
204/300 Write block 2091 len 40960 data: 5 3 4 4 4 5 6 6 5 5
205/300 Write block 2694 len 8192 data: 4 4
206/300 Read block 2589 len 20480
207/300 Write block 4609 len 24576 data: 2 2 2 2 3 3
208/300 Write block 2442 len 20480 data: 5 4 4 5 5
209/300 Read block 3601 len 28672
210/300 Write block 1511 len 24576 data: 2 3 3 4 3 3
211/300 Read block 462 len 16384
212/300 Write block 2923 len 20480 data: 4 4 4 4 3
213/300 Read block 1724 len 12288
214/300 Read block 492 len 24576
215/300 Read block 1535 len 40960
216/300 Read block 2547 len 16384
217/300 Flush
218/300 Write block 2819 len 32768 data: 5 4 3 2 2 4 4 4
219/300 Write block 3207 len 20480 data: 4 3 3 2 2
220/300 Flush
221/300 Read block 4016 len 8192
222/300 Read block 4733 len 32768
223/300 Write block 4882 len 36864 data: 5 5 4 4 4 6 7 8 7
224/300 Write block 2174 len 16384 data: 3 3 4 4
225/300 Read block 3517 len 20480
226/300 Flush
227/300 Write block 2472 len 24576 data: 5 5 5 5 5 5
228/300 Write block 120 len 12288 data: 3 3 4
229/300 Write block 3143 len 16384 data: 4 4 3 2
230/300 Write block 3335 len 4096 data: 4
231/300 Write block 2657 len 16384 data: 4 4 4 3
232/300 Write block 3093 len 28672 data: 3 3 2 3 3 4 4
233/300 Write block 664 len 40960 data: 6 6 4 5 5 5 6 6 7 6
234/300 Write block 1019 len 32768 data: 6 6 6 5 4 4 6 6
235/300 Flush
236/300 Write block 3842 len 28672 data: 4 3 4 4 3 3 4
237/300 Write block 2455 len 28672 data: 5 4 4 4 3 3 2
238/300 Read block 2506 len 24576
239/300 Write block 215 len 4096 data: 4
240/300 Read block 1016 len 32768
241/300 Write block 4345 len 36864 data: 5 6 5 5 5 4 4 5 5
242/300 Read block 3175 len 12288
243/300 Read block 2439 len 8192
244/300 Write block 3936 len 4096 data: 4
245/300 Flush
246/300 Read block 586 len 4096
247/300 Write block 2011 len 16384 data: 3 3 3 3
248/300 Read block 2803 len 24576
249/300 Write block 3201 len 28672 data: 4 4 4 4 5 5 5
250/300 Read block 431 len 32768
251/300 Read block 4085 len 4096
252/300 Write block 2937 len 32768 data: 4 5 5 5 4 4 3 2
253/300 Write block 1049 len 36864 data: 5 5 4 5 5 6 5 4 3
254/300 Flush
255/300 Write block 1253 len 12288 data: 5 6 7
256/300 Write block 2393 len 12288 data: 4 3 2
257/300 Write block 4342 len 40960 data: 6 5 5 6 7 6 6 6 5 5
258/300 Read block 4046 len 8192
259/300 Read block 1227 len 20480
260/300 Write block 1256 len 4096 data: 7
261/300 Write block 4135 len 40960 data: 2 2 2 3 3 3 4 4 5 4
262/300 Flush
263/300 Write block 319 len 4096 data: 5
264/300 Write block 3468 len 4096 data: 2
265/300 Write block 2318 len 40960 data: 6 6 5 4 4 4 3 3 4 3
266/300 Read block 2980 len 8192
267/300 Read block 2460 len 36864
268/300 Flush
269/300 Write block 2256 len 20480 data: 3 4 3 4 4
270/300 Read block 2740 len 24576
271/300 Read block 2460 len 16384
272/300 Write block 2739 len 12288 data: 3 5 5
273/300 Flush
274/300 Read block 1428 len 12288
275/300 Read block 4913 len 28672
276/300 Read block 1159 len 32768
277/300 Write block 1018 len 36864 data: 5 7 7 7 6 5 5 7 7
278/300 Write block 3507 len 12288 data: 2 2 2
279/300 Read block 1972 len 40960
280/300 Read block 2980 len 28672
281/300 Write block 2385 len 24576 data: 2 2 2 2 3 3
282/300 Write block 876 len 16384 data: 3 3 2 2
283/300 Read block 3256 len 32768
284/300 Write block 775 len 32768 data: 3 3 3 2 3 3 3 3
285/300 Read block 4573 len 20480
286/300 Write block 3480 len 8192 data: 7 7
287/300 Write block 735 len 40960 data: 3 4 3 3 4 4 3 3 3 4
288/300 Flush
289/300 Read block 447 len 40960
290/300 Read block 4570 len 32768
291/300 Write block 4171 len 32768 data: 3 3 3 4 3 3 2 3
292/300 Read block 151 len 24576
293/300 Write block 4032 len 40960 data: 4 3 3 3 4 4 5 5 4 4
294/300 Flush
295/300 Write block 4086 len 40960 data: 3 3 3 3 3 3 2 2 2 4
296/300 Write block 1836 len 24576 data: 3 3 4 5 5 3
297/300 Write block 2986 len 32768 data: 4 4 4 4 6 6 6 3
298/300 Write block 2992 len 40960 data: 7 4 3 3 4 4 4 4 3 3
299/300 Write block 4849 len 12288 data: 2 2 3
300/300 Write block 1722 len 16384 data: 4 5 4 4
301/300 Write block 4076 len 20480 data: 4 4 4 5 4
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:04:27.121 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5e59e4b7-331f-488b-8747-b0f6f13e196c, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4621 Acked Write 4 Sent Done Done false
4622 Acked Write 10 Sent Done Done false
4623 Acked Write 1 Sent Done Done false
4624 Acked Flush 0 Sent Done Done false
4625 Acked Flush 0 Sent Done Done false
4626 Acked Read 6 Sent Done Done false
4627 Acked Write 4 Sent Done Done false
4628 Acked Write 6 Sent Done Done false
4629 Acked Write 6 Sent Done Done false
4630 Acked Write 6 Sent Done Done false
4631 Acked Write 2 Sent Done Done false
4632 Acked Read 10 Sent Done Done false
4633 Acked Read 9 Sent Done Done false
4634 Acked Read 5 Sent Done Done false
4635 Acked Read 5 Sent Done Done false
4636 Acked Read 5 Sent Done Done false
4637 Acked Read 1 Sent Done Done false
4638 Acked Read 4 Sent Done Done false
4639 Acked Read 6 Sent Done Done false
4640 Acked Read 4 Sent Done Done false
4641 Acked Read 6 Sent Done Done false
4642 Acked Write 10 Sent Done Done false
4643 Acked Read 1 Sent Done Done false
4644 Acked Write 7 Sent Done Done false
4645 Acked Read 2 Sent Done Done false
4646 Acked Read 6 Sent Done Done false
4647 Acked Write 1 Sent Done Done false
4648 Acked Read 10 Sent Done Done false
4649 Acked Read 1 Sent Done Done false
4650 Acked Read 7 Sent Done Done false
4651 Acked Write 3 Sent Done Done false
4652 Acked Write 1 Sent Done Done false
4653 Acked Read 10 Sent Done Done false
4654 Acked Write 3 Sent Done Done false
4655 Acked Flush 0 Sent Done Done false
4656 Acked Flush 0 Sent Done Done false
4657 Acked Read 10 Sent Done Done false
4658 Acked Write 1 Sent Done Done false
4659 Acked Write 5 Sent Done Done false
4660 Acked Read 6 Sent Done Done false
4661 Acked Read 2 Sent Done Done false
4662 Acked Read 3 Sent Done Done false
4663 Acked Write 5 Sent Done Done false
4664 Acked Read 8 Sent Done Done false
4665 Acked Read 4 Sent Done Done false
4666 Acked Write 5 Sent Done Done false
4667 Acked Read 3 Sent Done Done false
4668 Acked Write 10 Sent Done Done false
4669 Acked Write 1 Sent Done Done false
4670 Acked Flush 0 Sent Done Done false
4671 Acked Write 6 Sent Done Done false
4672 Acked Write 7 Sent Done Done false
4673 Acked Flush 0 Sent Done Done false
4674 Acked Read 8 Sent Done Done false
4675 Acked Flush 0 Sent Done Done false
4676 Acked Flush 0 Sent Done Done false
4677 Acked Read 10 Sent Done Done false
4678 Acked Write 2 Sent Done Done false
4679 Acked Write 7 Sent Done Done false
4680 Acked Write 5 Sent Done Done false
4681 Acked Write 6 Sent Done Done false
4682 Acked Read 4 Sent Done Done false
4683 Acked Read 4 Sent Done Done false
4684 Acked Read 4 Sent Done Done false
4685 Acked Read 8 Sent Done Done false
4686 Acked Write 6 Sent Done Done false
4687 Acked Write 5 Sent Done Done false
4688 Acked Read 4 Sent Done Done false
4689 Acked Read 2 Sent Done Done false
4690 Acked Read 9 Sent Done Done false
4691 Acked Write 6 Sent Done Done false
4692 Acked Write 2 Sent Done Done false
4693 Acked Read 5 Sent Done Done false
4694 Acked Write 5 Sent Done Done false
4695 Acked Flush 0 Sent Done Done false
4696 Acked Read 4 Sent Done Done false
4697 Acked Write 10 Sent Done Done false
4698 Acked Read 4 Sent Done Done false
4699 Acked Read 6 Sent Done Done false
4700 Acked Read 7 Sent Done Done false
4701 Acked Flush 0 Sent Done Done false
4702 Acked Write 6 Sent Done Done false
4703 Acked Write 10 Sent Done Done false
4704 Acked Write 7 Sent Done Done false
4705 Acked Flush 0 Sent Done Done false
4706 Acked Write 3 Sent Done Done false
4707 Acked Write 10 Sent Done Done false
4708 Acked Read 2 Sent Done Done false
4709 Acked Read 8 Sent Done Done false
4710 Acked Flush 0 Sent Done Done false
4711 Acked Read 1 Sent Done Done false
4712 Acked Write 10 Sent Done Done false
4713 Acked Read 2 Sent Done Done false
4714 Acked Read 2 Sent Done Done false
4715 Acked Flush 0 Sent Done Done false
4716 Acked Write 10 Sent Done Done false
4717 Acked Read 1 Sent Done Done false
4718 Acked Write 8 Sent Done Done false
4719 Acked Read 10 Sent Done Done false
4720 Acked Flush 0 Sent Done Done false
4721 Acked Write 10 Sent Done Done false
4722 Acked Read 3 Sent Done Done false
4723 Acked Flush 0 Sent Done Done false
4724 Acked Read 9 Sent Done Done false
4725 Acked Read 1 Sent Done Done false
4726 Acked Read 10 Sent Done Done false
4727 Acked Write 5 Sent Done Done false
4728 Acked Write 10 Sent Done Done false
4729 Acked Write 4 Sent Done Done false
4730 Acked Read 2 Sent Done Done false
4731 Acked Write 6 Sent Done Done false
4732 Acked Write 1 Sent Done Done false
4733 Acked Write 3 Sent Done Done false
4734 Acked Write 1 Sent Done Done false
4735 Acked Read 1 Sent Done Done false
4736 Acked Read 9 Sent Done Done false
4737 Acked Flush 0 Sent Done Done false
4738 Acked Read 8 Sent Done Done false
4739 Acked Read 8 Sent Done Done false
4740 Acked Write 7 Sent Done Done false
4741 Acked Write 1 Sent Done Done false
4742 Acked Write 6 Sent Done Done false
4743 Acked Read 4 Sent Done Done false
4744 Acked Read 2 Sent Done Done false
4745 Acked Read 7 Sent Done Done false
4746 Acked Write 8 Sent Done Done false
4747 Acked Read 8 Sent Done Done false
4748 Acked Write 8 Sent Done Done false
4749 Acked Read 10 Sent Done Done false
4750 Acked Write 9 Sent Done Done false
4751 Acked Write 9 Sent Done Done false
4752 Acked Read 7 Sent Done Done false
4753 Acked Write 6 Sent Done Done false
4754 Acked Read 5 Sent Done Done false
4755 Acked Read 9 Sent Done Done false
4756 Acked Write 10 Sent Done Done false
4757 Acked Read 6 Sent Done Done false
4758 Acked Flush 0 Sent Done Done false
4759 Acked Read 3 Sent Done Done false
4760 Acked Write 10 Sent Done Done false
4761 Acked Read 7 Sent Done Done false
4762 Acked Write 8 Sent Done Done false
4763 Acked Read 7 Sent Done Done false
4764 Acked Read 1 Sent Done Done false
4765 Acked Write 1 Sent Done Done false
4766 Acked Read 10 Sent Done Done false
4767 Acked Read 4 Sent Done Done false
4768 Acked Flush 0 Sent Done Done false
4769 Acked Read 8 Sent Done Done false
4770 Acked Write 8 Sent Done Done false
4771 Acked Read 7 Sent Done Done false
4772 Acked Read 7 Sent Done Done false
4773 Acked Read 6 Sent Done Done false
4774 Acked Read 7 Sent Done Done false
4775 Acked Read 7 Sent Done Done false
4776 Acked Write 8 Sent Done Done false
4777 Acked Write 5 Sent Done Done false
4778 Acked Write 5 Sent Done Done false
4779 Acked Read 8 Sent Done Done false
4780 Acked Read 6 Sent Done Done false
4781 Acked Read 7 Sent Done Done false
4782 Acked Write 2 Sent Done Done false
4783 Acked Read 4 Sent Done Done false
4784 Acked Write 9 Sent Done Done false
4785 Acked Write 4 Sent Done Done false
4786 Acked Read 4 Sent Done Done false
4787 Acked Write 4 Sent Done Done false
4788 Acked Write 6 Sent Done Done false
4789 Acked Read 10 Sent Done Done false
4790 Acked Read 10 Sent Done Done false
4791 Acked Read 4 Sent Done Done false
4792 Acked Read 8 Sent Done Done false
4793 Acked Read 9 Sent Done Done false
4794 Acked Read 3 Sent Done Done false
4795 Acked Read 2 Sent Done Done false
4796 Acked Read 9 Sent Done Done false
4797 Acked Write 1 Sent Done Done false
4798 Acked Read 7 Sent Done Done false
4799 Acked Write 4 Sent Done Done false
4800 Acked Read 6 Sent Done Done false
4801 Acked Read 3 Sent Done Done false
4802 Acked Write 8 Sent Done Done false
4803 Acked Write 8 Sent Done Done false
4804 Acked Write 3 Sent Done Done false
4805 Acked Read 6 Sent Done Done false
4806 Acked Read 3 Sent Done Done false
4807 Acked Read 7 Sent Done Done false
4808 Acked Read 1 Sent Done Done false
4809 Acked Read 1 Sent Done Done false
4810 Acked Write 8 Sent Done Done false
4811 Acked Read 3 Sent Done Done false
4812 Acked Read 5 Sent Done Done false
4813 Acked Read 2 Sent Done Done false
4814 Acked Read 1 Sent Done Done false
4815 Acked Flush 0 Sent Done Done false
4816 Acked Write 5 Sent Done Done false
4817 Acked Read 2 Sent Done Done false
4818 Acked Write 10 Sent Done Done false
4819 Acked Write 2 Sent Done Done false
4820 Acked Write 10 Sent Done Done false
4821 Acked Write 3 Sent Done Done false
4822 Acked Write 1 Sent Done Done false
4823 Acked Write 1 Sent Done Done false
4824 Acked Write 10 Sent Done Done false
4825 Acked Write 2 Sent Done Done false
4826 Acked Read 5 Sent Done Done false
4827 Acked Write 6 Sent Done Done false
4828 Acked Write 5 Sent Done Done false
4829 Acked Read 7 Sent Done Done false
4830 Acked Write 6 Sent Done Done false
4831 Acked Read 4 Sent Done Done false
4832 Acked Write 5 Sent Done Done false
4833 Acked Read 3 Sent Done Done false
4834 Acked Read 6 Sent Done Done false
4835 Acked Read 10 Sent Done Done false
4836 Acked Read 4 Sent Done Done false
4837 Acked Flush 0 Sent Done Done false
4838 Acked Write 8 Sent Done Done false
4839 Acked Write 5 Sent Done Done false
4840 Acked Flush 0 Sent Done Done false
4841 Acked Read 2 Sent Done Done false
4842 Acked Read 8 Sent Done Done false
4843 Acked Write 9 Sent Done Done false
4844 Acked Write 4 Sent Done Done false
4845 Acked Read 5 Sent Done Done false
4846 Acked Flush 0 Sent Done Done false
4847 Acked Write 6 Sent Done Done false
4848 Acked Write 3 Sent Done Done false
4849 Acked Write 4 Sent Done Done false
4850 Acked Write 1 Sent Done Done false
4851 Acked Write 4 Sent Done Done false
4852 Acked Write 7 Sent Done Done false
4853 Acked Write 10 Sent Done Done false
4854 Acked Write 8 Sent Done Done false
4855 Acked Flush 0 Sent Done Done false
4856 Acked Write 7 Sent Done Done false
4857 Acked Write 7 Sent Done Done false
4858 Acked Read 6 Sent Done Done false
4859 Acked Write 1 Sent Done Done false
4860 Acked Read 8 Sent Done Done false
4861 Acked Write 9 Sent Done Done false
4862 Acked Read 3 Sent Done Done false
4863 Acked Read 2 Sent Done Done false
4864 Acked Write 1 Sent Done Done false
4865 Acked Flush 0 Sent Done Done false
4866 Acked Read 1 Sent Done Done false
4867 Acked Write 4 Sent Done Done false
4868 Acked Read 6 Sent Done Done false
4869 Acked Write 7 Sent Done Done false
4870 Acked Read 8 Sent Done Done false
4871 Acked Read 1 Sent Done Done false
4872 Acked Write 8 Sent Done Done false
4873 Acked Write 9 Sent Done Done false
4874 Acked Flush 0 Sent Done Done false
4875 Acked Write 3 Sent Done Done false
4876 Acked Write 3 Sent Done Done false
4877 Acked Write 10 Sent Done Done false
4878 Acked Read 2 Sent Done Done false
4879 Acked Read 5 Sent Done Done false
4880 Acked Write 1 Sent Done Done false
4881 Acked Write 10 Sent Done Done false
4882 Acked Flush 0 Sent Done Done false
4883 Acked Write 1 Sent Done Done false
4884 Acked Write 1 Sent Done Done false
4885 Acked Write 10 Sent Done Done false
4886 Acked Read 2 Sent Done Done false
4887 Acked Read 9 Sent Done Done false
4888 Acked Flush 0 Sent Done Done false
4889 Acked Write 5 Sent Done Done false
4890 Acked Read 6 Sent Done Done false
4891 Acked Read 4 Sent Done Done false
4892 Acked Write 3 Sent Done Done false
4893 Acked Flush 0 Sent Done Done false
4894 Acked Read 3 Sent Done Done false
4895 Acked Read 7 Sent Done Done false
4896 Acked Read 8 Sent Done Done false
4897 Acked Write 9 Sent Done Done false
4898 Acked Write 3 Sent Done Done false
4899 Acked Read 10 Sent Done Done false
4900 Acked Read 7 Sent Done Done false
4901 Acked Write 6 Sent Done Done false
4902 Acked Write 4 Sent Done Done false
4903 Acked Read 8 Sent Done Done false
4904 Acked Write 8 Sent Done Done false
4905 Acked Read 5 Sent Done Done false
4906 Acked Write 2 Sent Done Done false
4907 Acked Write 10 Sent Done Done false
4908 Acked Flush 0 Sent Done Done false
4909 Acked Read 10 Sent Done Done false
4910 Acked Read 8 Sent Done Done false
4911 Acked Write 8 Sent Done Done false
4912 Acked Read 6 Sent Done Done false
4913 Acked Write 10 Sent Done Done false
4914 Acked Flush 0 Sent Done Done false
4915 Acked Write 10 Sent Done Done false
4916 Acked Write 6 Sent Done Done false
4917 Acked Write 8 Sent Done Done false
4918 Acked Write 10 Sent Done Done false
4919 Acked Write 3 Sent Done Done false
4920 Acked Write 4 Sent Done Done false
4921 Acked Write 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4620)) Some(JobId(4914)) Some(JobId(4914))
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4921 4920 4919 4918 4917
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:27.681866806Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:28.681922016Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:29.684092739Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4621 Acked Write 4 Sent Done Done false
4622 Acked Write 10 Sent Done Done false
4623 Acked Write 1 Sent Done Done false
4624 Acked Flush 0 Sent Done Done false
4625 Acked Flush 0 Sent Done Done false
4626 Acked Read 6 Sent Done Done false
4627 Acked Write 4 Sent Done Done false
4628 Acked Write 6 Sent Done Done false
4629 Acked Write 6 Sent Done Done false
4630 Acked Write 6 Sent Done Done false
4631 Acked Write 2 Sent Done Done false
4632 Acked Read 10 Sent Done Done false
4633 Acked Read 9 Sent Done Done false
4634 Acked Read 5 Sent Done Done false
4635 Acked Read 5 Sent Done Done false
4636 Acked Read 5 Sent Done Done false
4637 Acked Read 1 Sent Done Done false
4638 Acked Read 4 Sent Done Done false
4639 Acked Read 6 Sent Done Done false
4640 Acked Read 4 Sent Done Done false
4641 Acked Read 6 Sent Done Done false
4642 Acked Write 10 Sent Done Done false
4643 Acked Read 1 Sent Done Done false
4644 Acked Write 7 Sent Done Done false
4645 Acked Read 2 Sent Done Done false
4646 Acked Read 6 Sent Done Done false
4647 Acked Write 1 Sent Done Done false
4648 Acked Read 10 Sent Done Done false
4649 Acked Read 1 Sent Done Done false
4650 Acked Read 7 Sent Done Done false
4651 Acked Write 3 Sent Done Done false
4652 Acked Write 1 Sent Done Done false
4653 Acked Read 10 Sent Done Done false
4654 Acked Write 3 Sent Done Done false
4655 Acked Flush 0 Sent Done Done false
4656 Acked Flush 0 Sent Done Done false
4657 Acked Read 10 Sent Done Done false
4658 Acked Write 1 Sent Done Done false
4659 Acked Write 5 Sent Done Done false
4660 Acked Read 6 Sent Done Done false
4661 Acked Read 2 Sent Done Done false
4662 Acked Read 3 Sent Done Done false
4663 Acked Write 5 Sent Done Done false
4664 Acked Read 8 Sent Done Done false
4665 Acked Read 4 Sent Done Done false
4666 Acked Write 5 Sent Done Done false
4667 Acked Read 3 Sent Done Done false
4668 Acked Write 10 Sent Done Done false
4669 Acked Write 1 Sent Done Done false
4670 Acked Flush 0 Sent Done Done false
4671 Acked Write 6 Sent Done Done false
4672 Acked Write 7 Sent Done Done false
4673 Acked Flush 0 Sent Done Done false
4674 Acked Read 8 Sent Done Done false
4675 Acked Flush 0 Sent Done Done false
4676 Acked Flush 0 Sent Done Done false
4677 Acked Read 10 Sent Done Done false
4678 Acked Write 2 Sent Done Done false
4679 Acked Write 7 Sent Done Done false
4680 Acked Write 5 Sent Done Done false
4681 Acked Write 6 Sent Done Done false
4682 Acked Read 4 Sent Done Done false
4683 Acked Read 4 Sent Done Done false
4684 Acked Read 4 Sent Done Done false
4685 Acked Read 8 Sent Done Done false
4686 Acked Write 6 Sent Done Done false
4687 Acked Write 5 Sent Done Done false
4688 Acked Read 4 Sent Done Done false
4689 Acked Read 2 Sent Done Done false
4690 Acked Read 9 Sent Done Done false
4691 Acked Write 6 Sent Done Done false
4692 Acked Write 2 Sent Done Done false
4693 Acked Read 5 Sent Done Done false
4694 Acked Write 5 Sent Done Done false
4695 Acked Flush 0 Sent Done Done false
4696 Acked Read 4 Sent Done Done false
4697 Acked Write 10 Sent Done Done false
4698 Acked Read 4 Sent Done Done false
4699 Acked Read 6 Sent Done Done false
4700 Acked Read 7 Sent Done Done false
4701 Acked Flush 0 Sent Done Done false
4702 Acked Write 6 Sent Done Done false
4703 Acked Write 10 Sent Done Done false
4704 Acked Write 7 Sent Done Done false
4705 Acked Flush 0 Sent Done Done false
4706 Acked Write 3 Sent Done Done false
4707 Acked Write 10 Sent Done Done false
4708 Acked Read 2 Sent Done Done false
4709 Acked Read 8 Sent Done Done false
4710 Acked Flush 0 Sent Done Done false
4711 Acked Read 1 Sent Done Done false
4712 Acked Write 10 Sent Done Done false
4713 Acked Read 2 Sent Done Done false
4714 Acked Read 2 Sent Done Done false
4715 Acked Flush 0 Sent Done Done false
4716 Acked Write 10 Sent Done Done false
4717 Acked Read 1 Sent Done Done false
4718 Acked Write 8 Sent Done Done false
4719 Acked Read 10 Sent Done Done false
4720 Acked Flush 0 Sent Done Done false
4721 Acked Write 10 Sent Done Done false
4722 Acked Read 3 Sent Done Done false
4723 Acked Flush 0 Sent Done Done false
4724 Acked Read 9 Sent Done Done false
4725 Acked Read 1 Sent Done Done false
4726 Acked Read 10 Sent Done Done false
4727 Acked Write 5 Sent Done Done false
4728 Acked Write 10 Sent Done Done false
4729 Acked Write 4 Sent Done Done false
4730 Acked Read 2 Sent Done Done false
4731 Acked Write 6 Sent Done Done false
4732 Acked Write 1 Sent Done Done false
4733 Acked Write 3 Sent Done Done false
4734 Acked Write 1 Sent Done Done false
4735 Acked Read 1 Sent Done Done false
4736 Acked Read 9 Sent Done Done false
4737 Acked Flush 0 Sent Done Done false
4738 Acked Read 8 Sent Done Done false
4739 Acked Read 8 Sent Done Done false
4740 Acked Write 7 Sent Done Done false
4741 Acked Write 1 Sent Done Done false
4742 Acked Write 6 Sent Done Done false
4743 Acked Read 4 Sent Done Done false
4744 Acked Read 2 Sent Done Done false
4745 Acked Read 7 Sent Done Done false
4746 Acked Write 8 Sent Done Done false
4747 Acked Read 8 Sent Done Done false
4748 Acked Write 8 Sent Done Done false
4749 Acked Read 10 Sent Done Done false
4750 Acked Write 9 Sent Done Done false
4751 Acked Write 9 Sent Done Done false
4752 Acked Read 7 Sent Done Done false
4753 Acked Write 6 Sent Done Done false
4754 Acked Read 5 Sent Done Done false
4755 Acked Read 9 Sent Done Done false
4756 Acked Write 10 Sent Done Done false
4757 Acked Read 6 Sent Done Done false
4758 Acked Flush 0 Sent Done Done false
4759 Acked Read 3 Sent Done Done false
4760 Acked Write 10 Sent Done Done false
4761 Acked Read 7 Sent Done Done false
4762 Acked Write 8 Sent Done Done false
4763 Acked Read 7 Sent Done Done false
4764 Acked Read 1 Sent Done Done false
4765 Acked Write 1 Sent Done Done false
4766 Acked Read 10 Sent Done Done false
4767 Acked Read 4 Sent Done Done false
4768 Acked Flush 0 Sent Done Done false
4769 Acked Read 8 Sent Done Done false
4770 Acked Write 8 Sent Done Done false
4771 Acked Read 7 Sent Done Done false
4772 Acked Read 7 Sent Done Done false
4773 Acked Read 6 Sent Done Done false
4774 Acked Read 7 Sent Done Done false
4775 Acked Read 7 Sent Done Done false
4776 Acked Write 8 Sent Done Done false
4777 Acked Write 5 Sent Done Done false
4778 Acked Write 5 Sent Done Done false
4779 Acked Read 8 Sent Done Done false
4780 Acked Read 6 Sent Done Done false
4781 Acked Read 7 Sent Done Done false
4782 Acked Write 2 Sent Done Done false
4783 Acked Read 4 Sent Done Done false
4784 Acked Write 9 Sent Done Done false
4785 Acked Write 4 Sent Done Done false
4786 Acked Read 4 Sent Done Done false
4787 Acked Write 4 Sent Done Done false
4788 Acked Write 6 Sent Done Done false
4789 Acked Read 10 Sent Done Done false
4790 Acked Read 10 Sent Done Done false
4791 Acked Read 4 Sent Done Done false
4792 Acked Read 8 Sent Done Done false
4793 Acked Read 9 Sent Done Done false
4794 Acked Read 3 Sent Done Done false
4795 Acked Read 2 Sent Done Done false
4796 Acked Read 9 Sent Done Done false
4797 Acked Write 1 Sent Done Done false
4798 Acked Read 7 Sent Done Done false
4799 Acked Write 4 Sent Done Done false
4800 Acked Read 6 Sent Done Done false
4801 Acked Read 3 Sent Done Done false
4802 Acked Write 8 Sent Done Done false
4803 Acked Write 8 Sent Done Done false
4804 Acked Write 3 Sent Done Done false
4805 Acked Read 6 Sent Done Done false
4806 Acked Read 3 Sent Done Done false
4807 Acked Read 7 Sent Done Done false
4808 Acked Read 1 Sent Done Done false
4809 Acked Read 1 Sent Done Done false
4810 Acked Write 8 Sent Done Done false
4811 Acked Read 3 Sent Done Done false
4812 Acked Read 5 Sent Done Done false
4813 Acked Read 2 Sent Done Done false
4814 Acked Read 1 Sent Done Done false
4815 Acked Flush 0 Sent Done Done false
4816 Acked Write 5 Sent Done Done false
4817 Acked Read 2 Sent Done Done false
4818 Acked Write 10 Sent Done Done false
4819 Acked Write 2 Sent Done Done false
4820 Acked Write 10 Sent Done Done false
4821 Acked Write 3 Sent Done Done false
4822 Acked Write 1 Sent Done Done false
4823 Acked Write 1 Sent Done Done false
4824 Acked Write 10 Sent Done Done false
4825 Acked Write 2 Sent Done Done false
4826 Acked Read 5 Sent Done Done false
4827 Acked Write 6 Sent Done Done false
4828 Acked Write 5 Sent Done Done false
4829 Acked Read 7 Sent Done Done false
4830 Acked Write 6 Sent Done Done false
4831 Acked Read 4 Sent Done Done false
4832 Acked Write 5 Sent Done Done false
4833 Acked Read 3 Sent Done Done false
4834 Acked Read 6 Sent Done Done false
4835 Acked Read 10 Sent Done Done false
4836 Acked Read 4 Sent Done Done false
4837 Acked Flush 0 Sent Done Done false
4838 Acked Write 8 Sent Done Done false
4839 Acked Write 5 Sent Done Done false
4840 Acked Flush 0 Sent Done Done false
4841 Acked Read 2 Sent Done Done false
4842 Acked Read 8 Sent Done Done false
4843 Acked Write 9 Sent Done Done false
4844 Acked Write 4 Sent Done Done false
4845 Acked Read 5 Sent Done Done false
4846 Acked Flush 0 Sent Done Done false
4847 Acked Write 6 Sent Done Done false
4848 Acked Write 3 Sent Done Done false
4849 Acked Write 4 Sent Done Done false
4850 Acked Write 1 Sent Done Done false
4851 Acked Write 4 Sent Done Done false
4852 Acked Write 7 Sent Done Done false
4853 Acked Write 10 Sent Done Done false
4854 Acked Write 8 Sent Done Done false
4855 Acked Flush 0 Sent Done Done false
4856 Acked Write 7 Sent Done Done false
4857 Acked Write 7 Sent Done Done false
4858 Acked Read 6 Sent Done Done false
4859 Acked Write 1 Sent Done Done false
4860 Acked Read 8 Sent Done Done false
4861 Acked Write 9 Sent Done Done false
4862 Acked Read 3 Sent Done Done false
4863 Acked Read 2 Sent Done Done false
4864 Acked Write 1 Sent Done Done false
4865 Acked Flush 0 Sent Done Done false
4866 Acked Read 1 Sent Done Done false
4867 Acked Write 4 Sent Done Done false
4868 Acked Read 6 Sent Done Done false
4869 Acked Write 7 Sent Done Done false
4870 Acked Read 8 Sent Done Done false
4871 Acked Read 1 Sent Done Done false
4872 Acked Write 8 Sent Done Done false
4873 Acked Write 9 Sent Done Done false
4874 Acked Flush 0 Sent Done Done false
4875 Acked Write 3 Sent Done Done false
4876 Acked Write 3 Sent Done Done false
4877 Acked Write 10 Sent Done Done false
4878 Acked Read 2 Sent Done Done false
4879 Acked Read 5 Sent Done Done false
4880 Acked Write 1 Sent Done Done false
4881 Acked Write 10 Sent Done Done false
4882 Acked Flush 0 Sent Done Done false
4883 Acked Write 1 Sent Done Done false
4884 Acked Write 1 Sent Done Done false
4885 Acked Write 10 Sent Done Done false
4886 Acked Read 2 Sent Done Done false
4887 Acked Read 9 Sent Done Done false
4888 Acked Flush 0 Sent Done Done false
4889 Acked Write 5 Sent Done Done false
4890 Acked Read 6 Sent Done Done false
4891 Acked Read 4 Sent Done Done false
4892 Acked Write 3 Sent Done Done false
4893 Acked Flush 0 Sent Done Done false
4894 Acked Read 3 Sent Done Done false
4895 Acked Read 7 Sent Done Done false
4896 Acked Read 8 Sent Done Done false
4897 Acked Write 9 Sent Done Done false
4898 Acked Write 3 Sent Done Done false
4899 Acked Read 10 Sent Done Done false
4900 Acked Read 7 Sent Done Done false
4901 Acked Write 6 Sent Done Done false
4902 Acked Write 4 Sent Done Done false
4903 Acked Read 8 Sent Done Done false
4904 Acked Write 8 Sent Done Done false
4905 Acked Read 5 Sent Done Done false
4906 Acked Write 2 Sent Done Done false
4907 Acked Write 10 Sent Done Done false
4908 Acked Flush 0 Sent Done Done false
4909 Acked Read 10 Sent Done Done false
4910 Acked Read 8 Sent Done Done false
4911 Acked Write 8 Sent Done Done false
4912 Acked Read 6 Sent Done Done false
4913 Acked Write 10 Sent Done Done false
4914 Acked Flush 0 Sent Done Done false
4915 Acked Write 10 Sent Done Done false
4916 Acked Write 6 Sent Done Done false
4917 Acked Write 8 Sent Done Done false
4918 Acked Write 10 Sent Done Done false
4919 Acked Write 3 Sent Done Done false
4920 Acked Write 4 Sent Done Done false
4921 Acked Write 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(4620)) Some(JobId(4914)) Some(JobId(4914))
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4921 4920 4919 4918 4917
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:31.890335297Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:32.689639769Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.881875398Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50445)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.882014545Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883069402Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883603004Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883630357Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(4620))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883645594Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883663369Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(4620))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.883677386Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(4620))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:32.88390964Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 4922 4921 4920 4919 4918
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 28 06:04:35.127 INFO request completed, latency_us: 356, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 78ace500-74f4-4c8e-9e1c-c591b6972b07, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:35.127989901Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:35.128055635Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:04:35.128 INFO request completed, latency_us: 130{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:35.128086618Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
, response_code: 200, uri: /state/cid/2, method: GET, req_id: 27226116-49bb-4742-9c7b-947b5c117b32, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:35.128195482Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:04:39.130 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6e150c67-ae22-4f3b-a755-facf37086e18, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 4451 len 16384
003/300 Read block 4164 len 12288
004/300 Write block 2561 len 36864 data: 3 3 4 4 3 2 2 3 4
005/300 Read block 1304 len 36864
006/300 Read block 4109 len 28672
007/300 Write block 1432 len 28672 data: 2 3 3 3 3 4 3
008/300 Read block 2570 len 4096
009/300 Write block 1489 len 24576 data: 7 7 7 7 7 6
010/300 Read block 681 len 20480
011/300 Write block 4260 len 4096 data: 3
012/300 Read block 1256 len 24576
013/300 Read block 2788 len 4096
014/300 Read block 3205 len 20480
015/300 Read block 1342 len 12288
016/300 Flush
017/300 Read block 2403 len 20480
018/300 Read block 3766 len 36864
019/300 Flush
020/300 Read block 4364 len 8192
021/300 Read block 1422 len 24576
022/300 Flush
023/300 Write block 1421 len 28672 data: 4 4 5 4 4 3 3
024/300 Write block 1982 len 4096 data: 3
025/300 Write block 4228 len 8192 data: 8 9
026/300 Write block 570 len 24576 data: 3 3 4 4 4 4
027/300 Read block 3676 len 32768
028/300 Write block 422 len 20480 data: 5 4 3 2 2
029/300 Write block 2109 len 8192 data: 3 3
030/300 Write block 4227 len 28672 data: 7 9 10 8 8 4 4
031/300 Read block 2560 len 32768
032/300 Read block 133 len 36864
033/300 Flush
034/300 Write block 702 len 12288 data: 6 6 7
035/300 Write block 1372 len 36864 data: 3 4 4 3 3 3 3 3 3
036/300 Read block 1486 len 4096
037/300 Read block 2660 len 32768
038/300 Read block 4301 len 40960
039/300 Read block 693 len 16384
040/300 Write block 93 len 16384 data: 5 6 7 4
041/300 Write block 2450 len 40960 data: 2 2 3 3 4 6 5 5 5 4
042/300 Read block 4877 len 40960
043/300 Write block 2529 len 8192 data: 4 3
044/300 Read block 3023 len 24576
045/300 Flush
046/300 Read block 3470 len 24576
047/300 Write block 4595 len 16384 data: 6 6 6 5
048/300 Read block 2388 len 8192
049/300 Write block 1219 len 28672 data: 3 3 3 3 3 3 2
050/300 Read block 4645 len 4096
051/300 Write block 915 len 36864 data: 4 2 2 2 2 2 2 2 2
052/300 Read block 371 len 32768
053/300 Read block 3478 len 16384
054/300 Write block 2166 len 8192 data: 6 6
055/300 Write block 2339 len 24576 data: 4 4 4 4 4 3
056/300 Write block 618 len 32768 data: 2 2 3 3 3 3 3 4
057/300 Write block 1769 len 28672 data: 2 2 2 2 2 3 3
058/300 Write block 4378 len 24576 data: 6 6 6 5 3 5
059/300 Flush
060/300 Read block 2815 len 40960
061/300 Write block 2388 len 4096 data: 3
062/300 Read block 2068 len 20480
063/300 Read block 4900 len 8192
064/300 Read block 1730 len 28672
065/300 Read block 3013 len 24576
066/300 Read block 2248 len 8192
067/300 Write block 3373 len 4096 data: 3
068/300 Write block 3742 len 32768 data: 5 6 6 6 4 4 4 4
069/300 Read block 575 len 32768
070/300 Write block 4913 len 28672 data: 3 3 3 2 2 2 2
071/300 Write block 297 len 12288 data: 4 4 4
072/300 Write block 2208 len 20480 data: 3 5 4 4 5
073/300 Read block 1657 len 8192
074/300 Flush
075/300 Write block 4443 len 12288 data: 4 4 4
076/300 Read block 3646 len 40960
077/300 Write block 2733 len 16384 data: 5 3 4 4
078/300 Write block 4839 len 24576 data: 5 5 4 5 4 4
079/300 Write block 1724 len 40960 data: 5 5 4 5 4 4 5 4 4 4
080/300 Write block 3843 len 28672 data: 4 5 5 4 4 5 4
081/300 Write block 1810 len 8192 data: 3 3
082/300 Write block 4133 len 24576 data: 4 2 3 3 3 4
083/300 Flush
084/300 Write block 4756 len 16384 data: 5 5 5 5
085/300 Write block 2833 len 4096 data: 2
086/300 Write block 2183 len 32768 data: 4 3 3 4 3 4 4 5
087/300 Write block 4828 len 24576 data: 6 6 8 8 7 4
088/300 Read block 2107 len 28672
089/300 Write block 599 len 8192 data: 3 3
090/300 Read block 1421 len 28672
091/300 Read block 1251 len 32768
092/300 Read block 4875 len 16384
093/300 Write block 1312 len 24576 data: 5 5 2 2 2 2
094/300 Read block 3674 len 16384
095/300 Write block 491 len 16384 data: 2 2 3 3
096/300 Write block 4693 len 36864 data: 3 3 3 4 4 3 5 3 3
097/300 Write block 3918 len 16384 data: 3 3 3 2
098/300 Read block 4397 len 20480
099/300 Write block 3236 len 16384 data: 2 2 2 2
100/300 Read block 4949 len 32768
101/300 Read block 2893 len 20480
102/300 Write block 2931 len 4096 data: 3
103/300 Read block 3179 len 12288
104/300 Read block 75 len 16384
105/300 Flush
106/300 Read block 808 len 4096
107/300 Write block 4861 len 20480 data: 2 3 3 3 3
108/300 Flush
109/300 Read block 1310 len 40960
110/300 Read block 456 len 16384
111/300 Read block 962 len 4096
112/300 Read block 552 len 8192
113/300 Read block 4754 len 8192
114/300 Write block 144 len 24576 data: 4 4 4 4 4 5
115/300 Write block 273 len 24576 data: 3 4 4 4 5 6
116/300 Write block 1151 len 20480 data: 4 4 5 4 5
117/300 Write block 4227 len 8192 data: 8 10
118/300 Write block 2384 len 8192 data: 3 3
119/300 Read block 1770 len 20480
120/300 Write block 2672 len 20480 data: 6 5 6 6 5
121/300 Read block 976 len 4096
122/300 Read block 3303 len 8192
123/300 Write block 1400 len 4096 data: 5
124/300 Read block 317 len 24576
125/300 Write block 3985 len 12288 data: 2 3 3
126/300 Read block 4204 len 20480
127/300 Write block 4093 len 36864 data: 3 3 5 4 4 4 4 4 5
128/300 Read block 3413 len 32768
129/300 Write block 567 len 8192 data: 2 2
130/300 Read block 2940 len 16384
131/300 Read block 340 len 4096
132/300 Write block 783 len 24576 data: 3 3 4 3 3 2
133/300 Read block 4931 len 20480
134/300 Write block 2003 len 4096 data: 5
135/300 Write block 2713 len 20480 data: 8 7 6 6 6
136/300 Write block 720 len 36864 data: 3 2 2 2 2 2 2 2 2
137/300 Write block 1586 len 12288 data: 4 4 3
138/300 Read block 4379 len 8192
139/300 Flush
140/300 Write block 3773 len 8192 data: 2 2
141/300 Write block 1816 len 16384 data: 2 2 2 2
142/300 Flush
143/300 Read block 3298 len 16384
144/300 Read block 1803 len 40960
145/300 Read block 1696 len 32768
146/300 Write block 4461 len 8192 data: 4 4
147/300 Flush
148/300 Write block 194 len 28672 data: 6 6 6 5 4 4 4
149/300 Read block 3474 len 36864
150/300 Read block 4712 len 40960
151/300 Read block 2137 len 28672
152/300 Write block 35 len 28672 data: 3 3 3 3 3 3 4
153/300 Read block 975 len 28672
154/300 Read block 472 len 32768
155/300 Write block 1250 len 4096 data: 4
156/300 Write block 1793 len 40960 data: 3 2 3 3 3 3 3 3 3 3
157/300 Read block 4841 len 8192
158/300 Write block 681 len 16384 data: 5 4 4 4
159/300 Write block 3807 len 8192 data: 4 4
160/300 Write block 2594 len 40960 data: 3 3 3 3 3 4 4 4 3 4
161/300 Read block 2576 len 40960
162/300 Read block 2042 len 40960
163/300 Write block 1927 len 28672 data: 2 2 2 2 2 3 3
164/300 Write block 1179 len 28672 data: 8 7 6 4 3 4 4
165/300 Read block 4471 len 4096
166/300 Flush
167/300 Read block 1007 len 40960
168/300 Flush
169/300 Read block 567 len 40960
170/300 Flush
171/300 Write block 1197 len 12288 data: 2 3 2
172/300 Write block 680 len 16384 data: 5 6 5 5
173/300 Write block 965 len 24576 data: 4 3 3 3 2 2
174/300 Write block 1837 len 36864 data: 4 5 6 6 4 3 5 5 5
175/300 Read block 2742 len 24576
176/300 Read block 616 len 24576
177/300 Flush
178/300 Write block 1934 len 4096 data: 3
179/300 Write block 1835 len 8192 data: 3 4
180/300 Read block 4852 len 12288
181/300 Read block 2695 len 8192
182/300 Write block 2447 len 20480 data: 5 4 3 3 3
183/300 Read block 1653 len 36864
184/300 Read block 1157 len 24576
185/300 Read block 1891 len 24576
186/300 Read block 4238 len 4096
187/300 Write block 268 len 28672 data: 3 2 2 3 3 4 5
188/300 Read block 2510 len 8192
189/300 Read block 2762 len 36864
190/300 Read block 4919 len 24576
191/300 Read block 4695 len 4096
192/300 Flush
193/300 Flush
194/300 Read block 4154 len 28672
195/300 Write block 2110 len 20480 data: 4 3 3 3 4
196/300 Read block 3013 len 32768
197/300 Read block 982 len 8192
198/300 Read block 4771 len 36864
199/300 Read block 340 len 4096
200/300 Read block 4762 len 8192
201/300 Read block 4582 len 32768
202/300 Read block 4747 len 16384
203/300 Read block 4744 len 16384
204/300 Read block 2780 len 8192
205/300 Read block 1917 len 32768
206/300 Read block 4795 len 24576
207/300 Write block 1335 len 32768 data: 4 4 4 3 3 4 4 4
208/300 Write block 4805 len 12288 data: 2 2 3
209/300 Write block 3705 len 12288 data: 3 4 5
210/300 Read block 4669 len 16384
211/300 Flush
212/300 Read block 1129 len 36864
213/300 Read block 2195 len 16384
214/300 Write block 687 len 8192 data: 2 2
215/300 Write block 1709 len 36864 data: 6 6 5 6 6 6 5 3 3
216/300 Write block 3986 len 40960 data: 4 4 2 2 2 2 2 2 2 2
217/300 Read block 2960 len 4096
218/300 Write block 4482 len 28672 data: 5 5 5 5 5 5 4
219/300 Read block 855 len 32768
220/300 Read block 1830 len 28672
221/300 Write block 716 len 24576 data: 4 4 4 3 4 3
222/300 Read block 778 len 20480
223/300 Write block 4770 len 32768 data: 4 4 3 2 2 3 3 4
224/300 Read block 1445 len 16384
225/300 Read block 3584 len 16384
226/300 Read block 1441 len 12288
227/300 Read block 391 len 36864
228/300 Read block 1440 len 24576
229/300 Flush
230/300 Write block 3762 len 32768 data: 4 3 2 2 2 2 2 2
231/300 Flush
232/300 Write block 1653 len 40960 data: 3 4 3 4 3 3 3 5 3 3
233/300 Write block 3398 len 36864 data: 2 2 3 3 3 3 2 3 3
234/300 Write block 956 len 8192 data: 3 3
235/300 Read block 3907 len 28672
236/300 Write block 2519 len 36864 data: 5 5 5 5 6 7 6 4 4
237/300 Read block 3590 len 28672
238/300 Flush
239/300 Write block 3545 len 16384 data: 3 4 4 5
240/300 Write block 2742 len 8192 data: 4 4
241/300 Read block 2703 len 24576
242/300 Read block 2891 len 4096
243/300 Read block 2867 len 4096
244/300 Flush
245/300 Read block 2920 len 32768
246/300 Flush
247/300 Flush
248/300 Read block 900 len 16384
249/300 Write block 3249 len 28672 data: 5 5 4 5 3 3 3
250/300 Flush
251/300 Flush
252/300 Read block 751 len 40960
253/300 Write block 3608 len 4096 data: 4
254/300 Flush
255/300 Write block 3823 len 4096 data: 2
256/300 Write block 3395 len 24576 data: 2 2 2 3 3 4
257/300 Write block 2240 len 4096 data: 3
258/300 Read block 3321 len 16384
259/300 Read block 2645 len 4096
260/300 Write block 1081 len 12288 data: 5 4 3
261/300 Read block 369 len 36864
262/300 Read block 1161 len 16384
263/300 Write block 2549 len 40960 data: 4 4 4 3 3 3 3 3 3 3
264/300 Write block 3355 len 12288 data: 4 4 3
265/300 Read block 1177 len 28672
266/300 Write block 808 len 12288 data: 3 3 3
267/300 Read block 2713 len 40960
268/300 Write block 678 len 8192 data: 5 6
269/300 Read block 3694 len 12288
270/300 Read block 1356 len 28672
271/300 Read block 2402 len 24576
272/300 Flush
273/300 Read block 1653 len 24576
274/300 Read block 1163 len 32768
275/300 Read block 560 len 8192
276/300 Write block 373 len 8192 data: 4 4
277/300 Write block 2446 len 28672 data: 6 6 5 4 4 4 4
278/300 Flush
279/300 Read block 2858 len 28672
280/300 Read block 556 len 12288
281/300 Write block 3285 len 36864 data: 2 3 3 3 3 3 2 2 5
282/300 Write block 1565 len 40960 data: 3 4 3 4 4 3 3 3 3 3
283/300 Write block 3497 len 12288 data: 3 3 3
284/300 Read block 2555 len 28672
285/300 Write block 772 len 36864 data: 4 3 3 4 4 4 3 4 4
286/300 Read block 3877 len 24576
287/300 Flush
288/300 Flush
289/300 Read block 2516 len 8192
290/300 Read block 4778 len 20480
291/300 Read block 2606 len 28672
292/300 Write block 2876 len 16384 data: 2 2 2 2
293/300 Read block 169 len 8192
294/300 Read block 1248 len 12288
295/300 Read block 4848 len 32768
296/300 Write block 3154 len 24576 data: 3 3 5 6 6 7
297/300 Read block 1322 len 28672
298/300 Write block 1337 len 4096 data: 5
299/300 Read block 3588 len 20480
300/300 Read block 1786 len 28672
301/300 Write block 2895 len 24576 data: 2 2 3 4 3 3
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:04:39.363 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/2, method: POST, req_id: eaccfbff-40a6-44c8-8200-62dd7390c4b5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Flush 0 Done Done Sent false
4924 Acked Read 4 Done Done Sent false
4925 Acked Read 3 Done Done Sent false
4926 Acked Write 9 Done Done Sent false
4927 Acked Read 9 Done Done Sent false
4928 Acked Read 7 Done Done Sent false
4929 Acked Write 7 Done Done Sent false
4930 Acked Read 1 Done Done Sent false
4931 Acked Write 6 Done Done Sent false
4932 Acked Read 5 Done Done Sent false
4933 Acked Write 1 Done Done Sent false
4934 Acked Read 6 Done Done Sent false
4935 Acked Read 1 Done Done Sent false
4936 Acked Read 5 Done Done Sent false
4937 Acked Read 3 Done Done Sent false
4938 Acked Flush 0 Done Done Sent false
4939 Acked Read 5 Done Done Sent false
4940 Acked Read 9 Done Done Sent false
4941 Acked Flush 0 Done Done Sent false
4942 Acked Read 2 Done Done Sent false
4943 Acked Read 6 Done Done Sent false
4944 Acked Flush 0 Done Done Sent false
4945 Acked Write 7 Done Done Sent false
4946 Acked Write 1 Done Done Sent false
4947 Acked Write 2 Done Done Sent false
4948 Acked Write 6 Done Done Sent false
4949 Acked Read 8 Done Done Sent false
4950 Acked Write 5 Done Done Sent false
4951 Acked Write 2 Done Done Sent false
4952 Acked Write 7 Done Done Sent false
4953 Acked Read 8 Done Done Sent false
4954 Acked Read 9 Done Done Sent false
4955 Acked Flush 0 Done Done Sent false
4956 Acked Write 3 Done Done Sent false
4957 Acked Write 9 Done Done Sent false
4958 Acked Read 1 Done Done Sent false
4959 Acked Read 8 Done Done Sent false
4960 Acked Read 10 Done Done Sent false
4961 Acked Read 4 Done Done Sent false
4962 Acked Write 4 Done Done Sent false
4963 Acked Write 10 Done Done Sent false
4964 Acked Read 10 Done Done Sent false
4965 Acked Write 2 Done Done Sent false
4966 Acked Read 6 Done Done Sent false
4967 Acked Flush 0 Done Done Sent false
4968 Acked Read 6 Done Done Sent false
4969 Acked Write 4 Done Done Sent false
4970 Acked Read 2 Done Done Sent false
4971 Acked Write 7 Done Done Sent false
4972 Acked Read 1 Done Done Sent false
4973 Acked Write 9 Done Done Sent false
4974 Acked Read 8 Done Done Sent false
4975 Acked Read 4 Done Done Sent false
4976 Acked Write 2 Done Done Sent false
4977 Acked Write 6 Done Done Sent false
4978 Acked Write 8 Done Done Sent false
4979 Acked Write 7 Done Done Sent false
4980 Acked Write 6 Done Done Sent false
4981 Acked Flush 0 Done Done Sent false
4982 Acked Read 10 Done Done Sent false
4983 Acked Write 1 Done Done Sent false
4984 Acked Read 5 Done Done Sent false
4985 Acked Read 2 Done Done Sent false
4986 Acked Read 7 Done Done Sent false
4987 Acked Read 6 Done Done Sent false
4988 Acked Read 2 Done Done Sent false
4989 Acked Write 1 Done Done Sent false
4990 Acked Write 8 Done Done Sent false
4991 Acked Read 8 Done Done Sent false
4992 Acked Write 7 Done Done Sent false
4993 Acked Write 3 Done Done Sent false
4994 Acked Write 5 Done Done Sent false
4995 Acked Read 2 Done Done Sent false
4996 Acked Flush 0 Done Done Sent false
4997 Acked Write 3 Done Done Sent false
4998 Acked Read 10 Done Done Sent false
4999 Acked Write 4 Done Done Sent false
5000 Acked Write 6 Done Done Sent false
5001 Acked Write 10 Done Done Sent false
5002 Acked Write 7 Done Done Sent false
5003 Acked Write 2 Done Done Sent false
5004 Acked Write 6 Done Done Sent false
5005 Acked Flush 0 Done Done Sent false
5006 Acked Write 4 Done Done Sent false
5007 Acked Write 1 Done Done Sent false
5008 Acked Write 8 Done Done Sent false
5009 Acked Write 6 Done Done Sent false
5010 Acked Read 7 Done Done Sent false
5011 Acked Write 2 Done Done Sent false
5012 Acked Read 7 Done Done Sent false
5013 Acked Read 8 Done Done Sent false
5014 Acked Read 4 Done Done Sent false
5015 Acked Write 6 Done Done Sent false
5016 Acked Read 4 Done Done Sent false
5017 Acked Write 4 Done Done Sent false
5018 Acked Write 9 Done Done Sent false
5019 Acked Write 4 Done Done Sent false
5020 Acked Read 5 Done Done Sent false
5021 Acked Write 4 Done Done Sent false
5022 Acked Read 8 Done Done Sent false
5023 Acked Read 5 Done Done Sent false
5024 Acked Write 1 Done Done Sent false
5025 Acked Read 3 Done Done Sent false
5026 Acked Read 4 Done Done Sent false
5027 Acked Flush 0 Done Done Sent false
5028 Acked Read 1 Done Done Sent false
5029 Acked Write 5 Done Done Sent false
5030 Acked Flush 0 Done Done Sent false
5031 Acked Read 10 Done Done Sent false
5032 Acked Read 4 Done Done Sent false
5033 Acked Read 1 Done Done Sent false
5034 Acked Read 2 Done Done Sent false
5035 Acked Read 2 Done Done Sent false
5036 Acked Write 6 Done Done Sent false
5037 Acked Write 6 Done Done Sent false
5038 Acked Write 5 Done Done Sent false
5039 Acked Write 2 Done Done Sent false
5040 Acked Write 2 Done Done Sent false
5041 Acked Read 5 Done Done Sent false
5042 Acked Write 5 Done Done Sent false
5043 Acked Read 1 Done Done Sent false
5044 Acked Read 2 Done Done Sent false
5045 Acked Write 1 Done Done Sent false
5046 Acked Read 6 Done Done Sent false
5047 Acked Write 3 Done Done Sent false
5048 Acked Read 5 Done Done Sent false
5049 Acked Write 9 Done Done Sent false
5050 Acked Read 8 Done Done Sent false
5051 Acked Write 2 Done Done Sent false
5052 Acked Read 4 Done Done Sent false
5053 Acked Read 1 Done Done Sent false
5054 Acked Write 6 Done Done Sent false
5055 Acked Read 5 Done Done Sent false
5056 Acked Write 1 Done Done Sent false
5057 Acked Write 5 Done Done Sent false
5058 Acked Write 9 Done Done Sent false
5059 Acked Write 3 Done Done Sent false
5060 Acked Read 2 Done Done Sent false
5061 Acked Flush 0 Done Done Sent false
5062 Acked Write 2 Done Done Sent false
5063 Acked Write 4 Done Done Sent false
5064 Acked Flush 0 Done Done Sent false
5065 Acked Read 4 Done Done Sent false
5066 Acked Read 10 Done Done Sent false
5067 Acked Read 8 Done Done Sent false
5068 Acked Write 2 Done Done Sent false
5069 Acked Flush 0 Done Done Sent false
5070 Acked Write 7 Done Done Sent false
5071 Acked Read 9 Done Done Sent false
5072 Acked Read 10 Done Done Sent false
5073 Acked Read 7 Done Done Sent false
5074 Acked Write 7 Done Done Sent false
5075 Acked Read 7 Done Done Sent false
5076 Acked Read 8 Done Done Sent false
5077 Acked Write 1 Done Done Sent false
5078 Acked Write 10 Done Done Sent false
5079 Acked Read 2 Done Done Sent false
5080 Acked Write 4 Done Done Sent false
5081 Acked Write 2 Done Done Sent false
5082 Acked Write 10 Done Done Sent false
5083 Acked Read 10 Done Done Sent false
5084 Acked Read 10 Done Done Sent false
5085 Acked Write 7 Done Done Sent false
5086 Acked Write 7 Done Done Sent false
5087 Acked Read 1 Done Done Sent false
5088 Acked Flush 0 Done Done Sent false
5089 Acked Read 10 Done Done Sent false
5090 Acked Flush 0 Done Done Sent false
5091 Acked Read 10 Done Done Sent false
5092 Acked Flush 0 Done Done Sent false
5093 Acked Write 3 Done Done Sent false
5094 Acked Write 4 Done Done Sent false
5095 Acked Write 6 Done Done Sent false
5096 Acked Write 9 Done Done Sent false
5097 Acked Read 6 Done Done Sent false
5098 Acked Read 6 Done Done Sent false
5099 Acked Flush 0 Done Done Sent false
5100 Acked Write 1 Done Done Sent false
5101 Acked Write 2 Done Done Sent false
5102 Acked Read 3 Done Done Sent false
5103 Acked Read 2 Done Done Sent false
5104 Acked Write 5 Done Done Sent false
5105 Acked Read 9 Done Done Sent false
5106 Acked Read 6 Done Done Sent false
5107 Acked Read 6 Done Done Sent false
5108 Acked Read 1 Done Done Sent false
5109 Acked Write 7 Done Done Sent false
5110 Acked Read 2 Done Done Sent false
5111 Acked Read 9 Done Done Sent false
5112 Acked Read 6 Done Done Sent false
5113 Acked Read 1 Done Done Sent false
5114 Acked Flush 0 Done Done Sent false
5115 Acked Flush 0 Done Done Sent false
5116 Acked Read 7 Done Done Sent false
5117 Acked Write 5 Done Done Sent false
5118 Acked Read 8 Done Done Sent false
5119 Acked Read 2 Done Done Sent false
5120 Acked Read 9 Done Done Sent false
5121 Acked Read 1 Done Done Sent false
5122 Acked Read 2 Done Done Sent false
5123 Acked Read 8 Done Done Sent false
5124 Acked Read 4 Done Done Sent false
5125 Acked Read 4 Done Done Sent false
5126 Acked Read 2 Done Done Sent false
5127 Acked Read 8 Done Done Sent false
5128 Acked Read 6 Done Done Sent false
5129 Acked Write 8 Done Done Sent false
5130 Acked Write 3 Done Done Sent false
5131 Acked Write 3 Done Done Sent false
5132 Acked Read 4 Done Done Sent false
5133 Acked Flush 0 Done Done Sent false
5134 Acked Read 9 Done Done Sent false
5135 Acked Read 4 Done Done Sent false
5136 Acked Write 2 Done Done Sent false
5137 Acked Write 9 Done Done Sent false
5138 Acked Write 10 Done Done Sent false
5139 Acked Read 1 Done Done Sent false
5140 Acked Write 7 Done Done Sent false
5141 Acked Read 8 Done Done Sent false
5142 Acked Read 7 Done Done Sent false
5143 Acked Write 6 Done Done Sent false
5144 Acked Read 5 Done Done Sent false
5145 Acked Write 8 Done Done Sent false
5146 Acked Read 4 Done Done Sent false
5147 Acked Read 4 Done Done Sent false
5148 Acked Read 3 Done Done Sent false
5149 Acked Read 9 Done Done Sent false
5150 Acked Read 6 Done Done Sent false
5151 Acked Flush 0 Done Done Sent false
5152 Acked Write 8 Done Done Sent false
5153 Acked Flush 0 Done Done Sent false
5154 Acked Write 10 Done Done Sent false
5155 Acked Write 9 Done Done Sent false
5156 Acked Write 2 Done Done Sent false
5157 Acked Read 7 Done Done Sent false
5158 Acked Write 9 Done Done Sent false
5159 Acked Read 7 Done Done Sent false
5160 Acked Flush 0 Done Done Sent false
5161 Acked Write 4 Done Done Sent false
5162 Acked Write 2 Done Done Sent false
5163 Acked Read 6 Done Done Sent false
5164 Acked Read 1 Done Done Sent false
5165 Acked Read 1 Done Done Sent false
5166 Acked Flush 0 Done Done Sent false
5167 Acked Read 8 Done Done Sent false
5168 Acked Flush 0 Done Done Sent false
5169 Acked Flush 0 Done Done Sent false
5170 Acked Read 4 Done Done Sent false
5171 Acked Write 7 Done Done Sent false
5172 Acked Flush 0 Done Done Sent false
5173 Acked Flush 0 Done Done Sent false
5174 Acked Read 10 Done Done Sent false
5175 Acked Write 1 Done Done Sent false
5176 Acked Flush 0 Done Done Sent false
5177 Acked Write 1 Done Done Sent false
5178 Acked Write 6 Done Done Sent false
5179 Acked Write 1 Done Done Sent false
5180 Acked Read 4 Done Done Sent false
5181 Acked Read 1 Done Done Sent false
5182 Acked Write 3 Done Done Sent false
5183 Acked Read 9 Done Done Sent false
5184 Acked Read 4 Done Done Sent false
5185 Acked Write 10 Done Done Sent false
5186 Acked Write 3 Done Done Sent false
5187 Acked Read 7 Done Done Sent false
5188 Acked Write 3 Done Done Sent false
5189 Acked Read 10 Done Done Sent false
5190 Acked Write 2 Done Done Sent false
5191 Acked Read 3 Done Done Sent false
5192 Acked Read 7 Done Done Sent false
5193 Acked Read 6 Done Done Sent false
5194 Acked Flush 0 Done Done Sent false
5195 Acked Read 6 Done Done Sent false
5196 Acked Read 8 Done Done Sent false
5197 Acked Read 2 Done Done Sent false
5198 Acked Write 2 Done Done Sent false
5199 Acked Write 7 Done Done Sent false
5200 Acked Flush 0 Done Done Sent false
5201 Acked Read 7 Done Done Sent false
5202 Acked Read 3 Done Done Sent false
5203 Acked Write 9 Done Done Sent false
5204 Acked Write 10 Done Done Sent false
5205 Acked Write 3 Done Done Sent false
5206 Acked Read 7 Done Done Sent false
5207 Acked Write 9 Done Done Sent false
5208 Acked Read 6 Done Done Sent false
5209 Acked Flush 0 Done Done Sent false
5210 Acked Flush 0 Done Done Sent false
5211 Acked Read 2 Done Done Sent false
5212 Acked Read 5 Done Done Sent false
5213 Acked Read 7 Done Done Sent false
5214 Acked Write 4 Done Done Sent false
5215 Acked Read 2 Done Done Sent false
5216 Acked Read 3 Done Done Sent false
5217 Acked Read 8 Done Done Sent false
5218 Acked Write 6 Done Done Sent false
5219 Acked Read 7 Done Done Sent false
5220 Acked Write 1 Done Done Sent false
5221 Acked Read 5 Done Done Sent false
5222 Acked Read 7 Done Done Sent false
5223 Acked Write 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:39.864904058Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:40.703084803Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:41.892667281Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:42.705758048Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Flush 0 Done Done Sent false
4924 Acked Read 4 Done Done Sent false
4925 Acked Read 3 Done Done Sent false
4926 Acked Write 9 Done Done Sent false
4927 Acked Read 9 Done Done Sent false
4928 Acked Read 7 Done Done Sent false
4929 Acked Write 7 Done Done Sent false
4930 Acked Read 1 Done Done Sent false
4931 Acked Write 6 Done Done Sent false
4932 Acked Read 5 Done Done Sent false
4933 Acked Write 1 Done Done Sent false
4934 Acked Read 6 Done Done Sent false
4935 Acked Read 1 Done Done Sent false
4936 Acked Read 5 Done Done Sent false
4937 Acked Read 3 Done Done Sent false
4938 Acked Flush 0 Done Done Sent false
4939 Acked Read 5 Done Done Sent false
4940 Acked Read 9 Done Done Sent false
4941 Acked Flush 0 Done Done Sent false
4942 Acked Read 2 Done Done Sent false
4943 Acked Read 6 Done Done Sent false
4944 Acked Flush 0 Done Done Sent false
4945 Acked Write 7 Done Done Sent false
4946 Acked Write 1 Done Done Sent false
4947 Acked Write 2 Done Done Sent false
4948 Acked Write 6 Done Done Sent false
4949 Acked Read 8 Done Done Sent false
4950 Acked Write 5 Done Done Sent false
4951 Acked Write 2 Done Done Sent false
4952 Acked Write 7 Done Done Sent false
4953 Acked Read 8 Done Done Sent false
4954 Acked Read 9 Done Done Sent false
4955 Acked Flush 0 Done Done Sent false
4956 Acked Write 3 Done Done Sent false
4957 Acked Write 9 Done Done Sent false
4958 Acked Read 1 Done Done Sent false
4959 Acked Read 8 Done Done Sent false
4960 Acked Read 10 Done Done Sent false
4961 Acked Read 4 Done Done Sent false
4962 Acked Write 4 Done Done Sent false
4963 Acked Write 10 Done Done Sent false
4964 Acked Read 10 Done Done Sent false
4965 Acked Write 2 Done Done Sent false
4966 Acked Read 6 Done Done Sent false
4967 Acked Flush 0 Done Done Sent false
4968 Acked Read 6 Done Done Sent false
4969 Acked Write 4 Done Done Sent false
4970 Acked Read 2 Done Done Sent false
4971 Acked Write 7 Done Done Sent false
4972 Acked Read 1 Done Done Sent false
4973 Acked Write 9 Done Done Sent false
4974 Acked Read 8 Done Done Sent false
4975 Acked Read 4 Done Done Sent false
4976 Acked Write 2 Done Done Sent false
4977 Acked Write 6 Done Done Sent false
4978 Acked Write 8 Done Done Sent false
4979 Acked Write 7 Done Done Sent false
4980 Acked Write 6 Done Done Sent false
4981 Acked Flush 0 Done Done Sent false
4982 Acked Read 10 Done Done Sent false
4983 Acked Write 1 Done Done Sent false
4984 Acked Read 5 Done Done Sent false
4985 Acked Read 2 Done Done Sent false
4986 Acked Read 7 Done Done Sent false
4987 Acked Read 6 Done Done Sent false
4988 Acked Read 2 Done Done Sent false
4989 Acked Write 1 Done Done Sent false
4990 Acked Write 8 Done Done Sent false
4991 Acked Read 8 Done Done Sent false
4992 Acked Write 7 Done Done Sent false
4993 Acked Write 3 Done Done Sent false
4994 Acked Write 5 Done Done Sent false
4995 Acked Read 2 Done Done Sent false
4996 Acked Flush 0 Done Done Sent false
4997 Acked Write 3 Done Done Sent false
4998 Acked Read 10 Done Done Sent false
4999 Acked Write 4 Done Done Sent false
5000 Acked Write 6 Done Done Sent false
5001 Acked Write 10 Done Done Sent false
5002 Acked Write 7 Done Done Sent false
5003 Acked Write 2 Done Done Sent false
5004 Acked Write 6 Done Done Sent false
5005 Acked Flush 0 Done Done Sent false
5006 Acked Write 4 Done Done Sent false
5007 Acked Write 1 Done Done Sent false
5008 Acked Write 8 Done Done Sent false
5009 Acked Write 6 Done Done Sent false
5010 Acked Read 7 Done Done Sent false
5011 Acked Write 2 Done Done Sent false
5012 Acked Read 7 Done Done Sent false
5013 Acked Read 8 Done Done Sent false
5014 Acked Read 4 Done Done Sent false
5015 Acked Write 6 Done Done Sent false
5016 Acked Read 4 Done Done Sent false
5017 Acked Write 4 Done Done Sent false
5018 Acked Write 9 Done Done Sent false
5019 Acked Write 4 Done Done Sent false
5020 Acked Read 5 Done Done Sent false
5021 Acked Write 4 Done Done Sent false
5022 Acked Read 8 Done Done Sent false
5023 Acked Read 5 Done Done Sent false
5024 Acked Write 1 Done Done Sent false
5025 Acked Read 3 Done Done Sent false
5026 Acked Read 4 Done Done Sent false
5027 Acked Flush 0 Done Done Sent false
5028 Acked Read 1 Done Done Sent false
5029 Acked Write 5 Done Done Sent false
5030 Acked Flush 0 Done Done Sent false
5031 Acked Read 10 Done Done Sent false
5032 Acked Read 4 Done Done Sent false
5033 Acked Read 1 Done Done Sent false
5034 Acked Read 2 Done Done Sent false
5035 Acked Read 2 Done Done Sent false
5036 Acked Write 6 Done Done Sent false
5037 Acked Write 6 Done Done Sent false
5038 Acked Write 5 Done Done Sent false
5039 Acked Write 2 Done Done Sent false
5040 Acked Write 2 Done Done Sent false
5041 Acked Read 5 Done Done Sent false
5042 Acked Write 5 Done Done Sent false
5043 Acked Read 1 Done Done Sent false
5044 Acked Read 2 Done Done Sent false
5045 Acked Write 1 Done Done Sent false
5046 Acked Read 6 Done Done Sent false
5047 Acked Write 3 Done Done Sent false
5048 Acked Read 5 Done Done Sent false
5049 Acked Write 9 Done Done Sent false
5050 Acked Read 8 Done Done Sent false
5051 Acked Write 2 Done Done Sent false
5052 Acked Read 4 Done Done Sent false
5053 Acked Read 1 Done Done Sent false
5054 Acked Write 6 Done Done Sent false
5055 Acked Read 5 Done Done Sent false
5056 Acked Write 1 Done Done Sent false
5057 Acked Write 5 Done Done Sent false
5058 Acked Write 9 Done Done Sent false
5059 Acked Write 3 Done Done Sent false
5060 Acked Read 2 Done Done Sent false
5061 Acked Flush 0 Done Done Sent false
5062 Acked Write 2 Done Done Sent false
5063 Acked Write 4 Done Done Sent false
5064 Acked Flush 0 Done Done Sent false
5065 Acked Read 4 Done Done Sent false
5066 Acked Read 10 Done Done Sent false
5067 Acked Read 8 Done Done Sent false
5068 Acked Write 2 Done Done Sent false
5069 Acked Flush 0 Done Done Sent false
5070 Acked Write 7 Done Done Sent false
5071 Acked Read 9 Done Done Sent false
5072 Acked Read 10 Done Done Sent false
5073 Acked Read 7 Done Done Sent false
5074 Acked Write 7 Done Done Sent false
5075 Acked Read 7 Done Done Sent false
5076 Acked Read 8 Done Done Sent false
5077 Acked Write 1 Done Done Sent false
5078 Acked Write 10 Done Done Sent false
5079 Acked Read 2 Done Done Sent false
5080 Acked Write 4 Done Done Sent false
5081 Acked Write 2 Done Done Sent false
5082 Acked Write 10 Done Done Sent false
5083 Acked Read 10 Done Done Sent false
5084 Acked Read 10 Done Done Sent false
5085 Acked Write 7 Done Done Sent false
5086 Acked Write 7 Done Done Sent false
5087 Acked Read 1 Done Done Sent false
5088 Acked Flush 0 Done Done Sent false
5089 Acked Read 10 Done Done Sent false
5090 Acked Flush 0 Done Done Sent false
5091 Acked Read 10 Done Done Sent false
5092 Acked Flush 0 Done Done Sent false
5093 Acked Write 3 Done Done Sent false
5094 Acked Write 4 Done Done Sent false
5095 Acked Write 6 Done Done Sent false
5096 Acked Write 9 Done Done Sent false
5097 Acked Read 6 Done Done Sent false
5098 Acked Read 6 Done Done Sent false
5099 Acked Flush 0 Done Done Sent false
5100 Acked Write 1 Done Done Sent false
5101 Acked Write 2 Done Done Sent false
5102 Acked Read 3 Done Done Sent false
5103 Acked Read 2 Done Done Sent false
5104 Acked Write 5 Done Done Sent false
5105 Acked Read 9 Done Done Sent false
5106 Acked Read 6 Done Done Sent false
5107 Acked Read 6 Done Done Sent false
5108 Acked Read 1 Done Done Sent false
5109 Acked Write 7 Done Done Sent false
5110 Acked Read 2 Done Done Sent false
5111 Acked Read 9 Done Done Sent false
5112 Acked Read 6 Done Done Sent false
5113 Acked Read 1 Done Done Sent false
5114 Acked Flush 0 Done Done Sent false
5115 Acked Flush 0 Done Done Sent false
5116 Acked Read 7 Done Done Sent false
5117 Acked Write 5 Done Done Sent false
5118 Acked Read 8 Done Done Sent false
5119 Acked Read 2 Done Done Sent false
5120 Acked Read 9 Done Done Sent false
5121 Acked Read 1 Done Done Sent false
5122 Acked Read 2 Done Done Sent false
5123 Acked Read 8 Done Done Sent false
5124 Acked Read 4 Done Done Sent false
5125 Acked Read 4 Done Done Sent false
5126 Acked Read 2 Done Done Sent false
5127 Acked Read 8 Done Done Sent false
5128 Acked Read 6 Done Done Sent false
5129 Acked Write 8 Done Done Sent false
5130 Acked Write 3 Done Done Sent false
5131 Acked Write 3 Done Done Sent false
5132 Acked Read 4 Done Done Sent false
5133 Acked Flush 0 Done Done Sent false
5134 Acked Read 9 Done Done Sent false
5135 Acked Read 4 Done Done Sent false
5136 Acked Write 2 Done Done Sent false
5137 Acked Write 9 Done Done Sent false
5138 Acked Write 10 Done Done Sent false
5139 Acked Read 1 Done Done Sent false
5140 Acked Write 7 Done Done Sent false
5141 Acked Read 8 Done Done Sent false
5142 Acked Read 7 Done Done Sent false
5143 Acked Write 6 Done Done Sent false
5144 Acked Read 5 Done Done Sent false
5145 Acked Write 8 Done Done Sent false
5146 Acked Read 4 Done Done Sent false
5147 Acked Read 4 Done Done Sent false
5148 Acked Read 3 Done Done Sent false
5149 Acked Read 9 Done Done Sent false
5150 Acked Read 6 Done Done Sent false
5151 Acked Flush 0 Done Done Sent false
5152 Acked Write 8 Done Done Sent false
5153 Acked Flush 0 Done Done Sent false
5154 Acked Write 10 Done Done Sent false
5155 Acked Write 9 Done Done Sent false
5156 Acked Write 2 Done Done Sent false
5157 Acked Read 7 Done Done Sent false
5158 Acked Write 9 Done Done Sent false
5159 Acked Read 7 Done Done Sent false
5160 Acked Flush 0 Done Done Sent false
5161 Acked Write 4 Done Done Sent false
5162 Acked Write 2 Done Done Sent false
5163 Acked Read 6 Done Done Sent false
5164 Acked Read 1 Done Done Sent false
5165 Acked Read 1 Done Done Sent false
5166 Acked Flush 0 Done Done Sent false
5167 Acked Read 8 Done Done Sent false
5168 Acked Flush 0 Done Done Sent false
5169 Acked Flush 0 Done Done Sent false
5170 Acked Read 4 Done Done Sent false
5171 Acked Write 7 Done Done Sent false
5172 Acked Flush 0 Done Done Sent false
5173 Acked Flush 0 Done Done Sent false
5174 Acked Read 10 Done Done Sent false
5175 Acked Write 1 Done Done Sent false
5176 Acked Flush 0 Done Done Sent false
5177 Acked Write 1 Done Done Sent false
5178 Acked Write 6 Done Done Sent false
5179 Acked Write 1 Done Done Sent false
5180 Acked Read 4 Done Done Sent false
5181 Acked Read 1 Done Done Sent false
5182 Acked Write 3 Done Done Sent false
5183 Acked Read 9 Done Done Sent false
5184 Acked Read 4 Done Done Sent false
5185 Acked Write 10 Done Done Sent false
5186 Acked Write 3 Done Done Sent false
5187 Acked Read 7 Done Done Sent false
5188 Acked Write 3 Done Done Sent false
5189 Acked Read 10 Done Done Sent false
5190 Acked Write 2 Done Done Sent false
5191 Acked Read 3 Done Done Sent false
5192 Acked Read 7 Done Done Sent false
5193 Acked Read 6 Done Done Sent false
5194 Acked Flush 0 Done Done Sent false
5195 Acked Read 6 Done Done Sent false
5196 Acked Read 8 Done Done Sent false
5197 Acked Read 2 Done Done Sent false
5198 Acked Write 2 Done Done Sent false
5199 Acked Write 7 Done Done Sent false
5200 Acked Flush 0 Done Done Sent false
5201 Acked Read 7 Done Done Sent false
5202 Acked Read 3 Done Done Sent false
5203 Acked Write 9 Done Done Sent false
5204 Acked Write 10 Done Done Sent false
5205 Acked Write 3 Done Done Sent false
5206 Acked Read 7 Done Done Sent false
5207 Acked Write 9 Done Done Sent false
5208 Acked Read 6 Done Done Sent false
5209 Acked Flush 0 Done Done Sent false
5210 Acked Flush 0 Done Done Sent false
5211 Acked Read 2 Done Done Sent false
5212 Acked Read 5 Done Done Sent false
5213 Acked Read 7 Done Done Sent false
5214 Acked Write 4 Done Done Sent false
5215 Acked Read 2 Done Done Sent false
5216 Acked Read 3 Done Done Sent false
5217 Acked Read 8 Done Done Sent false
5218 Acked Write 6 Done Done Sent false
5219 Acked Read 7 Done Done Sent false
5220 Acked Write 1 Done Done Sent false
5221 Acked Read 5 Done Done Sent false
5222 Acked Read 7 Done Done Sent false
5223 Acked Write 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:43.868231053Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:44.708175985Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:45.129480786Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:45.129527474Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:45.129548909Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:45.129568585Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:45.129706742Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:45.709527024Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:46.209824805Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:46.894400541Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Flush 0 Done Done Sent false
4924 Acked Read 4 Done Done Sent false
4925 Acked Read 3 Done Done Sent false
4926 Acked Write 9 Done Done Sent false
4927 Acked Read 9 Done Done Sent false
4928 Acked Read 7 Done Done Sent false
4929 Acked Write 7 Done Done Sent false
4930 Acked Read 1 Done Done Sent false
4931 Acked Write 6 Done Done Sent false
4932 Acked Read 5 Done Done Sent false
4933 Acked Write 1 Done Done Sent false
4934 Acked Read 6 Done Done Sent false
4935 Acked Read 1 Done Done Sent false
4936 Acked Read 5 Done Done Sent false
4937 Acked Read 3 Done Done Sent false
4938 Acked Flush 0 Done Done Sent false
4939 Acked Read 5 Done Done Sent false
4940 Acked Read 9 Done Done Sent false
4941 Acked Flush 0 Done Done Sent false
4942 Acked Read 2 Done Done Sent false
4943 Acked Read 6 Done Done Sent false
4944 Acked Flush 0 Done Done Sent false
4945 Acked Write 7 Done Done Sent false
4946 Acked Write 1 Done Done Sent false
4947 Acked Write 2 Done Done Sent false
4948 Acked Write 6 Done Done Sent false
4949 Acked Read 8 Done Done Sent false
4950 Acked Write 5 Done Done Sent false
4951 Acked Write 2 Done Done Sent false
4952 Acked Write 7 Done Done Sent false
4953 Acked Read 8 Done Done Sent false
4954 Acked Read 9 Done Done Sent false
4955 Acked Flush 0 Done Done Sent false
4956 Acked Write 3 Done Done Sent false
4957 Acked Write 9 Done Done Sent false
4958 Acked Read 1 Done Done Sent false
4959 Acked Read 8 Done Done Sent false
4960 Acked Read 10 Done Done Sent false
4961 Acked Read 4 Done Done Sent false
4962 Acked Write 4 Done Done Sent false
4963 Acked Write 10 Done Done Sent false
4964 Acked Read 10 Done Done Sent false
4965 Acked Write 2 Done Done Sent false
4966 Acked Read 6 Done Done Sent false
4967 Acked Flush 0 Done Done Sent false
4968 Acked Read 6 Done Done Sent false
4969 Acked Write 4 Done Done Sent false
4970 Acked Read 2 Done Done Sent false
4971 Acked Write 7 Done Done Sent false
4972 Acked Read 1 Done Done Sent false
4973 Acked Write 9 Done Done Sent false
4974 Acked Read 8 Done Done Sent false
4975 Acked Read 4 Done Done Sent false
4976 Acked Write 2 Done Done Sent false
4977 Acked Write 6 Done Done Sent false
4978 Acked Write 8 Done Done Sent false
4979 Acked Write 7 Done Done Sent false
4980 Acked Write 6 Done Done Sent false
4981 Acked Flush 0 Done Done Sent false
4982 Acked Read 10 Done Done Sent false
4983 Acked Write 1 Done Done Sent false
4984 Acked Read 5 Done Done Sent false
4985 Acked Read 2 Done Done Sent false
4986 Acked Read 7 Done Done Sent false
4987 Acked Read 6 Done Done Sent false
4988 Acked Read 2 Done Done Sent false
4989 Acked Write 1 Done Done Sent false
4990 Acked Write 8 Done Done Sent false
4991 Acked Read 8 Done Done Sent false
4992 Acked Write 7 Done Done Sent false
4993 Acked Write 3 Done Done Sent false
4994 Acked Write 5 Done Done Sent false
4995 Acked Read 2 Done Done Sent false
4996 Acked Flush 0 Done Done Sent false
4997 Acked Write 3 Done Done Sent false
4998 Acked Read 10 Done Done Sent false
4999 Acked Write 4 Done Done Sent false
5000 Acked Write 6 Done Done Sent false
5001 Acked Write 10 Done Done Sent false
5002 Acked Write 7 Done Done Sent false
5003 Acked Write 2 Done Done Sent false
5004 Acked Write 6 Done Done Sent false
5005 Acked Flush 0 Done Done Sent false
5006 Acked Write 4 Done Done Sent false
5007 Acked Write 1 Done Done Sent false
5008 Acked Write 8 Done Done Sent false
5009 Acked Write 6 Done Done Sent false
5010 Acked Read 7 Done Done Sent false
5011 Acked Write 2 Done Done Sent false
5012 Acked Read 7 Done Done Sent false
5013 Acked Read 8 Done Done Sent false
5014 Acked Read 4 Done Done Sent false
5015 Acked Write 6 Done Done Sent false
5016 Acked Read 4 Done Done Sent false
5017 Acked Write 4 Done Done Sent false
5018 Acked Write 9 Done Done Sent false
5019 Acked Write 4 Done Done Sent false
5020 Acked Read 5 Done Done Sent false
5021 Acked Write 4 Done Done Sent false
5022 Acked Read 8 Done Done Sent false
5023 Acked Read 5 Done Done Sent false
5024 Acked Write 1 Done Done Sent false
5025 Acked Read 3 Done Done Sent false
5026 Acked Read 4 Done Done Sent false
5027 Acked Flush 0 Done Done Sent false
5028 Acked Read 1 Done Done Sent false
5029 Acked Write 5 Done Done Sent false
5030 Acked Flush 0 Done Done Sent false
5031 Acked Read 10 Done Done Sent false
5032 Acked Read 4 Done Done Sent false
5033 Acked Read 1 Done Done Sent false
5034 Acked Read 2 Done Done Sent false
5035 Acked Read 2 Done Done Sent false
5036 Acked Write 6 Done Done Sent false
5037 Acked Write 6 Done Done Sent false
5038 Acked Write 5 Done Done Sent false
5039 Acked Write 2 Done Done Sent false
5040 Acked Write 2 Done Done Sent false
5041 Acked Read 5 Done Done Sent false
5042 Acked Write 5 Done Done Sent false
5043 Acked Read 1 Done Done Sent false
5044 Acked Read 2 Done Done Sent false
5045 Acked Write 1 Done Done Sent false
5046 Acked Read 6 Done Done Sent false
5047 Acked Write 3 Done Done Sent false
5048 Acked Read 5 Done Done Sent false
5049 Acked Write 9 Done Done Sent false
5050 Acked Read 8 Done Done Sent false
5051 Acked Write 2 Done Done Sent false
5052 Acked Read 4 Done Done Sent false
5053 Acked Read 1 Done Done Sent false
5054 Acked Write 6 Done Done Sent false
5055 Acked Read 5 Done Done Sent false
5056 Acked Write 1 Done Done Sent false
5057 Acked Write 5 Done Done Sent false
5058 Acked Write 9 Done Done Sent false
5059 Acked Write 3 Done Done Sent false
5060 Acked Read 2 Done Done Sent false
5061 Acked Flush 0 Done Done Sent false
5062 Acked Write 2 Done Done Sent false
5063 Acked Write 4 Done Done Sent false
5064 Acked Flush 0 Done Done Sent false
5065 Acked Read 4 Done Done Sent false
5066 Acked Read 10 Done Done Sent false
5067 Acked Read 8 Done Done Sent false
5068 Acked Write 2 Done Done Sent false
5069 Acked Flush 0 Done Done Sent false
5070 Acked Write 7 Done Done Sent false
5071 Acked Read 9 Done Done Sent false
5072 Acked Read 10 Done Done Sent false
5073 Acked Read 7 Done Done Sent false
5074 Acked Write 7 Done Done Sent false
5075 Acked Read 7 Done Done Sent false
5076 Acked Read 8 Done Done Sent false
5077 Acked Write 1 Done Done Sent false
5078 Acked Write 10 Done Done Sent false
5079 Acked Read 2 Done Done Sent false
5080 Acked Write 4 Done Done Sent false
5081 Acked Write 2 Done Done Sent false
5082 Acked Write 10 Done Done Sent false
5083 Acked Read 10 Done Done Sent false
5084 Acked Read 10 Done Done Sent false
5085 Acked Write 7 Done Done Sent false
5086 Acked Write 7 Done Done Sent false
5087 Acked Read 1 Done Done Sent false
5088 Acked Flush 0 Done Done Sent false
5089 Acked Read 10 Done Done Sent false
5090 Acked Flush 0 Done Done Sent false
5091 Acked Read 10 Done Done Sent false
5092 Acked Flush 0 Done Done Sent false
5093 Acked Write 3 Done Done Sent false
5094 Acked Write 4 Done Done Sent false
5095 Acked Write 6 Done Done Sent false
5096 Acked Write 9 Done Done Sent false
5097 Acked Read 6 Done Done Sent false
5098 Acked Read 6 Done Done Sent false
5099 Acked Flush 0 Done Done Sent false
5100 Acked Write 1 Done Done Sent false
5101 Acked Write 2 Done Done Sent false
5102 Acked Read 3 Done Done Sent false
5103 Acked Read 2 Done Done Sent false
5104 Acked Write 5 Done Done Sent false
5105 Acked Read 9 Done Done Sent false
5106 Acked Read 6 Done Done Sent false
5107 Acked Read 6 Done Done Sent false
5108 Acked Read 1 Done Done Sent false
5109 Acked Write 7 Done Done Sent false
5110 Acked Read 2 Done Done Sent false
5111 Acked Read 9 Done Done Sent false
5112 Acked Read 6 Done Done Sent false
5113 Acked Read 1 Done Done Sent false
5114 Acked Flush 0 Done Done Sent false
5115 Acked Flush 0 Done Done Sent false
5116 Acked Read 7 Done Done Sent false
5117 Acked Write 5 Done Done Sent false
5118 Acked Read 8 Done Done Sent false
5119 Acked Read 2 Done Done Sent false
5120 Acked Read 9 Done Done Sent false
5121 Acked Read 1 Done Done Sent false
5122 Acked Read 2 Done Done Sent false
5123 Acked Read 8 Done Done Sent false
5124 Acked Read 4 Done Done Sent false
5125 Acked Read 4 Done Done Sent false
5126 Acked Read 2 Done Done Sent false
5127 Acked Read 8 Done Done Sent false
5128 Acked Read 6 Done Done Sent false
5129 Acked Write 8 Done Done Sent false
5130 Acked Write 3 Done Done Sent false
5131 Acked Write 3 Done Done Sent false
5132 Acked Read 4 Done Done Sent false
5133 Acked Flush 0 Done Done Sent false
5134 Acked Read 9 Done Done Sent false
5135 Acked Read 4 Done Done Sent false
5136 Acked Write 2 Done Done Sent false
5137 Acked Write 9 Done Done Sent false
5138 Acked Write 10 Done Done Sent false
5139 Acked Read 1 Done Done Sent false
5140 Acked Write 7 Done Done Sent false
5141 Acked Read 8 Done Done Sent false
5142 Acked Read 7 Done Done Sent false
5143 Acked Write 6 Done Done Sent false
5144 Acked Read 5 Done Done Sent false
5145 Acked Write 8 Done Done Sent false
5146 Acked Read 4 Done Done Sent false
5147 Acked Read 4 Done Done Sent false
5148 Acked Read 3 Done Done Sent false
5149 Acked Read 9 Done Done Sent false
5150 Acked Read 6 Done Done Sent false
5151 Acked Flush 0 Done Done Sent false
5152 Acked Write 8 Done Done Sent false
5153 Acked Flush 0 Done Done Sent false
5154 Acked Write 10 Done Done Sent false
5155 Acked Write 9 Done Done Sent false
5156 Acked Write 2 Done Done Sent false
5157 Acked Read 7 Done Done Sent false
5158 Acked Write 9 Done Done Sent false
5159 Acked Read 7 Done Done Sent false
5160 Acked Flush 0 Done Done Sent false
5161 Acked Write 4 Done Done Sent false
5162 Acked Write 2 Done Done Sent false
5163 Acked Read 6 Done Done Sent false
5164 Acked Read 1 Done Done Sent false
5165 Acked Read 1 Done Done Sent false
5166 Acked Flush 0 Done Done Sent false
5167 Acked Read 8 Done Done Sent false
5168 Acked Flush 0 Done Done Sent false
5169 Acked Flush 0 Done Done Sent false
5170 Acked Read 4 Done Done Sent false
5171 Acked Write 7 Done Done Sent false
5172 Acked Flush 0 Done Done Sent false
5173 Acked Flush 0 Done Done Sent false
5174 Acked Read 10 Done Done Sent false
5175 Acked Write 1 Done Done Sent false
5176 Acked Flush 0 Done Done Sent false
5177 Acked Write 1 Done Done Sent false
5178 Acked Write 6 Done Done Sent false
5179 Acked Write 1 Done Done Sent false
5180 Acked Read 4 Done Done Sent false
5181 Acked Read 1 Done Done Sent false
5182 Acked Write 3 Done Done Sent false
5183 Acked Read 9 Done Done Sent false
5184 Acked Read 4 Done Done Sent false
5185 Acked Write 10 Done Done Sent false
5186 Acked Write 3 Done Done Sent false
5187 Acked Read 7 Done Done Sent false
5188 Acked Write 3 Done Done Sent false
5189 Acked Read 10 Done Done Sent false
5190 Acked Write 2 Done Done Sent false
5191 Acked Read 3 Done Done Sent false
5192 Acked Read 7 Done Done Sent false
5193 Acked Read 6 Done Done Sent false
5194 Acked Flush 0 Done Done Sent false
5195 Acked Read 6 Done Done Sent false
5196 Acked Read 8 Done Done Sent false
5197 Acked Read 2 Done Done Sent false
5198 Acked Write 2 Done Done Sent false
5199 Acked Write 7 Done Done Sent false
5200 Acked Flush 0 Done Done Sent false
5201 Acked Read 7 Done Done Sent false
5202 Acked Read 3 Done Done Sent false
5203 Acked Write 9 Done Done Sent false
5204 Acked Write 10 Done Done Sent false
5205 Acked Write 3 Done Done Sent false
5206 Acked Read 7 Done Done Sent false
5207 Acked Write 9 Done Done Sent false
5208 Acked Read 6 Done Done Sent false
5209 Acked Flush 0 Done Done Sent false
5210 Acked Flush 0 Done Done Sent false
5211 Acked Read 2 Done Done Sent false
5212 Acked Read 5 Done Done Sent false
5213 Acked Read 7 Done Done Sent false
5214 Acked Write 4 Done Done Sent false
5215 Acked Read 2 Done Done Sent false
5216 Acked Read 3 Done Done Sent false
5217 Acked Read 8 Done Done Sent false
5218 Acked Write 6 Done Done Sent false
5219 Acked Read 7 Done Done Sent false
5220 Acked Write 1 Done Done Sent false
5221 Acked Read 5 Done Done Sent false
5222 Acked Read 7 Done Done Sent false
5223 Acked Write 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:47.869959892Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:48.714976699Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:49.715888123Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:50.717722957Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Flush 0 Done Done Sent false
4924 Acked Read 4 Done Done Sent false
4925 Acked Read 3 Done Done Sent false
4926 Acked Write 9 Done Done Sent false
4927 Acked Read 9 Done Done Sent false
4928 Acked Read 7 Done Done Sent false
4929 Acked Write 7 Done Done Sent false
4930 Acked Read 1 Done Done Sent false
4931 Acked Write 6 Done Done Sent false
4932 Acked Read 5 Done Done Sent false
4933 Acked Write 1 Done Done Sent false
4934 Acked Read 6 Done Done Sent false
4935 Acked Read 1 Done Done Sent false
4936 Acked Read 5 Done Done Sent false
4937 Acked Read 3 Done Done Sent false
4938 Acked Flush 0 Done Done Sent false
4939 Acked Read 5 Done Done Sent false
4940 Acked Read 9 Done Done Sent false
4941 Acked Flush 0 Done Done Sent false
4942 Acked Read 2 Done Done Sent false
4943 Acked Read 6 Done Done Sent false
4944 Acked Flush 0 Done Done Sent false
4945 Acked Write 7 Done Done Sent false
4946 Acked Write 1 Done Done Sent false
4947 Acked Write 2 Done Done Sent false
4948 Acked Write 6 Done Done Sent false
4949 Acked Read 8 Done Done Sent false
4950 Acked Write 5 Done Done Sent false
4951 Acked Write 2 Done Done Sent false
4952 Acked Write 7 Done Done Sent false
4953 Acked Read 8 Done Done Sent false
4954 Acked Read 9 Done Done Sent false
4955 Acked Flush 0 Done Done Sent false
4956 Acked Write 3 Done Done Sent false
4957 Acked Write 9 Done Done Sent false
4958 Acked Read 1 Done Done Sent false
4959 Acked Read 8 Done Done Sent false
4960 Acked Read 10 Done Done Sent false
4961 Acked Read 4 Done Done Sent false
4962 Acked Write 4 Done Done Sent false
4963 Acked Write 10 Done Done Sent false
4964 Acked Read 10 Done Done Sent false
4965 Acked Write 2 Done Done Sent false
4966 Acked Read 6 Done Done Sent false
4967 Acked Flush 0 Done Done Sent false
4968 Acked Read 6 Done Done Sent false
4969 Acked Write 4 Done Done Sent false
4970 Acked Read 2 Done Done Sent false
4971 Acked Write 7 Done Done Sent false
4972 Acked Read 1 Done Done Sent false
4973 Acked Write 9 Done Done Sent false
4974 Acked Read 8 Done Done Sent false
4975 Acked Read 4 Done Done Sent false
4976 Acked Write 2 Done Done Sent false
4977 Acked Write 6 Done Done Sent false
4978 Acked Write 8 Done Done Sent false
4979 Acked Write 7 Done Done Sent false
4980 Acked Write 6 Done Done Sent false
4981 Acked Flush 0 Done Done Sent false
4982 Acked Read 10 Done Done Sent false
4983 Acked Write 1 Done Done Sent false
4984 Acked Read 5 Done Done Sent false
4985 Acked Read 2 Done Done Sent false
4986 Acked Read 7 Done Done Sent false
4987 Acked Read 6 Done Done Sent false
4988 Acked Read 2 Done Done Sent false
4989 Acked Write 1 Done Done Sent false
4990 Acked Write 8 Done Done Sent false
4991 Acked Read 8 Done Done Sent false
4992 Acked Write 7 Done Done Sent false
4993 Acked Write 3 Done Done Sent false
4994 Acked Write 5 Done Done Sent false
4995 Acked Read 2 Done Done Sent false
4996 Acked Flush 0 Done Done Sent false
4997 Acked Write 3 Done Done Sent false
4998 Acked Read 10 Done Done Sent false
4999 Acked Write 4 Done Done Sent false
5000 Acked Write 6 Done Done Sent false
5001 Acked Write 10 Done Done Sent false
5002 Acked Write 7 Done Done Sent false
5003 Acked Write 2 Done Done Sent false
5004 Acked Write 6 Done Done Sent false
5005 Acked Flush 0 Done Done Sent false
5006 Acked Write 4 Done Done Sent false
5007 Acked Write 1 Done Done Sent false
5008 Acked Write 8 Done Done Sent false
5009 Acked Write 6 Done Done Sent false
5010 Acked Read 7 Done Done Sent false
5011 Acked Write 2 Done Done Sent false
5012 Acked Read 7 Done Done Sent false
5013 Acked Read 8 Done Done Sent false
5014 Acked Read 4 Done Done Sent false
5015 Acked Write 6 Done Done Sent false
5016 Acked Read 4 Done Done Sent false
5017 Acked Write 4 Done Done Sent false
5018 Acked Write 9 Done Done Sent false
5019 Acked Write 4 Done Done Sent false
5020 Acked Read 5 Done Done Sent false
5021 Acked Write 4 Done Done Sent false
5022 Acked Read 8 Done Done Sent false
5023 Acked Read 5 Done Done Sent false
5024 Acked Write 1 Done Done Sent false
5025 Acked Read 3 Done Done Sent false
5026 Acked Read 4 Done Done Sent false
5027 Acked Flush 0 Done Done Sent false
5028 Acked Read 1 Done Done Sent false
5029 Acked Write 5 Done Done Sent false
5030 Acked Flush 0 Done Done Sent false
5031 Acked Read 10 Done Done Sent false
5032 Acked Read 4 Done Done Sent false
5033 Acked Read 1 Done Done Sent false
5034 Acked Read 2 Done Done Sent false
5035 Acked Read 2 Done Done Sent false
5036 Acked Write 6 Done Done Sent false
5037 Acked Write 6 Done Done Sent false
5038 Acked Write 5 Done Done Sent false
5039 Acked Write 2 Done Done Sent false
5040 Acked Write 2 Done Done Sent false
5041 Acked Read 5 Done Done Sent false
5042 Acked Write 5 Done Done Sent false
5043 Acked Read 1 Done Done Sent false
5044 Acked Read 2 Done Done Sent false
5045 Acked Write 1 Done Done Sent false
5046 Acked Read 6 Done Done Sent false
5047 Acked Write 3 Done Done Sent false
5048 Acked Read 5 Done Done Sent false
5049 Acked Write 9 Done Done Sent false
5050 Acked Read 8 Done Done Sent false
5051 Acked Write 2 Done Done Sent false
5052 Acked Read 4 Done Done Sent false
5053 Acked Read 1 Done Done Sent false
5054 Acked Write 6 Done Done Sent false
5055 Acked Read 5 Done Done Sent false
5056 Acked Write 1 Done Done Sent false
5057 Acked Write 5 Done Done Sent false
5058 Acked Write 9 Done Done Sent false
5059 Acked Write 3 Done Done Sent false
5060 Acked Read 2 Done Done Sent false
5061 Acked Flush 0 Done Done Sent false
5062 Acked Write 2 Done Done Sent false
5063 Acked Write 4 Done Done Sent false
5064 Acked Flush 0 Done Done Sent false
5065 Acked Read 4 Done Done Sent false
5066 Acked Read 10 Done Done Sent false
5067 Acked Read 8 Done Done Sent false
5068 Acked Write 2 Done Done Sent false
5069 Acked Flush 0 Done Done Sent false
5070 Acked Write 7 Done Done Sent false
5071 Acked Read 9 Done Done Sent false
5072 Acked Read 10 Done Done Sent false
5073 Acked Read 7 Done Done Sent false
5074 Acked Write 7 Done Done Sent false
5075 Acked Read 7 Done Done Sent false
5076 Acked Read 8 Done Done Sent false
5077 Acked Write 1 Done Done Sent false
5078 Acked Write 10 Done Done Sent false
5079 Acked Read 2 Done Done Sent false
5080 Acked Write 4 Done Done Sent false
5081 Acked Write 2 Done Done Sent false
5082 Acked Write 10 Done Done Sent false
5083 Acked Read 10 Done Done Sent false
5084 Acked Read 10 Done Done Sent false
5085 Acked Write 7 Done Done Sent false
5086 Acked Write 7 Done Done Sent false
5087 Acked Read 1 Done Done Sent false
5088 Acked Flush 0 Done Done Sent false
5089 Acked Read 10 Done Done Sent false
5090 Acked Flush 0 Done Done Sent false
5091 Acked Read 10 Done Done Sent false
5092 Acked Flush 0 Done Done Sent false
5093 Acked Write 3 Done Done Sent false
5094 Acked Write 4 Done Done Sent false
5095 Acked Write 6 Done Done Sent false
5096 Acked Write 9 Done Done Sent false
5097 Acked Read 6 Done Done Sent false
5098 Acked Read 6 Done Done Sent false
5099 Acked Flush 0 Done Done Sent false
5100 Acked Write 1 Done Done Sent false
5101 Acked Write 2 Done Done Sent false
5102 Acked Read 3 Done Done Sent false
5103 Acked Read 2 Done Done Sent false
5104 Acked Write 5 Done Done Sent false
5105 Acked Read 9 Done Done Sent false
5106 Acked Read 6 Done Done Sent false
5107 Acked Read 6 Done Done Sent false
5108 Acked Read 1 Done Done Sent false
5109 Acked Write 7 Done Done Sent false
5110 Acked Read 2 Done Done Sent false
5111 Acked Read 9 Done Done Sent false
5112 Acked Read 6 Done Done Sent false
5113 Acked Read 1 Done Done Sent false
5114 Acked Flush 0 Done Done Sent false
5115 Acked Flush 0 Done Done Sent false
5116 Acked Read 7 Done Done Sent false
5117 Acked Write 5 Done Done Sent false
5118 Acked Read 8 Done Done Sent false
5119 Acked Read 2 Done Done Sent false
5120 Acked Read 9 Done Done Sent false
5121 Acked Read 1 Done Done Sent false
5122 Acked Read 2 Done Done Sent false
5123 Acked Read 8 Done Done Sent false
5124 Acked Read 4 Done Done Sent false
5125 Acked Read 4 Done Done Sent false
5126 Acked Read 2 Done Done Sent false
5127 Acked Read 8 Done Done Sent false
5128 Acked Read 6 Done Done Sent false
5129 Acked Write 8 Done Done Sent false
5130 Acked Write 3 Done Done Sent false
5131 Acked Write 3 Done Done Sent false
5132 Acked Read 4 Done Done Sent false
5133 Acked Flush 0 Done Done Sent false
5134 Acked Read 9 Done Done Sent false
5135 Acked Read 4 Done Done Sent false
5136 Acked Write 2 Done Done Sent false
5137 Acked Write 9 Done Done Sent false
5138 Acked Write 10 Done Done Sent false
5139 Acked Read 1 Done Done Sent false
5140 Acked Write 7 Done Done Sent false
5141 Acked Read 8 Done Done Sent false
5142 Acked Read 7 Done Done Sent false
5143 Acked Write 6 Done Done Sent false
5144 Acked Read 5 Done Done Sent false
5145 Acked Write 8 Done Done Sent false
5146 Acked Read 4 Done Done Sent false
5147 Acked Read 4 Done Done Sent false
5148 Acked Read 3 Done Done Sent false
5149 Acked Read 9 Done Done Sent false
5150 Acked Read 6 Done Done Sent false
5151 Acked Flush 0 Done Done Sent false
5152 Acked Write 8 Done Done Sent false
5153 Acked Flush 0 Done Done Sent false
5154 Acked Write 10 Done Done Sent false
5155 Acked Write 9 Done Done Sent false
5156 Acked Write 2 Done Done Sent false
5157 Acked Read 7 Done Done Sent false
5158 Acked Write 9 Done Done Sent false
5159 Acked Read 7 Done Done Sent false
5160 Acked Flush 0 Done Done Sent false
5161 Acked Write 4 Done Done Sent false
5162 Acked Write 2 Done Done Sent false
5163 Acked Read 6 Done Done Sent false
5164 Acked Read 1 Done Done Sent false
5165 Acked Read 1 Done Done Sent false
5166 Acked Flush 0 Done Done Sent false
5167 Acked Read 8 Done Done Sent false
5168 Acked Flush 0 Done Done Sent false
5169 Acked Flush 0 Done Done Sent false
5170 Acked Read 4 Done Done Sent false
5171 Acked Write 7 Done Done Sent false
5172 Acked Flush 0 Done Done Sent false
5173 Acked Flush 0 Done Done Sent false
5174 Acked Read 10 Done Done Sent false
5175 Acked Write 1 Done Done Sent false
5176 Acked Flush 0 Done Done Sent false
5177 Acked Write 1 Done Done Sent false
5178 Acked Write 6 Done Done Sent false
5179 Acked Write 1 Done Done Sent false
5180 Acked Read 4 Done Done Sent false
5181 Acked Read 1 Done Done Sent false
5182 Acked Write 3 Done Done Sent false
5183 Acked Read 9 Done Done Sent false
5184 Acked Read 4 Done Done Sent false
5185 Acked Write 10 Done Done Sent false
5186 Acked Write 3 Done Done Sent false
5187 Acked Read 7 Done Done Sent false
5188 Acked Write 3 Done Done Sent false
5189 Acked Read 10 Done Done Sent false
5190 Acked Write 2 Done Done Sent false
5191 Acked Read 3 Done Done Sent false
5192 Acked Read 7 Done Done Sent false
5193 Acked Read 6 Done Done Sent false
5194 Acked Flush 0 Done Done Sent false
5195 Acked Read 6 Done Done Sent false
5196 Acked Read 8 Done Done Sent false
5197 Acked Read 2 Done Done Sent false
5198 Acked Write 2 Done Done Sent false
5199 Acked Write 7 Done Done Sent false
5200 Acked Flush 0 Done Done Sent false
5201 Acked Read 7 Done Done Sent false
5202 Acked Read 3 Done Done Sent false
5203 Acked Write 9 Done Done Sent false
5204 Acked Write 10 Done Done Sent false
5205 Acked Write 3 Done Done Sent false
5206 Acked Read 7 Done Done Sent false
5207 Acked Write 9 Done Done Sent false
5208 Acked Read 6 Done Done Sent false
5209 Acked Flush 0 Done Done Sent false
5210 Acked Flush 0 Done Done Sent false
5211 Acked Read 2 Done Done Sent false
5212 Acked Read 5 Done Done Sent false
5213 Acked Read 7 Done Done Sent false
5214 Acked Write 4 Done Done Sent false
5215 Acked Read 2 Done Done Sent false
5216 Acked Read 3 Done Done Sent false
5217 Acked Read 8 Done Done Sent false
5218 Acked Write 6 Done Done Sent false
5219 Acked Read 7 Done Done Sent false
5220 Acked Write 1 Done Done Sent false
5221 Acked Read 5 Done Done Sent false
5222 Acked Read 7 Done Done Sent false
5223 Acked Write 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:51.89530795Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:52.721279721Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:53.72180039Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:54.722908957Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.131166086Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36327)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.131284067Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132249636Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132804373Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132828157Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(4922))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132842514Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132860689Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(4922))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.132874206Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(4922))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:55.133124566Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:13
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5211 Acked Read 2 Done Done Done true
5212 Acked Read 5 Done Done Done true
5213 Acked Read 7 Done Done Done true
5214 Acked Write 4 Done Done Done true
5215 Acked Read 2 Done Done Done true
5216 Acked Read 3 Done Done Done true
5217 Acked Read 8 Done Done Done true
5218 Acked Write 6 Done Done Done true
5219 Acked Read 7 Done Done Done true
5220 Acked Write 1 Done Done Done true
5221 Acked Read 5 Done Done Done true
5222 Acked Read 7 Done Done Done true
5223 Acked Write 6 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 13 13 13 39
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(5210))
Downstairs last five completed: 5210 5209 5208 5207 5206
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:13 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5224 5223 5222 5221 5220
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 28 06:04:59.374 [8830] Got stop action so:false kr:false
INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f249af23-ed04-4480-b732-3e37c5d68655, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:59.375227447Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:59.375276895Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:59.375293252Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:04:59.375 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5a2bac78-f902-4b6a-add9-09ed6bfb084e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:59.375378671Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:05:03.377 INFO request completed, latency_us: 314, response_code: 200, uri: /state/cid/2, method: GET, req_id: f0f67ab0-86d2-4253-b700-9d3711c09ec3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 2627 len 32768
002/300 Write block 1290 len 8192 data: 4 5
003/300 Write block 418 len 28672 data: 6 7 5 5 6 5 4
004/300 Read block 3520 len 12288
005/300 Write block 4513 len 8192 data: 4 5
006/300 Write block 1894 len 8192 data: 6 6
007/300 Read block 488 len 12288
008/300 Write block 1355 len 4096 data: 4
009/300 Write block 208 len 32768 data: 3 4 4 4 4 3 5 5
010/300 Read block 2048 len 8192
011/300 Write block 1656 len 24576 data: 5 4 4 4 6 4
012/300 Read block 925 len 20480
013/300 Read block 702 len 16384
014/300 Read block 3215 len 20480
015/300 Read block 4259 len 20480
016/300 Write block 2542 len 20480 data: 3 3 3 3 3
017/300 Read block 3065 len 20480
018/300 Write block 997 len 36864 data: 3 3 3 4 5 5 4 3 3
019/300 Read block 752 len 4096
020/300 Write block 1794 len 40960 data: 3 4 4 4 4 4 4 4 4 2
021/300 Write block 256 len 24576 data: 3 2 2 2 3 3
022/300 Read block 953 len 20480
023/300 Write block 3380 len 24576 data: 2 2 2 2 2 2
024/300 Write block 4006 len 32768 data: 4 4 4 3 3 3 2 2
025/300 Read block 4162 len 28672
026/300 Read block 147 len 16384
027/300 Read block 3865 len 24576
028/300 Read block 3471 len 8192
029/300 Read block 2930 len 36864
030/300 Write block 4695 len 36864 data: 4 5 5 4 6 4 4 3 3
031/300 Read block 271 len 24576
032/300 Flush
033/300 Write block 4649 len 32768 data: 5 4 4 6 6 6 5 3
034/300 Read block 2826 len 40960
035/300 Write block 2467 len 8192 data: 2 3
036/300 Flush
037/300 Read block 1197 len 20480
038/300 Read block 460 len 12288
039/300 Read block 447 len 20480
040/300 Flush
041/300 Write block 2132 len 28672 data: 4 4 4 3 2 4 3
042/300 Flush
043/300 Write block 141 len 4096 data: 2
044/300 Read block 1001 len 20480
045/300 Write block 1904 len 12288 data: 2 2 3
046/300 Read block 262 len 20480
047/300 Write block 3465 len 24576 data: 4 4 4 3 2 2
048/300 Read block 3333 len 36864
049/300 Write block 4810 len 8192 data: 3 3
050/300 Read block 1338 len 16384
051/300 Read block 2448 len 4096
052/300 Read block 565 len 16384
053/300 Read block 3217 len 16384
054/300 Write block 438 len 24576 data: 6 6 6 5 7 7
055/300 Write block 4883 len 20480 data: 6 5 5 5 7
056/300 Write block 3096 len 24576 data: 4 4 5 5 5 6
057/300 Flush
058/300 Write block 4323 len 12288 data: 4 3 3
059/300 Write block 3740 len 32768 data: 3 4 6 7 7 7 5 5
060/300 Write block 4268 len 32768 data: 4 4 4 6 6 5 5 4
061/300 Write block 812 len 16384 data: 5 5 4 3
062/300 Flush
063/300 Read block 3749 len 32768
064/300 Write block 1819 len 4096 data: 3
065/300 Read block 3255 len 20480
066/300 Write block 4388 len 4096 data: 5
067/300 Read block 1013 len 40960
068/300 Read block 1028 len 16384
069/300 Read block 1555 len 28672
070/300 Read block 1308 len 4096
071/300 Read block 1550 len 32768
072/300 Write block 3142 len 20480 data: 3 5 5 4 3
073/300 Read block 4786 len 40960
074/300 Read block 2329 len 20480
075/300 Flush
076/300 Flush
077/300 Read block 4192 len 20480
078/300 Read block 947 len 8192
079/300 Write block 2884 len 4096 data: 4
080/300 Read block 3564 len 4096
081/300 Flush
082/300 Read block 1231 len 8192
083/300 Read block 2412 len 32768
084/300 Read block 2531 len 32768
085/300 Write block 3184 len 20480 data: 3 3 3 3 3
086/300 Write block 3522 len 8192 data: 5 5
087/300 Read block 1361 len 24576
088/300 Read block 4308 len 24576
089/300 Read block 1804 len 36864
090/300 Write block 1414 len 32768 data: 3 2 2 2 2 3 4 5
091/300 Write block 1947 len 8192 data: 4 3
092/300 Flush
093/300 Read block 774 len 36864
094/300 Read block 1682 len 8192
095/300 Read block 3556 len 28672
096/300 Read block 3711 len 36864
097/300 Write block 3088 len 24576 data: 2 2 3 3 3 4
098/300 Read block 633 len 8192
099/300 Write block 1809 len 32768 data: 2 4 4 3 3 3 3 3
100/300 Read block 4536 len 32768
101/300 Read block 2752 len 36864
102/300 Write block 709 len 40960 data: 6 6 5 4 4 3 3 5 5 5
103/300 Read block 2992 len 24576
104/300 Read block 2344 len 40960
105/300 Read block 1249 len 32768
106/300 Read block 4908 len 28672
107/300 Read block 545 len 12288
108/300 Read block 4513 len 28672
109/300 Read block 3543 len 32768
110/300 Write block 1365 len 28672 data: 4 3 3 4 2 2 3
111/300 Read block 1555 len 4096
112/300 Write block 3811 len 12288 data: 2 2 2
113/300 Read block 900 len 12288
114/300 Write block 3348 len 28672 data: 5 4 4 4 3 3 4
115/300 Read block 3273 len 8192
116/300 Write block 2675 len 28672 data: 7 6 4 5 5 5 4
117/300 Read block 4243 len 28672
118/300 Write block 467 len 8192 data: 3 4
119/300 Read block 813 len 16384
120/300 Write block 133 len 40960 data: 5 4 5 5 5 4 4 3 3 2
121/300 Read block 1277 len 12288
122/300 Write block 3843 len 4096 data: 5
123/300 Write block 3056 len 24576 data: 4 2 3 3 3 5
124/300 Write block 4756 len 4096 data: 6
125/300 Read block 1155 len 4096
126/300 Read block 4988 len 32768
127/300 Flush
128/300 Read block 3099 len 20480
129/300 Flush
130/300 Write block 3456 len 4096 data: 5
131/300 Write block 4563 len 12288 data: 3 3 3
132/300 Write block 3731 len 20480 data: 4 3 4 5 5
133/300 Flush
134/300 Read block 4059 len 4096
135/300 Read block 4157 len 36864
136/300 Write block 3610 len 8192 data: 4 4
137/300 Read block 840 len 24576
138/300 Read block 1733 len 24576
139/300 Write block 2684 len 40960 data: 5 6 5 6 5 6 5 5 4 4
140/300 Read block 1536 len 36864
141/300 Flush
142/300 Read block 1921 len 36864
143/300 Write block 3611 len 20480 data: 5 4 3 3 3
144/300 Read block 879 len 28672
145/300 Read block 893 len 16384
146/300 Write block 4591 len 24576 data: 4 5 5 5 7 7
147/300 Read block 1994 len 8192
148/300 Read block 3376 len 20480
149/300 Flush
150/300 Write block 2625 len 32768 data: 3 4 3 3 3 4 4 3
151/300 Write block 2457 len 32768 data: 6 6 5 4 3 2 2 2
152/300 Read block 4388 len 36864
153/300 Read block 3948 len 20480
154/300 Flush
155/300 Read block 293 len 36864
156/300 Read block 4860 len 28672
157/300 Read block 3175 len 12288
158/300 Flush
159/300 Write block 1122 len 36864 data: 2 2 3 4 4 4 3 3 3
160/300 Write block 2083 len 36864 data: 3 3 3 5 5 5 5 5 6
161/300 Read block 228 len 36864
162/300 Write block 424 len 32768 data: 5 3 3 3 4 4 5 5
163/300 Write block 4757 len 12288 data: 6 6 6
164/300 Flush
165/300 Read block 4718 len 24576
166/300 Write block 4913 len 4096 data: 4
167/300 Write block 4709 len 32768 data: 2 2 3 3 3 3 3 3
168/300 Write block 1071 len 20480 data: 3 3 3 3 3
169/300 Read block 4238 len 24576
170/300 Flush
171/300 Read block 3694 len 40960
172/300 Read block 642 len 24576
173/300 Read block 3368 len 20480
174/300 Write block 4834 len 40960 data: 4 5 5 6 7 6 6 5 6 5
175/300 Write block 4295 len 28672 data: 4 4 4 4 2 2 2
176/300 Write block 3630 len 16384 data: 3 3 3 4
177/300 Read block 3960 len 4096
178/300 Write block 921 len 20480 data: 3 3 3 3 3
179/300 Flush
180/300 Write block 1459 len 4096 data: 3
181/300 Write block 1658 len 36864 data: 5 5 7 5 4 4 4 5 5
182/300 Flush
183/300 Flush
184/300 Read block 2064 len 40960
185/300 Write block 2291 len 4096 data: 4
186/300 Read block 2944 len 20480
187/300 Read block 2594 len 28672
188/300 Read block 3513 len 16384
189/300 Read block 1828 len 24576
190/300 Write block 4003 len 40960 data: 2 3 4 5 5 5 4 4 4 3
191/300 Read block 1457 len 36864
192/300 Write block 4428 len 36864 data: 6 6 6 6 4 3 2 2 4
193/300 Read block 155 len 16384
194/300 Flush
195/300 Write block 1222 len 24576 data: 4 4 4 3 2 3
196/300 Write block 3647 len 12288 data: 2 4 4
197/300 Read block 3041 len 4096
198/300 Read block 1415 len 8192
199/300 Write block 10 len 4096 data: 4
200/300 Write block 4089 len 16384 data: 4 4 4 3
201/300 Read block 2161 len 36864
202/300 Write block 2811 len 24576 data: 5 4 5 6 6 6
203/300 Write block 2522 len 20480 data: 6 7 8 7 5
204/300 Read block 2078 len 12288
205/300 Write block 2307 len 32768 data: 4 4 4 3 4 4 4 4
206/300 Read block 2156 len 40960
207/300 Read block 3485 len 20480
208/300 Write block 2068 len 28672 data: 4 3 3 4 4 3 3
209/300 Read block 551 len 28672
210/300 Write block 1297 len 32768 data: 4 4 3 3 3 3 3 3
211/300 Write block 715 len 4096 data: 4
212/300 Write block 3938 len 12288 data: 5 6 7
213/300 Write block 2684 len 20480 data: 6 7 6 7 6
214/300 Read block 3745 len 8192
215/300 Read block 2885 len 20480
216/300 Write block 1862 len 12288 data: 2 2 3
217/300 Write block 140 len 20480 data: 4 4 3 2 5
218/300 Write block 4428 len 32768 data: 7 7 7 7 5 4 3 3
219/300 Read block 1571 len 28672
220/300 Write block 4355 len 4096 data: 6
221/300 Read block 4447 len 28672
222/300 Read block 2238 len 36864
223/300 Read block 2934 len 8192
224/300 Read block 1230 len 28672
225/300 Flush
226/300 Write block 949 len 28672 data: 4 4 4 3 3 3 3
227/300 Read block 2000 len 12288
228/300 Flush
229/300 Read block 3009 len 24576
230/300 Read block 1547 len 8192
231/300 Write block 1377 len 28672 data: 4 4 4 4 4 4 5
232/300 Write block 1577 len 28672 data: 4 6 6 6 6 6 6
233/300 Read block 1472 len 24576
234/300 Read block 1354 len 32768
235/300 Write block 4252 len 16384 data: 2 2 2 3
236/300 Write block 3179 len 28672 data: 2 3 3 3 3 4 4
237/300 Write block 286 len 4096 data: 3
238/300 Write block 653 len 4096 data: 3
239/300 Write block 2913 len 12288 data: 3 3 3
240/300 Write block 1425 len 32768 data: 5 4 4 2 2 2 2 3
241/300 Flush
242/300 Read block 4944 len 24576
243/300 Write block 1912 len 40960 data: 2 2 2 2 2 2 2 2 2 2
244/300 Read block 748 len 36864
245/300 Flush
246/300 Flush
247/300 Read block 719 len 24576
248/300 Read block 2560 len 20480
249/300 Flush
250/300 Write block 747 len 20480 data: 4 5 6 6 4
251/300 Write block 4563 len 20480 data: 4 4 4 3 4
252/300 Read block 3059 len 12288
253/300 Read block 3907 len 24576
254/300 Write block 676 len 16384 data: 6 4 6 7
255/300 Flush
256/300 Read block 1527 len 24576
257/300 Read block 1431 len 36864
258/300 Read block 3376 len 20480
259/300 Read block 3868 len 16384
260/300 Write block 597 len 20480 data: 2 3 4 4 3
261/300 Flush
262/300 Write block 2571 len 16384 data: 5 5 5 3
263/300 Read block 1332 len 32768
264/300 Read block 3808 len 16384
265/300 Read block 10 len 12288
266/300 Flush
267/300 Read block 4052 len 36864
268/300 Flush
269/300 Read block 4204 len 40960
270/300 Read block 2035 len 24576
271/300 Read block 1853 len 32768
272/300 Flush
273/300 Read block 3836 len 8192
274/300 Write block 3293 len 4096 data: 6
275/300 Flush
276/300 Read block 3562 len 24576
277/300 Read block 1284 len 32768
278/300 Read block 3024 len 24576
279/300 Read block 3766 len 4096
280/300 Write block 2748 len 12288 data: 3 3 4
281/300 Read block 143 len 32768
282/300 Write block 4605 len 32768 data: 3 3 3 2 3 3 3 3
283/300 Write block 4823 len 32768 data: 4 4 4 4 4 7 7 9
284/300 Read block 4443 len 24576
285/300 Read block 567 len 20480
286/300 Read block 1226 len 16384
287/300 Read block 4191 len 20480
288/300 Read block 2563 len 40960
289/300 Read block 476 len 16384
290/300 Flush
291/300 Write block 4186 len 32768 data: 5 5 3 3 5 5 4 4
292/300 Write block 1608 len 8192 data: 3 3
293/300 Write block 3869 len 8192 data: 6 5
294/300 Write block 4043 len 16384 data: 5 5 5 4
295/300 Write block 2503 len 4096 data: 2
296/300 Flush
297/300 Flush
298/300 Read block 744 len 32768
299/300 Read block 3585 len 40960
300/300 Write block 941 len 8192 data: 5 5
301/300 Read block 2994 len 24576
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:05:03.619 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/2, method: POST, req_id: dad4bdbe-1952-4efd-b269-7b029b015be1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5225 Acked Read 8 Done Done Sent false
5226 Acked Write 2 Done Done Sent false
5227 Acked Write 7 Done Done Sent false
5228 Acked Read 3 Done Done Sent false
5229 Acked Write 2 Done Done Sent false
5230 Acked Write 2 Done Done Sent false
5231 Acked Read 3 Done Done Sent false
5232 Acked Write 1 Done Done Sent false
5233 Acked Write 8 Done Done Sent false
5234 Acked Read 2 Done Done Sent false
5235 Acked Write 6 Done Done Sent false
5236 Acked Read 5 Done Done Sent false
5237 Acked Read 4 Done Done Sent false
5238 Acked Read 5 Done Done Sent false
5239 Acked Read 5 Done Done Sent false
5240 Acked Write 5 Done Done Sent false
5241 Acked Read 5 Done Done Sent false
5242 Acked Write 9 Done Done Sent false
5243 Acked Read 1 Done Done Sent false
5244 Acked Write 10 Done Done Sent false
5245 Acked Write 6 Done Done Sent false
5246 Acked Read 5 Done Done Sent false
5247 Acked Write 6 Done Done Sent false
5248 Acked Write 8 Done Done Sent false
5249 Acked Read 7 Done Done Sent false
5250 Acked Read 4 Done Done Sent false
5251 Acked Read 6 Done Done Sent false
5252 Acked Read 2 Done Done Sent false
5253 Acked Read 9 Done Done Sent false
5254 Acked Write 9 Done Done Sent false
5255 Acked Read 6 Done Done Sent false
5256 Acked Flush 0 Done Done Sent false
5257 Acked Write 8 Done Done Sent false
5258 Acked Read 10 Done Done Sent false
5259 Acked Write 2 Done Done Sent false
5260 Acked Flush 0 Done Done Sent false
5261 Acked Read 5 Done Done Sent false
5262 Acked Read 3 Done Done Sent false
5263 Acked Read 5 Done Done Sent false
5264 Acked Flush 0 Done Done Sent false
5265 Acked Write 7 Done Done Sent false
5266 Acked Flush 0 Done Done Sent false
5267 Acked Write 1 Done Done Sent false
5268 Acked Read 5 Done Done Sent false
5269 Acked Write 3 Done Done Sent false
5270 Acked Read 5 Done Done Sent false
5271 Acked Write 6 Done Done Sent false
5272 Acked Read 9 Done Done Sent false
5273 Acked Write 2 Done Done Sent false
5274 Acked Read 4 Done Done Sent false
5275 Acked Read 1 Done Done Sent false
5276 Acked Read 4 Done Done Sent false
5277 Acked Read 4 Done Done Sent false
5278 Acked Write 6 Done Done Sent false
5279 Acked Write 5 Done Done Sent false
5280 Acked Write 6 Done Done Sent false
5281 Acked Flush 0 Done Done Sent false
5282 Acked Write 3 Done Done Sent false
5283 Acked Write 8 Done Done Sent false
5284 Acked Write 8 Done Done Sent false
5285 Acked Write 4 Done Done Sent false
5286 Acked Flush 0 Done Done Sent false
5287 Acked Read 8 Done Done Sent false
5288 Acked Write 1 Done Done Sent false
5289 Acked Read 5 Done Done Sent false
5290 Acked Write 1 Done Done Sent false
5291 Acked Read 10 Done Done Sent false
5292 Acked Read 4 Done Done Sent false
5293 Acked Read 7 Done Done Sent false
5294 Acked Read 1 Done Done Sent false
5295 Acked Read 8 Done Done Sent false
5296 Acked Write 5 Done Done Sent false
5297 Acked Read 10 Done Done Sent false
5298 Acked Read 5 Done Done Sent false
5299 Acked Flush 0 Done Done Sent false
5300 Acked Flush 0 Done Done Sent false
5301 Acked Read 5 Done Done Sent false
5302 Acked Read 2 Done Done Sent false
5303 Acked Write 1 Done Done Sent false
5304 Acked Read 1 Done Done Sent false
5305 Acked Flush 0 Done Done Sent false
5306 Acked Read 2 Done Done Sent false
5307 Acked Read 8 Done Done Sent false
5308 Acked Read 8 Done Done Sent false
5309 Acked Write 5 Done Done Sent false
5310 Acked Write 2 Done Done Sent false
5311 Acked Read 6 Done Done Sent false
5312 Acked Read 6 Done Done Sent false
5313 Acked Read 9 Done Done Sent false
5314 Acked Write 8 Done Done Sent false
5315 Acked Write 2 Done Done Sent false
5316 Acked Flush 0 Done Done Sent false
5317 Acked Read 9 Done Done Sent false
5318 Acked Read 2 Done Done Sent false
5319 Acked Read 7 Done Done Sent false
5320 Acked Read 9 Done Done Sent false
5321 Acked Write 6 Done Done Sent false
5322 Acked Read 2 Done Done Sent false
5323 Acked Write 8 Done Done Sent false
5324 Acked Read 8 Done Done Sent false
5325 Acked Read 9 Done Done Sent false
5326 Acked Write 10 Done Done Sent false
5327 Acked Read 6 Done Done Sent false
5328 Acked Read 10 Done Done Sent false
5329 Acked Read 8 Done Done Sent false
5330 Acked Read 7 Done Done Sent false
5331 Acked Read 3 Done Done Sent false
5332 Acked Read 7 Done Done Sent false
5333 Acked Read 8 Done Done Sent false
5334 Acked Write 7 Done Done Sent false
5335 Acked Read 1 Done Done Sent false
5336 Acked Write 3 Done Done Sent false
5337 Acked Read 3 Done Done Sent false
5338 Acked Write 7 Done Done Sent false
5339 Acked Read 2 Done Done Sent false
5340 Acked Write 7 Done Done Sent false
5341 Acked Read 7 Done Done Sent false
5342 Acked Write 2 Done Done Sent false
5343 Acked Read 4 Done Done Sent false
5344 Acked Write 10 Done Done Sent false
5345 Acked Read 3 Done Done Sent false
5346 Acked Write 1 Done Done Sent false
5347 Acked Write 6 Done Done Sent false
5348 Acked Write 1 Done Done Sent false
5349 Acked Read 1 Done Done Sent false
5350 Acked Read 8 Done Done Sent false
5351 Acked Flush 0 Done Done Sent false
5352 Acked Read 5 Done Done Sent false
5353 Acked Flush 0 Done Done Sent false
5354 Acked Write 1 Done Done Sent false
5355 Acked Write 3 Done Done Sent false
5356 Acked Write 5 Done Done Sent false
5357 Acked Flush 0 Done Done Sent false
5358 Acked Read 1 Done Done Sent false
5359 Acked Read 9 Done Done Sent false
5360 Acked Write 2 Done Done Sent false
5361 Acked Read 6 Done Done Sent false
5362 Acked Read 6 Done Done Sent false
5363 Acked Write 10 Done Done Sent false
5364 Acked Read 9 Done Done Sent false
5365 Acked Flush 0 Done Done Sent false
5366 Acked Read 9 Done Done Sent false
5367 Acked Write 5 Done Done Sent false
5368 Acked Read 7 Done Done Sent false
5369 Acked Read 4 Done Done Sent false
5370 Acked Write 6 Done Done Sent false
5371 Acked Read 2 Done Done Sent false
5372 Acked Read 5 Done Done Sent false
5373 Acked Flush 0 Done Done Sent false
5374 Acked Write 8 Done Done Sent false
5375 Acked Write 8 Done Done Sent false
5376 Acked Read 9 Done Done Sent false
5377 Acked Read 5 Done Done Sent false
5378 Acked Flush 0 Done Done Sent false
5379 Acked Read 9 Done Done Sent false
5380 Acked Read 7 Done Done Sent false
5381 Acked Read 3 Done Done Sent false
5382 Acked Flush 0 Done Done Sent false
5383 Acked Write 9 Done Done Sent false
5384 Acked Write 9 Done Done Sent false
5385 Acked Read 9 Done Done Sent false
5386 Acked Write 8 Done Done Sent false
5387 Acked Write 3 Done Done Sent false
5388 Acked Flush 0 Done Done Sent false
5389 Acked Read 6 Done Done Sent false
5390 Acked Write 1 Done Done Sent false
5391 Acked Write 8 Done Done Sent false
5392 Acked Write 5 Done Done Sent false
5393 Acked Read 6 Done Done Sent false
5394 Acked Flush 0 Done Done Sent false
5395 Acked Read 10 Done Done Sent false
5396 Acked Read 6 Done Done Sent false
5397 Acked Read 5 Done Done Sent false
5398 Acked Write 10 Done Done Sent false
5399 Acked Write 7 Done Done Sent false
5400 Acked Write 4 Done Done Sent false
5401 Acked Read 1 Done Done Sent false
5402 Acked Write 5 Done Done Sent false
5403 Acked Flush 0 Done Done Sent false
5404 Acked Write 1 Done Done Sent false
5405 Acked Write 9 Done Done Sent false
5406 Acked Flush 0 Done Done Sent false
5407 Acked Flush 0 Done Done Sent false
5408 Acked Read 10 Done Done Sent false
5409 Acked Write 1 Done Done Sent false
5410 Acked Read 5 Done Done Sent false
5411 Acked Read 7 Done Done Sent false
5412 Acked Read 4 Done Done Sent false
5413 Acked Read 6 Done Done Sent false
5414 Acked Write 10 Done Done Sent false
5415 Acked Read 9 Done Done Sent false
5416 Acked Write 9 Done Done Sent false
5417 Acked Read 4 Done Done Sent false
5418 Acked Flush 0 Done Done Sent false
5419 Acked Write 6 Done Done Sent false
5420 Acked Write 3 Done Done Sent false
5421 Acked Read 1 Done Done Sent false
5422 Acked Read 2 Done Done Sent false
5423 Acked Write 1 Done Done Sent false
5424 Acked Write 4 Done Done Sent false
5425 Acked Read 9 Done Done Sent false
5426 Acked Write 6 Done Done Sent false
5427 Acked Write 5 Done Done Sent false
5428 Acked Read 3 Done Done Sent false
5429 Acked Write 8 Done Done Sent false
5430 Acked Read 10 Done Done Sent false
5431 Acked Read 5 Done Done Sent false
5432 Acked Write 7 Done Done Sent false
5433 Acked Read 7 Done Done Sent false
5434 Acked Write 8 Done Done Sent false
5435 Acked Write 1 Done Done Sent false
5436 Acked Write 3 Done Done Sent false
5437 Acked Write 5 Done Done Sent false
5438 Acked Read 2 Done Done Sent false
5439 Acked Read 5 Done Done Sent false
5440 Acked Write 3 Done Done Sent false
5441 Acked Write 5 Done Done Sent false
5442 Acked Write 8 Done Done Sent false
5443 Acked Read 7 Done Done Sent false
5444 Acked Write 1 Done Done Sent false
5445 Acked Read 7 Done Done Sent false
5446 Acked Read 9 Done Done Sent false
5447 Acked Read 2 Done Done Sent false
5448 Acked Read 7 Done Done Sent false
5449 Acked Flush 0 Done Done Sent false
5450 Acked Write 7 Done Done Sent false
5451 Acked Read 3 Done Done Sent false
5452 Acked Flush 0 Done Done Sent false
5453 Acked Read 6 Done Done Sent false
5454 Acked Read 2 Done Done Sent false
5455 Acked Write 7 Done Done Sent false
5456 Acked Write 7 Done Done Sent false
5457 Acked Read 6 Done Done Sent false
5458 Acked Read 8 Done Done Sent false
5459 Acked Write 4 Done Done Sent false
5460 Acked Write 7 Done Done Sent false
5461 Acked Write 1 Done Done Sent false
5462 Acked Write 1 Done Done Sent false
5463 Acked Write 3 Done Done Sent false
5464 Acked Write 8 Done Done Sent false
5465 Acked Flush 0 Done Done Sent false
5466 Acked Read 6 Done Done Sent false
5467 Acked Write 10 Done Done Sent false
5468 Acked Read 9 Done Done Sent false
5469 Acked Flush 0 Done Done Sent false
5470 Acked Flush 0 Done Done Sent false
5471 Acked Read 6 Done Done Sent false
5472 Acked Read 5 Done Done Sent false
5473 Acked Flush 0 Done Done Sent false
5474 Acked Write 5 Done Done Sent false
5475 Acked Write 5 Done Done Sent false
5476 Acked Read 3 Done Done Sent false
5477 Acked Read 6 Done Done Sent false
5478 Acked Write 4 Done Done Sent false
5479 Acked Flush 0 Done Done Sent false
5480 Acked Read 6 Done Done Sent false
5481 Acked Read 9 Done Done Sent false
5482 Acked Read 5 Done Done Sent false
5483 Acked Read 4 Done Done Sent false
5484 Acked Write 5 Done Done Sent false
5485 Acked Flush 0 Done Done Sent false
5486 Acked Write 4 Done Done Sent false
5487 Acked Read 8 Done Done Sent false
5488 Acked Read 4 Done Done Sent false
5489 Acked Read 3 Done Done Sent false
5490 Acked Flush 0 Done Done Sent false
5491 Acked Read 9 Done Done Sent false
5492 Acked Flush 0 Done Done Sent false
5493 Acked Read 10 Done Done Sent false
5494 Acked Read 6 Done Done Sent false
5495 Acked Read 8 Done Done Sent false
5496 Acked Flush 0 Done Done Sent false
5497 Acked Read 2 Done Done Sent false
5498 Acked Write 1 Done Done Sent false
5499 Acked Flush 0 Done Done Sent false
5500 Acked Read 6 Done Done Sent false
5501 Acked Read 8 Done Done Sent false
5502 Acked Read 6 Done Done Sent false
5503 Acked Read 1 Done Done Sent false
5504 Acked Write 3 Done Done Sent false
5505 Acked Read 8 Done Done Sent false
5506 Acked Write 8 Done Done Sent false
5507 Acked Write 8 Done Done Sent false
5508 Acked Read 6 Done Done Sent false
5509 Acked Read 5 Done Done Sent false
5510 Acked Read 4 Done Done Sent false
5511 Acked Read 5 Done Done Sent false
5512 Acked Read 10 Done Done Sent false
5513 Acked Read 4 Done Done Sent false
5514 Acked Flush 0 Done Done Sent false
5515 Acked Write 8 Done Done Sent false
5516 Acked Write 2 Done Done Sent false
5517 Acked Write 2 Done Done Sent false
5518 Acked Write 4 Done Done Sent false
5519 Acked Write 1 Done Done Sent false
5520 Acked Flush 0 Done Done Sent false
5521 Acked Flush 0 Done Done Sent false
5522 Acked Read 8 Done Done Sent false
5523 Acked Read 10 Done Done Sent false
5524 Acked Write 2 Done Done Sent false
5525 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5521)) Some(JobId(5521)) Some(JobId(5224))
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5525 5524 5523 5522 5521
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:04.121551729Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:04.742580336Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:05.743000198Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:06.896813969Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5225 Acked Read 8 Done Done Sent false
5226 Acked Write 2 Done Done Sent false
5227 Acked Write 7 Done Done Sent false
5228 Acked Read 3 Done Done Sent false
5229 Acked Write 2 Done Done Sent false
5230 Acked Write 2 Done Done Sent false
5231 Acked Read 3 Done Done Sent false
5232 Acked Write 1 Done Done Sent false
5233 Acked Write 8 Done Done Sent false
5234 Acked Read 2 Done Done Sent false
5235 Acked Write 6 Done Done Sent false
5236 Acked Read 5 Done Done Sent false
5237 Acked Read 4 Done Done Sent false
5238 Acked Read 5 Done Done Sent false
5239 Acked Read 5 Done Done Sent false
5240 Acked Write 5 Done Done Sent false
5241 Acked Read 5 Done Done Sent false
5242 Acked Write 9 Done Done Sent false
5243 Acked Read 1 Done Done Sent false
5244 Acked Write 10 Done Done Sent false
5245 Acked Write 6 Done Done Sent false
5246 Acked Read 5 Done Done Sent false
5247 Acked Write 6 Done Done Sent false
5248 Acked Write 8 Done Done Sent false
5249 Acked Read 7 Done Done Sent false
5250 Acked Read 4 Done Done Sent false
5251 Acked Read 6 Done Done Sent false
5252 Acked Read 2 Done Done Sent false
5253 Acked Read 9 Done Done Sent false
5254 Acked Write 9 Done Done Sent false
5255 Acked Read 6 Done Done Sent false
5256 Acked Flush 0 Done Done Sent false
5257 Acked Write 8 Done Done Sent false
5258 Acked Read 10 Done Done Sent false
5259 Acked Write 2 Done Done Sent false
5260 Acked Flush 0 Done Done Sent false
5261 Acked Read 5 Done Done Sent false
5262 Acked Read 3 Done Done Sent false
5263 Acked Read 5 Done Done Sent false
5264 Acked Flush 0 Done Done Sent false
5265 Acked Write 7 Done Done Sent false
5266 Acked Flush 0 Done Done Sent false
5267 Acked Write 1 Done Done Sent false
5268 Acked Read 5 Done Done Sent false
5269 Acked Write 3 Done Done Sent false
5270 Acked Read 5 Done Done Sent false
5271 Acked Write 6 Done Done Sent false
5272 Acked Read 9 Done Done Sent false
5273 Acked Write 2 Done Done Sent false
5274 Acked Read 4 Done Done Sent false
5275 Acked Read 1 Done Done Sent false
5276 Acked Read 4 Done Done Sent false
5277 Acked Read 4 Done Done Sent false
5278 Acked Write 6 Done Done Sent false
5279 Acked Write 5 Done Done Sent false
5280 Acked Write 6 Done Done Sent false
5281 Acked Flush 0 Done Done Sent false
5282 Acked Write 3 Done Done Sent false
5283 Acked Write 8 Done Done Sent false
5284 Acked Write 8 Done Done Sent false
5285 Acked Write 4 Done Done Sent false
5286 Acked Flush 0 Done Done Sent false
5287 Acked Read 8 Done Done Sent false
5288 Acked Write 1 Done Done Sent false
5289 Acked Read 5 Done Done Sent false
5290 Acked Write 1 Done Done Sent false
5291 Acked Read 10 Done Done Sent false
5292 Acked Read 4 Done Done Sent false
5293 Acked Read 7 Done Done Sent false
5294 Acked Read 1 Done Done Sent false
5295 Acked Read 8 Done Done Sent false
5296 Acked Write 5 Done Done Sent false
5297 Acked Read 10 Done Done Sent false
5298 Acked Read 5 Done Done Sent false
5299 Acked Flush 0 Done Done Sent false
5300 Acked Flush 0 Done Done Sent false
5301 Acked Read 5 Done Done Sent false
5302 Acked Read 2 Done Done Sent false
5303 Acked Write 1 Done Done Sent false
5304 Acked Read 1 Done Done Sent false
5305 Acked Flush 0 Done Done Sent false
5306 Acked Read 2 Done Done Sent false
5307 Acked Read 8 Done Done Sent false
5308 Acked Read 8 Done Done Sent false
5309 Acked Write 5 Done Done Sent false
5310 Acked Write 2 Done Done Sent false
5311 Acked Read 6 Done Done Sent false
5312 Acked Read 6 Done Done Sent false
5313 Acked Read 9 Done Done Sent false
5314 Acked Write 8 Done Done Sent false
5315 Acked Write 2 Done Done Sent false
5316 Acked Flush 0 Done Done Sent false
5317 Acked Read 9 Done Done Sent false
5318 Acked Read 2 Done Done Sent false
5319 Acked Read 7 Done Done Sent false
5320 Acked Read 9 Done Done Sent false
5321 Acked Write 6 Done Done Sent false
5322 Acked Read 2 Done Done Sent false
5323 Acked Write 8 Done Done Sent false
5324 Acked Read 8 Done Done Sent false
5325 Acked Read 9 Done Done Sent false
5326 Acked Write 10 Done Done Sent false
5327 Acked Read 6 Done Done Sent false
5328 Acked Read 10 Done Done Sent false
5329 Acked Read 8 Done Done Sent false
5330 Acked Read 7 Done Done Sent false
5331 Acked Read 3 Done Done Sent false
5332 Acked Read 7 Done Done Sent false
5333 Acked Read 8 Done Done Sent false
5334 Acked Write 7 Done Done Sent false
5335 Acked Read 1 Done Done Sent false
5336 Acked Write 3 Done Done Sent false
5337 Acked Read 3 Done Done Sent false
5338 Acked Write 7 Done Done Sent false
5339 Acked Read 2 Done Done Sent false
5340 Acked Write 7 Done Done Sent false
5341 Acked Read 7 Done Done Sent false
5342 Acked Write 2 Done Done Sent false
5343 Acked Read 4 Done Done Sent false
5344 Acked Write 10 Done Done Sent false
5345 Acked Read 3 Done Done Sent false
5346 Acked Write 1 Done Done Sent false
5347 Acked Write 6 Done Done Sent false
5348 Acked Write 1 Done Done Sent false
5349 Acked Read 1 Done Done Sent false
5350 Acked Read 8 Done Done Sent false
5351 Acked Flush 0 Done Done Sent false
5352 Acked Read 5 Done Done Sent false
5353 Acked Flush 0 Done Done Sent false
5354 Acked Write 1 Done Done Sent false
5355 Acked Write 3 Done Done Sent false
5356 Acked Write 5 Done Done Sent false
5357 Acked Flush 0 Done Done Sent false
5358 Acked Read 1 Done Done Sent false
5359 Acked Read 9 Done Done Sent false
5360 Acked Write 2 Done Done Sent false
5361 Acked Read 6 Done Done Sent false
5362 Acked Read 6 Done Done Sent false
5363 Acked Write 10 Done Done Sent false
5364 Acked Read 9 Done Done Sent false
5365 Acked Flush 0 Done Done Sent false
5366 Acked Read 9 Done Done Sent false
5367 Acked Write 5 Done Done Sent false
5368 Acked Read 7 Done Done Sent false
5369 Acked Read 4 Done Done Sent false
5370 Acked Write 6 Done Done Sent false
5371 Acked Read 2 Done Done Sent false
5372 Acked Read 5 Done Done Sent false
5373 Acked Flush 0 Done Done Sent false
5374 Acked Write 8 Done Done Sent false
5375 Acked Write 8 Done Done Sent false
5376 Acked Read 9 Done Done Sent false
5377 Acked Read 5 Done Done Sent false
5378 Acked Flush 0 Done Done Sent false
5379 Acked Read 9 Done Done Sent false
5380 Acked Read 7 Done Done Sent false
5381 Acked Read 3 Done Done Sent false
5382 Acked Flush 0 Done Done Sent false
5383 Acked Write 9 Done Done Sent false
5384 Acked Write 9 Done Done Sent false
5385 Acked Read 9 Done Done Sent false
5386 Acked Write 8 Done Done Sent false
5387 Acked Write 3 Done Done Sent false
5388 Acked Flush 0 Done Done Sent false
5389 Acked Read 6 Done Done Sent false
5390 Acked Write 1 Done Done Sent false
5391 Acked Write 8 Done Done Sent false
5392 Acked Write 5 Done Done Sent false
5393 Acked Read 6 Done Done Sent false
5394 Acked Flush 0 Done Done Sent false
5395 Acked Read 10 Done Done Sent false
5396 Acked Read 6 Done Done Sent false
5397 Acked Read 5 Done Done Sent false
5398 Acked Write 10 Done Done Sent false
5399 Acked Write 7 Done Done Sent false
5400 Acked Write 4 Done Done Sent false
5401 Acked Read 1 Done Done Sent false
5402 Acked Write 5 Done Done Sent false
5403 Acked Flush 0 Done Done Sent false
5404 Acked Write 1 Done Done Sent false
5405 Acked Write 9 Done Done Sent false
5406 Acked Flush 0 Done Done Sent false
5407 Acked Flush 0 Done Done Sent false
5408 Acked Read 10 Done Done Sent false
5409 Acked Write 1 Done Done Sent false
5410 Acked Read 5 Done Done Sent false
5411 Acked Read 7 Done Done Sent false
5412 Acked Read 4 Done Done Sent false
5413 Acked Read 6 Done Done Sent false
5414 Acked Write 10 Done Done Sent false
5415 Acked Read 9 Done Done Sent false
5416 Acked Write 9 Done Done Sent false
5417 Acked Read 4 Done Done Sent false
5418 Acked Flush 0 Done Done Sent false
5419 Acked Write 6 Done Done Sent false
5420 Acked Write 3 Done Done Sent false
5421 Acked Read 1 Done Done Sent false
5422 Acked Read 2 Done Done Sent false
5423 Acked Write 1 Done Done Sent false
5424 Acked Write 4 Done Done Sent false
5425 Acked Read 9 Done Done Sent false
5426 Acked Write 6 Done Done Sent false
5427 Acked Write 5 Done Done Sent false
5428 Acked Read 3 Done Done Sent false
5429 Acked Write 8 Done Done Sent false
5430 Acked Read 10 Done Done Sent false
5431 Acked Read 5 Done Done Sent false
5432 Acked Write 7 Done Done Sent false
5433 Acked Read 7 Done Done Sent false
5434 Acked Write 8 Done Done Sent false
5435 Acked Write 1 Done Done Sent false
5436 Acked Write 3 Done Done Sent false
5437 Acked Write 5 Done Done Sent false
5438 Acked Read 2 Done Done Sent false
5439 Acked Read 5 Done Done Sent false
5440 Acked Write 3 Done Done Sent false
5441 Acked Write 5 Done Done Sent false
5442 Acked Write 8 Done Done Sent false
5443 Acked Read 7 Done Done Sent false
5444 Acked Write 1 Done Done Sent false
5445 Acked Read 7 Done Done Sent false
5446 Acked Read 9 Done Done Sent false
5447 Acked Read 2 Done Done Sent false
5448 Acked Read 7 Done Done Sent false
5449 Acked Flush 0 Done Done Sent false
5450 Acked Write 7 Done Done Sent false
5451 Acked Read 3 Done Done Sent false
5452 Acked Flush 0 Done Done Sent false
5453 Acked Read 6 Done Done Sent false
5454 Acked Read 2 Done Done Sent false
5455 Acked Write 7 Done Done Sent false
5456 Acked Write 7 Done Done Sent false
5457 Acked Read 6 Done Done Sent false
5458 Acked Read 8 Done Done Sent false
5459 Acked Write 4 Done Done Sent false
5460 Acked Write 7 Done Done Sent false
5461 Acked Write 1 Done Done Sent false
5462 Acked Write 1 Done Done Sent false
5463 Acked Write 3 Done Done Sent false
5464 Acked Write 8 Done Done Sent false
5465 Acked Flush 0 Done Done Sent false
5466 Acked Read 6 Done Done Sent false
5467 Acked Write 10 Done Done Sent false
5468 Acked Read 9 Done Done Sent false
5469 Acked Flush 0 Done Done Sent false
5470 Acked Flush 0 Done Done Sent false
5471 Acked Read 6 Done Done Sent false
5472 Acked Read 5 Done Done Sent false
5473 Acked Flush 0 Done Done Sent false
5474 Acked Write 5 Done Done Sent false
5475 Acked Write 5 Done Done Sent false
5476 Acked Read 3 Done Done Sent false
5477 Acked Read 6 Done Done Sent false
5478 Acked Write 4 Done Done Sent false
5479 Acked Flush 0 Done Done Sent false
5480 Acked Read 6 Done Done Sent false
5481 Acked Read 9 Done Done Sent false
5482 Acked Read 5 Done Done Sent false
5483 Acked Read 4 Done Done Sent false
5484 Acked Write 5 Done Done Sent false
5485 Acked Flush 0 Done Done Sent false
5486 Acked Write 4 Done Done Sent false
5487 Acked Read 8 Done Done Sent false
5488 Acked Read 4 Done Done Sent false
5489 Acked Read 3 Done Done Sent false
5490 Acked Flush 0 Done Done Sent false
5491 Acked Read 9 Done Done Sent false
5492 Acked Flush 0 Done Done Sent false
5493 Acked Read 10 Done Done Sent false
5494 Acked Read 6 Done Done Sent false
5495 Acked Read 8 Done Done Sent false
5496 Acked Flush 0 Done Done Sent false
5497 Acked Read 2 Done Done Sent false
5498 Acked Write 1 Done Done Sent false
5499 Acked Flush 0 Done Done Sent false
5500 Acked Read 6 Done Done Sent false
5501 Acked Read 8 Done Done Sent false
5502 Acked Read 6 Done Done Sent false
5503 Acked Read 1 Done Done Sent false
5504 Acked Write 3 Done Done Sent false
5505 Acked Read 8 Done Done Sent false
5506 Acked Write 8 Done Done Sent false
5507 Acked Write 8 Done Done Sent false
5508 Acked Read 6 Done Done Sent false
5509 Acked Read 5 Done Done Sent false
5510 Acked Read 4 Done Done Sent false
5511 Acked Read 5 Done Done Sent false
5512 Acked Read 10 Done Done Sent false
5513 Acked Read 4 Done Done Sent false
5514 Acked Flush 0 Done Done Sent false
5515 Acked Write 8 Done Done Sent false
5516 Acked Write 2 Done Done Sent false
5517 Acked Write 2 Done Done Sent false
5518 Acked Write 4 Done Done Sent false
5519 Acked Write 1 Done Done Sent false
5520 Acked Flush 0 Done Done Sent false
5521 Acked Flush 0 Done Done Sent false
5522 Acked Read 8 Done Done Sent false
5523 Acked Read 10 Done Done Sent false
5524 Acked Write 2 Done Done Sent false
5525 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5521)) Some(JobId(5521)) Some(JobId(5224))
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5525 5524 5523 5522 5521
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:08.748260577Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.376183761Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48058)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.376347492Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.377418425Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.377973312Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.378006635Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(5224))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.37802683Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.378043326Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(5224))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.378059932Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(5224))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:09.378223952Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5526 5525 5524 5523 5522
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:05:11.624 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1bc2441d-e1b1-416c-9d10-f7a0acbf9204[8820] Got stop action so:false kr:false
, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:11.625455779Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:11.625504197Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:11.625520253Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:11.625594995Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:05:11.625 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/1, method: GET, req_id: df4f2adf-9855-45d6-91cf-77908657d784, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:05:15.627 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/1, method: GET, req_id: 459c192d-0563-446d-a0f4-645b3f45b2fd, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 1408 len 40960
002/300 Read block 2016 len 36864
003/300 Read block 3268 len 36864
004/300 Write block 2843 len 40960 data: 5 5 4 5 5 4 4 4 4 5
005/300 Read block 3798 len 24576
006/300 Read block 1515 len 16384
007/300 Read block 4250 len 36864
008/300 Read block 3195 len 4096
009/300 Read block 4669 len 40960
010/300 Write block 899 len 40960 data: 4 4 4 5 3 2 3 3 3 4
011/300 Read block 638 len 8192
012/300 Read block 4820 len 24576
013/300 Write block 3492 len 36864 data: 3 3 2 2 2 4 4 4 3
014/300 Read block 1249 len 28672
015/300 Read block 4124 len 40960
016/300 Read block 883 len 40960
017/300 Read block 3212 len 8192
018/300 Read block 1276 len 40960
019/300 Read block 4830 len 20480
020/300 Flush
021/300 Read block 2764 len 24576
022/300 Flush
023/300 Flush
024/300 Read block 2568 len 32768
025/300 Read block 4900 len 40960
026/300 Read block 4391 len 12288
027/300 Write block 4770 len 4096 data: 5
028/300 Write block 4246 len 40960 data: 2 2 2 3 3 3 3 3 3 4
029/300 Read block 1068 len 36864
030/300 Read block 2665 len 16384
031/300 Read block 2953 len 20480
032/300 Read block 1277 len 16384
033/300 Write block 1065 len 32768 data: 3 4 3 3 4 4 4 4
034/300 Write block 4440 len 20480 data: 3 4 5 5 5
035/300 Read block 3705 len 12288
036/300 Write block 3017 len 32768 data: 3 3 3 3 3 3 3 3
037/300 Write block 558 len 32768 data: 3 3 3 3 3 3 3 3
038/300 Read block 4905 len 16384
039/300 Read block 937 len 28672
040/300 Read block 1510 len 36864
041/300 Write block 3881 len 40960 data: 4 5 6 4 4 4 4 5 4 4
042/300 Write block 370 len 4096 data: 2
043/300 Write block 2444 len 4096 data: 5
044/300 Write block 2854 len 20480 data: 4 4 4 4 4
045/300 Write block 1783 len 4096 data: 2
046/300 Write block 1836 len 12288 data: 5 5 6
047/300 Write block 1153 len 36864 data: 6 5 6 5 5 4 4 4 4
048/300 Read block 2086 len 24576
049/300 Write block 4612 len 32768 data: 4 4 4 3 3 3 3 4
050/300 Read block 33 len 16384
051/300 Read block 3772 len 40960
052/300 Write block 4342 len 24576 data: 7 6 6 7 8 7
053/300 Read block 4206 len 28672
054/300 Write block 4077 len 24576 data: 5 5 6 5 5 2
055/300 Write block 409 len 28672 data: 3 4 3 3 4 6 5
056/300 Read block 469 len 20480
057/300 Read block 4000 len 32768
058/300 Read block 2473 len 40960
059/300 Read block 346 len 20480
060/300 Flush
061/300 Read block 1490 len 12288
062/300 Write block 2900 len 20480 data: 4 2 2 2 2
063/300 Read block 234 len 24576
064/300 Read block 1158 len 4096
065/300 Write block 2472 len 8192 data: 6 6
066/300 Read block 1186 len 36864
067/300 Read block 1845 len 32768
068/300 Write block 256 len 8192 data: 4 3
069/300 Read block 3705 len 40960
070/300 Write block 4194 len 24576 data: 3 3 3 4 4 4
071/300 Read block 4171 len 8192
072/300 Read block 3034 len 4096
073/300 Write block 91 len 28672 data: 4 5 6 7 8 5 3
074/300 Write block 2014 len 20480 data: 4 3 2 2 2
075/300 Write block 3933 len 36864 data: 3 4 4 5 4 6 7 8 6
076/300 Write block 550 len 24576 data: 4 5 6 7 3 3
077/300 Flush
078/300 Write block 3619 len 32768 data: 2 2 3 3 3 3 4 3
079/300 Read block 516 len 36864
080/300 Write block 265 len 28672 data: 4 4 3 4 3 3 4
081/300 Read block 3754 len 4096
082/300 Write block 2922 len 16384 data: 3 5 5 5
083/300 Read block 3997 len 8192
084/300 Write block 918 len 20480 data: 3 3 3 4 4
085/300 Read block 1675 len 12288
086/300 Read block 3656 len 16384
087/300 Flush
088/300 Write block 2086 len 20480 data: 6 6 6 6 6
089/300 Flush
090/300 Read block 2858 len 36864
091/300 Read block 2006 len 28672
092/300 Write block 1152 len 4096 data: 5
093/300 Write block 4455 len 40960 data: 2 3 3 3 3 3 5 5 4 4
094/300 Read block 3522 len 40960
095/300 Read block 4260 len 8192
096/300 Read block 3342 len 28672
097/300 Read block 4194 len 8192
098/300 Write block 4725 len 32768 data: 4 4 4 4 5 5 5 5
099/300 Read block 4338 len 16384
100/300 Write block 4843 len 20480 data: 6 5 4 2 2
101/300 Flush
102/300 Write block 1784 len 8192 data: 2 2
103/300 Read block 4000 len 40960
104/300 Read block 4921 len 28672
105/300 Read block 450 len 20480
106/300 Read block 2204 len 12288
107/300 Read block 1791 len 24576
108/300 Read block 1368 len 40960
109/300 Read block 3897 len 4096
110/300 Write block 4756 len 32768 data: 7 7 7 7 5 4 5 5
111/300 Write block 4055 len 40960 data: 4 4 3 3 3 3 2 4 3 3
112/300 Write block 3997 len 32768 data: 3 3 3 3 3 3 3 4
113/300 Write block 4529 len 36864 data: 6 6 5 3 3 3 4 4 3
114/300 Read block 2346 len 4096
115/300 Write block 4669 len 36864 data: 3 3 3 2 2 2 2 2 3
116/300 Write block 1631 len 12288 data: 4 3 5
117/300 Read block 1878 len 16384
118/300 Write block 459 len 16384 data: 4 4 3 2
119/300 Read block 3862 len 4096
120/300 Write block 1792 len 4096 data: 4
121/300 Read block 1043 len 20480
122/300 Write block 476 len 36864 data: 6 5 6 3 3 4 4 4 4
123/300 Read block 2477 len 12288
124/300 Read block 3103 len 8192
125/300 Read block 2277 len 16384
126/300 Write block 4528 len 12288 data: 6 7 7
127/300 Read block 1207 len 12288
128/300 Read block 648 len 20480
129/300 Write block 1534 len 32768 data: 4 4 4 4 5 4 4 4
130/300 Read block 452 len 4096
131/300 Read block 3938 len 24576
132/300 Read block 2219 len 4096
133/300 Flush
134/300 Read block 4491 len 40960
135/300 Write block 1175 len 16384 data: 6 6 7 8
136/300 Flush
137/300 Read block 1462 len 24576
138/300 Read block 1200 len 4096
139/300 Read block 4421 len 36864
140/300 Read block 40 len 40960
141/300 Write block 1642 len 40960 data: 3 2 2 2 2 3 3 3 3 3
142/300 Read block 2920 len 8192
143/300 Read block 3866 len 32768
144/300 Write block 2926 len 4096 data: 5
145/300 Write block 4921 len 28672 data: 2 2 3 3 5 6 5
146/300 Flush
147/300 Write block 1162 len 32768 data: 3 3 4 4 4 4 4 5
148/300 Read block 1242 len 12288
149/300 Write block 4266 len 28672 data: 3 4 5 5 5 7 7
150/300 Write block 2142 len 16384 data: 3 3 3 3
151/300 Read block 356 len 16384
152/300 Write block 4066 len 4096 data: 4
153/300 Write block 2669 len 32768 data: 6 7 4 7 6 7 8 7
154/300 Write block 203 len 36864 data: 2 2 2 2 3 4 5 5 5
155/300 Read block 1222 len 28672
156/300 Read block 732 len 4096
157/300 Flush
158/300 Read block 668 len 8192
159/300 Flush
160/300 Write block 2135 len 12288 data: 4 3 5
161/300 Read block 701 len 24576
162/300 Read block 2991 len 16384
163/300 Write block 3083 len 24576 data: 3 3 2 2 2 3
164/300 Read block 1526 len 8192
165/300 Read block 4704 len 24576
166/300 Flush
167/300 Read block 166 len 20480
168/300 Write block 2342 len 16384 data: 5 5 4 2
169/300 Write block 413 len 40960 data: 5 7 6 6 6 7 8 6 6 7
170/300 Write block 829 len 12288 data: 4 4 4
171/300 Read block 1581 len 24576
172/300 Write block 2483 len 20480 data: 5 6 6 5 5
173/300 Read block 3018 len 12288
174/300 Flush
175/300 Write block 277 len 16384 data: 6 7 5 5
176/300 Write block 3715 len 4096 data: 2
177/300 Write block 4139 len 24576 data: 4 4 5 5 6 5
178/300 Read block 709 len 24576
179/300 Read block 2449 len 32768
180/300 Flush
181/300 Read block 1902 len 20480
182/300 Write block 2795 len 12288 data: 4 3 3
183/300 Read block 596 len 24576
184/300 Write block 4769 len 36864 data: 4 6 5 4 3 3 4 4 5
185/300 Read block 3320 len 20480
186/300 Read block 322 len 16384
187/300 Flush
188/300 Write block 1131 len 28672 data: 4 3 2 2 2 3 3
189/300 Write block 4520 len 32768 data: 5 6 5 5 7 6 7 7
190/300 Write block 980 len 20480 data: 4 3 3 4 4
191/300 Read block 2598 len 24576
192/300 Read block 1954 len 28672
193/300 Write block 2636 len 20480 data: 2 3 4 4 4
194/300 Flush
195/300 Write block 403 len 4096 data: 3
196/300 Write block 3107 len 24576 data: 5 6 6 6 6 7
197/300 Read block 429 len 8192
198/300 Read block 4677 len 8192
199/300 Write block 2794 len 32768 data: 4 5 4 4 3 4 4 4
200/300 Write block 800 len 28672 data: 5 4 3 3 3 4 4
201/300 Read block 508 len 8192
202/300 Read block 2364 len 24576
203/300 Flush
204/300 Read block 4784 len 4096
205/300 Write block 3715 len 20480 data: 3 3 5 7 6
206/300 Read block 4072 len 28672
207/300 Write block 4629 len 28672 data: 6 6 7 7 7 6 4
208/300 Read block 1799 len 16384
209/300 Read block 1215 len 8192
210/300 Read block 1627 len 4096
211/300 Read block 2932 len 20480
212/300 Read block 1029 len 28672
213/300 Read block 4597 len 24576
214/300 Read block 2747 len 32768
215/300 Read block 2213 len 4096
216/300 Read block 4769 len 36864
217/300 Read block 4580 len 32768
218/300 Write block 3788 len 4096 data: 3
219/300 Read block 3612 len 28672
220/300 Read block 1542 len 28672
221/300 Write block 759 len 4096 data: 2
222/300 Write block 2267 len 32768 data: 5 5 4 4 3 3 5 5
223/300 Read block 1554 len 12288
224/300 Read block 731 len 40960
225/300 Read block 716 len 12288
226/300 Read block 4471 len 12288
227/300 Read block 2838 len 16384
228/300 Write block 1364 len 16384 data: 4 5 4 4
229/300 Write block 4092 len 32768 data: 4 4 4 6 5 5 5 5
230/300 Write block 2344 len 28672 data: 5 3 2 2 3 3 2
231/300 Write block 4626 len 40960 data: 5 7 7 7 7 8 8 8 7 5
232/300 Read block 2028 len 8192
233/300 Write block 4051 len 8192 data: 2 3
234/300 Read block 1613 len 28672
235/300 Write block 1981 len 28672 data: 3 4 5 5 4 4 4
236/300 Flush
237/300 Read block 3636 len 16384
238/300 Write block 1686 len 32768 data: 5 4 3 3 3 4 5 4
239/300 Read block 539 len 32768
240/300 Read block 4284 len 24576
241/300 Read block 464 len 36864
242/300 Write block 4804 len 20480 data: 2 3 3 4 4
243/300 Write block 3610 len 40960 data: 5 6 5 4 4 4 3 3 2 3
244/300 Write block 3431 len 36864 data: 3 3 2 3 5 6 6 6 4
245/300 Write block 4212 len 8192 data: 6 6
246/300 Read block 843 len 36864
247/300 Read block 3326 len 20480
248/300 Read block 4919 len 28672
249/300 Write block 4038 len 40960 data: 6 6 5 5 3 6 6 6 5 3
250/300 Read block 3141 len 24576
251/300 Write block 2184 len 36864 data: 4 4 5 4 5 5 6 4 4
252/300 Write block 3636 len 8192 data: 3 3
253/300 Write block 3804 len 32768 data: 4 4 4 5 5 5 4 3
254/300 Write block 2015 len 24576 data: 4 3 3 3 3 4
255/300 Write block 3492 len 32768 data: 4 4 3 3 3 5 5 5
256/300 Write block 2527 len 40960 data: 5 4 5 4 2 2 2 2 2 2
257/300 Read block 4955 len 24576
258/300 Read block 2711 len 24576
259/300 Write block 2181 len 24576 data: 5 5 5 5 5 6
260/300 Read block 2711 len 20480
261/300 Read block 1705 len 28672
262/300 Read block 416 len 16384
263/300 Write block 165 len 12288 data: 3 3 2
264/300 Read block 3124 len 40960
265/300 Write block 3561 len 24576 data: 3 2 3 3 3 3
266/300 Read block 2564 len 20480
267/300 Read block 1794 len 28672
268/300 Read block 3942 len 32768
269/300 Write block 2148 len 36864 data: 4 4 4 4 5 4 3 3 3
270/300 Read block 1567 len 4096
271/300 Flush
272/300 Write block 2259 len 24576 data: 5 5 3 4 4 5
273/300 Write block 1399 len 32768 data: 4 6 4 5 4 4 4 3
274/300 Write block 3396 len 4096 data: 3
275/300 Flush
276/300 Read block 607 len 32768
277/300 Read block 1255 len 8192
278/300 Write block 2120 len 16384 data: 4 5 5 4
279/300 Read block 4964 len 36864
280/300 Read block 3765 len 36864
281/300 Write block 4018 len 8192 data: 3 3
282/300 Write block 4874 len 28672 data: 3 4 5 5 6 7 5
283/300 Write block 678 len 12288 data: 7 8 6
284/300 Flush
285/300 Write block 4659 len 12288 data: 3 3 3
286/300 Write block 3129 len 20480 data: 3 3 3 3 3
287/300 Read block 2357 len 36864
288/300 Write block 1699 len 20480 data: 5 6 6 4 5
289/300 Read block 1402 len 32768
290/300 Read block 3642 len 40960
291/300 Read block 4859 len 8192
292/300 Flush
293/300 Write block 4353 len 4096 data: 6
294/300 Read block 1781 len 40960
295/300 Write block 2560 len 28672 data: 3 4 4 5 5 4 3
296/300 Read block 3553 len 12288
297/300 Read block 4033 len 24576
298/300 Write block 1693 len 8192 data: 5 5
299/300 Write block 2884 len 4096 data: 5
300/300 Flush
301/300 Read block 2649 len 28672
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:05:15.875 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3bcf45c2-782f-4504-a362-19c27f935d2d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5527 Acked Read 10 Done Sent Done false
5528 Acked Read 9 Done Sent Done false
5529 Acked Read 9 Done Sent Done false
5530 Acked Write 10 Done Sent Done false
5531 Acked Read 6 Done Sent Done false
5532 Acked Read 4 Done Sent Done false
5533 Acked Read 9 Done Sent Done false
5534 Acked Read 1 Done Sent Done false
5535 Acked Read 10 Done Sent Done false
5536 Acked Write 10 Done Sent Done false
5537 Acked Read 2 Done Sent Done false
5538 Acked Read 6 Done Sent Done false
5539 Acked Write 9 Done Sent Done false
5540 Acked Read 7 Done Sent Done false
5541 Acked Read 10 Done Sent Done false
5542 Acked Read 10 Done Sent Done false
5543 Acked Read 2 Done Sent Done false
5544 Acked Read 10 Done Sent Done false
5545 Acked Read 5 Done Sent Done false
5546 Acked Flush 0 Done Sent Done false
5547 Acked Read 6 Done Sent Done false
5548 Acked Flush 0 Done Sent Done false
5549 Acked Flush 0 Done Sent Done false
5550 Acked Read 8 Done Sent Done false
5551 Acked Read 10 Done Sent Done false
5552 Acked Read 3 Done Sent Done false
5553 Acked Write 1 Done Sent Done false
5554 Acked Write 10 Done Sent Done false
5555 Acked Read 9 Done Sent Done false
5556 Acked Read 4 Done Sent Done false
5557 Acked Read 5 Done Sent Done false
5558 Acked Read 4 Done Sent Done false
5559 Acked Write 8 Done Sent Done false
5560 Acked Write 5 Done Sent Done false
5561 Acked Read 3 Done Sent Done false
5562 Acked Write 8 Done Sent Done false
5563 Acked Write 8 Done Sent Done false
5564 Acked Read 4 Done Sent Done false
5565 Acked Read 7 Done Sent Done false
5566 Acked Read 9 Done Sent Done false
5567 Acked Write 10 Done Sent Done false
5568 Acked Write 1 Done Sent Done false
5569 Acked Write 1 Done Sent Done false
5570 Acked Write 5 Done Sent Done false
5571 Acked Write 1 Done Sent Done false
5572 Acked Write 3 Done Sent Done false
5573 Acked Write 9 Done Sent Done false
5574 Acked Read 6 Done Sent Done false
5575 Acked Write 8 Done Sent Done false
5576 Acked Read 4 Done Sent Done false
5577 Acked Read 10 Done Sent Done false
5578 Acked Write 6 Done Sent Done false
5579 Acked Read 7 Done Sent Done false
5580 Acked Write 6 Done Sent Done false
5581 Acked Write 7 Done Sent Done false
5582 Acked Read 5 Done Sent Done false
5583 Acked Read 8 Done Sent Done false
5584 Acked Read 10 Done Sent Done false
5585 Acked Read 5 Done Sent Done false
5586 Acked Flush 0 Done Sent Done false
5587 Acked Read 3 Done Sent Done false
5588 Acked Write 5 Done Sent Done false
5589 Acked Read 6 Done Sent Done false
5590 Acked Read 1 Done Sent Done false
5591 Acked Write 2 Done Sent Done false
5592 Acked Read 9 Done Sent Done false
5593 Acked Read 8 Done Sent Done false
5594 Acked Write 2 Done Sent Done false
5595 Acked Read 10 Done Sent Done false
5596 Acked Write 6 Done Sent Done false
5597 Acked Read 2 Done Sent Done false
5598 Acked Read 1 Done Sent Done false
5599 Acked Write 7 Done Sent Done false
5600 Acked Write 5 Done Sent Done false
5601 Acked Write 9 Done Sent Done false
5602 Acked Write 6 Done Sent Done false
5603 Acked Flush 0 Done Sent Done false
5604 Acked Write 8 Done Sent Done false
5605 Acked Read 9 Done Sent Done false
5606 Acked Write 7 Done Sent Done false
5607 Acked Read 1 Done Sent Done false
5608 Acked Write 4 Done Sent Done false
5609 Acked Read 2 Done Sent Done false
5610 Acked Write 5 Done Sent Done false
5611 Acked Read 3 Done Sent Done false
5612 Acked Read 4 Done Sent Done false
5613 Acked Flush 0 Done Sent Done false
5614 Acked Write 5 Done Sent Done false
5615 Acked Flush 0 Done Sent Done false
5616 Acked Read 9 Done Sent Done false
5617 Acked Read 7 Done Sent Done false
5618 Acked Write 1 Done Sent Done false
5619 Acked Write 10 Done Sent Done false
5620 Acked Read 10 Done Sent Done false
5621 Acked Read 2 Done Sent Done false
5622 Acked Read 7 Done Sent Done false
5623 Acked Read 2 Done Sent Done false
5624 Acked Write 8 Done Sent Done false
5625 Acked Read 4 Done Sent Done false
5626 Acked Write 5 Done Sent Done false
5627 Acked Flush 0 Done Sent Done false
5628 Acked Write 2 Done Sent Done false
5629 Acked Read 10 Done Sent Done false
5630 Acked Read 7 Done Sent Done false
5631 Acked Read 5 Done Sent Done false
5632 Acked Read 3 Done Sent Done false
5633 Acked Read 6 Done Sent Done false
5634 Acked Read 10 Done Sent Done false
5635 Acked Read 1 Done Sent Done false
5636 Acked Write 8 Done Sent Done false
5637 Acked Write 10 Done Sent Done false
5638 Acked Write 8 Done Sent Done false
5639 Acked Write 9 Done Sent Done false
5640 Acked Read 1 Done Sent Done false
5641 Acked Write 9 Done Sent Done false
5642 Acked Write 3 Done Sent Done false
5643 Acked Read 4 Done Sent Done false
5644 Acked Write 4 Done Sent Done false
5645 Acked Read 1 Done Sent Done false
5646 Acked Write 1 Done Sent Done false
5647 Acked Read 5 Done Sent Done false
5648 Acked Write 9 Done Sent Done false
5649 Acked Read 3 Done Sent Done false
5650 Acked Read 2 Done Sent Done false
5651 Acked Read 4 Done Sent Done false
5652 Acked Write 3 Done Sent Done false
5653 Acked Read 3 Done Sent Done false
5654 Acked Read 5 Done Sent Done false
5655 Acked Write 8 Done Sent Done false
5656 Acked Read 1 Done Sent Done false
5657 Acked Read 6 Done Sent Done false
5658 Acked Read 1 Done Sent Done false
5659 Acked Flush 0 Done Sent Done false
5660 Acked Read 10 Done Sent Done false
5661 Acked Write 4 Done Sent Done false
5662 Acked Flush 0 Done Sent Done false
5663 Acked Read 6 Done Sent Done false
5664 Acked Read 1 Done Sent Done false
5665 Acked Read 9 Done Sent Done false
5666 Acked Read 10 Done Sent Done false
5667 Acked Write 10 Done Sent Done false
5668 Acked Read 2 Done Sent Done false
5669 Acked Read 8 Done Sent Done false
5670 Acked Write 1 Done Sent Done false
5671 Acked Write 7 Done Sent Done false
5672 Acked Flush 0 Done Sent Done false
5673 Acked Write 8 Done Sent Done false
5674 Acked Read 3 Done Sent Done false
5675 Acked Write 7 Done Sent Done false
5676 Acked Write 4 Done Sent Done false
5677 Acked Read 4 Done Sent Done false
5678 Acked Write 1 Done Sent Done false
5679 Acked Write 8 Done Sent Done false
5680 Acked Write 9 Done Sent Done false
5681 Acked Read 7 Done Sent Done false
5682 Acked Read 1 Done Sent Done false
5683 Acked Flush 0 Done Sent Done false
5684 Acked Read 2 Done Sent Done false
5685 Acked Flush 0 Done Sent Done false
5686 Acked Write 3 Done Sent Done false
5687 Acked Read 6 Done Sent Done false
5688 Acked Read 4 Done Sent Done false
5689 Acked Write 6 Done Sent Done false
5690 Acked Read 2 Done Sent Done false
5691 Acked Read 6 Done Sent Done false
5692 Acked Flush 0 Done Sent Done false
5693 Acked Read 5 Done Sent Done false
5694 Acked Write 4 Done Sent Done false
5695 Acked Write 10 Done Sent Done false
5696 Acked Write 3 Done Sent Done false
5697 Acked Read 6 Done Sent Done false
5698 Acked Write 5 Done Sent Done false
5699 Acked Read 3 Done Sent Done false
5700 Acked Flush 0 Done Sent Done false
5701 Acked Write 4 Done Sent Done false
5702 Acked Write 1 Done Sent Done false
5703 Acked Write 6 Done Sent Done false
5704 Acked Read 6 Done Sent Done false
5705 Acked Read 8 Done Sent Done false
5706 Acked Flush 0 Done Sent Done false
5707 Acked Read 5 Done Sent Done false
5708 Acked Write 3 Done Sent Done false
5709 Acked Read 6 Done Sent Done false
5710 Acked Write 9 Done Sent Done false
5711 Acked Read 5 Done Sent Done false
5712 Acked Read 4 Done Sent Done false
5713 Acked Flush 0 Done Sent Done false
5714 Acked Write 7 Done Sent Done false
5715 Acked Write 8 Done Sent Done false
5716 Acked Write 5 Done Sent Done false
5717 Acked Read 6 Done Sent Done false
5718 Acked Read 7 Done Sent Done false
5719 Acked Write 5 Done Sent Done false
5720 Acked Flush 0 Done Sent Done false
5721 Acked Write 1 Done Sent Done false
5722 Acked Write 6 Done Sent Done false
5723 Acked Read 2 Done Sent Done false
5724 Acked Read 2 Done Sent Done false
5725 Acked Write 8 Done Sent Done false
5726 Acked Write 7 Done Sent Done false
5727 Acked Read 2 Done Sent Done false
5728 Acked Read 6 Done Sent Done false
5729 Acked Flush 0 Done Sent Done false
5730 Acked Read 1 Done Sent Done false
5731 Acked Write 5 Done Sent Done false
5732 Acked Read 7 Done Sent Done false
5733 Acked Write 7 Done Sent Done false
5734 Acked Read 4 Done Sent Done false
5735 Acked Read 2 Done Sent Done false
5736 Acked Read 1 Done Sent Done false
5737 Acked Read 5 Done Sent Done false
5738 Acked Read 7 Done Sent Done false
5739 Acked Read 6 Done Sent Done false
5740 Acked Read 8 Done Sent Done false
5741 Acked Read 1 Done Sent Done false
5742 Acked Read 9 Done Sent Done false
5743 Acked Read 8 Done Sent Done false
5744 Acked Write 1 Done Sent Done false
5745 Acked Read 7 Done Sent Done false
5746 Acked Read 7 Done Sent Done false
5747 Acked Write 1 Done Sent Done false
5748 Acked Write 8 Done Sent Done false
5749 Acked Read 3 Done Sent Done false
5750 Acked Read 10 Done Sent Done false
5751 Acked Read 3 Done Sent Done false
5752 Acked Read 3 Done Sent Done false
5753 Acked Read 4 Done Sent Done false
5754 Acked Write 4 Done Sent Done false
5755 Acked Write 8 Done Sent Done false
5756 Acked Write 7 Done Sent Done false
5757 Acked Write 10 Done Sent Done false
5758 Acked Read 2 Done Sent Done false
5759 Acked Write 2 Done Sent Done false
5760 Acked Read 7 Done Sent Done false
5761 Acked Write 7 Done Sent Done false
5762 Acked Flush 0 Done Sent Done false
5763 Acked Read 4 Done Sent Done false
5764 Acked Write 8 Done Sent Done false
5765 Acked Read 8 Done Sent Done false
5766 Acked Read 6 Done Sent Done false
5767 Acked Read 9 Done Sent Done false
5768 Acked Write 5 Done Sent Done false
5769 Acked Write 10 Done Sent Done false
5770 Acked Write 9 Done Sent Done false
5771 Acked Write 2 Done Sent Done false
5772 Acked Read 9 Done Sent Done false
5773 Acked Read 5 Done Sent Done false
5774 Acked Read 7 Done Sent Done false
5775 Acked Write 10 Done Sent Done false
5776 Acked Read 6 Done Sent Done false
5777 Acked Write 9 Done Sent Done false
5778 Acked Write 2 Done Sent Done false
5779 Acked Write 8 Done Sent Done false
5780 Acked Write 6 Done Sent Done false
5781 Acked Write 8 Done Sent Done false
5782 Acked Write 10 Done Sent Done false
5783 Acked Read 6 Done Sent Done false
5784 Acked Read 6 Done Sent Done false
5785 Acked Write 6 Done Sent Done false
5786 Acked Read 5 Done Sent Done false
5787 Acked Read 7 Done Sent Done false
5788 Acked Read 4 Done Sent Done false
5789 Acked Write 3 Done Sent Done false
5790 Acked Read 10 Done Sent Done false
5791 Acked Write 6 Done Sent Done false
5792 Acked Read 5 Done Sent Done false
5793 Acked Read 7 Done Sent Done false
5794 Acked Read 8 Done Sent Done false
5795 Acked Write 9 Done Sent Done false
5796 Acked Read 1 Done Sent Done false
5797 Acked Flush 0 Done Sent Done false
5798 Acked Write 6 Done Sent Done false
5799 Acked Write 8 Done Sent Done false
5800 Acked Write 1 Done Sent Done false
5801 Acked Flush 0 Done Sent Done false
5802 Acked Read 8 Done Sent Done false
5803 Acked Read 2 Done Sent Done false
5804 Acked Write 4 Done Sent Done false
5805 Acked Read 9 Done Sent Done false
5806 Acked Read 9 Done Sent Done false
5807 Acked Write 2 Done Sent Done false
5808 Acked Write 7 Done Sent Done false
5809 Acked Write 3 Done Sent Done false
5810 Acked Flush 0 Done Sent Done false
5811 Acked Write 3 Done Sent Done false
5812 Acked Write 5 Done Sent Done false
5813 Acked Read 9 Done Sent Done false
5814 Acked Write 5 Done Sent Done false
5815 Acked Read 8 Done Sent Done false
5816 Acked Read 10 Done Sent Done false
5817 Acked Read 2 Done Sent Done false
5818 Acked Flush 0 Done Sent Done false
5819 Acked Write 1 Done Sent Done false
5820 Acked Read 10 Done Sent Done false
5821 Acked Write 7 Done Sent Done false
5822 Acked Read 3 Done Sent Done false
5823 Acked Read 6 Done Sent Done false
5824 Acked Write 2 Done Sent Done false
5825 Acked Write 1 Done Sent Done false
5826 Acked Flush 0 Done Sent Done false
5827 Acked Read 7 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5826)) Some(JobId(5526)) Some(JobId(5826))
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5827 5826 5825 5824 5823
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:16.761165941Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:17.763323977Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:18.765178346Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5527 Acked Read 10 Done Sent Done false
5528 Acked Read 9 Done Sent Done false
5529 Acked Read 9 Done Sent Done false
5530 Acked Write 10 Done Sent Done false
5531 Acked Read 6 Done Sent Done false
5532 Acked Read 4 Done Sent Done false
5533 Acked Read 9 Done Sent Done false
5534 Acked Read 1 Done Sent Done false
5535 Acked Read 10 Done Sent Done false
5536 Acked Write 10 Done Sent Done false
5537 Acked Read 2 Done Sent Done false
5538 Acked Read 6 Done Sent Done false
5539 Acked Write 9 Done Sent Done false
5540 Acked Read 7 Done Sent Done false
5541 Acked Read 10 Done Sent Done false
5542 Acked Read 10 Done Sent Done false
5543 Acked Read 2 Done Sent Done false
5544 Acked Read 10 Done Sent Done false
5545 Acked Read 5 Done Sent Done false
5546 Acked Flush 0 Done Sent Done false
5547 Acked Read 6 Done Sent Done false
5548 Acked Flush 0 Done Sent Done false
5549 Acked Flush 0 Done Sent Done false
5550 Acked Read 8 Done Sent Done false
5551 Acked Read 10 Done Sent Done false
5552 Acked Read 3 Done Sent Done false
5553 Acked Write 1 Done Sent Done false
5554 Acked Write 10 Done Sent Done false
5555 Acked Read 9 Done Sent Done false
5556 Acked Read 4 Done Sent Done false
5557 Acked Read 5 Done Sent Done false
5558 Acked Read 4 Done Sent Done false
5559 Acked Write 8 Done Sent Done false
5560 Acked Write 5 Done Sent Done false
5561 Acked Read 3 Done Sent Done false
5562 Acked Write 8 Done Sent Done false
5563 Acked Write 8 Done Sent Done false
5564 Acked Read 4 Done Sent Done false
5565 Acked Read 7 Done Sent Done false
5566 Acked Read 9 Done Sent Done false
5567 Acked Write 10 Done Sent Done false
5568 Acked Write 1 Done Sent Done false
5569 Acked Write 1 Done Sent Done false
5570 Acked Write 5 Done Sent Done false
5571 Acked Write 1 Done Sent Done false
5572 Acked Write 3 Done Sent Done false
5573 Acked Write 9 Done Sent Done false
5574 Acked Read 6 Done Sent Done false
5575 Acked Write 8 Done Sent Done false
5576 Acked Read 4 Done Sent Done false
5577 Acked Read 10 Done Sent Done false
5578 Acked Write 6 Done Sent Done false
5579 Acked Read 7 Done Sent Done false
5580 Acked Write 6 Done Sent Done false
5581 Acked Write 7 Done Sent Done false
5582 Acked Read 5 Done Sent Done false
5583 Acked Read 8 Done Sent Done false
5584 Acked Read 10 Done Sent Done false
5585 Acked Read 5 Done Sent Done false
5586 Acked Flush 0 Done Sent Done false
5587 Acked Read 3 Done Sent Done false
5588 Acked Write 5 Done Sent Done false
5589 Acked Read 6 Done Sent Done false
5590 Acked Read 1 Done Sent Done false
5591 Acked Write 2 Done Sent Done false
5592 Acked Read 9 Done Sent Done false
5593 Acked Read 8 Done Sent Done false
5594 Acked Write 2 Done Sent Done false
5595 Acked Read 10 Done Sent Done false
5596 Acked Write 6 Done Sent Done false
5597 Acked Read 2 Done Sent Done false
5598 Acked Read 1 Done Sent Done false
5599 Acked Write 7 Done Sent Done false
5600 Acked Write 5 Done Sent Done false
5601 Acked Write 9 Done Sent Done false
5602 Acked Write 6 Done Sent Done false
5603 Acked Flush 0 Done Sent Done false
5604 Acked Write 8 Done Sent Done false
5605 Acked Read 9 Done Sent Done false
5606 Acked Write 7 Done Sent Done false
5607 Acked Read 1 Done Sent Done false
5608 Acked Write 4 Done Sent Done false
5609 Acked Read 2 Done Sent Done false
5610 Acked Write 5 Done Sent Done false
5611 Acked Read 3 Done Sent Done false
5612 Acked Read 4 Done Sent Done false
5613 Acked Flush 0 Done Sent Done false
5614 Acked Write 5 Done Sent Done false
5615 Acked Flush 0 Done Sent Done false
5616 Acked Read 9 Done Sent Done false
5617 Acked Read 7 Done Sent Done false
5618 Acked Write 1 Done Sent Done false
5619 Acked Write 10 Done Sent Done false
5620 Acked Read 10 Done Sent Done false
5621 Acked Read 2 Done Sent Done false
5622 Acked Read 7 Done Sent Done false
5623 Acked Read 2 Done Sent Done false
5624 Acked Write 8 Done Sent Done false
5625 Acked Read 4 Done Sent Done false
5626 Acked Write 5 Done Sent Done false
5627 Acked Flush 0 Done Sent Done false
5628 Acked Write 2 Done Sent Done false
5629 Acked Read 10 Done Sent Done false
5630 Acked Read 7 Done Sent Done false
5631 Acked Read 5 Done Sent Done false
5632 Acked Read 3 Done Sent Done false
5633 Acked Read 6 Done Sent Done false
5634 Acked Read 10 Done Sent Done false
5635 Acked Read 1 Done Sent Done false
5636 Acked Write 8 Done Sent Done false
5637 Acked Write 10 Done Sent Done false
5638 Acked Write 8 Done Sent Done false
5639 Acked Write 9 Done Sent Done false
5640 Acked Read 1 Done Sent Done false
5641 Acked Write 9 Done Sent Done false
5642 Acked Write 3 Done Sent Done false
5643 Acked Read 4 Done Sent Done false
5644 Acked Write 4 Done Sent Done false
5645 Acked Read 1 Done Sent Done false
5646 Acked Write 1 Done Sent Done false
5647 Acked Read 5 Done Sent Done false
5648 Acked Write 9 Done Sent Done false
5649 Acked Read 3 Done Sent Done false
5650 Acked Read 2 Done Sent Done false
5651 Acked Read 4 Done Sent Done false
5652 Acked Write 3 Done Sent Done false
5653 Acked Read 3 Done Sent Done false
5654 Acked Read 5 Done Sent Done false
5655 Acked Write 8 Done Sent Done false
5656 Acked Read 1 Done Sent Done false
5657 Acked Read 6 Done Sent Done false
5658 Acked Read 1 Done Sent Done false
5659 Acked Flush 0 Done Sent Done false
5660 Acked Read 10 Done Sent Done false
5661 Acked Write 4 Done Sent Done false
5662 Acked Flush 0 Done Sent Done false
5663 Acked Read 6 Done Sent Done false
5664 Acked Read 1 Done Sent Done false
5665 Acked Read 9 Done Sent Done false
5666 Acked Read 10 Done Sent Done false
5667 Acked Write 10 Done Sent Done false
5668 Acked Read 2 Done Sent Done false
5669 Acked Read 8 Done Sent Done false
5670 Acked Write 1 Done Sent Done false
5671 Acked Write 7 Done Sent Done false
5672 Acked Flush 0 Done Sent Done false
5673 Acked Write 8 Done Sent Done false
5674 Acked Read 3 Done Sent Done false
5675 Acked Write 7 Done Sent Done false
5676 Acked Write 4 Done Sent Done false
5677 Acked Read 4 Done Sent Done false
5678 Acked Write 1 Done Sent Done false
5679 Acked Write 8 Done Sent Done false
5680 Acked Write 9 Done Sent Done false
5681 Acked Read 7 Done Sent Done false
5682 Acked Read 1 Done Sent Done false
5683 Acked Flush 0 Done Sent Done false
5684 Acked Read 2 Done Sent Done false
5685 Acked Flush 0 Done Sent Done false
5686 Acked Write 3 Done Sent Done false
5687 Acked Read 6 Done Sent Done false
5688 Acked Read 4 Done Sent Done false
5689 Acked Write 6 Done Sent Done false
5690 Acked Read 2 Done Sent Done false
5691 Acked Read 6 Done Sent Done false
5692 Acked Flush 0 Done Sent Done false
5693 Acked Read 5 Done Sent Done false
5694 Acked Write 4 Done Sent Done false
5695 Acked Write 10 Done Sent Done false
5696 Acked Write 3 Done Sent Done false
5697 Acked Read 6 Done Sent Done false
5698 Acked Write 5 Done Sent Done false
5699 Acked Read 3 Done Sent Done false
5700 Acked Flush 0 Done Sent Done false
5701 Acked Write 4 Done Sent Done false
5702 Acked Write 1 Done Sent Done false
5703 Acked Write 6 Done Sent Done false
5704 Acked Read 6 Done Sent Done false
5705 Acked Read 8 Done Sent Done false
5706 Acked Flush 0 Done Sent Done false
5707 Acked Read 5 Done Sent Done false
5708 Acked Write 3 Done Sent Done false
5709 Acked Read 6 Done Sent Done false
5710 Acked Write 9 Done Sent Done false
5711 Acked Read 5 Done Sent Done false
5712 Acked Read 4 Done Sent Done false
5713 Acked Flush 0 Done Sent Done false
5714 Acked Write 7 Done Sent Done false
5715 Acked Write 8 Done Sent Done false
5716 Acked Write 5 Done Sent Done false
5717 Acked Read 6 Done Sent Done false
5718 Acked Read 7 Done Sent Done false
5719 Acked Write 5 Done Sent Done false
5720 Acked Flush 0 Done Sent Done false
5721 Acked Write 1 Done Sent Done false
5722 Acked Write 6 Done Sent Done false
5723 Acked Read 2 Done Sent Done false
5724 Acked Read 2 Done Sent Done false
5725 Acked Write 8 Done Sent Done false
5726 Acked Write 7 Done Sent Done false
5727 Acked Read 2 Done Sent Done false
5728 Acked Read 6 Done Sent Done false
5729 Acked Flush 0 Done Sent Done false
5730 Acked Read 1 Done Sent Done false
5731 Acked Write 5 Done Sent Done false
5732 Acked Read 7 Done Sent Done false
5733 Acked Write 7 Done Sent Done false
5734 Acked Read 4 Done Sent Done false
5735 Acked Read 2 Done Sent Done false
5736 Acked Read 1 Done Sent Done false
5737 Acked Read 5 Done Sent Done false
5738 Acked Read 7 Done Sent Done false
5739 Acked Read 6 Done Sent Done false
5740 Acked Read 8 Done Sent Done false
5741 Acked Read 1 Done Sent Done false
5742 Acked Read 9 Done Sent Done false
5743 Acked Read 8 Done Sent Done false
5744 Acked Write 1 Done Sent Done false
5745 Acked Read 7 Done Sent Done false
5746 Acked Read 7 Done Sent Done false
5747 Acked Write 1 Done Sent Done false
5748 Acked Write 8 Done Sent Done false
5749 Acked Read 3 Done Sent Done false
5750 Acked Read 10 Done Sent Done false
5751 Acked Read 3 Done Sent Done false
5752 Acked Read 3 Done Sent Done false
5753 Acked Read 4 Done Sent Done false
5754 Acked Write 4 Done Sent Done false
5755 Acked Write 8 Done Sent Done false
5756 Acked Write 7 Done Sent Done false
5757 Acked Write 10 Done Sent Done false
5758 Acked Read 2 Done Sent Done false
5759 Acked Write 2 Done Sent Done false
5760 Acked Read 7 Done Sent Done false
5761 Acked Write 7 Done Sent Done false
5762 Acked Flush 0 Done Sent Done false
5763 Acked Read 4 Done Sent Done false
5764 Acked Write 8 Done Sent Done false
5765 Acked Read 8 Done Sent Done false
5766 Acked Read 6 Done Sent Done false
5767 Acked Read 9 Done Sent Done false
5768 Acked Write 5 Done Sent Done false
5769 Acked Write 10 Done Sent Done false
5770 Acked Write 9 Done Sent Done false
5771 Acked Write 2 Done Sent Done false
5772 Acked Read 9 Done Sent Done false
5773 Acked Read 5 Done Sent Done false
5774 Acked Read 7 Done Sent Done false
5775 Acked Write 10 Done Sent Done false
5776 Acked Read 6 Done Sent Done false
5777 Acked Write 9 Done Sent Done false
5778 Acked Write 2 Done Sent Done false
5779 Acked Write 8 Done Sent Done false
5780 Acked Write 6 Done Sent Done false
5781 Acked Write 8 Done Sent Done false
5782 Acked Write 10 Done Sent Done false
5783 Acked Read 6 Done Sent Done false
5784 Acked Read 6 Done Sent Done false
5785 Acked Write 6 Done Sent Done false
5786 Acked Read 5 Done Sent Done false
5787 Acked Read 7 Done Sent Done false
5788 Acked Read 4 Done Sent Done false
5789 Acked Write 3 Done Sent Done false
5790 Acked Read 10 Done Sent Done false
5791 Acked Write 6 Done Sent Done false
5792 Acked Read 5 Done Sent Done false
5793 Acked Read 7 Done Sent Done false
5794 Acked Read 8 Done Sent Done false
5795 Acked Write 9 Done Sent Done false
5796 Acked Read 1 Done Sent Done false
5797 Acked Flush 0 Done Sent Done false
5798 Acked Write 6 Done Sent Done false
5799 Acked Write 8 Done Sent Done false
5800 Acked Write 1 Done Sent Done false
5801 Acked Flush 0 Done Sent Done false
5802 Acked Read 8 Done Sent Done false
5803 Acked Read 2 Done Sent Done false
5804 Acked Write 4 Done Sent Done false
5805 Acked Read 9 Done Sent Done false
5806 Acked Read 9 Done Sent Done false
5807 Acked Write 2 Done Sent Done false
5808 Acked Write 7 Done Sent Done false
5809 Acked Write 3 Done Sent Done false
5810 Acked Flush 0 Done Sent Done false
5811 Acked Write 3 Done Sent Done false
5812 Acked Write 5 Done Sent Done false
5813 Acked Read 9 Done Sent Done false
5814 Acked Write 5 Done Sent Done false
5815 Acked Read 8 Done Sent Done false
5816 Acked Read 10 Done Sent Done false
5817 Acked Read 2 Done Sent Done false
5818 Acked Flush 0 Done Sent Done false
5819 Acked Write 1 Done Sent Done false
5820 Acked Read 10 Done Sent Done false
5821 Acked Write 7 Done Sent Done false
5822 Acked Read 3 Done Sent Done false
5823 Acked Read 6 Done Sent Done false
5824 Acked Write 2 Done Sent Done false
5825 Acked Write 1 Done Sent Done false
5826 Acked Flush 0 Done Sent Done false
5827 Acked Read 7 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5826)) Some(JobId(5526)) Some(JobId(5826))
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5827 5826 5825 5824 5823
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:20.768808912Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.626071283Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33576)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.626267856Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.627608884Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.62816757Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.628189855Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(5526))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.62820947Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.628222917Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(5526))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.628235934Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(5526))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:21.628445644Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 5828 5827 5826 5825 5824
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:05:23.881 INFO request completed, [8810] Got stop action so:false kr:false
latency_us: 358, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 41f01ae6-ef21-4a44-b4b0-1a036aa56905, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:23.882296863Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:23.882366267Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:23.882387951Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:23.882484038Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:05:23.882 INFO request completed, latency_us: 136, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1380ef48-91a6-4c8d-bebb-1f2f892c9341, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:05:27.885 INFO request completed, latency_us: 381, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6056b0a6-aa70-4a55-b2da-f227162966f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 3439 len 4096
002/300 Write block 215 len 36864 data: 6 4 4 5 7 4 5 4 4
003/300 Write block 3215 len 8192 data: 3 3
004/300 Read block 2278 len 12288
005/300 Read block 767 len 12288
006/300 Read block 3140 len 24576
007/300 Flush
008/300 Read block 4535 len 28672
009/300 Flush
010/300 Write block 3271 len 40960 data: 4 4 3 4 4 4 3 3 3 2
011/300 Read block 86 len 20480
012/300 Read block 2710 len 12288
013/300 Write block 483 len 20480 data: 5 5 4 5 4
014/300 Read block 3424 len 8192
015/300 Write block 3690 len 20480 data: 3 3 5 3 2
016/300 Read block 1464 len 8192
017/300 Read block 1095 len 16384
018/300 Read block 1229 len 16384
019/300 Write block 421 len 12288 data: 7 8 6
020/300 Write block 4387 len 40960 data: 5 6 5 7 7 5 4 4 4 4
021/300 Read block 4344 len 8192
022/300 Write block 3903 len 8192 data: 5 5
023/300 Write block 3324 len 16384 data: 4 3 2 2
024/300 Read block 3918 len 8192
025/300 Write block 3208 len 40960 data: 4 4 3 3 3 3 3 4 4 3
026/300 Write block 1072 len 20480 data: 5 4 4 4 3
027/300 Read block 4057 len 8192
028/300 Read block 3777 len 20480
029/300 Read block 1429 len 20480
030/300 Read block 4382 len 40960
031/300 Read block 1237 len 12288
032/300 Write block 942 len 40960 data: 6 5 5 4 3 3 3 5 5 5
033/300 Write block 1734 len 32768 data: 3 3 3 3 3 4 4 6
034/300 Read block 1567 len 32768
035/300 Write block 3944 len 40960 data: 4 3 3 2 2 2 2 2 2 2
036/300 Read block 3542 len 4096
037/300 Write block 812 len 16384 data: 6 6 5 4
038/300 Write block 3179 len 36864 data: 3 4 4 4 4 5 5 4 4
039/300 Read block 3193 len 36864
040/300 Read block 333 len 24576
041/300 Flush
042/300 Flush
043/300 Flush
044/300 Read block 2798 len 24576
045/300 Read block 1903 len 20480
046/300 Write block 4467 len 40960 data: 3 3 3 3 2 2 3 3 4 4
047/300 Read block 24 len 36864
048/300 Read block 1687 len 28672
049/300 Write block 3993 len 28672 data: 3 3 3 3 4 4 4
050/300 Write block 4902 len 32768 data: 5 6 5 4 4 3 3 3
051/300 Write block 3867 len 4096 data: 5
052/300 Write block 1196 len 32768 data: 2 3 4 3 2 3 3 4
053/300 Write block 4687 len 4096 data: 3
054/300 Write block 3580 len 12288 data: 2 2 2
055/300 Flush
056/300 Read block 4293 len 20480
057/300 Read block 1182 len 8192
058/300 Read block 3100 len 8192
059/300 Read block 715 len 40960
060/300 Flush
061/300 Read block 2781 len 32768
062/300 Read block 1845 len 12288
063/300 Write block 2661 len 36864 data: 2 4 5 6 6 6 6 6 7
064/300 Read block 1054 len 20480
065/300 Read block 886 len 24576
066/300 Write block 3947 len 8192 data: 3 3
067/300 Read block 915 len 28672
068/300 Write block 3612 len 16384 data: 6 5 5 5
069/300 Flush
070/300 Read block 1013 len 36864
071/300 Flush
072/300 Read block 4680 len 4096
073/300 Read block 1228 len 32768
074/300 Flush
075/300 Read block 2673 len 16384
076/300 Write block 3862 len 36864 data: 5 5 6 5 5 6 5 7 6
077/300 Read block 54 len 12288
078/300 Read block 2095 len 36864
079/300 Write block 3164 len 36864 data: 3 4 4 4 4 4 4 2 2
080/300 Flush
081/300 Write block 2647 len 20480 data: 4 4 4 4 4
082/300 Write block 4375 len 32768 data: 5 5 6 7 7 7 6 4
083/300 Read block 4882 len 36864
084/300 Write block 483 len 24576 data: 6 6 5 6 5 3
085/300 Write block 3623 len 20480 data: 4 4 5 4 3
086/300 Read block 2425 len 20480
087/300 Read block 2492 len 28672
088/300 Write block 2089 len 12288 data: 7 7 7
089/300 Write block 4951 len 4096 data: 4
090/300 Flush
091/300 Read block 820 len 12288
092/300 Write block 2748 len 16384 data: 4 4 5 4
093/300 Flush
094/300 Read block 967 len 36864
095/300 Read block 4852 len 32768
096/300 Write block 1261 len 24576 data: 6 6 5 6 4 3
097/300 Flush
098/300 Flush
099/300 Write block 3350 len 4096 data: 5
100/300 Write block 1331 len 8192 data: 5 4
101/300 Flush
102/300 Write block 601 len 12288 data: 4 4 4
103/300 Read block 209 len 28672
104/300 Write block 4419 len 12288 data: 3 3 3
105/300 Write block 2012 len 24576 data: 4 4 5 5 4 4
106/300 Read block 679 len 40960
107/300 Write block 820 len 20480 data: 2 2 2 3 2
108/300 Write block 2857 len 32768 data: 5 5 3 2 4 4 5 4
109/300 Write block 4662 len 12288 data: 3 3 3
110/300 Read block 2088 len 32768
111/300 Read block 64 len 28672
112/300 Write block 2076 len 8192 data: 3 3
113/300 Write block 441 len 20480 data: 6 8 8 9 9
114/300 Write block 4818 len 24576 data: 3 3 3 3 3 5
115/300 Read block 4988 len 32768
116/300 Read block 1000 len 4096
117/300 Read block 1923 len 16384
118/300 Flush
119/300 Write block 4355 len 40960 data: 7 5 6 6 5 3 2 2 2 3
120/300 Write block 1097 len 36864 data: 2 3 3 2 2 2 2 2 2
121/300 Write block 480 len 4096 data: 4
122/300 Write block 2051 len 32768 data: 4 4 4 2 2 3 3 4
123/300 Read block 3928 len 28672
124/300 Read block 244 len 40960
125/300 Read block 819 len 20480
126/300 Write block 486 len 32768 data: 7 6 4 3 3 3 3 4
127/300 Flush
128/300 Read block 73 len 36864
129/300 Read block 4359 len 24576
130/300 Flush
131/300 Write block 1806 len 16384 data: 2 2 2 3
132/300 Read block 4103 len 36864
133/300 Write block 860 len 24576 data: 3 3 4 4 4 3
134/300 Read block 301 len 24576
135/300 Write block 4159 len 4096 data: 6
136/300 Read block 3188 len 16384
137/300 Read block 2137 len 20480
138/300 Write block 946 len 24576 data: 4 4 4 6 6 6
139/300 Read block 3354 len 4096
140/300 Read block 3794 len 28672
141/300 Write block 1737 len 20480 data: 4 4 5 5 7
142/300 Write block 201 len 20480 data: 3 2 3 3 3
143/300 Read block 3687 len 16384
144/300 Read block 2536 len 28672
145/300 Read block 3551 len 12288
146/300 Write block 4963 len 36864 data: 4 4 5 4 4 4 4 4 5
147/300 Write block 693 len 12288 data: 2 2 2
148/300 Write block 1891 len 4096 data: 6
149/300 Read block 4091 len 40960
150/300 Read block 4579 len 32768
151/300 Read block 3769 len 16384
152/300 Write block 3200 len 24576 data: 4 5 5 5 5 6
153/300 Write block 1817 len 4096 data: 3
154/300 Read block 4370 len 20480
155/300 Write block 1349 len 20480 data: 3 4 4 4 4
156/300 Write block 417 len 24576 data: 7 8 9 7 8 9
157/300 Read block 3619 len 28672
158/300 Read block 2210 len 12288
159/300 Write block 3151 len 36864 data: 2 2 2 4 4 6 7 7 8
160/300 Read block 389 len 24576
161/300 Read block 576 len 28672
162/300 Read block 1278 len 28672
163/300 Write block 1827 len 24576 data: 4 5 5 3 3 3
164/300 Read block 3071 len 32768
165/300 Read block 2154 len 12288
166/300 Read block 512 len 32768
167/300 Read block 1752 len 28672
168/300 Write block 3933 len 28672 data: 4 5 5 6 5 7 8
169/300 Write block 4041 len 16384 data: 6 4 7 7
170/300 Write block 4155 len 40960 data: 4 4 4 5 7 6 4 4 4 4
171/300 Read block 2839 len 24576
172/300 Read block 1676 len 36864
173/300 Write block 1788 len 12288 data: 2 3 3
174/300 Read block 2574 len 8192
175/300 Write block 4100 len 40960 data: 5 6 5 5 4 2 2 2 3 3
176/300 Write block 3114 len 28672 data: 8 6 6 6 5 5 3
177/300 Write block 1964 len 20480 data: 2 2 3 3 2
178/300 Write block 90 len 24576 data: 3 5 6 7 8 9
179/300 Read block 3158 len 28672
180/300 Write block 4321 len 32768 data: 5 4 5 4 4 3 3 3
181/300 Read block 2037 len 16384
182/300 Read block 2620 len 20480
183/300 Write block 721 len 24576 data: 4 3 3 3 3 3
184/300 Read block 3184 len 8192
185/300 Read block 3410 len 8192
186/300 Write block 2917 len 12288 data: 5 5 4
187/300 Read block 994 len 40960
188/300 Write block 3919 len 16384 data: 4 4 3 2
189/300 Write block 1023 len 8192 data: 6 6
190/300 Read block 302 len 12288
191/300 Flush
192/300 Write block 4770 len 36864 data: 7 6 5 4 4 5 5 6 4
193/300 Read block 65 len 20480
194/300 Read block 4079 len 24576
195/300 Write block 2456 len 32768 data: 6 7 7 6 5 4 3 3
196/300 Write block 938 len 4096 data: 5
197/300 Read block 375 len 12288
198/300 Read block 3958 len 4096
199/300 Read block 774 len 24576
200/300 Write block 2579 len 36864 data: 3 3 3 2 3 3 3 3 3
201/300 Read block 3092 len 24576
202/300 Read block 4153 len 4096
203/300 Read block 3070 len 40960
204/300 Read block 1580 len 40960
205/300 Read block 578 len 24576
206/300 Write block 1540 len 28672 data: 5 5 4 3 3 3 4
207/300 Read block 2822 len 28672
208/300 Read block 1874 len 4096
209/300 Write block 2552 len 40960 data: 4 4 4 4 4 4 4 3 4 5
210/300 Read block 2860 len 12288
211/300 Read block 170 len 4096
212/300 Write block 1125 len 16384 data: 5 5 5 4
213/300 Read block 4293 len 16384
214/300 Write block 1320 len 16384 data: 3 3 4 4
215/300 Write block 1128 len 36864 data: 5 4 4 5 4 3 3 3 4
216/300 Write block 4599 len 40960 data: 3 3 3 3 3 3 4 4 4 3
217/300 Read block 1094 len 12288
218/300 Read block 744 len 20480
219/300 Read block 4020 len 12288
220/300 Write block 2759 len 4096 data: 5
221/300 Read block 4977 len 28672
222/300 Read block 976 len 20480
223/300 Write block 4510 len 24576 data: 5 5 4 5 6 5
224/300 Write block 1065 len 32768 data: 4 5 4 4 5 5 5 6
225/300 Read block 3001 len 16384
226/300 Write block 1225 len 24576 data: 4 3 4 3 4 4
227/300 Read block 2452 len 24576
228/300 Write block 1399 len 4096 data: 5
229/300 Write block 1173 len 8192 data: 6 7
230/300 Write block 3180 len 4096 data: 5
231/300 Read block 2485 len 12288
232/300 Read block 3367 len 20480
233/300 Read block 785 len 28672
234/300 Read block 958 len 40960
235/300 Read block 4013 len 32768
236/300 Write block 2012 len 40960 data: 5 5 6 6 5 5 4 4 5 4
237/300 Read block 711 len 16384
238/300 Read block 443 len 32768
239/300 Write block 3101 len 40960 data: 7 6 6 5 5 4 6 7 7 7
240/300 Write block 3531 len 4096 data: 3
241/300 Write block 1682 len 28672 data: 5 6 6 5 6 5 4
242/300 Write block 3707 len 20480 data: 6 4 4 3 3
243/300 Write block 2495 len 36864 data: 3 3 3 3 3 3 2 2 3
244/300 Write block 577 len 40960 data: 5 5 5 6 5 5 5 3 3 3
245/300 Read block 2382 len 8192
246/300 Write block 230 len 20480 data: 4 4 3 3 4
247/300 Read block 349 len 28672
248/300 Write block 2431 len 24576 data: 4 4 3 3 3 3
249/300 Read block 4176 len 8192
250/300 Write block 257 len 16384 data: 4 3 3 4
251/300 Write block 1557 len 20480 data: 3 3 3 3 4
252/300 Read block 1668 len 20480
253/300 Write block 4809 len 20480 data: 3 4 4 2 2
254/300 Write block 4155 len 32768 data: 5 5 5 6 8 7 5 5
255/300 Write block 769 len 12288 data: 4 4 3
256/300 Read block 747 len 4096
257/300 Read block 1760 len 16384
258/300 Read block 4756 len 36864
259/300 Read block 2792 len 8192
260/300 Flush
261/300 Read block 1001 len 16384
262/300 Write block 3470 len 32768 data: 3 3 3 4 4 5 4 4
263/300 Read block 980 len 36864
264/300 Read block 2087 len 20480
265/300 Write block 3814 len 8192 data: 2 2
266/300 Read block 4959 len 24576
267/300 Read block 1982 len 20480
268/300 Read block 2355 len 24576
269/300 Read block 3047 len 20480
270/300 Flush
271/300 Read block 1535 len 12288
272/300 Read block 63 len 28672
273/300 Write block 2758 len 28672 data: 3 6 5 4 4 4 3
274/300 Flush
275/300 Read block 2099 len 40960
276/300 Read block 2659 len 20480
277/300 Flush
278/300 Read block 1722 len 24576
279/300 Write block 1703 len 40960 data: 6 4 3 4 6 6 7 7 6 7
280/300 Write block 4187 len 20480 data: 6 4 4 6 6
281/300 Flush
282/300 Write block 202 len 12288 data: 3 4 4
283/300 Read block 512 len 40960
284/300 Flush
285/300 Read block 1271 len 20480
286/300 Flush
287/300 Write block 2634 len 32768 data: 2 2 3 4 5 5 5 5
288/300 Read block 1692 len 32768
289/300 Write block 356 len 4096 data: 5
290/300 Read block 3158 len 20480
291/300 Write block 970 len 8192 data: 3 3
292/300 Read block 4992 len 12288
293/300 Read block 4111 len 8192
294/300 Flush
295/300 Read block 4844 len 36864
296/300 Write block 2150 len 8192 data: 5 5
297/300 Flush
298/300 Write block 83 len 32768 data: 4 3 3 3 2 2 2 4
299/300 Write block 413 len 8192 data: 6 8
300/300 Read block 4110 len 20480
301/300 Read block 835 len 20480
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:05:28.126 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3f28db01-95c5-4034-823c-fd3b930a7a3e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 1 Sent Done Done false
5830 Acked Write 9 Sent Done Done false
5831 Acked Write 2 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 3 Sent Done Done false
5834 Acked Read 6 Sent Done Done false
5835 Acked Flush 0 Sent Done Done false
5836 Acked Read 7 Sent Done Done false
5837 Acked Flush 0 Sent Done Done false
5838 Acked Write 10 Sent Done Done false
5839 Acked Read 5 Sent Done Done false
5840 Acked Read 3 Sent Done Done false
5841 Acked Write 5 Sent Done Done false
5842 Acked Read 2 Sent Done Done false
5843 Acked Write 5 Sent Done Done false
5844 Acked Read 2 Sent Done Done false
5845 Acked Read 4 Sent Done Done false
5846 Acked Read 4 Sent Done Done false
5847 Acked Write 3 Sent Done Done false
5848 Acked Write 10 Sent Done Done false
5849 Acked Read 2 Sent Done Done false
5850 Acked Write 2 Sent Done Done false
5851 Acked Write 4 Sent Done Done false
5852 Acked Read 2 Sent Done Done false
5853 Acked Write 10 Sent Done Done false
5854 Acked Write 5 Sent Done Done false
5855 Acked Read 2 Sent Done Done false
5856 Acked Read 5 Sent Done Done false
5857 Acked Read 5 Sent Done Done false
5858 Acked Read 10 Sent Done Done false
5859 Acked Read 3 Sent Done Done false
5860 Acked Write 10 Sent Done Done false
5861 Acked Write 8 Sent Done Done false
5862 Acked Read 8 Sent Done Done false
5863 Acked Write 10 Sent Done Done false
5864 Acked Read 1 Sent Done Done false
5865 Acked Write 4 Sent Done Done false
5866 Acked Write 9 Sent Done Done false
5867 Acked Read 9 Sent Done Done false
5868 Acked Read 6 Sent Done Done false
5869 Acked Flush 0 Sent Done Done false
5870 Acked Flush 0 Sent Done Done false
5871 Acked Flush 0 Sent Done Done false
5872 Acked Read 6 Sent Done Done false
5873 Acked Read 5 Sent Done Done false
5874 Acked Write 10 Sent Done Done false
5875 Acked Read 9 Sent Done Done false
5876 Acked Read 7 Sent Done Done false
5877 Acked Write 7 Sent Done Done false
5878 Acked Write 8 Sent Done Done false
5879 Acked Write 1 Sent Done Done false
5880 Acked Write 8 Sent Done Done false
5881 Acked Write 1 Sent Done Done false
5882 Acked Write 3 Sent Done Done false
5883 Acked Flush 0 Sent Done Done false
5884 Acked Read 5 Sent Done Done false
5885 Acked Read 2 Sent Done Done false
5886 Acked Read 2 Sent Done Done false
5887 Acked Read 10 Sent Done Done false
5888 Acked Flush 0 Sent Done Done false
5889 Acked Read 8 Sent Done Done false
5890 Acked Read 3 Sent Done Done false
5891 Acked Write 9 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Read 6 Sent Done Done false
5894 Acked Write 2 Sent Done Done false
5895 Acked Read 7 Sent Done Done false
5896 Acked Write 4 Sent Done Done false
5897 Acked Flush 0 Sent Done Done false
5898 Acked Read 9 Sent Done Done false
5899 Acked Flush 0 Sent Done Done false
5900 Acked Read 1 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Flush 0 Sent Done Done false
5903 Acked Read 4 Sent Done Done false
5904 Acked Write 9 Sent Done Done false
5905 Acked Read 3 Sent Done Done false
5906 Acked Read 9 Sent Done Done false
5907 Acked Write 9 Sent Done Done false
5908 Acked Flush 0 Sent Done Done false
5909 Acked Write 5 Sent Done Done false
5910 Acked Write 8 Sent Done Done false
5911 Acked Read 9 Sent Done Done false
5912 Acked Write 6 Sent Done Done false
5913 Acked Write 5 Sent Done Done false
5914 Acked Read 5 Sent Done Done false
5915 Acked Read 7 Sent Done Done false
5916 Acked Write 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Flush 0 Sent Done Done false
5919 Acked Read 3 Sent Done Done false
5920 Acked Write 4 Sent Done Done false
5921 Acked Flush 0 Sent Done Done false
5922 Acked Read 9 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Write 6 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Flush 0 Sent Done Done false
5927 Acked Write 1 Sent Done Done false
5928 Acked Write 2 Sent Done Done false
5929 Acked Flush 0 Sent Done Done false
5930 Acked Write 3 Sent Done Done false
5931 Acked Read 7 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Write 6 Sent Done Done false
5934 Acked Read 10 Sent Done Done false
5935 Acked Write 5 Sent Done Done false
5936 Acked Write 8 Sent Done Done false
5937 Acked Write 3 Sent Done Done false
5938 Acked Read 8 Sent Done Done false
5939 Acked Read 7 Sent Done Done false
5940 Acked Write 2 Sent Done Done false
5941 Acked Write 5 Sent Done Done false
5942 Acked Write 6 Sent Done Done false
5943 Acked Read 8 Sent Done Done false
5944 Acked Read 1 Sent Done Done false
5945 Acked Read 4 Sent Done Done false
5946 Acked Flush 0 Sent Done Done false
5947 Acked Write 10 Sent Done Done false
5948 Acked Write 9 Sent Done Done false
5949 Acked Write 1 Sent Done Done false
5950 Acked Write 8 Sent Done Done false
5951 Acked Read 7 Sent Done Done false
5952 Acked Read 10 Sent Done Done false
5953 Acked Read 5 Sent Done Done false
5954 Acked Write 8 Sent Done Done false
5955 Acked Flush 0 Sent Done Done false
5956 Acked Read 9 Sent Done Done false
5957 Acked Read 6 Sent Done Done false
5958 Acked Flush 0 Sent Done Done false
5959 Acked Write 4 Sent Done Done false
5960 Acked Read 9 Sent Done Done false
5961 Acked Write 6 Sent Done Done false
5962 Acked Read 6 Sent Done Done false
5963 Acked Write 1 Sent Done Done false
5964 Acked Read 4 Sent Done Done false
5965 Acked Read 5 Sent Done Done false
5966 Acked Write 6 Sent Done Done false
5967 Acked Read 1 Sent Done Done false
5968 Acked Read 7 Sent Done Done false
5969 Acked Write 5 Sent Done Done false
5970 Acked Write 5 Sent Done Done false
5971 Acked Read 4 Sent Done Done false
5972 Acked Read 7 Sent Done Done false
5973 Acked Read 3 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Write 3 Sent Done Done false
5976 Acked Write 1 Sent Done Done false
5977 Acked Read 10 Sent Done Done false
5978 Acked Read 8 Sent Done Done false
5979 Acked Read 4 Sent Done Done false
5980 Acked Write 6 Sent Done Done false
5981 Acked Write 1 Sent Done Done false
5982 Acked Read 5 Sent Done Done false
5983 Acked Write 5 Sent Done Done false
5984 Acked Write 6 Sent Done Done false
5985 Acked Read 7 Sent Done Done false
5986 Acked Read 3 Sent Done Done false
5987 Acked Write 9 Sent Done Done false
5988 Acked Read 6 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 7 Sent Done Done false
5991 Acked Write 6 Sent Done Done false
5992 Acked Read 8 Sent Done Done false
5993 Acked Read 3 Sent Done Done false
5994 Acked Read 8 Sent Done Done false
5995 Acked Read 7 Sent Done Done false
5996 Acked Write 7 Sent Done Done false
5997 Acked Write 4 Sent Done Done false
5998 Acked Write 10 Sent Done Done false
5999 Acked Read 6 Sent Done Done false
6000 Acked Read 9 Sent Done Done false
6001 Acked Write 3 Sent Done Done false
6002 Acked Read 2 Sent Done Done false
6003 Acked Write 10 Sent Done Done false
6004 Acked Write 7 Sent Done Done false
6005 Acked Write 5 Sent Done Done false
6006 Acked Write 6 Sent Done Done false
6007 Acked Read 7 Sent Done Done false
6008 Acked Write 8 Sent Done Done false
6009 Acked Read 4 Sent Done Done false
6010 Acked Read 5 Sent Done Done false
6011 Acked Write 6 Sent Done Done false
6012 Acked Read 2 Sent Done Done false
6013 Acked Read 2 Sent Done Done false
6014 Acked Write 3 Sent Done Done false
6015 Acked Read 10 Sent Done Done false
6016 Acked Write 4 Sent Done Done false
6017 Acked Write 2 Sent Done Done false
6018 Acked Read 3 Sent Done Done false
6019 Acked Flush 0 Sent Done Done false
6020 Acked Write 9 Sent Done Done false
6021 Acked Read 5 Sent Done Done false
6022 Acked Read 6 Sent Done Done false
6023 Acked Write 8 Sent Done Done false
6024 Acked Write 1 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 1 Sent Done Done false
6027 Acked Read 6 Sent Done Done false
6028 Acked Write 9 Sent Done Done false
6029 Acked Read 6 Sent Done Done false
6030 Acked Read 1 Sent Done Done false
6031 Acked Read 10 Sent Done Done false
6032 Acked Read 10 Sent Done Done false
6033 Acked Read 6 Sent Done Done false
6034 Acked Write 7 Sent Done Done false
6035 Acked Read 7 Sent Done Done false
6036 Acked Read 1 Sent Done Done false
6037 Acked Write 10 Sent Done Done false
6038 Acked Read 3 Sent Done Done false
6039 Acked Read 1 Sent Done Done false
6040 Acked Write 4 Sent Done Done false
6041 Acked Read 4 Sent Done Done false
6042 Acked Write 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 10 Sent Done Done false
6045 Acked Read 3 Sent Done Done false
6046 Acked Read 5 Sent Done Done false
6047 Acked Read 3 Sent Done Done false
6048 Acked Write 1 Sent Done Done false
6049 Acked Read 7 Sent Done Done false
6050 Acked Read 5 Sent Done Done false
6051 Acked Write 6 Sent Done Done false
6052 Acked Write 8 Sent Done Done false
6053 Acked Read 4 Sent Done Done false
6054 Acked Write 6 Sent Done Done false
6055 Acked Read 6 Sent Done Done false
6056 Acked Write 1 Sent Done Done false
6057 Acked Write 2 Sent Done Done false
6058 Acked Write 1 Sent Done Done false
6059 Acked Read 3 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 7 Sent Done Done false
6062 Acked Read 10 Sent Done Done false
6063 Acked Read 8 Sent Done Done false
6064 Acked Write 10 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 8 Sent Done Done false
6067 Acked Write 10 Sent Done Done false
6068 Acked Write 1 Sent Done Done false
6069 Acked Write 7 Sent Done Done false
6070 Acked Write 5 Sent Done Done false
6071 Acked Write 9 Sent Done Done false
6072 Acked Write 10 Sent Done Done false
6073 Acked Read 2 Sent Done Done false
6074 Acked Write 5 Sent Done Done false
6075 Acked Read 7 Sent Done Done false
6076 Acked Write 6 Sent Done Done false
6077 Acked Read 2 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Write 5 Sent Done Done false
6080 Acked Read 5 Sent Done Done false
6081 Acked Write 5 Sent Done Done false
6082 Acked Write 8 Sent Done Done false
6083 Acked Write 3 Sent Done Done false
6084 Acked Read 1 Sent Done Done false
6085 Acked Read 4 Sent Done Done false
6086 Acked Read 9 Sent Done Done false
6087 Acked Read 2 Sent Done Done false
6088 Acked Flush 0 Sent Done Done false
6089 Acked Read 4 Sent Done Done false
6090 Acked Write 8 Sent Done Done false
6091 Acked Read 9 Sent Done Done false
6092 Acked Read 5 Sent Done Done false
6093 Acked Write 2 Sent Done Done false
6094 Acked Read 6 Sent Done Done false
6095 Acked Read 5 Sent Done Done false
6096 Acked Read 6 Sent Done Done false
6097 Acked Read 5 Sent Done Done false
6098 Acked Flush 0 Sent Done Done false
6099 Acked Read 3 Sent Done Done false
6100 Acked Read 7 Sent Done Done false
6101 Acked Write 7 Sent Done Done false
6102 Acked Flush 0 Sent Done Done false
6103 Acked Read 10 Sent Done Done false
6104 Acked Read 5 Sent Done Done false
6105 Acked Flush 0 Sent Done Done false
6106 Acked Read 6 Sent Done Done false
6107 Acked Write 10 Sent Done Done false
6108 Acked Write 5 Sent Done Done false
6109 Acked Flush 0 Sent Done Done false
6110 Acked Write 3 Sent Done Done false
6111 Acked Read 10 Sent Done Done false
6112 Acked Flush 0 Sent Done Done false
6113 Acked Read 5 Sent Done Done false
6114 Acked Flush 0 Sent Done Done false
6115 Acked Write 8 Sent Done Done false
6116 Acked Read 8 Sent Done Done false
6117 Acked Write 1 Sent Done Done false
6118 Acked Read 5 Sent Done Done false
6119 Acked Write 2 Sent Done Done false
6120 Acked Read 3 Sent Done Done false
6121 Acked Read 2 Sent Done Done false
6122 Acked Flush 0 Sent Done Done false
6123 Acked Read 9 Sent Done Done false
6124 Acked Write 2 Sent Done Done false
6125 Acked Flush 0 Sent Done Done false
6126 Acked Write 8 Sent Done Done false
6127 Acked Write 2 Sent Done Done false
6128 Acked Read 5 Sent Done Done false
6129 Acked Read 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:28.784353582Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:29.882824856Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:30.788274408Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 1 Sent Done Done false
5830 Acked Write 9 Sent Done Done false
5831 Acked Write 2 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 3 Sent Done Done false
5834 Acked Read 6 Sent Done Done false
5835 Acked Flush 0 Sent Done Done false
5836 Acked Read 7 Sent Done Done false
5837 Acked Flush 0 Sent Done Done false
5838 Acked Write 10 Sent Done Done false
5839 Acked Read 5 Sent Done Done false
5840 Acked Read 3 Sent Done Done false
5841 Acked Write 5 Sent Done Done false
5842 Acked Read 2 Sent Done Done false
5843 Acked Write 5 Sent Done Done false
5844 Acked Read 2 Sent Done Done false
5845 Acked Read 4 Sent Done Done false
5846 Acked Read 4 Sent Done Done false
5847 Acked Write 3 Sent Done Done false
5848 Acked Write 10 Sent Done Done false
5849 Acked Read 2 Sent Done Done false
5850 Acked Write 2 Sent Done Done false
5851 Acked Write 4 Sent Done Done false
5852 Acked Read 2 Sent Done Done false
5853 Acked Write 10 Sent Done Done false
5854 Acked Write 5 Sent Done Done false
5855 Acked Read 2 Sent Done Done false
5856 Acked Read 5 Sent Done Done false
5857 Acked Read 5 Sent Done Done false
5858 Acked Read 10 Sent Done Done false
5859 Acked Read 3 Sent Done Done false
5860 Acked Write 10 Sent Done Done false
5861 Acked Write 8 Sent Done Done false
5862 Acked Read 8 Sent Done Done false
5863 Acked Write 10 Sent Done Done false
5864 Acked Read 1 Sent Done Done false
5865 Acked Write 4 Sent Done Done false
5866 Acked Write 9 Sent Done Done false
5867 Acked Read 9 Sent Done Done false
5868 Acked Read 6 Sent Done Done false
5869 Acked Flush 0 Sent Done Done false
5870 Acked Flush 0 Sent Done Done false
5871 Acked Flush 0 Sent Done Done false
5872 Acked Read 6 Sent Done Done false
5873 Acked Read 5 Sent Done Done false
5874 Acked Write 10 Sent Done Done false
5875 Acked Read 9 Sent Done Done false
5876 Acked Read 7 Sent Done Done false
5877 Acked Write 7 Sent Done Done false
5878 Acked Write 8 Sent Done Done false
5879 Acked Write 1 Sent Done Done false
5880 Acked Write 8 Sent Done Done false
5881 Acked Write 1 Sent Done Done false
5882 Acked Write 3 Sent Done Done false
5883 Acked Flush 0 Sent Done Done false
5884 Acked Read 5 Sent Done Done false
5885 Acked Read 2 Sent Done Done false
5886 Acked Read 2 Sent Done Done false
5887 Acked Read 10 Sent Done Done false
5888 Acked Flush 0 Sent Done Done false
5889 Acked Read 8 Sent Done Done false
5890 Acked Read 3 Sent Done Done false
5891 Acked Write 9 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Read 6 Sent Done Done false
5894 Acked Write 2 Sent Done Done false
5895 Acked Read 7 Sent Done Done false
5896 Acked Write 4 Sent Done Done false
5897 Acked Flush 0 Sent Done Done false
5898 Acked Read 9 Sent Done Done false
5899 Acked Flush 0 Sent Done Done false
5900 Acked Read 1 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Flush 0 Sent Done Done false
5903 Acked Read 4 Sent Done Done false
5904 Acked Write 9 Sent Done Done false
5905 Acked Read 3 Sent Done Done false
5906 Acked Read 9 Sent Done Done false
5907 Acked Write 9 Sent Done Done false
5908 Acked Flush 0 Sent Done Done false
5909 Acked Write 5 Sent Done Done false
5910 Acked Write 8 Sent Done Done false
5911 Acked Read 9 Sent Done Done false
5912 Acked Write 6 Sent Done Done false
5913 Acked Write 5 Sent Done Done false
5914 Acked Read 5 Sent Done Done false
5915 Acked Read 7 Sent Done Done false
5916 Acked Write 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Flush 0 Sent Done Done false
5919 Acked Read 3 Sent Done Done false
5920 Acked Write 4 Sent Done Done false
5921 Acked Flush 0 Sent Done Done false
5922 Acked Read 9 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Write 6 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Flush 0 Sent Done Done false
5927 Acked Write 1 Sent Done Done false
5928 Acked Write 2 Sent Done Done false
5929 Acked Flush 0 Sent Done Done false
5930 Acked Write 3 Sent Done Done false
5931 Acked Read 7 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Write 6 Sent Done Done false
5934 Acked Read 10 Sent Done Done false
5935 Acked Write 5 Sent Done Done false
5936 Acked Write 8 Sent Done Done false
5937 Acked Write 3 Sent Done Done false
5938 Acked Read 8 Sent Done Done false
5939 Acked Read 7 Sent Done Done false
5940 Acked Write 2 Sent Done Done false
5941 Acked Write 5 Sent Done Done false
5942 Acked Write 6 Sent Done Done false
5943 Acked Read 8 Sent Done Done false
5944 Acked Read 1 Sent Done Done false
5945 Acked Read 4 Sent Done Done false
5946 Acked Flush 0 Sent Done Done false
5947 Acked Write 10 Sent Done Done false
5948 Acked Write 9 Sent Done Done false
5949 Acked Write 1 Sent Done Done false
5950 Acked Write 8 Sent Done Done false
5951 Acked Read 7 Sent Done Done false
5952 Acked Read 10 Sent Done Done false
5953 Acked Read 5 Sent Done Done false
5954 Acked Write 8 Sent Done Done false
5955 Acked Flush 0 Sent Done Done false
5956 Acked Read 9 Sent Done Done false
5957 Acked Read 6 Sent Done Done false
5958 Acked Flush 0 Sent Done Done false
5959 Acked Write 4 Sent Done Done false
5960 Acked Read 9 Sent Done Done false
5961 Acked Write 6 Sent Done Done false
5962 Acked Read 6 Sent Done Done false
5963 Acked Write 1 Sent Done Done false
5964 Acked Read 4 Sent Done Done false
5965 Acked Read 5 Sent Done Done false
5966 Acked Write 6 Sent Done Done false
5967 Acked Read 1 Sent Done Done false
5968 Acked Read 7 Sent Done Done false
5969 Acked Write 5 Sent Done Done false
5970 Acked Write 5 Sent Done Done false
5971 Acked Read 4 Sent Done Done false
5972 Acked Read 7 Sent Done Done false
5973 Acked Read 3 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Write 3 Sent Done Done false
5976 Acked Write 1 Sent Done Done false
5977 Acked Read 10 Sent Done Done false
5978 Acked Read 8 Sent Done Done false
5979 Acked Read 4 Sent Done Done false
5980 Acked Write 6 Sent Done Done false
5981 Acked Write 1 Sent Done Done false
5982 Acked Read 5 Sent Done Done false
5983 Acked Write 5 Sent Done Done false
5984 Acked Write 6 Sent Done Done false
5985 Acked Read 7 Sent Done Done false
5986 Acked Read 3 Sent Done Done false
5987 Acked Write 9 Sent Done Done false
5988 Acked Read 6 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 7 Sent Done Done false
5991 Acked Write 6 Sent Done Done false
5992 Acked Read 8 Sent Done Done false
5993 Acked Read 3 Sent Done Done false
5994 Acked Read 8 Sent Done Done false
5995 Acked Read 7 Sent Done Done false
5996 Acked Write 7 Sent Done Done false
5997 Acked Write 4 Sent Done Done false
5998 Acked Write 10 Sent Done Done false
5999 Acked Read 6 Sent Done Done false
6000 Acked Read 9 Sent Done Done false
6001 Acked Write 3 Sent Done Done false
6002 Acked Read 2 Sent Done Done false
6003 Acked Write 10 Sent Done Done false
6004 Acked Write 7 Sent Done Done false
6005 Acked Write 5 Sent Done Done false
6006 Acked Write 6 Sent Done Done false
6007 Acked Read 7 Sent Done Done false
6008 Acked Write 8 Sent Done Done false
6009 Acked Read 4 Sent Done Done false
6010 Acked Read 5 Sent Done Done false
6011 Acked Write 6 Sent Done Done false
6012 Acked Read 2 Sent Done Done false
6013 Acked Read 2 Sent Done Done false
6014 Acked Write 3 Sent Done Done false
6015 Acked Read 10 Sent Done Done false
6016 Acked Write 4 Sent Done Done false
6017 Acked Write 2 Sent Done Done false
6018 Acked Read 3 Sent Done Done false
6019 Acked Flush 0 Sent Done Done false
6020 Acked Write 9 Sent Done Done false
6021 Acked Read 5 Sent Done Done false
6022 Acked Read 6 Sent Done Done false
6023 Acked Write 8 Sent Done Done false
6024 Acked Write 1 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 1 Sent Done Done false
6027 Acked Read 6 Sent Done Done false
6028 Acked Write 9 Sent Done Done false
6029 Acked Read 6 Sent Done Done false
6030 Acked Read 1 Sent Done Done false
6031 Acked Read 10 Sent Done Done false
6032 Acked Read 10 Sent Done Done false
6033 Acked Read 6 Sent Done Done false
6034 Acked Write 7 Sent Done Done false
6035 Acked Read 7 Sent Done Done false
6036 Acked Read 1 Sent Done Done false
6037 Acked Write 10 Sent Done Done false
6038 Acked Read 3 Sent Done Done false
6039 Acked Read 1 Sent Done Done false
6040 Acked Write 4 Sent Done Done false
6041 Acked Read 4 Sent Done Done false
6042 Acked Write 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 10 Sent Done Done false
6045 Acked Read 3 Sent Done Done false
6046 Acked Read 5 Sent Done Done false
6047 Acked Read 3 Sent Done Done false
6048 Acked Write 1 Sent Done Done false
6049 Acked Read 7 Sent Done Done false
6050 Acked Read 5 Sent Done Done false
6051 Acked Write 6 Sent Done Done false
6052 Acked Write 8 Sent Done Done false
6053 Acked Read 4 Sent Done Done false
6054 Acked Write 6 Sent Done Done false
6055 Acked Read 6 Sent Done Done false
6056 Acked Write 1 Sent Done Done false
6057 Acked Write 2 Sent Done Done false
6058 Acked Write 1 Sent Done Done false
6059 Acked Read 3 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 7 Sent Done Done false
6062 Acked Read 10 Sent Done Done false
6063 Acked Read 8 Sent Done Done false
6064 Acked Write 10 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 8 Sent Done Done false
6067 Acked Write 10 Sent Done Done false
6068 Acked Write 1 Sent Done Done false
6069 Acked Write 7 Sent Done Done false
6070 Acked Write 5 Sent Done Done false
6071 Acked Write 9 Sent Done Done false
6072 Acked Write 10 Sent Done Done false
6073 Acked Read 2 Sent Done Done false
6074 Acked Write 5 Sent Done Done false
6075 Acked Read 7 Sent Done Done false
6076 Acked Write 6 Sent Done Done false
6077 Acked Read 2 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Write 5 Sent Done Done false
6080 Acked Read 5 Sent Done Done false
6081 Acked Write 5 Sent Done Done false
6082 Acked Write 8 Sent Done Done false
6083 Acked Write 3 Sent Done Done false
6084 Acked Read 1 Sent Done Done false
6085 Acked Read 4 Sent Done Done false
6086 Acked Read 9 Sent Done Done false
6087 Acked Read 2 Sent Done Done false
6088 Acked Flush 0 Sent Done Done false
6089 Acked Read 4 Sent Done Done false
6090 Acked Write 8 Sent Done Done false
6091 Acked Read 9 Sent Done Done false
6092 Acked Read 5 Sent Done Done false
6093 Acked Write 2 Sent Done Done false
6094 Acked Read 6 Sent Done Done false
6095 Acked Read 5 Sent Done Done false
6096 Acked Read 6 Sent Done Done false
6097 Acked Read 5 Sent Done Done false
6098 Acked Flush 0 Sent Done Done false
6099 Acked Read 3 Sent Done Done false
6100 Acked Read 7 Sent Done Done false
6101 Acked Write 7 Sent Done Done false
6102 Acked Flush 0 Sent Done Done false
6103 Acked Read 10 Sent Done Done false
6104 Acked Read 5 Sent Done Done false
6105 Acked Flush 0 Sent Done Done false
6106 Acked Read 6 Sent Done Done false
6107 Acked Write 10 Sent Done Done false
6108 Acked Write 5 Sent Done Done false
6109 Acked Flush 0 Sent Done Done false
6110 Acked Write 3 Sent Done Done false
6111 Acked Read 10 Sent Done Done false
6112 Acked Flush 0 Sent Done Done false
6113 Acked Read 5 Sent Done Done false
6114 Acked Flush 0 Sent Done Done false
6115 Acked Write 8 Sent Done Done false
6116 Acked Read 8 Sent Done Done false
6117 Acked Write 1 Sent Done Done false
6118 Acked Read 5 Sent Done Done false
6119 Acked Write 2 Sent Done Done false
6120 Acked Read 3 Sent Done Done false
6121 Acked Read 2 Sent Done Done false
6122 Acked Flush 0 Sent Done Done false
6123 Acked Read 9 Sent Done Done false
6124 Acked Write 2 Sent Done Done false
6125 Acked Flush 0 Sent Done Done false
6126 Acked Write 8 Sent Done Done false
6127 Acked Write 2 Sent Done Done false
6128 Acked Read 5 Sent Done Done false
6129 Acked Read 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:32.792658554Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:33.793759523Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:33.883183252Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:33.883240029Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:33.883260654Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:33.883280029Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:33.883381155Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:34.884244431Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:35.79726208Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1228)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 1 Sent Done Done false
5830 Acked Write 9 Sent Done Done false
5831 Acked Write 2 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 3 Sent Done Done false
5834 Acked Read 6 Sent Done Done false
5835 Acked Flush 0 Sent Done Done false
5836 Acked Read 7 Sent Done Done false
5837 Acked Flush 0 Sent Done Done false
5838 Acked Write 10 Sent Done Done false
5839 Acked Read 5 Sent Done Done false
5840 Acked Read 3 Sent Done Done false
5841 Acked Write 5 Sent Done Done false
5842 Acked Read 2 Sent Done Done false
5843 Acked Write 5 Sent Done Done false
5844 Acked Read 2 Sent Done Done false
5845 Acked Read 4 Sent Done Done false
5846 Acked Read 4 Sent Done Done false
5847 Acked Write 3 Sent Done Done false
5848 Acked Write 10 Sent Done Done false
5849 Acked Read 2 Sent Done Done false
5850 Acked Write 2 Sent Done Done false
5851 Acked Write 4 Sent Done Done false
5852 Acked Read 2 Sent Done Done false
5853 Acked Write 10 Sent Done Done false
5854 Acked Write 5 Sent Done Done false
5855 Acked Read 2 Sent Done Done false
5856 Acked Read 5 Sent Done Done false
5857 Acked Read 5 Sent Done Done false
5858 Acked Read 10 Sent Done Done false
5859 Acked Read 3 Sent Done Done false
5860 Acked Write 10 Sent Done Done false
5861 Acked Write 8 Sent Done Done false
5862 Acked Read 8 Sent Done Done false
5863 Acked Write 10 Sent Done Done false
5864 Acked Read 1 Sent Done Done false
5865 Acked Write 4 Sent Done Done false
5866 Acked Write 9 Sent Done Done false
5867 Acked Read 9 Sent Done Done false
5868 Acked Read 6 Sent Done Done false
5869 Acked Flush 0 Sent Done Done false
5870 Acked Flush 0 Sent Done Done false
5871 Acked Flush 0 Sent Done Done false
5872 Acked Read 6 Sent Done Done false
5873 Acked Read 5 Sent Done Done false
5874 Acked Write 10 Sent Done Done false
5875 Acked Read 9 Sent Done Done false
5876 Acked Read 7 Sent Done Done false
5877 Acked Write 7 Sent Done Done false
5878 Acked Write 8 Sent Done Done false
5879 Acked Write 1 Sent Done Done false
5880 Acked Write 8 Sent Done Done false
5881 Acked Write 1 Sent Done Done false
5882 Acked Write 3 Sent Done Done false
5883 Acked Flush 0 Sent Done Done false
5884 Acked Read 5 Sent Done Done false
5885 Acked Read 2 Sent Done Done false
5886 Acked Read 2 Sent Done Done false
5887 Acked Read 10 Sent Done Done false
5888 Acked Flush 0 Sent Done Done false
5889 Acked Read 8 Sent Done Done false
5890 Acked Read 3 Sent Done Done false
5891 Acked Write 9 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Read 6 Sent Done Done false
5894 Acked Write 2 Sent Done Done false
5895 Acked Read 7 Sent Done Done false
5896 Acked Write 4 Sent Done Done false
5897 Acked Flush 0 Sent Done Done false
5898 Acked Read 9 Sent Done Done false
5899 Acked Flush 0 Sent Done Done false
5900 Acked Read 1 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Flush 0 Sent Done Done false
5903 Acked Read 4 Sent Done Done false
5904 Acked Write 9 Sent Done Done false
5905 Acked Read 3 Sent Done Done false
5906 Acked Read 9 Sent Done Done false
5907 Acked Write 9 Sent Done Done false
5908 Acked Flush 0 Sent Done Done false
5909 Acked Write 5 Sent Done Done false
5910 Acked Write 8 Sent Done Done false
5911 Acked Read 9 Sent Done Done false
5912 Acked Write 6 Sent Done Done false
5913 Acked Write 5 Sent Done Done false
5914 Acked Read 5 Sent Done Done false
5915 Acked Read 7 Sent Done Done false
5916 Acked Write 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Flush 0 Sent Done Done false
5919 Acked Read 3 Sent Done Done false
5920 Acked Write 4 Sent Done Done false
5921 Acked Flush 0 Sent Done Done false
5922 Acked Read 9 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Write 6 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Flush 0 Sent Done Done false
5927 Acked Write 1 Sent Done Done false
5928 Acked Write 2 Sent Done Done false
5929 Acked Flush 0 Sent Done Done false
5930 Acked Write 3 Sent Done Done false
5931 Acked Read 7 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Write 6 Sent Done Done false
5934 Acked Read 10 Sent Done Done false
5935 Acked Write 5 Sent Done Done false
5936 Acked Write 8 Sent Done Done false
5937 Acked Write 3 Sent Done Done false
5938 Acked Read 8 Sent Done Done false
5939 Acked Read 7 Sent Done Done false
5940 Acked Write 2 Sent Done Done false
5941 Acked Write 5 Sent Done Done false
5942 Acked Write 6 Sent Done Done false
5943 Acked Read 8 Sent Done Done false
5944 Acked Read 1 Sent Done Done false
5945 Acked Read 4 Sent Done Done false
5946 Acked Flush 0 Sent Done Done false
5947 Acked Write 10 Sent Done Done false
5948 Acked Write 9 Sent Done Done false
5949 Acked Write 1 Sent Done Done false
5950 Acked Write 8 Sent Done Done false
5951 Acked Read 7 Sent Done Done false
5952 Acked Read 10 Sent Done Done false
5953 Acked Read 5 Sent Done Done false
5954 Acked Write 8 Sent Done Done false
5955 Acked Flush 0 Sent Done Done false
5956 Acked Read 9 Sent Done Done false
5957 Acked Read 6 Sent Done Done false
5958 Acked Flush 0 Sent Done Done false
5959 Acked Write 4 Sent Done Done false
5960 Acked Read 9 Sent Done Done false
5961 Acked Write 6 Sent Done Done false
5962 Acked Read 6 Sent Done Done false
5963 Acked Write 1 Sent Done Done false
5964 Acked Read 4 Sent Done Done false
5965 Acked Read 5 Sent Done Done false
5966 Acked Write 6 Sent Done Done false
5967 Acked Read 1 Sent Done Done false
5968 Acked Read 7 Sent Done Done false
5969 Acked Write 5 Sent Done Done false
5970 Acked Write 5 Sent Done Done false
5971 Acked Read 4 Sent Done Done false
5972 Acked Read 7 Sent Done Done false
5973 Acked Read 3 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Write 3 Sent Done Done false
5976 Acked Write 1 Sent Done Done false
5977 Acked Read 10 Sent Done Done false
5978 Acked Read 8 Sent Done Done false
5979 Acked Read 4 Sent Done Done false
5980 Acked Write 6 Sent Done Done false
5981 Acked Write 1 Sent Done Done false
5982 Acked Read 5 Sent Done Done false
5983 Acked Write 5 Sent Done Done false
5984 Acked Write 6 Sent Done Done false
5985 Acked Read 7 Sent Done Done false
5986 Acked Read 3 Sent Done Done false
5987 Acked Write 9 Sent Done Done false
5988 Acked Read 6 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 7 Sent Done Done false
5991 Acked Write 6 Sent Done Done false
5992 Acked Read 8 Sent Done Done false
5993 Acked Read 3 Sent Done Done false
5994 Acked Read 8 Sent Done Done false
5995 Acked Read 7 Sent Done Done false
5996 Acked Write 7 Sent Done Done false
5997 Acked Write 4 Sent Done Done false
5998 Acked Write 10 Sent Done Done false
5999 Acked Read 6 Sent Done Done false
6000 Acked Read 9 Sent Done Done false
6001 Acked Write 3 Sent Done Done false
6002 Acked Read 2 Sent Done Done false
6003 Acked Write 10 Sent Done Done false
6004 Acked Write 7 Sent Done Done false
6005 Acked Write 5 Sent Done Done false
6006 Acked Write 6 Sent Done Done false
6007 Acked Read 7 Sent Done Done false
6008 Acked Write 8 Sent Done Done false
6009 Acked Read 4 Sent Done Done false
6010 Acked Read 5 Sent Done Done false
6011 Acked Write 6 Sent Done Done false
6012 Acked Read 2 Sent Done Done false
6013 Acked Read 2 Sent Done Done false
6014 Acked Write 3 Sent Done Done false
6015 Acked Read 10 Sent Done Done false
6016 Acked Write 4 Sent Done Done false
6017 Acked Write 2 Sent Done Done false
6018 Acked Read 3 Sent Done Done false
6019 Acked Flush 0 Sent Done Done false
6020 Acked Write 9 Sent Done Done false
6021 Acked Read 5 Sent Done Done false
6022 Acked Read 6 Sent Done Done false
6023 Acked Write 8 Sent Done Done false
6024 Acked Write 1 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 1 Sent Done Done false
6027 Acked Read 6 Sent Done Done false
6028 Acked Write 9 Sent Done Done false
6029 Acked Read 6 Sent Done Done false
6030 Acked Read 1 Sent Done Done false
6031 Acked Read 10 Sent Done Done false
6032 Acked Read 10 Sent Done Done false
6033 Acked Read 6 Sent Done Done false
6034 Acked Write 7 Sent Done Done false
6035 Acked Read 7 Sent Done Done false
6036 Acked Read 1 Sent Done Done false
6037 Acked Write 10 Sent Done Done false
6038 Acked Read 3 Sent Done Done false
6039 Acked Read 1 Sent Done Done false
6040 Acked Write 4 Sent Done Done false
6041 Acked Read 4 Sent Done Done false
6042 Acked Write 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 10 Sent Done Done false
6045 Acked Read 3 Sent Done Done false
6046 Acked Read 5 Sent Done Done false
6047 Acked Read 3 Sent Done Done false
6048 Acked Write 1 Sent Done Done false
6049 Acked Read 7 Sent Done Done false
6050 Acked Read 5 Sent Done Done false
6051 Acked Write 6 Sent Done Done false
6052 Acked Write 8 Sent Done Done false
6053 Acked Read 4 Sent Done Done false
6054 Acked Write 6 Sent Done Done false
6055 Acked Read 6 Sent Done Done false
6056 Acked Write 1 Sent Done Done false
6057 Acked Write 2 Sent Done Done false
6058 Acked Write 1 Sent Done Done false
6059 Acked Read 3 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 7 Sent Done Done false
6062 Acked Read 10 Sent Done Done false
6063 Acked Read 8 Sent Done Done false
6064 Acked Write 10 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 8 Sent Done Done false
6067 Acked Write 10 Sent Done Done false
6068 Acked Write 1 Sent Done Done false
6069 Acked Write 7 Sent Done Done false
6070 Acked Write 5 Sent Done Done false
6071 Acked Write 9 Sent Done Done false
6072 Acked Write 10 Sent Done Done false
6073 Acked Read 2 Sent Done Done false
6074 Acked Write 5 Sent Done Done false
6075 Acked Read 7 Sent Done Done false
6076 Acked Write 6 Sent Done Done false
6077 Acked Read 2 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Write 5 Sent Done Done false
6080 Acked Read 5 Sent Done Done false
6081 Acked Write 5 Sent Done Done false
6082 Acked Write 8 Sent Done Done false
6083 Acked Write 3 Sent Done Done false
6084 Acked Read 1 Sent Done Done false
6085 Acked Read 4 Sent Done Done false
6086 Acked Read 9 Sent Done Done false
6087 Acked Read 2 Sent Done Done false
6088 Acked Flush 0 Sent Done Done false
6089 Acked Read 4 Sent Done Done false
6090 Acked Write 8 Sent Done Done false
6091 Acked Read 9 Sent Done Done false
6092 Acked Read 5 Sent Done Done false
6093 Acked Write 2 Sent Done Done false
6094 Acked Read 6 Sent Done Done false
6095 Acked Read 5 Sent Done Done false
6096 Acked Read 6 Sent Done Done false
6097 Acked Read 5 Sent Done Done false
6098 Acked Flush 0 Sent Done Done false
6099 Acked Read 3 Sent Done Done false
6100 Acked Read 7 Sent Done Done false
6101 Acked Write 7 Sent Done Done false
6102 Acked Flush 0 Sent Done Done false
6103 Acked Read 10 Sent Done Done false
6104 Acked Read 5 Sent Done Done false
6105 Acked Flush 0 Sent Done Done false
6106 Acked Read 6 Sent Done Done false
6107 Acked Write 10 Sent Done Done false
6108 Acked Write 5 Sent Done Done false
6109 Acked Flush 0 Sent Done Done false
6110 Acked Write 3 Sent Done Done false
6111 Acked Read 10 Sent Done Done false
6112 Acked Flush 0 Sent Done Done false
6113 Acked Read 5 Sent Done Done false
6114 Acked Flush 0 Sent Done Done false
6115 Acked Write 8 Sent Done Done false
6116 Acked Read 8 Sent Done Done false
6117 Acked Write 1 Sent Done Done false
6118 Acked Read 5 Sent Done Done false
6119 Acked Write 2 Sent Done Done false
6120 Acked Read 3 Sent Done Done false
6121 Acked Read 2 Sent Done Done false
6122 Acked Flush 0 Sent Done Done false
6123 Acked Read 9 Sent Done Done false
6124 Acked Write 2 Sent Done Done false
6125 Acked Flush 0 Sent Done Done false
6126 Acked Write 8 Sent Done Done false
6127 Acked Write 2 Sent Done Done false
6128 Acked Read 5 Sent Done Done false
6129 Acked Read 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:37.130550358Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:37.801735254Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:38.803243156Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:39.885089115Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 1 Sent Done Done false
5830 Acked Write 9 Sent Done Done false
5831 Acked Write 2 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 3 Sent Done Done false
5834 Acked Read 6 Sent Done Done false
5835 Acked Flush 0 Sent Done Done false
5836 Acked Read 7 Sent Done Done false
5837 Acked Flush 0 Sent Done Done false
5838 Acked Write 10 Sent Done Done false
5839 Acked Read 5 Sent Done Done false
5840 Acked Read 3 Sent Done Done false
5841 Acked Write 5 Sent Done Done false
5842 Acked Read 2 Sent Done Done false
5843 Acked Write 5 Sent Done Done false
5844 Acked Read 2 Sent Done Done false
5845 Acked Read 4 Sent Done Done false
5846 Acked Read 4 Sent Done Done false
5847 Acked Write 3 Sent Done Done false
5848 Acked Write 10 Sent Done Done false
5849 Acked Read 2 Sent Done Done false
5850 Acked Write 2 Sent Done Done false
5851 Acked Write 4 Sent Done Done false
5852 Acked Read 2 Sent Done Done false
5853 Acked Write 10 Sent Done Done false
5854 Acked Write 5 Sent Done Done false
5855 Acked Read 2 Sent Done Done false
5856 Acked Read 5 Sent Done Done false
5857 Acked Read 5 Sent Done Done false
5858 Acked Read 10 Sent Done Done false
5859 Acked Read 3 Sent Done Done false
5860 Acked Write 10 Sent Done Done false
5861 Acked Write 8 Sent Done Done false
5862 Acked Read 8 Sent Done Done false
5863 Acked Write 10 Sent Done Done false
5864 Acked Read 1 Sent Done Done false
5865 Acked Write 4 Sent Done Done false
5866 Acked Write 9 Sent Done Done false
5867 Acked Read 9 Sent Done Done false
5868 Acked Read 6 Sent Done Done false
5869 Acked Flush 0 Sent Done Done false
5870 Acked Flush 0 Sent Done Done false
5871 Acked Flush 0 Sent Done Done false
5872 Acked Read 6 Sent Done Done false
5873 Acked Read 5 Sent Done Done false
5874 Acked Write 10 Sent Done Done false
5875 Acked Read 9 Sent Done Done false
5876 Acked Read 7 Sent Done Done false
5877 Acked Write 7 Sent Done Done false
5878 Acked Write 8 Sent Done Done false
5879 Acked Write 1 Sent Done Done false
5880 Acked Write 8 Sent Done Done false
5881 Acked Write 1 Sent Done Done false
5882 Acked Write 3 Sent Done Done false
5883 Acked Flush 0 Sent Done Done false
5884 Acked Read 5 Sent Done Done false
5885 Acked Read 2 Sent Done Done false
5886 Acked Read 2 Sent Done Done false
5887 Acked Read 10 Sent Done Done false
5888 Acked Flush 0 Sent Done Done false
5889 Acked Read 8 Sent Done Done false
5890 Acked Read 3 Sent Done Done false
5891 Acked Write 9 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Read 6 Sent Done Done false
5894 Acked Write 2 Sent Done Done false
5895 Acked Read 7 Sent Done Done false
5896 Acked Write 4 Sent Done Done false
5897 Acked Flush 0 Sent Done Done false
5898 Acked Read 9 Sent Done Done false
5899 Acked Flush 0 Sent Done Done false
5900 Acked Read 1 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Flush 0 Sent Done Done false
5903 Acked Read 4 Sent Done Done false
5904 Acked Write 9 Sent Done Done false
5905 Acked Read 3 Sent Done Done false
5906 Acked Read 9 Sent Done Done false
5907 Acked Write 9 Sent Done Done false
5908 Acked Flush 0 Sent Done Done false
5909 Acked Write 5 Sent Done Done false
5910 Acked Write 8 Sent Done Done false
5911 Acked Read 9 Sent Done Done false
5912 Acked Write 6 Sent Done Done false
5913 Acked Write 5 Sent Done Done false
5914 Acked Read 5 Sent Done Done false
5915 Acked Read 7 Sent Done Done false
5916 Acked Write 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Flush 0 Sent Done Done false
5919 Acked Read 3 Sent Done Done false
5920 Acked Write 4 Sent Done Done false
5921 Acked Flush 0 Sent Done Done false
5922 Acked Read 9 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Write 6 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Flush 0 Sent Done Done false
5927 Acked Write 1 Sent Done Done false
5928 Acked Write 2 Sent Done Done false
5929 Acked Flush 0 Sent Done Done false
5930 Acked Write 3 Sent Done Done false
5931 Acked Read 7 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Write 6 Sent Done Done false
5934 Acked Read 10 Sent Done Done false
5935 Acked Write 5 Sent Done Done false
5936 Acked Write 8 Sent Done Done false
5937 Acked Write 3 Sent Done Done false
5938 Acked Read 8 Sent Done Done false
5939 Acked Read 7 Sent Done Done false
5940 Acked Write 2 Sent Done Done false
5941 Acked Write 5 Sent Done Done false
5942 Acked Write 6 Sent Done Done false
5943 Acked Read 8 Sent Done Done false
5944 Acked Read 1 Sent Done Done false
5945 Acked Read 4 Sent Done Done false
5946 Acked Flush 0 Sent Done Done false
5947 Acked Write 10 Sent Done Done false
5948 Acked Write 9 Sent Done Done false
5949 Acked Write 1 Sent Done Done false
5950 Acked Write 8 Sent Done Done false
5951 Acked Read 7 Sent Done Done false
5952 Acked Read 10 Sent Done Done false
5953 Acked Read 5 Sent Done Done false
5954 Acked Write 8 Sent Done Done false
5955 Acked Flush 0 Sent Done Done false
5956 Acked Read 9 Sent Done Done false
5957 Acked Read 6 Sent Done Done false
5958 Acked Flush 0 Sent Done Done false
5959 Acked Write 4 Sent Done Done false
5960 Acked Read 9 Sent Done Done false
5961 Acked Write 6 Sent Done Done false
5962 Acked Read 6 Sent Done Done false
5963 Acked Write 1 Sent Done Done false
5964 Acked Read 4 Sent Done Done false
5965 Acked Read 5 Sent Done Done false
5966 Acked Write 6 Sent Done Done false
5967 Acked Read 1 Sent Done Done false
5968 Acked Read 7 Sent Done Done false
5969 Acked Write 5 Sent Done Done false
5970 Acked Write 5 Sent Done Done false
5971 Acked Read 4 Sent Done Done false
5972 Acked Read 7 Sent Done Done false
5973 Acked Read 3 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Write 3 Sent Done Done false
5976 Acked Write 1 Sent Done Done false
5977 Acked Read 10 Sent Done Done false
5978 Acked Read 8 Sent Done Done false
5979 Acked Read 4 Sent Done Done false
5980 Acked Write 6 Sent Done Done false
5981 Acked Write 1 Sent Done Done false
5982 Acked Read 5 Sent Done Done false
5983 Acked Write 5 Sent Done Done false
5984 Acked Write 6 Sent Done Done false
5985 Acked Read 7 Sent Done Done false
5986 Acked Read 3 Sent Done Done false
5987 Acked Write 9 Sent Done Done false
5988 Acked Read 6 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 7 Sent Done Done false
5991 Acked Write 6 Sent Done Done false
5992 Acked Read 8 Sent Done Done false
5993 Acked Read 3 Sent Done Done false
5994 Acked Read 8 Sent Done Done false
5995 Acked Read 7 Sent Done Done false
5996 Acked Write 7 Sent Done Done false
5997 Acked Write 4 Sent Done Done false
5998 Acked Write 10 Sent Done Done false
5999 Acked Read 6 Sent Done Done false
6000 Acked Read 9 Sent Done Done false
6001 Acked Write 3 Sent Done Done false
6002 Acked Read 2 Sent Done Done false
6003 Acked Write 10 Sent Done Done false
6004 Acked Write 7 Sent Done Done false
6005 Acked Write 5 Sent Done Done false
6006 Acked Write 6 Sent Done Done false
6007 Acked Read 7 Sent Done Done false
6008 Acked Write 8 Sent Done Done false
6009 Acked Read 4 Sent Done Done false
6010 Acked Read 5 Sent Done Done false
6011 Acked Write 6 Sent Done Done false
6012 Acked Read 2 Sent Done Done false
6013 Acked Read 2 Sent Done Done false
6014 Acked Write 3 Sent Done Done false
6015 Acked Read 10 Sent Done Done false
6016 Acked Write 4 Sent Done Done false
6017 Acked Write 2 Sent Done Done false
6018 Acked Read 3 Sent Done Done false
6019 Acked Flush 0 Sent Done Done false
6020 Acked Write 9 Sent Done Done false
6021 Acked Read 5 Sent Done Done false
6022 Acked Read 6 Sent Done Done false
6023 Acked Write 8 Sent Done Done false
6024 Acked Write 1 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 1 Sent Done Done false
6027 Acked Read 6 Sent Done Done false
6028 Acked Write 9 Sent Done Done false
6029 Acked Read 6 Sent Done Done false
6030 Acked Read 1 Sent Done Done false
6031 Acked Read 10 Sent Done Done false
6032 Acked Read 10 Sent Done Done false
6033 Acked Read 6 Sent Done Done false
6034 Acked Write 7 Sent Done Done false
6035 Acked Read 7 Sent Done Done false
6036 Acked Read 1 Sent Done Done false
6037 Acked Write 10 Sent Done Done false
6038 Acked Read 3 Sent Done Done false
6039 Acked Read 1 Sent Done Done false
6040 Acked Write 4 Sent Done Done false
6041 Acked Read 4 Sent Done Done false
6042 Acked Write 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 10 Sent Done Done false
6045 Acked Read 3 Sent Done Done false
6046 Acked Read 5 Sent Done Done false
6047 Acked Read 3 Sent Done Done false
6048 Acked Write 1 Sent Done Done false
6049 Acked Read 7 Sent Done Done false
6050 Acked Read 5 Sent Done Done false
6051 Acked Write 6 Sent Done Done false
6052 Acked Write 8 Sent Done Done false
6053 Acked Read 4 Sent Done Done false
6054 Acked Write 6 Sent Done Done false
6055 Acked Read 6 Sent Done Done false
6056 Acked Write 1 Sent Done Done false
6057 Acked Write 2 Sent Done Done false
6058 Acked Write 1 Sent Done Done false
6059 Acked Read 3 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 7 Sent Done Done false
6062 Acked Read 10 Sent Done Done false
6063 Acked Read 8 Sent Done Done false
6064 Acked Write 10 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 8 Sent Done Done false
6067 Acked Write 10 Sent Done Done false
6068 Acked Write 1 Sent Done Done false
6069 Acked Write 7 Sent Done Done false
6070 Acked Write 5 Sent Done Done false
6071 Acked Write 9 Sent Done Done false
6072 Acked Write 10 Sent Done Done false
6073 Acked Read 2 Sent Done Done false
6074 Acked Write 5 Sent Done Done false
6075 Acked Read 7 Sent Done Done false
6076 Acked Write 6 Sent Done Done false
6077 Acked Read 2 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Write 5 Sent Done Done false
6080 Acked Read 5 Sent Done Done false
6081 Acked Write 5 Sent Done Done false
6082 Acked Write 8 Sent Done Done false
6083 Acked Write 3 Sent Done Done false
6084 Acked Read 1 Sent Done Done false
6085 Acked Read 4 Sent Done Done false
6086 Acked Read 9 Sent Done Done false
6087 Acked Read 2 Sent Done Done false
6088 Acked Flush 0 Sent Done Done false
6089 Acked Read 4 Sent Done Done false
6090 Acked Write 8 Sent Done Done false
6091 Acked Read 9 Sent Done Done false
6092 Acked Read 5 Sent Done Done false
6093 Acked Write 2 Sent Done Done false
6094 Acked Read 6 Sent Done Done false
6095 Acked Read 5 Sent Done Done false
6096 Acked Read 6 Sent Done Done false
6097 Acked Read 5 Sent Done Done false
6098 Acked Flush 0 Sent Done Done false
6099 Acked Read 3 Sent Done Done false
6100 Acked Read 7 Sent Done Done false
6101 Acked Write 7 Sent Done Done false
6102 Acked Flush 0 Sent Done Done false
6103 Acked Read 10 Sent Done Done false
6104 Acked Read 5 Sent Done Done false
6105 Acked Flush 0 Sent Done Done false
6106 Acked Read 6 Sent Done Done false
6107 Acked Write 10 Sent Done Done false
6108 Acked Write 5 Sent Done Done false
6109 Acked Flush 0 Sent Done Done false
6110 Acked Write 3 Sent Done Done false
6111 Acked Read 10 Sent Done Done false
6112 Acked Flush 0 Sent Done Done false
6113 Acked Read 5 Sent Done Done false
6114 Acked Flush 0 Sent Done Done false
6115 Acked Write 8 Sent Done Done false
6116 Acked Read 8 Sent Done Done false
6117 Acked Write 1 Sent Done Done false
6118 Acked Read 5 Sent Done Done false
6119 Acked Write 2 Sent Done Done false
6120 Acked Read 3 Sent Done Done false
6121 Acked Read 2 Sent Done Done false
6122 Acked Flush 0 Sent Done Done false
6123 Acked Read 9 Sent Done Done false
6124 Acked Write 2 Sent Done Done false
6125 Acked Flush 0 Sent Done Done false
6126 Acked Write 8 Sent Done Done false
6127 Acked Write 2 Sent Done Done false
6128 Acked Read 5 Sent Done Done false
6129 Acked Read 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:40.807191727Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:42.132130647Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:42.811437255Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:43.813488117Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.884763525Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63838)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.885001348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.885985563Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.886549188Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.886569992Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(5828))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.886583809Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.886601875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(5828))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.886615332Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(5828))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:43.88691476Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6126 Acked Write 8 Done Done Done true
6127 Acked Write 2 Done Done Done true
6128 Acked Read 5 Done Done Done true
6129 Acked Read 5 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(6125)) Some(JobId(6125)) Some(JobId(6125))
Downstairs last five completed: 6125 6124 6123 6122 6121
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:4 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6130 6129 6128 6127 6126
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:05:48.138 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0[8810] Got stop action so:false kr:false
, method: POST, req_id: deb6970b-f511-4973-8371-0585464063f2, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:48.139219456Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:48.139268364Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:48.13928522Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:05:48.139 INFO request completed, latency_us: 143, response_code:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:48.139384317Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
200, uri: /state/cid/0, method: GET, req_id: 2ebecbeb-dc79-4a8c-b558-38da0b3c2e77, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:05:52.141 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/0, method: GET, req_id: f77cd126-d788-46f0-b5bb-3eacd146d0f5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 3985 len 8192
002/300 Read block 4255 len 16384
003/300 Flush
004/300 Read block 1637 len 8192
005/300 Write block 3919 len 16384 data: 5 5 4 3
006/300 Read block 1575 len 36864
007/300 Flush
008/300 Write block 1120 len 36864 data: 3 2 3 3 4 6 6 6 6
009/300 Write block 3428 len 36864 data: 3 3 4 4 4 3 4 6 7
010/300 Flush
011/300 Read block 2349 len 4096
012/300 Flush
013/300 Read block 2131 len 28672
014/300 Write block 3325 len 20480 data: 4 3 3 2 2
015/300 Write block 1554 len 16384 data: 3 3 3 4
016/300 Write block 295 len 4096 data: 5
017/300 Write block 1967 len 40960 data: 4 3 4 4 5 5 5 5 5 3
018/300 Write block 3585 len 20480 data: 3 3 3 3 4
019/300 Flush
020/300 Write block 4303 len 28672 data: 2 2 2 3 4 3 3
021/300 Read block 4595 len 16384
022/300 Write block 4776 len 20480 data: 6 7 5 5 6
023/300 Read block 1340 len 32768
024/300 Read block 909 len 24576
025/300 Read block 2552 len 32768
026/300 Write block 4347 len 28672 data: 8 7 7 6 6 6 7
027/300 Read block 1814 len 4096
028/300 Read block 214 len 20480
029/300 Read block 1603 len 36864
030/300 Write block 4882 len 24576 data: 6 7 6 6 6 8
031/300 Write block 4813 len 24576 data: 3 2 3 3 3 4
032/300 Read block 4754 len 36864
033/300 Read block 4318 len 20480
034/300 Read block 1617 len 8192
035/300 Read block 1253 len 8192
036/300 Write block 4075 len 12288 data: 3 5 6
037/300 Write block 3974 len 40960 data: 5 5 4 4 4 4 3 3 3 2
038/300 Read block 1723 len 20480
039/300 Flush
040/300 Flush
041/300 Read block 74 len 12288
042/300 Write block 4730 len 24576 data: 6 6 6 5 5 4
043/300 Write block 719 len 12288 data: 4 5 5
044/300 Read block 801 len 32768
045/300 Write block 4783 len 16384 data: 5 4 4 4
046/300 Read block 4768 len 36864
047/300 Write block 813 len 8192 data: 7 6
048/300 Read block 3509 len 4096
049/300 Write block 4203 len 24576 data: 3 5 4 5 3 3
050/300 Read block 4260 len 36864
051/300 Read block 2483 len 32768
052/300 Read block 225 len 36864
053/300 Write block 20 len 28672 data: 4 4 4 4 4 6 4
054/300 Read block 3297 len 16384
055/300 Write block 234 len 16384 data: 5 4 5 7
056/300 Read block 3588 len 36864
057/300 Flush
058/300 Write block 1982 len 32768 data: 5 6 6 5 5 5 6 6
059/300 Read block 4581 len 32768
060/300 Read block 153 len 20480
061/300 Read block 1719 len 20480
062/300 Read block 3780 len 36864
063/300 Read block 4231 len 32768
064/300 Flush
065/300 Write block 3823 len 28672 data: 3 2 2 3 3 3 4
066/300 Flush
067/300 Flush
068/300 Flush
069/300 Read block 1399 len 36864
070/300 Write block 3986 len 36864 data: 5 5 3 3 3 3 3 4 4
071/300 Write block 3847 len 24576 data: 5 6 5 7 7 8
072/300 Read block 3819 len 12288
073/300 Write block 3662 len 12288 data: 3 3 3
074/300 Write block 2716 len 16384 data: 7 7 5 3
075/300 Write block 1716 len 36864 data: 4 4 3 2 3 4 5 6 6
076/300 Write block 3729 len 36864 data: 3 3 5 4 5 6 6 4 4
077/300 Write block 465 len 28672 data: 2 3 4 5 5 6 4
078/300 Write block 2172 len 36864 data: 4 3 4 4 5 5 5 7 5
079/300 Flush
080/300 Read block 2247 len 36864
081/300 Read block 4702 len 28672
082/300 Read block 1810 len 20480
083/300 Read block 421 len 4096
084/300 Write block 3340 len 36864 data: 5 6 5 6 6 6 6 6 6
085/300 Write block 4701 len 32768 data: 5 4 4 2 2 2 2 2
086/300 Flush
087/300 Write block 4066 len 24576 data: 5 4 4 4 4 4
088/300 Write block 423 len 4096 data: 7
089/300 Read block 4091 len 4096
090/300 Write block 1319 len 40960 data: 3 4 4 5 5 4 4 5 5 5
091/300 Read block 972 len 28672
092/300 Write block 4768 len 24576 data: 4 5 8 7 6 5
093/300 Read block 811 len 4096
094/300 Read block 3568 len 16384
095/300 Read block 2401 len 4096
096/300 Read block 4365 len 24576
097/300 Read block 330 len 4096
098/300 Read block 2056 len 8192
099/300 Flush
100/300 Read block 1558 len 32768
101/300 Flush
102/300 Read block 3245 len 28672
103/300 Read block 1322 len 32768
104/300 Read block 565 len 20480
105/300 Read block 894 len 4096
106/300 Read block 4158 len 28672
107/300 Write block 4967 len 8192 data: 5 5
108/300 Read block 2127 len 32768
109/300 Read block 426 len 4096
110/300 Write block 2685 len 12288 data: 8 7 8
111/300 Write block 3564 len 4096 data: 4
112/300 Read block 3645 len 24576
113/300 Read block 1225 len 8192
114/300 Write block 4114 len 28672 data: 5 5 6 6 4 4 3
115/300 Write block 848 len 8192 data: 2 2
116/300 Read block 778 len 32768
117/300 Flush
118/300 Write block 1877 len 8192 data: 6 6
119/300 Read block 1582 len 4096
120/300 Write block 4712 len 8192 data: 4 4
121/300 Read block 3247 len 8192
122/300 Write block 2818 len 20480 data: 5 6 5 4 3
123/300 Write block 3848 len 12288 data: 7 6 8
124/300 Write block 3529 len 32768 data: 6 4 4 4 4 4 4 6
125/300 Read block 1445 len 24576
126/300 Read block 4133 len 32768
127/300 Write block 1788 len 32768 data: 3 4 4 4 5 4 4 5
128/300 Write block 2045 len 12288 data: 3 3 3
129/300 Read block 3741 len 8192
130/300 Flush
131/300 Write block 4968 len 36864 data: 6 5 5 6 5 4 4 3 3
132/300 Read block 1495 len 28672
133/300 Read block 1494 len 28672
134/300 Read block 502 len 12288
135/300 Write block 3838 len 16384 data: 3 3 4 4
136/300 Read block 1675 len 24576
137/300 Read block 174 len 24576
138/300 Read block 2981 len 16384
139/300 Read block 3565 len 40960
140/300 Write block 4615 len 4096 data: 4
141/300 Write block 1436 len 8192 data: 4 5
142/300 Write block 217 len 32768 data: 5 6 8 5 6 5 5 4
143/300 Write block 549 len 16384 data: 4 5 6 7
144/300 Read block 2594 len 4096
145/300 Write block 3831 len 40960 data: 3 3 3 3 3 2 2 4 4 5
146/300 Write block 1874 len 36864 data: 5 5 5 7 7 5 5 5 4
147/300 Write block 292 len 8192 data: 5 4
148/300 Read block 2638 len 16384
149/300 Flush
150/300 Read block 3120 len 16384
151/300 Read block 1430 len 12288
152/300 Read block 2015 len 16384
153/300 Read block 1579 len 40960
154/300 Write block 3627 len 12288 data: 4 4 4
155/300 Write block 1676 len 36864 data: 5 4 4 4 4 4 6 7 7
156/300 Read block 4132 len 8192
157/300 Read block 2277 len 4096
158/300 Read block 773 len 8192
159/300 Write block 3328 len 36864 data: 3 3 3 3 3 2 3 5 5
160/300 Read block 989 len 12288
161/300 Write block 692 len 40960 data: 4 3 3 3 2 3 3 3 4 5
162/300 Read block 3211 len 28672
163/300 Write block 1151 len 24576 data: 5 6 7 6 7 6
164/300 Flush
165/300 Write block 2254 len 4096 data: 3
166/300 Read block 4104 len 40960
167/300 Read block 763 len 20480
168/300 Read block 1888 len 24576
169/300 Read block 2823 len 20480
170/300 Write block 1164 len 16384 data: 5 5 5 5
171/300 Read block 4715 len 16384
172/300 Flush
173/300 Read block 1708 len 28672
174/300 Read block 2791 len 4096
175/300 Write block 3676 len 12288 data: 2 2 2
176/300 Write block 4709 len 40960 data: 3 3 4 5 5 4 4 4 4 4
177/300 Read block 3956 len 8192
178/300 Read block 4542 len 8192
179/300 Read block 2532 len 12288
180/300 Write block 590 len 40960 data: 2 2 2 2 2 2 2 3 4 5
181/300 Write block 1157 len 12288 data: 6 5 5
182/300 Write block 3942 len 12288 data: 5 4 5
183/300 Read block 723 len 36864
184/300 Read block 2083 len 8192
185/300 Write block 1246 len 16384 data: 2 4 3 4
186/300 Read block 2752 len 24576
187/300 Read block 3637 len 36864
188/300 Read block 4143 len 32768
189/300 Read block 1438 len 16384
190/300 Read block 2819 len 20480
191/300 Read block 3368 len 24576
192/300 Read block 899 len 28672
193/300 Flush
194/300 Read block 1301 len 12288
195/300 Flush
196/300 Read block 4785 len 28672
197/300 Read block 4082 len 36864
198/300 Write block 2102 len 4096 data: 4
199/300 Flush
200/300 Flush
201/300 Read block 280 len 4096
202/300 Flush
203/300 Read block 4139 len 16384
204/300 Read block 861 len 12288
205/300 Write block 3001 len 24576 data: 4 2 2 2 2 3
206/300 Write block 1471 len 36864 data: 3 3 4 4 2 2 3 3 3
207/300 Write block 557 len 40960 data: 3 4 4 4 4 4 4 4 4 2
208/300 Read block 4410 len 4096
209/300 Flush
210/300 Read block 3109 len 36864
211/300 Read block 3996 len 32768
212/300 Write block 3363 len 8192 data: 3 4
213/300 Read block 1142 len 8192
214/300 Flush
215/300 Write block 2687 len 4096 data: 9
216/300 Write block 4320 len 20480 data: 5 6 5 6 5
217/300 Read block 4871 len 28672
218/300 Flush
219/300 Read block 4437 len 16384
220/300 Read block 3007 len 4096
221/300 Flush
222/300 Read block 1665 len 12288
223/300 Write block 2692 len 16384 data: 5 5 5 5
224/300 Read block 4092 len 36864
225/300 Flush
226/300 Write block 2986 len 36864 data: 5 5 5 5 7 7 8 5 4
227/300 Flush
228/300 Read block 2288 len 28672
229/300 Flush
230/300 Write block 2370 len 4096 data: 3
231/300 Read block 1237 len 36864
232/300 Write block 4716 len 12288 data: 5 5 5
233/300 Read block 1076 len 12288
234/300 Read block 3562 len 4096
235/300 Write block 1240 len 20480 data: 3 3 3 2 2
236/300 Read block 4159 len 24576
237/300 Read block 2711 len 32768
238/300 Flush
239/300 Write block 4864 len 28672 data: 4 4 3 4 4 3 3
240/300 Read block 2942 len 28672
241/300 Read block 922 len 16384
242/300 Write block 2600 len 12288 data: 5 5 4
243/300 Read block 2737 len 24576
244/300 Read block 2100 len 12288
245/300 Flush
246/300 Flush
247/300 Write block 1754 len 32768 data: 4 4 3 3 4 4 4 4
248/300 Flush
249/300 Write block 3156 len 24576 data: 7 8 8 9 7 4
250/300 Read block 584 len 32768
251/300 Read block 2530 len 8192
252/300 Read block 3458 len 28672
253/300 Write block 2764 len 24576 data: 4 2 2 2 2 3
254/300 Read block 4766 len 40960
255/300 Read block 2469 len 40960
256/300 Read block 4300 len 20480
257/300 Write block 3979 len 4096 data: 5
258/300 Write block 4550 len 24576 data: 3 3 2 2 2 2
259/300 Write block 4470 len 4096 data: 4
260/300 Read block 1576 len 12288
261/300 Read block 3079 len 40960
262/300 Read block 3515 len 16384
263/300 Read block 172 len 20480
264/300 Read block 2901 len 24576
265/300 Write block 474 len 40960 data: 5 6 7 6 7 4 5 5 5 7
266/300 Read block 1428 len 28672
267/300 Read block 4432 len 40960
268/300 Read block 284 len 8192
269/300 Write block 3593 len 32768 data: 4 3 3 3 4 3 3 2
270/300 Write block 206 len 12288 data: 3 4 5
271/300 Read block 419 len 20480
272/300 Read block 4718 len 28672
273/300 Write block 498 len 12288 data: 5 5 5
274/300 Write block 2501 len 36864 data: 3 3 4 2 2 2 2 2 3
275/300 Write block 1195 len 24576 data: 3 3 4 5 4 3
276/300 Read block 4025 len 28672
277/300 Write block 1187 len 32768 data: 4 3 3 3 3 3 3 3
278/300 Read block 3687 len 24576
279/300 Write block 4887 len 32768 data: 9 8 9 8 7 5 5 5
280/300 Write block 2288 len 40960 data: 2 2 3 5 4 4 2 2 2 2
281/300 Write block 4394 len 24576 data: 5 5 5 3 2 2
282/300 Flush
283/300 Write block 1273 len 16384 data: 6 6 4 4
284/300 Write block 1055 len 32768 data: 6 5 4 4 4 4 3 2
285/300 Read block 1292 len 20480
286/300 Write block 2921 len 40960 data: 4 4 6 6 6 6 4 2 3 3
287/300 Read block 2109 len 8192
288/300 Read block 1706 len 8192
289/300 Read block 1051 len 8192
290/300 Write block 2669 len 12288 data: 8 8 5
291/300 Write block 2184 len 28672 data: 6 6 7 5 6 6 7
292/300 Flush
293/300 Write block 4958 len 16384 data: 6 5 4 5
294/300 Write block 2021 len 40960 data: 5 3 3 3 3 4 4 6 6 5
295/300 Read block 164 len 12288
296/300 Write block 1467 len 40960 data: 4 4 4 3 4 4 5 5 3 3
297/300 Read block 4230 len 4096
298/300 Write block 2085 len 4096 data: 4
299/300 Write block 334 len 20480 data: 3 3 4 5 5
300/300 Read block 1112 len 24576
301/300 Write block 3343 len 32768 data: 7 7 7 7 7 7 5 6
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:05:52.376 INFO request completed, latency_us: 174, response_code: 204, uri: /start/cid/0, method: POST, req_id: 610ad222-6002-4f5c-aeaf-90e3d7435f4a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Read 2 Sent Done Done false
6132 Acked Read 4 Sent Done Done false
6133 Acked Flush 0 Sent Done Done false
6134 Acked Read 2 Sent Done Done false
6135 Acked Write 4 Sent Done Done false
6136 Acked Read 9 Sent Done Done false
6137 Acked Flush 0 Sent Done Done false
6138 Acked Write 9 Sent Done Done false
6139 Acked Write 9 Sent Done Done false
6140 Acked Flush 0 Sent Done Done false
6141 Acked Read 1 Sent Done Done false
6142 Acked Flush 0 Sent Done Done false
6143 Acked Read 7 Sent Done Done false
6144 Acked Write 5 Sent Done Done false
6145 Acked Write 4 Sent Done Done false
6146 Acked Write 1 Sent Done Done false
6147 Acked Write 10 Sent Done Done false
6148 Acked Write 5 Sent Done Done false
6149 Acked Flush 0 Sent Done Done false
6150 Acked Write 7 Sent Done Done false
6151 Acked Read 4 Sent Done Done false
6152 Acked Write 5 Sent Done Done false
6153 Acked Read 8 Sent Done Done false
6154 Acked Read 6 Sent Done Done false
6155 Acked Read 8 Sent Done Done false
6156 Acked Write 7 Sent Done Done false
6157 Acked Read 1 Sent Done Done false
6158 Acked Read 5 Sent Done Done false
6159 Acked Read 9 Sent Done Done false
6160 Acked Write 6 Sent Done Done false
6161 Acked Write 6 Sent Done Done false
6162 Acked Read 9 Sent Done Done false
6163 Acked Read 5 Sent Done Done false
6164 Acked Read 2 Sent Done Done false
6165 Acked Read 2 Sent Done Done false
6166 Acked Write 3 Sent Done Done false
6167 Acked Write 10 Sent Done Done false
6168 Acked Read 5 Sent Done Done false
6169 Acked Flush 0 Sent Done Done false
6170 Acked Flush 0 Sent Done Done false
6171 Acked Read 3 Sent Done Done false
6172 Acked Write 6 Sent Done Done false
6173 Acked Write 3 Sent Done Done false
6174 Acked Read 8 Sent Done Done false
6175 Acked Write 4 Sent Done Done false
6176 Acked Read 9 Sent Done Done false
6177 Acked Write 2 Sent Done Done false
6178 Acked Read 1 Sent Done Done false
6179 Acked Write 6 Sent Done Done false
6180 Acked Read 9 Sent Done Done false
6181 Acked Read 8 Sent Done Done false
6182 Acked Read 9 Sent Done Done false
6183 Acked Write 7 Sent Done Done false
6184 Acked Read 4 Sent Done Done false
6185 Acked Write 4 Sent Done Done false
6186 Acked Read 9 Sent Done Done false
6187 Acked Flush 0 Sent Done Done false
6188 Acked Write 8 Sent Done Done false
6189 Acked Read 8 Sent Done Done false
6190 Acked Read 5 Sent Done Done false
6191 Acked Read 5 Sent Done Done false
6192 Acked Read 9 Sent Done Done false
6193 Acked Read 8 Sent Done Done false
6194 Acked Flush 0 Sent Done Done false
6195 Acked Write 7 Sent Done Done false
6196 Acked Flush 0 Sent Done Done false
6197 Acked Flush 0 Sent Done Done false
6198 Acked Flush 0 Sent Done Done false
6199 Acked Read 9 Sent Done Done false
6200 Acked Write 9 Sent Done Done false
6201 Acked Write 6 Sent Done Done false
6202 Acked Read 3 Sent Done Done false
6203 Acked Write 3 Sent Done Done false
6204 Acked Write 4 Sent Done Done false
6205 Acked Write 9 Sent Done Done false
6206 Acked Write 9 Sent Done Done false
6207 Acked Write 7 Sent Done Done false
6208 Acked Write 9 Sent Done Done false
6209 Acked Flush 0 Sent Done Done false
6210 Acked Read 9 Sent Done Done false
6211 Acked Read 7 Sent Done Done false
6212 Acked Read 5 Sent Done Done false
6213 Acked Read 1 Sent Done Done false
6214 Acked Write 9 Sent Done Done false
6215 Acked Write 8 Sent Done Done false
6216 Acked Flush 0 Sent Done Done false
6217 Acked Write 6 Sent Done Done false
6218 Acked Write 1 Sent Done Done false
6219 Acked Read 1 Sent Done Done false
6220 Acked Write 10 Sent Done Done false
6221 Acked Read 7 Sent Done Done false
6222 Acked Write 6 Sent Done Done false
6223 Acked Read 1 Sent Done Done false
6224 Acked Read 4 Sent Done Done false
6225 Acked Read 1 Sent Done Done false
6226 Acked Read 6 Sent Done Done false
6227 Acked Read 1 Sent Done Done false
6228 Acked Read 2 Sent Done Done false
6229 Acked Flush 0 Sent Done Done false
6230 Acked Read 8 Sent Done Done false
6231 Acked Flush 0 Sent Done Done false
6232 Acked Read 7 Sent Done Done false
6233 Acked Read 8 Sent Done Done false
6234 Acked Read 5 Sent Done Done false
6235 Acked Read 1 Sent Done Done false
6236 Acked Read 7 Sent Done Done false
6237 Acked Write 2 Sent Done Done false
6238 Acked Read 8 Sent Done Done false
6239 Acked Read 1 Sent Done Done false
6240 Acked Write 3 Sent Done Done false
6241 Acked Write 1 Sent Done Done false
6242 Acked Read 6 Sent Done Done false
6243 Acked Read 2 Sent Done Done false
6244 Acked Write 7 Sent Done Done false
6245 Acked Write 2 Sent Done Done false
6246 Acked Read 8 Sent Done Done false
6247 Acked Flush 0 Sent Done Done false
6248 Acked Write 2 Sent Done Done false
6249 Acked Read 1 Sent Done Done false
6250 Acked Write 2 Sent Done Done false
6251 Acked Read 2 Sent Done Done false
6252 Acked Write 5 Sent Done Done false
6253 Acked Write 3 Sent Done Done false
6254 Acked Write 8 Sent Done Done false
6255 Acked Read 6 Sent Done Done false
6256 Acked Read 8 Sent Done Done false
6257 Acked Write 8 Sent Done Done false
6258 Acked Write 3 Sent Done Done false
6259 Acked Read 2 Sent Done Done false
6260 Acked Flush 0 Sent Done Done false
6261 Acked Write 9 Sent Done Done false
6262 Acked Read 7 Sent Done Done false
6263 Acked Read 7 Sent Done Done false
6264 Acked Read 3 Sent Done Done false
6265 Acked Write 4 Sent Done Done false
6266 Acked Read 6 Sent Done Done false
6267 Acked Read 6 Sent Done Done false
6268 Acked Read 4 Sent Done Done false
6269 Acked Read 10 Sent Done Done false
6270 Acked Write 1 Sent Done Done false
6271 Acked Write 2 Sent Done Done false
6272 Acked Write 8 Sent Done Done false
6273 Acked Write 4 Sent Done Done false
6274 Acked Read 1 Sent Done Done false
6275 Acked Write 10 Sent Done Done false
6276 Acked Write 9 Sent Done Done false
6277 Acked Write 2 Sent Done Done false
6278 Acked Read 4 Sent Done Done false
6279 Acked Flush 0 Sent Done Done false
6280 Acked Read 4 Sent Done Done false
6281 Acked Read 3 Sent Done Done false
6282 Acked Read 4 Sent Done Done false
6283 Acked Read 10 Sent Done Done false
6284 Acked Write 3 Sent Done Done false
6285 Acked Write 9 Sent Done Done false
6286 Acked Read 2 Sent Done Done false
6287 Acked Read 1 Sent Done Done false
6288 Acked Read 2 Sent Done Done false
6289 Acked Write 9 Sent Done Done false
6290 Acked Read 3 Sent Done Done false
6291 Acked Write 10 Sent Done Done false
6292 Acked Read 7 Sent Done Done false
6293 Acked Write 6 Sent Done Done false
6294 Acked Flush 0 Sent Done Done false
6295 Acked Write 1 Sent Done Done false
6296 Acked Read 10 Sent Done Done false
6297 Acked Read 5 Sent Done Done false
6298 Acked Read 6 Sent Done Done false
6299 Acked Read 5 Sent Done Done false
6300 Acked Write 4 Sent Done Done false
6301 Acked Read 4 Sent Done Done false
6302 Acked Flush 0 Sent Done Done false
6303 Acked Read 7 Sent Done Done false
6304 Acked Read 1 Sent Done Done false
6305 Acked Write 3 Sent Done Done false
6306 Acked Write 10 Sent Done Done false
6307 Acked Read 2 Sent Done Done false
6308 Acked Read 2 Sent Done Done false
6309 Acked Read 3 Sent Done Done false
6310 Acked Write 10 Sent Done Done false
6311 Acked Write 3 Sent Done Done false
6312 Acked Write 3 Sent Done Done false
6313 Acked Read 9 Sent Done Done false
6314 Acked Read 2 Sent Done Done false
6315 Acked Write 4 Sent Done Done false
6316 Acked Read 6 Sent Done Done false
6317 Acked Read 9 Sent Done Done false
6318 Acked Read 8 Sent Done Done false
6319 Acked Read 4 Sent Done Done false
6320 Acked Read 5 Sent Done Done false
6321 Acked Read 6 Sent Done Done false
6322 Acked Read 7 Sent Done Done false
6323 Acked Flush 0 Sent Done Done false
6324 Acked Read 3 Sent Done Done false
6325 Acked Flush 0 Sent Done Done false
6326 Acked Read 7 Sent Done Done false
6327 Acked Read 9 Sent Done Done false
6328 Acked Write 1 Sent Done Done false
6329 Acked Flush 0 Sent Done Done false
6330 Acked Flush 0 Sent Done Done false
6331 Acked Read 1 Sent Done Done false
6332 Acked Flush 0 Sent Done Done false
6333 Acked Read 4 Sent Done Done false
6334 Acked Read 3 Sent Done Done false
6335 Acked Write 6 Sent Done Done false
6336 Acked Write 9 Sent Done Done false
6337 Acked Write 10 Sent Done Done false
6338 Acked Read 1 Sent Done Done false
6339 Acked Flush 0 Sent Done Done false
6340 Acked Read 9 Sent Done Done false
6341 Acked Read 8 Sent Done Done false
6342 Acked Write 2 Sent Done Done false
6343 Acked Read 2 Sent Done Done false
6344 Acked Flush 0 Sent Done Done false
6345 Acked Write 1 Sent Done Done false
6346 Acked Write 5 Sent Done Done false
6347 Acked Read 7 Sent Done Done false
6348 Acked Flush 0 Sent Done Done false
6349 Acked Read 4 Sent Done Done false
6350 Acked Read 1 Sent Done Done false
6351 Acked Flush 0 Sent Done Done false
6352 Acked Read 3 Sent Done Done false
6353 Acked Write 4 Sent Done Done false
6354 Acked Read 9 Sent Done Done false
6355 Acked Flush 0 Sent Done Done false
6356 Acked Write 9 Sent Done Done false
6357 Acked Flush 0 Sent Done Done false
6358 Acked Read 7 Sent Done Done false
6359 Acked Flush 0 Sent Done Done false
6360 Acked Write 1 Sent Done Done false
6361 Acked Read 9 Sent Done Done false
6362 Acked Write 3 Sent Done Done false
6363 Acked Read 3 Sent Done Done false
6364 Acked Read 1 Sent Done Done false
6365 Acked Write 5 Sent Done Done false
6366 Acked Read 6 Sent Done Done false
6367 Acked Read 8 Sent Done Done false
6368 Acked Flush 0 Sent Done Done false
6369 Acked Write 7 Sent Done Done false
6370 Acked Read 7 Sent Done Done false
6371 Acked Read 4 Sent Done Done false
6372 Acked Write 3 Sent Done Done false
6373 Acked Read 6 Sent Done Done false
6374 Acked Read 3 Sent Done Done false
6375 Acked Flush 0 Sent Done Done false
6376 Acked Flush 0 Sent Done Done false
6377 Acked Write 8 Sent Done Done false
6378 Acked Flush 0 Sent Done Done false
6379 Acked Write 6 Sent Done Done false
6380 Acked Read 8 Sent Done Done false
6381 Acked Read 2 Sent Done Done false
6382 Acked Read 7 Sent Done Done false
6383 Acked Write 6 Sent Done Done false
6384 Acked Read 10 Sent Done Done false
6385 Acked Read 10 Sent Done Done false
6386 Acked Read 5 Sent Done Done false
6387 Acked Write 1 Sent Done Done false
6388 Acked Write 6 Sent Done Done false
6389 Acked Write 1 Sent Done Done false
6390 Acked Read 3 Sent Done Done false
6391 Acked Read 10 Sent Done Done false
6392 Acked Read 4 Sent Done Done false
6393 Acked Read 5 Sent Done Done false
6394 Acked Read 6 Sent Done Done false
6395 Acked Write 10 Sent Done Done false
6396 Acked Read 7 Sent Done Done false
6397 Acked Read 10 Sent Done Done false
6398 Acked Read 2 Sent Done Done false
6399 Acked Write 8 Sent Done Done false
6400 Acked Write 3 Sent Done Done false
6401 Acked Read 5 Sent Done Done false
6402 Acked Read 7 Sent Done Done false
6403 Acked Write 3 Sent Done Done false
6404 Acked Write 9 Sent Done Done false
6405 Acked Write 6 Sent Done Done false
6406 Acked Read 7 Sent Done Done false
6407 Acked Write 8 Sent Done Done false
6408 Acked Read 6 Sent Done Done false
6409 Acked Write 8 Sent Done Done false
6410 Acked Write 10 Sent Done Done false
6411 Acked Write 6 Sent Done Done false
6412 Acked Flush 0 Sent Done Done false
6413 Acked Write 4 Sent Done Done false
6414 Acked Write 8 Sent Done Done false
6415 Acked Read 5 Sent Done Done false
6416 Acked Write 10 Sent Done Done false
6417 Acked Read 2 Sent Done Done false
6418 Acked Read 2 Sent Done Done false
6419 Acked Read 2 Sent Done Done false
6420 Acked Write 3 Sent Done Done false
6421 Acked Write 7 Sent Done Done false
6422 Acked Flush 0 Sent Done Done false
6423 Acked Write 4 Sent Done Done false
6424 Acked Write 10 Sent Done Done false
6425 Acked Read 3 Sent Done Done false
6426 Acked Write 10 Sent Done Done false
6427 Acked Read 1 Sent Done Done false
6428 Acked Write 1 Sent Done Done false
6429 Acked Write 5 Sent Done Done false
6430 Acked Read 6 Sent Done Done false
6431 Acked Write 8 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(6130)) Some(JobId(6422)) Some(JobId(6422))
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6431 6430 6429 6428 6427
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:52.878776688Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:53.828952143Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:54.887883293Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:55.833102584Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1229)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Read 2 Sent Done Done false
6132 Acked Read 4 Sent Done Done false
6133 Acked Flush 0 Sent Done Done false
6134 Acked Read 2 Sent Done Done false
6135 Acked Write 4 Sent Done Done false
6136 Acked Read 9 Sent Done Done false
6137 Acked Flush 0 Sent Done Done false
6138 Acked Write 9 Sent Done Done false
6139 Acked Write 9 Sent Done Done false
6140 Acked Flush 0 Sent Done Done false
6141 Acked Read 1 Sent Done Done false
6142 Acked Flush 0 Sent Done Done false
6143 Acked Read 7 Sent Done Done false
6144 Acked Write 5 Sent Done Done false
6145 Acked Write 4 Sent Done Done false
6146 Acked Write 1 Sent Done Done false
6147 Acked Write 10 Sent Done Done false
6148 Acked Write 5 Sent Done Done false
6149 Acked Flush 0 Sent Done Done false
6150 Acked Write 7 Sent Done Done false
6151 Acked Read 4 Sent Done Done false
6152 Acked Write 5 Sent Done Done false
6153 Acked Read 8 Sent Done Done false
6154 Acked Read 6 Sent Done Done false
6155 Acked Read 8 Sent Done Done false
6156 Acked Write 7 Sent Done Done false
6157 Acked Read 1 Sent Done Done false
6158 Acked Read 5 Sent Done Done false
6159 Acked Read 9 Sent Done Done false
6160 Acked Write 6 Sent Done Done false
6161 Acked Write 6 Sent Done Done false
6162 Acked Read 9 Sent Done Done false
6163 Acked Read 5 Sent Done Done false
6164 Acked Read 2 Sent Done Done false
6165 Acked Read 2 Sent Done Done false
6166 Acked Write 3 Sent Done Done false
6167 Acked Write 10 Sent Done Done false
6168 Acked Read 5 Sent Done Done false
6169 Acked Flush 0 Sent Done Done false
6170 Acked Flush 0 Sent Done Done false
6171 Acked Read 3 Sent Done Done false
6172 Acked Write 6 Sent Done Done false
6173 Acked Write 3 Sent Done Done false
6174 Acked Read 8 Sent Done Done false
6175 Acked Write 4 Sent Done Done false
6176 Acked Read 9 Sent Done Done false
6177 Acked Write 2 Sent Done Done false
6178 Acked Read 1 Sent Done Done false
6179 Acked Write 6 Sent Done Done false
6180 Acked Read 9 Sent Done Done false
6181 Acked Read 8 Sent Done Done false
6182 Acked Read 9 Sent Done Done false
6183 Acked Write 7 Sent Done Done false
6184 Acked Read 4 Sent Done Done false
6185 Acked Write 4 Sent Done Done false
6186 Acked Read 9 Sent Done Done false
6187 Acked Flush 0 Sent Done Done false
6188 Acked Write 8 Sent Done Done false
6189 Acked Read 8 Sent Done Done false
6190 Acked Read 5 Sent Done Done false
6191 Acked Read 5 Sent Done Done false
6192 Acked Read 9 Sent Done Done false
6193 Acked Read 8 Sent Done Done false
6194 Acked Flush 0 Sent Done Done false
6195 Acked Write 7 Sent Done Done false
6196 Acked Flush 0 Sent Done Done false
6197 Acked Flush 0 Sent Done Done false
6198 Acked Flush 0 Sent Done Done false
6199 Acked Read 9 Sent Done Done false
6200 Acked Write 9 Sent Done Done false
6201 Acked Write 6 Sent Done Done false
6202 Acked Read 3 Sent Done Done false
6203 Acked Write 3 Sent Done Done false
6204 Acked Write 4 Sent Done Done false
6205 Acked Write 9 Sent Done Done false
6206 Acked Write 9 Sent Done Done false
6207 Acked Write 7 Sent Done Done false
6208 Acked Write 9 Sent Done Done false
6209 Acked Flush 0 Sent Done Done false
6210 Acked Read 9 Sent Done Done false
6211 Acked Read 7 Sent Done Done false
6212 Acked Read 5 Sent Done Done false
6213 Acked Read 1 Sent Done Done false
6214 Acked Write 9 Sent Done Done false
6215 Acked Write 8 Sent Done Done false
6216 Acked Flush 0 Sent Done Done false
6217 Acked Write 6 Sent Done Done false
6218 Acked Write 1 Sent Done Done false
6219 Acked Read 1 Sent Done Done false
6220 Acked Write 10 Sent Done Done false
6221 Acked Read 7 Sent Done Done false
6222 Acked Write 6 Sent Done Done false
6223 Acked Read 1 Sent Done Done false
6224 Acked Read 4 Sent Done Done false
6225 Acked Read 1 Sent Done Done false
6226 Acked Read 6 Sent Done Done false
6227 Acked Read 1 Sent Done Done false
6228 Acked Read 2 Sent Done Done false
6229 Acked Flush 0 Sent Done Done false
6230 Acked Read 8 Sent Done Done false
6231 Acked Flush 0 Sent Done Done false
6232 Acked Read 7 Sent Done Done false
6233 Acked Read 8 Sent Done Done false
6234 Acked Read 5 Sent Done Done false
6235 Acked Read 1 Sent Done Done false
6236 Acked Read 7 Sent Done Done false
6237 Acked Write 2 Sent Done Done false
6238 Acked Read 8 Sent Done Done false
6239 Acked Read 1 Sent Done Done false
6240 Acked Write 3 Sent Done Done false
6241 Acked Write 1 Sent Done Done false
6242 Acked Read 6 Sent Done Done false
6243 Acked Read 2 Sent Done Done false
6244 Acked Write 7 Sent Done Done false
6245 Acked Write 2 Sent Done Done false
6246 Acked Read 8 Sent Done Done false
6247 Acked Flush 0 Sent Done Done false
6248 Acked Write 2 Sent Done Done false
6249 Acked Read 1 Sent Done Done false
6250 Acked Write 2 Sent Done Done false
6251 Acked Read 2 Sent Done Done false
6252 Acked Write 5 Sent Done Done false
6253 Acked Write 3 Sent Done Done false
6254 Acked Write 8 Sent Done Done false
6255 Acked Read 6 Sent Done Done false
6256 Acked Read 8 Sent Done Done false
6257 Acked Write 8 Sent Done Done false
6258 Acked Write 3 Sent Done Done false
6259 Acked Read 2 Sent Done Done false
6260 Acked Flush 0 Sent Done Done false
6261 Acked Write 9 Sent Done Done false
6262 Acked Read 7 Sent Done Done false
6263 Acked Read 7 Sent Done Done false
6264 Acked Read 3 Sent Done Done false
6265 Acked Write 4 Sent Done Done false
6266 Acked Read 6 Sent Done Done false
6267 Acked Read 6 Sent Done Done false
6268 Acked Read 4 Sent Done Done false
6269 Acked Read 10 Sent Done Done false
6270 Acked Write 1 Sent Done Done false
6271 Acked Write 2 Sent Done Done false
6272 Acked Write 8 Sent Done Done false
6273 Acked Write 4 Sent Done Done false
6274 Acked Read 1 Sent Done Done false
6275 Acked Write 10 Sent Done Done false
6276 Acked Write 9 Sent Done Done false
6277 Acked Write 2 Sent Done Done false
6278 Acked Read 4 Sent Done Done false
6279 Acked Flush 0 Sent Done Done false
6280 Acked Read 4 Sent Done Done false
6281 Acked Read 3 Sent Done Done false
6282 Acked Read 4 Sent Done Done false
6283 Acked Read 10 Sent Done Done false
6284 Acked Write 3 Sent Done Done false
6285 Acked Write 9 Sent Done Done false
6286 Acked Read 2 Sent Done Done false
6287 Acked Read 1 Sent Done Done false
6288 Acked Read 2 Sent Done Done false
6289 Acked Write 9 Sent Done Done false
6290 Acked Read 3 Sent Done Done false
6291 Acked Write 10 Sent Done Done false
6292 Acked Read 7 Sent Done Done false
6293 Acked Write 6 Sent Done Done false
6294 Acked Flush 0 Sent Done Done false
6295 Acked Write 1 Sent Done Done false
6296 Acked Read 10 Sent Done Done false
6297 Acked Read 5 Sent Done Done false
6298 Acked Read 6 Sent Done Done false
6299 Acked Read 5 Sent Done Done false
6300 Acked Write 4 Sent Done Done false
6301 Acked Read 4 Sent Done Done false
6302 Acked Flush 0 Sent Done Done false
6303 Acked Read 7 Sent Done Done false
6304 Acked Read 1 Sent Done Done false
6305 Acked Write 3 Sent Done Done false
6306 Acked Write 10 Sent Done Done false
6307 Acked Read 2 Sent Done Done false
6308 Acked Read 2 Sent Done Done false
6309 Acked Read 3 Sent Done Done false
6310 Acked Write 10 Sent Done Done false
6311 Acked Write 3 Sent Done Done false
6312 Acked Write 3 Sent Done Done false
6313 Acked Read 9 Sent Done Done false
6314 Acked Read 2 Sent Done Done false
6315 Acked Write 4 Sent Done Done false
6316 Acked Read 6 Sent Done Done false
6317 Acked Read 9 Sent Done Done false
6318 Acked Read 8 Sent Done Done false
6319 Acked Read 4 Sent Done Done false
6320 Acked Read 5 Sent Done Done false
6321 Acked Read 6 Sent Done Done false
6322 Acked Read 7 Sent Done Done false
6323 Acked Flush 0 Sent Done Done false
6324 Acked Read 3 Sent Done Done false
6325 Acked Flush 0 Sent Done Done false
6326 Acked Read 7 Sent Done Done false
6327 Acked Read 9 Sent Done Done false
6328 Acked Write 1 Sent Done Done false
6329 Acked Flush 0 Sent Done Done false
6330 Acked Flush 0 Sent Done Done false
6331 Acked Read 1 Sent Done Done false
6332 Acked Flush 0 Sent Done Done false
6333 Acked Read 4 Sent Done Done false
6334 Acked Read 3 Sent Done Done false
6335 Acked Write 6 Sent Done Done false
6336 Acked Write 9 Sent Done Done false
6337 Acked Write 10 Sent Done Done false
6338 Acked Read 1 Sent Done Done false
6339 Acked Flush 0 Sent Done Done false
6340 Acked Read 9 Sent Done Done false
6341 Acked Read 8 Sent Done Done false
6342 Acked Write 2 Sent Done Done false
6343 Acked Read 2 Sent Done Done false
6344 Acked Flush 0 Sent Done Done false
6345 Acked Write 1 Sent Done Done false
6346 Acked Write 5 Sent Done Done false
6347 Acked Read 7 Sent Done Done false
6348 Acked Flush 0 Sent Done Done false
6349 Acked Read 4 Sent Done Done false
6350 Acked Read 1 Sent Done Done false
6351 Acked Flush 0 Sent Done Done false
6352 Acked Read 3 Sent Done Done false
6353 Acked Write 4 Sent Done Done false
6354 Acked Read 9 Sent Done Done false
6355 Acked Flush 0 Sent Done Done false
6356 Acked Write 9 Sent Done Done false
6357 Acked Flush 0 Sent Done Done false
6358 Acked Read 7 Sent Done Done false
6359 Acked Flush 0 Sent Done Done false
6360 Acked Write 1 Sent Done Done false
6361 Acked Read 9 Sent Done Done false
6362 Acked Write 3 Sent Done Done false
6363 Acked Read 3 Sent Done Done false
6364 Acked Read 1 Sent Done Done false
6365 Acked Write 5 Sent Done Done false
6366 Acked Read 6 Sent Done Done false
6367 Acked Read 8 Sent Done Done false
6368 Acked Flush 0 Sent Done Done false
6369 Acked Write 7 Sent Done Done false
6370 Acked Read 7 Sent Done Done false
6371 Acked Read 4 Sent Done Done false
6372 Acked Write 3 Sent Done Done false
6373 Acked Read 6 Sent Done Done false
6374 Acked Read 3 Sent Done Done false
6375 Acked Flush 0 Sent Done Done false
6376 Acked Flush 0 Sent Done Done false
6377 Acked Write 8 Sent Done Done false
6378 Acked Flush 0 Sent Done Done false
6379 Acked Write 6 Sent Done Done false
6380 Acked Read 8 Sent Done Done false
6381 Acked Read 2 Sent Done Done false
6382 Acked Read 7 Sent Done Done false
6383 Acked Write 6 Sent Done Done false
6384 Acked Read 10 Sent Done Done false
6385 Acked Read 10 Sent Done Done false
6386 Acked Read 5 Sent Done Done false
6387 Acked Write 1 Sent Done Done false
6388 Acked Write 6 Sent Done Done false
6389 Acked Write 1 Sent Done Done false
6390 Acked Read 3 Sent Done Done false
6391 Acked Read 10 Sent Done Done false
6392 Acked Read 4 Sent Done Done false
6393 Acked Read 5 Sent Done Done false
6394 Acked Read 6 Sent Done Done false
6395 Acked Write 10 Sent Done Done false
6396 Acked Read 7 Sent Done Done false
6397 Acked Read 10 Sent Done Done false
6398 Acked Read 2 Sent Done Done false
6399 Acked Write 8 Sent Done Done false
6400 Acked Write 3 Sent Done Done false
6401 Acked Read 5 Sent Done Done false
6402 Acked Read 7 Sent Done Done false
6403 Acked Write 3 Sent Done Done false
6404 Acked Write 9 Sent Done Done false
6405 Acked Write 6 Sent Done Done false
6406 Acked Read 7 Sent Done Done false
6407 Acked Write 8 Sent Done Done false
6408 Acked Read 6 Sent Done Done false
6409 Acked Write 8 Sent Done Done false
6410 Acked Write 10 Sent Done Done false
6411 Acked Write 6 Sent Done Done false
6412 Acked Flush 0 Sent Done Done false
6413 Acked Write 4 Sent Done Done false
6414 Acked Write 8 Sent Done Done false
6415 Acked Read 5 Sent Done Done false
6416 Acked Write 10 Sent Done Done false
6417 Acked Read 2 Sent Done Done false
6418 Acked Read 2 Sent Done Done false
6419 Acked Read 2 Sent Done Done false
6420 Acked Write 3 Sent Done Done false
6421 Acked Write 7 Sent Done Done false
6422 Acked Flush 0 Sent Done Done false
6423 Acked Write 4 Sent Done Done false
6424 Acked Write 10 Sent Done Done false
6425 Acked Read 3 Sent Done Done false
6426 Acked Write 10 Sent Done Done false
6427 Acked Read 1 Sent Done Done false
6428 Acked Write 1 Sent Done Done false
6429 Acked Write 5 Sent Done Done false
6430 Acked Read 6 Sent Done Done false
6431 Acked Write 8 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(6130)) Some(JobId(6422)) Some(JobId(6422))
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6431 6430 6429 6428 6427
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:57.135414227Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:05:57.837575119Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.139860754Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51046)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.14000344Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.140974607Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141482396Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141515748Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(6130))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141534643Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141551249Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(6130))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141567625Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(6130))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:58.141734316Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6432 6431 6430 6429 6428
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:06:00.381 [8820] Got stop action so:false kr:false
INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ef22cb81-e4ca-4eb4-ac79-bb55dcf4e66e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:00.382166861Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:00.382231086Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:00.382249601Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:00.382367963Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:06:00.382 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: 331cfa79-0187-40f3-8a86-720e79b66881, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:06:04.384 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/1, method: GET, req_id: 00ad2030-d195-45ad-a3ec-d8ca45201fd3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 1641 len 12288 data: 3 4 3
002/300 Write block 2423 len 24576 data: 4 5 4 4 4 3
003/300 Write block 3682 len 12288 data: 4 4 3
004/300 Flush
005/300 Read block 3304 len 4096
006/300 Read block 3139 len 8192
007/300 Write block 334 len 4096 data: 4
008/300 Write block 2494 len 16384 data: 3 4 4 4
009/300 Read block 1128 len 36864
010/300 Read block 2784 len 24576
011/300 Write block 1886 len 40960 data: 3 3 3 4 4 7 6 6 7 7
012/300 Read block 4819 len 28672
013/300 Write block 4893 len 40960 data: 6 6 6 6 6 7 6 6 5 6
014/300 Read block 424 len 8192
015/300 Read block 3599 len 8192
016/300 Write block 369 len 40960 data: 3 3 2 3 5 5 4 4 5 6
017/300 Read block 161 len 32768
018/300 Read block 142 len 40960
019/300 Write block 3389 len 12288 data: 2 2 2
020/300 Read block 3962 len 4096
021/300 Flush
022/300 Read block 1785 len 16384
023/300 Read block 4540 len 40960
024/300 Write block 650 len 12288 data: 3 3 3
025/300 Flush
026/300 Read block 1512 len 4096
027/300 Write block 3983 len 16384 data: 3 2 3 6
028/300 Write block 3144 len 20480 data: 6 5 4 2 2
029/300 Read block 4345 len 8192
030/300 Write block 2730 len 36864 data: 6 5 5 6 4 5 5 4 3
031/300 Write block 3089 len 24576 data: 3 4 4 4 5 4
032/300 Write block 2522 len 24576 data: 7 8 9 8 6 6
033/300 Read block 4183 len 16384
034/300 Read block 2850 len 4096
035/300 Write block 821 len 12288 data: 3 3 4
036/300 Write block 1999 len 32768 data: 3 5 4 5 6 4 4 4
037/300 Read block 105 len 36864
038/300 Read block 1161 len 36864
039/300 Write block 1865 len 36864 data: 3 3 3 3 4 3 3 4 4
040/300 Read block 1813 len 16384
041/300 Read block 1368 len 28672
042/300 Read block 1709 len 40960
043/300 Write block 2003 len 8192 data: 7 5
044/300 Write block 987 len 40960 data: 4 4 4 4 5 3 3 3 2 2
045/300 Flush
046/300 Read block 2131 len 32768
047/300 Write block 839 len 16384 data: 3 3 4 4
048/300 Flush
049/300 Read block 549 len 20480
050/300 Write block 1663 len 36864 data: 5 5 6 6 4 5 5 4 4
051/300 Read block 4421 len 36864
052/300 Read block 2651 len 28672
053/300 Read block 4196 len 16384
054/300 Write block 3493 len 12288 data: 5 4 4
055/300 Write block 3009 len 28672 data: 4 4 4 3 3 3 4
056/300 Read block 488 len 8192
057/300 Read block 1374 len 24576
058/300 Read block 1205 len 12288
059/300 Read block 114 len 12288
060/300 Write block 2974 len 28672 data: 2 3 4 4 4 4 4
061/300 Read block 1008 len 28672
062/300 Read block 3403 len 4096
063/300 Write block 4426 len 20480 data: 3 5 8 8 8
064/300 Read block 927 len 28672
065/300 Write block 3645 len 12288 data: 3 3 3
066/300 Write block 4464 len 36864 data: 5 4 3 4 4 4 5 3 3
067/300 Read block 1633 len 16384
068/300 Write block 4649 len 40960 data: 6 5 5 7 7 7 6 4 3 3
069/300 Write block 1600 len 40960 data: 2 2 2 2 3 3 3 2 4 4
070/300 Read block 1388 len 4096
071/300 Write block 152 len 24576 data: 4 5 5 4 4 2
072/300 Write block 4886 len 40960 data: 7 10 9 10 9 8 6 7 7 7
073/300 Read block 1308 len 40960
074/300 Read block 4856 len 12288
075/300 Flush
076/300 Read block 2815 len 36864
077/300 Read block 2873 len 28672
078/300 Write block 1960 len 24576 data: 2 2 2 2 3 3
079/300 Read block 3379 len 4096
080/300 Read block 2386 len 36864
081/300 Read block 4988 len 20480
082/300 Flush
083/300 Write block 4950 len 36864 data: 3 5 4 5 6 5 5 6 7
084/300 Read block 4875 len 12288
085/300 Read block 3826 len 40960
086/300 Read block 3837 len 24576
087/300 Write block 1623 len 36864 data: 3 6 5 5 5 5 5 5 5
088/300 Write block 1821 len 28672 data: 5 4 3 3 4 4 5
089/300 Write block 445 len 20480 data: 10 7 8 8 9
090/300 Flush
091/300 Flush
092/300 Write block 2788 len 28672 data: 4 5 5 4 4 3 5
093/300 Read block 2597 len 28672
094/300 Read block 1150 len 24576
095/300 Read block 957 len 4096
096/300 Flush
097/300 Write block 4312 len 12288 data: 3 3 3
098/300 Write block 124 len 36864 data: 4 3 4 5 5 5 5 4 3
099/300 Read block 4840 len 36864
100/300 Write block 1134 len 20480 data: 4 4 5 4 4
101/300 Read block 4568 len 28672
102/300 Read block 3620 len 16384
103/300 Write block 4266 len 8192 data: 4 5
104/300 Flush
105/300 Write block 321 len 12288 data: 5 5 5
106/300 Read block 4511 len 28672
107/300 Write block 436 len 36864 data: 7 6 7 7 7 7 9 9 10
108/300 Read block 4243 len 12288
109/300 Write block 73 len 40960 data: 3 3 6 5 5 5 5 4 5 4
110/300 Read block 2190 len 24576
111/300 Write block 767 len 4096 data: 2
112/300 Read block 2744 len 12288
113/300 Read block 4652 len 28672
114/300 Write block 3934 len 28672 data: 6 6 7 6 8 9 9
115/300 Write block 718 len 4096 data: 6
116/300 Read block 2847 len 20480
117/300 Write block 1873 len 20480 data: 5 6 6 6 8
118/300 Read block 736 len 32768
119/300 Read block 2127 len 4096
120/300 Read block 2342 len 28672
121/300 Read block 629 len 40960
122/300 Read block 4172 len 12288
123/300 Flush
124/300 Flush
125/300 Write block 4036 len 20480 data: 5 5 7 7 6
126/300 Read block 3978 len 28672
127/300 Read block 4846 len 4096
128/300 Flush
129/300 Read block 2799 len 28672
130/300 Write block 2755 len 16384 data: 3 3 3 4
131/300 Read block 2143 len 4096
132/300 Read block 141 len 24576
133/300 Read block 3906 len 16384
134/300 Read block 72 len 32768
135/300 Write block 2173 len 8192 data: 4 5
136/300 Read block 900 len 40960
137/300 Write block 4158 len 20480 data: 7 9 8 6 6
138/300 Write block 4683 len 40960 data: 4 4 3 3 4 2 2 2 2 3
139/300 Read block 1594 len 28672
140/300 Read block 1719 len 24576
141/300 Write block 1368 len 20480 data: 5 3 3 4 4
142/300 Read block 643 len 20480
143/300 Read block 1010 len 40960
144/300 Read block 4716 len 4096
145/300 Write block 3340 len 16384 data: 6 7 6 8
146/300 Read block 4742 len 32768
147/300 Write block 3731 len 4096 data: 6
148/300 Write block 3343 len 4096 data: 9
149/300 Flush
150/300 Write block 2715 len 32768 data: 7 8 8 6 4 4 3 2
151/300 Write block 3690 len 8192 data: 4 4
152/300 Read block 3324 len 32768
153/300 Read block 1354 len 36864
154/300 Write block 2803 len 28672 data: 4 4 5 6 6 5 5
155/300 Write block 4111 len 4096 data: 4
156/300 Read block 353 len 28672
157/300 Read block 4091 len 40960
158/300 Write block 804 len 40960 data: 4 5 5 4 4 4 4 4 7 8
159/300 Read block 1889 len 8192
160/300 Write block 378 len 40960 data: 7 6 7 6 7 7 5 5 5 2
161/300 Read block 2188 len 28672
162/300 Read block 1772 len 8192
163/300 Write block 1499 len 20480 data: 2 2 3 2 3
164/300 Write block 1783 len 32768 data: 3 3 3 2 2 4 5 5
165/300 Write block 4037 len 8192 data: 6 8
166/300 Read block 4174 len 8192
167/300 Write block 4819 len 40960 data: 4 4 4 4 6 5 5 5 5 8
168/300 Flush
169/300 Read block 134 len 24576
170/300 Read block 4974 len 8192
171/300 Read block 2680 len 16384
172/300 Read block 1544 len 16384
173/300 Write block 3256 len 20480 data: 3 3 3 2 2
174/300 Read block 1825 len 4096
175/300 Read block 3658 len 8192
176/300 Write block 2667 len 28672 data: 7 7 9 9 6 8 7
177/300 Read block 4669 len 36864
178/300 Write block 1351 len 40960 data: 5 5 5 4 5 4 4 5 6 6
179/300 Flush
180/300 Read block 4129 len 8192
181/300 Read block 4307 len 4096
182/300 Read block 1272 len 4096
183/300 Read block 1434 len 20480
184/300 Read block 2807 len 8192
185/300 Flush
186/300 Write block 829 len 24576 data: 5 5 5 3 4 4
187/300 Read block 2539 len 8192
188/300 Write block 1106 len 16384 data: 2 2 2 2
189/300 Write block 3366 len 40960 data: 4 3 4 3 3 4 3 4 3 2
190/300 Read block 3340 len 24576
191/300 Read block 1912 len 28672
192/300 Write block 3242 len 4096 data: 3
193/300 Write block 2931 len 20480 data: 4 3 4 4 2
194/300 Write block 1926 len 16384 data: 2 3 3 3
195/300 Read block 3395 len 28672
196/300 Read block 3799 len 40960
197/300 Read block 1776 len 36864
198/300 Read block 1563 len 4096
199/300 Write block 3771 len 16384 data: 2 2 3 3
200/300 Write block 859 len 28672 data: 3 4 4 5 5 5 4
201/300 Write block 2373 len 16384 data: 4 4 5 6
202/300 Read block 3460 len 8192
203/300 Flush
204/300 Write block 2138 len 24576 data: 4 4 5 4 4 4
205/300 Read block 468 len 32768
206/300 Write block 4539 len 12288 data: 3 3 2
207/300 Write block 1417 len 8192 data: 3 3
208/300 Flush
209/300 Write block 394 len 4096 data: 2
210/300 Write block 1410 len 36864 data: 4 4 3 3 4 3 3 4 4
211/300 Read block 4302 len 32768
212/300 Read block 110 len 8192
213/300 Flush
214/300 Read block 2215 len 32768
215/300 Write block 697 len 20480 data: 4 4 4 5 6
216/300 Read block 1147 len 28672
217/300 Read block 3245 len 4096
218/300 Write block 579 len 40960 data: 6 7 6 6 6 4 4 4 2 2
219/300 Write block 2116 len 16384 data: 4 5 5 4
220/300 Flush
221/300 Read block 402 len 16384
222/300 Read block 379 len 40960
223/300 Read block 840 len 36864
224/300 Write block 3373 len 8192 data: 5 4
225/300 Read block 3520 len 12288
226/300 Read block 2321 len 4096
227/300 Read block 4202 len 8192
228/300 Write block 935 len 40960 data: 4 4 5 6 5 5 6 7 6 6
229/300 Write block 823 len 4096 data: 5
230/300 Flush
231/300 Write block 3300 len 8192 data: 4 6
232/300 Flush
233/300 Write block 4909 len 28672 data: 4 3 3 3 5 4 4
234/300 Read block 18 len 4096
235/300 Write block 2693 len 32768 data: 6 6 6 4 4 4 2 3
236/300 Write block 941 len 16384 data: 7 8 7 7
237/300 Read block 1947 len 24576
238/300 Read block 1920 len 24576
239/300 Write block 2604 len 8192 data: 4 3
240/300 Read block 517 len 4096
241/300 Read block 908 len 8192
242/300 Read block 939 len 20480
243/300 Read block 2255 len 12288
244/300 Read block 793 len 24576
245/300 Write block 2582 len 16384 data: 3 4 4 4
246/300 Write block 1338 len 24576 data: 4 4 5 5 5 4
247/300 Read block 1483 len 16384
248/300 Flush
249/300 Read block 2305 len 4096
250/300 Read block 2654 len 20480
251/300 Read block 132 len 4096
252/300 Flush
253/300 Read block 88 len 28672
254/300 Read block 4452 len 4096
255/300 Write block 3332 len 12288 data: 4 3 4
256/300 Read block 2299 len 24576
257/300 Read block 2477 len 36864
258/300 Flush
259/300 Read block 4594 len 12288
260/300 Read block 4872 len 12288
261/300 Flush
262/300 Write block 4868 len 4096 data: 5
263/300 Write block 1913 len 20480 data: 3 3 3 3 3
264/300 Write block 93 len 28672 data: 8 9 10 6 4 3 3
265/300 Read block 434 len 40960
266/300 Write block 2185 len 28672 data: 7 8 6 7 7 8 5
267/300 Write block 4347 len 4096 data: 9
268/300 Read block 1945 len 40960
269/300 Read block 1095 len 28672
270/300 Write block 1408 len 32768 data: 4 5 5 5 4 4 5 4
271/300 Read block 2212 len 32768
272/300 Read block 202 len 32768
273/300 Read block 3122 len 36864
274/300 Write block 1015 len 4096 data: 4
275/300 Write block 1376 len 4096 data: 4
276/300 Read block 2341 len 24576
277/300 Flush
278/300 Read block 4088 len 8192
279/300 Write block 4750 len 16384 data: 2 2 2 2
280/300 Read block 2052 len 8192
281/300 Read block 4201 len 24576
282/300 Write block 1306 len 24576 data: 2 2 4 4 4 4
283/300 Read block 2569 len 40960
284/300 Read block 4584 len 8192
285/300 Read block 4729 len 24576
286/300 Read block 1361 len 24576
287/300 Write block 1762 len 20480 data: 4 4 4 4 4
288/300 Write block 1062 len 20480 data: 3 3 3 5 6
289/300 Read block 1613 len 28672
290/300 Write block 3105 len 36864 data: 6 5 7 8 8 8 7 8 6
291/300 Read block 4836 len 32768
292/300 Write block 1509 len 16384 data: 3 3 3 4
293/300 Read block 591 len 16384
294/300 Write block 4336 len 4096 data: 5
295/300 Write block 1802 len 16384 data: 5 3 2 2
296/300 Read block 3639 len 12288
297/300 Read block 4333 len 28672
298/300 Write block 1477 len 32768 data: 4 4 4 3 3 3 4 4
299/300 Write block 3602 len 4096 data: 2
300/300 Read block 1741 len 24576
301/300 Write block 418 len 24576 data: 9 10 8 9 10 8
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:06:04.625 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: da75fd40-c177-4014-b28f-1742a2f968d4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Done Sent Done false
6434 Acked Write 6 Done Sent Done false
6435 Acked Write 3 Done Sent Done false
6436 Acked Flush 0 Done Sent Done false
6437 Acked Read 1 Done Sent Done false
6438 Acked Read 2 Done Sent Done false
6439 Acked Write 1 Done Sent Done false
6440 Acked Write 4 Done Sent Done false
6441 Acked Read 9 Done Sent Done false
6442 Acked Read 6 Done Sent Done false
6443 Acked Write 10 Done Sent Done false
6444 Acked Read 7 Done Sent Done false
6445 Acked Write 10 Done Sent Done false
6446 Acked Read 2 Done Sent Done false
6447 Acked Read 2 Done Sent Done false
6448 Acked Write 10 Done Sent Done false
6449 Acked Read 8 Done Sent Done false
6450 Acked Read 10 Done Sent Done false
6451 Acked Write 3 Done Sent Done false
6452 Acked Read 1 Done Sent Done false
6453 Acked Flush 0 Done Sent Done false
6454 Acked Read 4 Done Sent Done false
6455 Acked Read 10 Done Sent Done false
6456 Acked Write 3 Done Sent Done false
6457 Acked Flush 0 Done Sent Done false
6458 Acked Read 1 Done Sent Done false
6459 Acked Write 4 Done Sent Done false
6460 Acked Write 5 Done Sent Done false
6461 Acked Read 2 Done Sent Done false
6462 Acked Write 9 Done Sent Done false
6463 Acked Write 6 Done Sent Done false
6464 Acked Write 6 Done Sent Done false
6465 Acked Read 4 Done Sent Done false
6466 Acked Read 1 Done Sent Done false
6467 Acked Write 3 Done Sent Done false
6468 Acked Write 8 Done Sent Done false
6469 Acked Read 9 Done Sent Done false
6470 Acked Read 9 Done Sent Done false
6471 Acked Write 9 Done Sent Done false
6472 Acked Read 4 Done Sent Done false
6473 Acked Read 7 Done Sent Done false
6474 Acked Read 10 Done Sent Done false
6475 Acked Write 2 Done Sent Done false
6476 Acked Write 10 Done Sent Done false
6477 Acked Flush 0 Done Sent Done false
6478 Acked Read 8 Done Sent Done false
6479 Acked Write 4 Done Sent Done false
6480 Acked Flush 0 Done Sent Done false
6481 Acked Read 5 Done Sent Done false
6482 Acked Write 9 Done Sent Done false
6483 Acked Read 9 Done Sent Done false
6484 Acked Read 7 Done Sent Done false
6485 Acked Read 4 Done Sent Done false
6486 Acked Write 3 Done Sent Done false
6487 Acked Write 7 Done Sent Done false
6488 Acked Read 2 Done Sent Done false
6489 Acked Read 6 Done Sent Done false
6490 Acked Read 3 Done Sent Done false
6491 Acked Read 3 Done Sent Done false
6492 Acked Write 7 Done Sent Done false
6493 Acked Read 7 Done Sent Done false
6494 Acked Read 1 Done Sent Done false
6495 Acked Write 5 Done Sent Done false
6496 Acked Read 7 Done Sent Done false
6497 Acked Write 3 Done Sent Done false
6498 Acked Write 9 Done Sent Done false
6499 Acked Read 4 Done Sent Done false
6500 Acked Write 10 Done Sent Done false
6501 Acked Write 10 Done Sent Done false
6502 Acked Read 1 Done Sent Done false
6503 Acked Write 6 Done Sent Done false
6504 Acked Write 10 Done Sent Done false
6505 Acked Read 10 Done Sent Done false
6506 Acked Read 3 Done Sent Done false
6507 Acked Flush 0 Done Sent Done false
6508 Acked Read 9 Done Sent Done false
6509 Acked Read 7 Done Sent Done false
6510 Acked Write 6 Done Sent Done false
6511 Acked Read 1 Done Sent Done false
6512 Acked Read 9 Done Sent Done false
6513 Acked Read 5 Done Sent Done false
6514 Acked Flush 0 Done Sent Done false
6515 Acked Write 9 Done Sent Done false
6516 Acked Read 3 Done Sent Done false
6517 Acked Read 10 Done Sent Done false
6518 Acked Read 6 Done Sent Done false
6519 Acked Write 9 Done Sent Done false
6520 Acked Write 7 Done Sent Done false
6521 Acked Write 5 Done Sent Done false
6522 Acked Flush 0 Done Sent Done false
6523 Acked Flush 0 Done Sent Done false
6524 Acked Write 7 Done Sent Done false
6525 Acked Read 7 Done Sent Done false
6526 Acked Read 6 Done Sent Done false
6527 Acked Read 1 Done Sent Done false
6528 Acked Flush 0 Done Sent Done false
6529 Acked Write 3 Done Sent Done false
6530 Acked Write 9 Done Sent Done false
6531 Acked Read 9 Done Sent Done false
6532 Acked Write 5 Done Sent Done false
6533 Acked Read 7 Done Sent Done false
6534 Acked Read 4 Done Sent Done false
6535 Acked Write 2 Done Sent Done false
6536 Acked Flush 0 Done Sent Done false
6537 Acked Write 3 Done Sent Done false
6538 Acked Read 7 Done Sent Done false
6539 Acked Write 9 Done Sent Done false
6540 Acked Read 3 Done Sent Done false
6541 Acked Write 10 Done Sent Done false
6542 Acked Read 6 Done Sent Done false
6543 Acked Write 1 Done Sent Done false
6544 Acked Read 3 Done Sent Done false
6545 Acked Read 7 Done Sent Done false
6546 Acked Write 7 Done Sent Done false
6547 Acked Write 1 Done Sent Done false
6548 Acked Read 5 Done Sent Done false
6549 Acked Write 5 Done Sent Done false
6550 Acked Read 8 Done Sent Done false
6551 Acked Read 1 Done Sent Done false
6552 Acked Read 7 Done Sent Done false
6553 Acked Read 10 Done Sent Done false
6554 Acked Read 3 Done Sent Done false
6555 Acked Flush 0 Done Sent Done false
6556 Acked Flush 0 Done Sent Done false
6557 Acked Write 5 Done Sent Done false
6558 Acked Read 7 Done Sent Done false
6559 Acked Read 1 Done Sent Done false
6560 Acked Flush 0 Done Sent Done false
6561 Acked Read 7 Done Sent Done false
6562 Acked Write 4 Done Sent Done false
6563 Acked Read 1 Done Sent Done false
6564 Acked Read 6 Done Sent Done false
6565 Acked Read 4 Done Sent Done false
6566 Acked Read 8 Done Sent Done false
6567 Acked Write 2 Done Sent Done false
6568 Acked Read 10 Done Sent Done false
6569 Acked Write 5 Done Sent Done false
6570 Acked Write 10 Done Sent Done false
6571 Acked Read 7 Done Sent Done false
6572 Acked Read 6 Done Sent Done false
6573 Acked Write 5 Done Sent Done false
6574 Acked Read 5 Done Sent Done false
6575 Acked Read 10 Done Sent Done false
6576 Acked Read 1 Done Sent Done false
6577 Acked Write 4 Done Sent Done false
6578 Acked Read 8 Done Sent Done false
6579 Acked Write 1 Done Sent Done false
6580 Acked Write 1 Done Sent Done false
6581 Acked Flush 0 Done Sent Done false
6582 Acked Write 8 Done Sent Done false
6583 Acked Write 2 Done Sent Done false
6584 Acked Read 8 Done Sent Done false
6585 Acked Read 9 Done Sent Done false
6586 Acked Write 7 Done Sent Done false
6587 Acked Write 1 Done Sent Done false
6588 Acked Read 7 Done Sent Done false
6589 Acked Read 10 Done Sent Done false
6590 Acked Write 10 Done Sent Done false
6591 Acked Read 2 Done Sent Done false
6592 Acked Write 10 Done Sent Done false
6593 Acked Read 7 Done Sent Done false
6594 Acked Read 2 Done Sent Done false
6595 Acked Write 5 Done Sent Done false
6596 Acked Write 8 Done Sent Done false
6597 Acked Write 2 Done Sent Done false
6598 Acked Read 2 Done Sent Done false
6599 Acked Write 10 Done Sent Done false
6600 Acked Flush 0 Done Sent Done false
6601 Acked Read 6 Done Sent Done false
6602 Acked Read 2 Done Sent Done false
6603 Acked Read 4 Done Sent Done false
6604 Acked Read 4 Done Sent Done false
6605 Acked Write 5 Done Sent Done false
6606 Acked Read 1 Done Sent Done false
6607 Acked Read 2 Done Sent Done false
6608 Acked Write 7 Done Sent Done false
6609 Acked Read 9 Done Sent Done false
6610 Acked Write 10 Done Sent Done false
6611 Acked Flush 0 Done Sent Done false
6612 Acked Read 2 Done Sent Done false
6613 Acked Read 1 Done Sent Done false
6614 Acked Read 1 Done Sent Done false
6615 Acked Read 5 Done Sent Done false
6616 Acked Read 2 Done Sent Done false
6617 Acked Flush 0 Done Sent Done false
6618 Acked Write 6 Done Sent Done false
6619 Acked Read 2 Done Sent Done false
6620 Acked Write 4 Done Sent Done false
6621 Acked Write 10 Done Sent Done false
6622 Acked Read 6 Done Sent Done false
6623 Acked Read 7 Done Sent Done false
6624 Acked Write 1 Done Sent Done false
6625 Acked Write 5 Done Sent Done false
6626 Acked Write 4 Done Sent Done false
6627 Acked Read 7 Done Sent Done false
6628 Acked Read 10 Done Sent Done false
6629 Acked Read 9 Done Sent Done false
6630 Acked Read 1 Done Sent Done false
6631 Acked Write 4 Done Sent Done false
6632 Acked Write 7 Done Sent Done false
6633 Acked Write 4 Done Sent Done false
6634 Acked Read 2 Done Sent Done false
6635 Acked Flush 0 Done Sent Done false
6636 Acked Write 6 Done Sent Done false
6637 Acked Read 8 Done Sent Done false
6638 Acked Write 3 Done Sent Done false
6639 Acked Write 2 Done Sent Done false
6640 Acked Flush 0 Done Sent Done false
6641 Acked Write 1 Done Sent Done false
6642 Acked Write 9 Done Sent Done false
6643 Acked Read 8 Done Sent Done false
6644 Acked Read 2 Done Sent Done false
6645 Acked Flush 0 Done Sent Done false
6646 Acked Read 8 Done Sent Done false
6647 Acked Write 5 Done Sent Done false
6648 Acked Read 7 Done Sent Done false
6649 Acked Read 1 Done Sent Done false
6650 Acked Write 10 Done Sent Done false
6651 Acked Write 4 Done Sent Done false
6652 Acked Flush 0 Done Sent Done false
6653 Acked Read 4 Done Sent Done false
6654 Acked Read 10 Done Sent Done false
6655 Acked Read 9 Done Sent Done false
6656 Acked Write 2 Done Sent Done false
6657 Acked Read 3 Done Sent Done false
6658 Acked Read 1 Done Sent Done false
6659 Acked Read 2 Done Sent Done false
6660 Acked Write 10 Done Sent Done false
6661 Acked Write 1 Done Sent Done false
6662 Acked Flush 0 Done Sent Done false
6663 Acked Write 2 Done Sent Done false
6664 Acked Flush 0 Done Sent Done false
6665 Acked Write 7 Done Sent Done false
6666 Acked Read 1 Done Sent Done false
6667 Acked Write 8 Done Sent Done false
6668 Acked Write 4 Done Sent Done false
6669 Acked Read 6 Done Sent Done false
6670 Acked Read 6 Done Sent Done false
6671 Acked Write 2 Done Sent Done false
6672 Acked Read 1 Done Sent Done false
6673 Acked Read 2 Done Sent Done false
6674 Acked Read 5 Done Sent Done false
6675 Acked Read 3 Done Sent Done false
6676 Acked Read 6 Done Sent Done false
6677 Acked Write 4 Done Sent Done false
6678 Acked Write 6 Done Sent Done false
6679 Acked Read 4 Done Sent Done false
6680 Acked Flush 0 Done Sent Done false
6681 Acked Read 1 Done Sent Done false
6682 Acked Read 5 Done Sent Done false
6683 Acked Read 1 Done Sent Done false
6684 Acked Flush 0 Done Sent Done false
6685 Acked Read 7 Done Sent Done false
6686 Acked Read 1 Done Sent Done false
6687 Acked Write 3 Done Sent Done false
6688 Acked Read 6 Done Sent Done false
6689 Acked Read 9 Done Sent Done false
6690 Acked Flush 0 Done Sent Done false
6691 Acked Read 3 Done Sent Done false
6692 Acked Read 3 Done Sent Done false
6693 Acked Flush 0 Done Sent Done false
6694 Acked Write 1 Done Sent Done false
6695 Acked Write 5 Done Sent Done false
6696 Acked Write 7 Done Sent Done false
6697 Acked Read 10 Done Sent Done false
6698 Acked Write 7 Done Sent Done false
6699 Acked Write 1 Done Sent Done false
6700 Acked Read 10 Done Sent Done false
6701 Acked Read 7 Done Sent Done false
6702 Acked Write 8 Done Sent Done false
6703 Acked Read 8 Done Sent Done false
6704 Acked Read 8 Done Sent Done false
6705 Acked Read 9 Done Sent Done false
6706 Acked Write 1 Done Sent Done false
6707 Acked Write 1 Done Sent Done false
6708 Acked Read 6 Done Sent Done false
6709 Acked Flush 0 Done Sent Done false
6710 Acked Read 2 Done Sent Done false
6711 Acked Write 4 Done Sent Done false
6712 Acked Read 2 Done Sent Done false
6713 Acked Read 6 Done Sent Done false
6714 Acked Write 6 Done Sent Done false
6715 Acked Read 10 Done Sent Done false
6716 Acked Read 2 Done Sent Done false
6717 Acked Read 6 Done Sent Done false
6718 Acked Read 6 Done Sent Done false
6719 Acked Write 5 Done Sent Done false
6720 Acked Write 5 Done Sent Done false
6721 Acked Read 7 Done Sent Done false
6722 Acked Write 9 Done Sent Done false
6723 Acked Read 8 Done Sent Done false
6724 Acked Write 4 Done Sent Done false
6725 Acked Read 4 Done Sent Done false
6726 Acked Write 1 Done Sent Done false
6727 Acked Write 4 Done Sent Done false
6728 Acked Read 3 Done Sent Done false
6729 Acked Read 7 Done Sent Done false
6730 Acked Write 8 Done Sent Done false
6731 Acked Write 1 Done Sent Done false
6732 Acked Read 6 Done Sent Done false
6733 Acked Write 6 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(6709)) Some(JobId(6432)) Some(JobId(6709))
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6733 6732 6731 6730 6729
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:05.127896444Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:05.848230911Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1230)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:06.850153833Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:07.852542083Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Done Sent Done false
6434 Acked Write 6 Done Sent Done false
6435 Acked Write 3 Done Sent Done false
6436 Acked Flush 0 Done Sent Done false
6437 Acked Read 1 Done Sent Done false
6438 Acked Read 2 Done Sent Done false
6439 Acked Write 1 Done Sent Done false
6440 Acked Write 4 Done Sent Done false
6441 Acked Read 9 Done Sent Done false
6442 Acked Read 6 Done Sent Done false
6443 Acked Write 10 Done Sent Done false
6444 Acked Read 7 Done Sent Done false
6445 Acked Write 10 Done Sent Done false
6446 Acked Read 2 Done Sent Done false
6447 Acked Read 2 Done Sent Done false
6448 Acked Write 10 Done Sent Done false
6449 Acked Read 8 Done Sent Done false
6450 Acked Read 10 Done Sent Done false
6451 Acked Write 3 Done Sent Done false
6452 Acked Read 1 Done Sent Done false
6453 Acked Flush 0 Done Sent Done false
6454 Acked Read 4 Done Sent Done false
6455 Acked Read 10 Done Sent Done false
6456 Acked Write 3 Done Sent Done false
6457 Acked Flush 0 Done Sent Done false
6458 Acked Read 1 Done Sent Done false
6459 Acked Write 4 Done Sent Done false
6460 Acked Write 5 Done Sent Done false
6461 Acked Read 2 Done Sent Done false
6462 Acked Write 9 Done Sent Done false
6463 Acked Write 6 Done Sent Done false
6464 Acked Write 6 Done Sent Done false
6465 Acked Read 4 Done Sent Done false
6466 Acked Read 1 Done Sent Done false
6467 Acked Write 3 Done Sent Done false
6468 Acked Write 8 Done Sent Done false
6469 Acked Read 9 Done Sent Done false
6470 Acked Read 9 Done Sent Done false
6471 Acked Write 9 Done Sent Done false
6472 Acked Read 4 Done Sent Done false
6473 Acked Read 7 Done Sent Done false
6474 Acked Read 10 Done Sent Done false
6475 Acked Write 2 Done Sent Done false
6476 Acked Write 10 Done Sent Done false
6477 Acked Flush 0 Done Sent Done false
6478 Acked Read 8 Done Sent Done false
6479 Acked Write 4 Done Sent Done false
6480 Acked Flush 0 Done Sent Done false
6481 Acked Read 5 Done Sent Done false
6482 Acked Write 9 Done Sent Done false
6483 Acked Read 9 Done Sent Done false
6484 Acked Read 7 Done Sent Done false
6485 Acked Read 4 Done Sent Done false
6486 Acked Write 3 Done Sent Done false
6487 Acked Write 7 Done Sent Done false
6488 Acked Read 2 Done Sent Done false
6489 Acked Read 6 Done Sent Done false
6490 Acked Read 3 Done Sent Done false
6491 Acked Read 3 Done Sent Done false
6492 Acked Write 7 Done Sent Done false
6493 Acked Read 7 Done Sent Done false
6494 Acked Read 1 Done Sent Done false
6495 Acked Write 5 Done Sent Done false
6496 Acked Read 7 Done Sent Done false
6497 Acked Write 3 Done Sent Done false
6498 Acked Write 9 Done Sent Done false
6499 Acked Read 4 Done Sent Done false
6500 Acked Write 10 Done Sent Done false
6501 Acked Write 10 Done Sent Done false
6502 Acked Read 1 Done Sent Done false
6503 Acked Write 6 Done Sent Done false
6504 Acked Write 10 Done Sent Done false
6505 Acked Read 10 Done Sent Done false
6506 Acked Read 3 Done Sent Done false
6507 Acked Flush 0 Done Sent Done false
6508 Acked Read 9 Done Sent Done false
6509 Acked Read 7 Done Sent Done false
6510 Acked Write 6 Done Sent Done false
6511 Acked Read 1 Done Sent Done false
6512 Acked Read 9 Done Sent Done false
6513 Acked Read 5 Done Sent Done false
6514 Acked Flush 0 Done Sent Done false
6515 Acked Write 9 Done Sent Done false
6516 Acked Read 3 Done Sent Done false
6517 Acked Read 10 Done Sent Done false
6518 Acked Read 6 Done Sent Done false
6519 Acked Write 9 Done Sent Done false
6520 Acked Write 7 Done Sent Done false
6521 Acked Write 5 Done Sent Done false
6522 Acked Flush 0 Done Sent Done false
6523 Acked Flush 0 Done Sent Done false
6524 Acked Write 7 Done Sent Done false
6525 Acked Read 7 Done Sent Done false
6526 Acked Read 6 Done Sent Done false
6527 Acked Read 1 Done Sent Done false
6528 Acked Flush 0 Done Sent Done false
6529 Acked Write 3 Done Sent Done false
6530 Acked Write 9 Done Sent Done false
6531 Acked Read 9 Done Sent Done false
6532 Acked Write 5 Done Sent Done false
6533 Acked Read 7 Done Sent Done false
6534 Acked Read 4 Done Sent Done false
6535 Acked Write 2 Done Sent Done false
6536 Acked Flush 0 Done Sent Done false
6537 Acked Write 3 Done Sent Done false
6538 Acked Read 7 Done Sent Done false
6539 Acked Write 9 Done Sent Done false
6540 Acked Read 3 Done Sent Done false
6541 Acked Write 10 Done Sent Done false
6542 Acked Read 6 Done Sent Done false
6543 Acked Write 1 Done Sent Done false
6544 Acked Read 3 Done Sent Done false
6545 Acked Read 7 Done Sent Done false
6546 Acked Write 7 Done Sent Done false
6547 Acked Write 1 Done Sent Done false
6548 Acked Read 5 Done Sent Done false
6549 Acked Write 5 Done Sent Done false
6550 Acked Read 8 Done Sent Done false
6551 Acked Read 1 Done Sent Done false
6552 Acked Read 7 Done Sent Done false
6553 Acked Read 10 Done Sent Done false
6554 Acked Read 3 Done Sent Done false
6555 Acked Flush 0 Done Sent Done false
6556 Acked Flush 0 Done Sent Done false
6557 Acked Write 5 Done Sent Done false
6558 Acked Read 7 Done Sent Done false
6559 Acked Read 1 Done Sent Done false
6560 Acked Flush 0 Done Sent Done false
6561 Acked Read 7 Done Sent Done false
6562 Acked Write 4 Done Sent Done false
6563 Acked Read 1 Done Sent Done false
6564 Acked Read 6 Done Sent Done false
6565 Acked Read 4 Done Sent Done false
6566 Acked Read 8 Done Sent Done false
6567 Acked Write 2 Done Sent Done false
6568 Acked Read 10 Done Sent Done false
6569 Acked Write 5 Done Sent Done false
6570 Acked Write 10 Done Sent Done false
6571 Acked Read 7 Done Sent Done false
6572 Acked Read 6 Done Sent Done false
6573 Acked Write 5 Done Sent Done false
6574 Acked Read 5 Done Sent Done false
6575 Acked Read 10 Done Sent Done false
6576 Acked Read 1 Done Sent Done false
6577 Acked Write 4 Done Sent Done false
6578 Acked Read 8 Done Sent Done false
6579 Acked Write 1 Done Sent Done false
6580 Acked Write 1 Done Sent Done false
6581 Acked Flush 0 Done Sent Done false
6582 Acked Write 8 Done Sent Done false
6583 Acked Write 2 Done Sent Done false
6584 Acked Read 8 Done Sent Done false
6585 Acked Read 9 Done Sent Done false
6586 Acked Write 7 Done Sent Done false
6587 Acked Write 1 Done Sent Done false
6588 Acked Read 7 Done Sent Done false
6589 Acked Read 10 Done Sent Done false
6590 Acked Write 10 Done Sent Done false
6591 Acked Read 2 Done Sent Done false
6592 Acked Write 10 Done Sent Done false
6593 Acked Read 7 Done Sent Done false
6594 Acked Read 2 Done Sent Done false
6595 Acked Write 5 Done Sent Done false
6596 Acked Write 8 Done Sent Done false
6597 Acked Write 2 Done Sent Done false
6598 Acked Read 2 Done Sent Done false
6599 Acked Write 10 Done Sent Done false
6600 Acked Flush 0 Done Sent Done false
6601 Acked Read 6 Done Sent Done false
6602 Acked Read 2 Done Sent Done false
6603 Acked Read 4 Done Sent Done false
6604 Acked Read 4 Done Sent Done false
6605 Acked Write 5 Done Sent Done false
6606 Acked Read 1 Done Sent Done false
6607 Acked Read 2 Done Sent Done false
6608 Acked Write 7 Done Sent Done false
6609 Acked Read 9 Done Sent Done false
6610 Acked Write 10 Done Sent Done false
6611 Acked Flush 0 Done Sent Done false
6612 Acked Read 2 Done Sent Done false
6613 Acked Read 1 Done Sent Done false
6614 Acked Read 1 Done Sent Done false
6615 Acked Read 5 Done Sent Done false
6616 Acked Read 2 Done Sent Done false
6617 Acked Flush 0 Done Sent Done false
6618 Acked Write 6 Done Sent Done false
6619 Acked Read 2 Done Sent Done false
6620 Acked Write 4 Done Sent Done false
6621 Acked Write 10 Done Sent Done false
6622 Acked Read 6 Done Sent Done false
6623 Acked Read 7 Done Sent Done false
6624 Acked Write 1 Done Sent Done false
6625 Acked Write 5 Done Sent Done false
6626 Acked Write 4 Done Sent Done false
6627 Acked Read 7 Done Sent Done false
6628 Acked Read 10 Done Sent Done false
6629 Acked Read 9 Done Sent Done false
6630 Acked Read 1 Done Sent Done false
6631 Acked Write 4 Done Sent Done false
6632 Acked Write 7 Done Sent Done false
6633 Acked Write 4 Done Sent Done false
6634 Acked Read 2 Done Sent Done false
6635 Acked Flush 0 Done Sent Done false
6636 Acked Write 6 Done Sent Done false
6637 Acked Read 8 Done Sent Done false
6638 Acked Write 3 Done Sent Done false
6639 Acked Write 2 Done Sent Done false
6640 Acked Flush 0 Done Sent Done false
6641 Acked Write 1 Done Sent Done false
6642 Acked Write 9 Done Sent Done false
6643 Acked Read 8 Done Sent Done false
6644 Acked Read 2 Done Sent Done false
6645 Acked Flush 0 Done Sent Done false
6646 Acked Read 8 Done Sent Done false
6647 Acked Write 5 Done Sent Done false
6648 Acked Read 7 Done Sent Done false
6649 Acked Read 1 Done Sent Done false
6650 Acked Write 10 Done Sent Done false
6651 Acked Write 4 Done Sent Done false
6652 Acked Flush 0 Done Sent Done false
6653 Acked Read 4 Done Sent Done false
6654 Acked Read 10 Done Sent Done false
6655 Acked Read 9 Done Sent Done false
6656 Acked Write 2 Done Sent Done false
6657 Acked Read 3 Done Sent Done false
6658 Acked Read 1 Done Sent Done false
6659 Acked Read 2 Done Sent Done false
6660 Acked Write 10 Done Sent Done false
6661 Acked Write 1 Done Sent Done false
6662 Acked Flush 0 Done Sent Done false
6663 Acked Write 2 Done Sent Done false
6664 Acked Flush 0 Done Sent Done false
6665 Acked Write 7 Done Sent Done false
6666 Acked Read 1 Done Sent Done false
6667 Acked Write 8 Done Sent Done false
6668 Acked Write 4 Done Sent Done false
6669 Acked Read 6 Done Sent Done false
6670 Acked Read 6 Done Sent Done false
6671 Acked Write 2 Done Sent Done false
6672 Acked Read 1 Done Sent Done false
6673 Acked Read 2 Done Sent Done false
6674 Acked Read 5 Done Sent Done false
6675 Acked Read 3 Done Sent Done false
6676 Acked Read 6 Done Sent Done false
6677 Acked Write 4 Done Sent Done false
6678 Acked Write 6 Done Sent Done false
6679 Acked Read 4 Done Sent Done false
6680 Acked Flush 0 Done Sent Done false
6681 Acked Read 1 Done Sent Done false
6682 Acked Read 5 Done Sent Done false
6683 Acked Read 1 Done Sent Done false
6684 Acked Flush 0 Done Sent Done false
6685 Acked Read 7 Done Sent Done false
6686 Acked Read 1 Done Sent Done false
6687 Acked Write 3 Done Sent Done false
6688 Acked Read 6 Done Sent Done false
6689 Acked Read 9 Done Sent Done false
6690 Acked Flush 0 Done Sent Done false
6691 Acked Read 3 Done Sent Done false
6692 Acked Read 3 Done Sent Done false
6693 Acked Flush 0 Done Sent Done false
6694 Acked Write 1 Done Sent Done false
6695 Acked Write 5 Done Sent Done false
6696 Acked Write 7 Done Sent Done false
6697 Acked Read 10 Done Sent Done false
6698 Acked Write 7 Done Sent Done false
6699 Acked Write 1 Done Sent Done false
6700 Acked Read 10 Done Sent Done false
6701 Acked Read 7 Done Sent Done false
6702 Acked Write 8 Done Sent Done false
6703 Acked Read 8 Done Sent Done false
6704 Acked Read 8 Done Sent Done false
6705 Acked Read 9 Done Sent Done false
6706 Acked Write 1 Done Sent Done false
6707 Acked Write 1 Done Sent Done false
6708 Acked Read 6 Done Sent Done false
6709 Acked Flush 0 Done Sent Done false
6710 Acked Read 2 Done Sent Done false
6711 Acked Write 4 Done Sent Done false
6712 Acked Read 2 Done Sent Done false
6713 Acked Read 6 Done Sent Done false
6714 Acked Write 6 Done Sent Done false
6715 Acked Read 10 Done Sent Done false
6716 Acked Read 2 Done Sent Done false
6717 Acked Read 6 Done Sent Done false
6718 Acked Read 6 Done Sent Done false
6719 Acked Write 5 Done Sent Done false
6720 Acked Write 5 Done Sent Done false
6721 Acked Read 7 Done Sent Done false
6722 Acked Write 9 Done Sent Done false
6723 Acked Read 8 Done Sent Done false
6724 Acked Write 4 Done Sent Done false
6725 Acked Read 4 Done Sent Done false
6726 Acked Write 1 Done Sent Done false
6727 Acked Write 4 Done Sent Done false
6728 Acked Read 3 Done Sent Done false
6729 Acked Read 7 Done Sent Done false
6730 Acked Write 8 Done Sent Done false
6731 Acked Write 1 Done Sent Done false
6732 Acked Read 6 Done Sent Done false
6733 Acked Write 6 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(6709)) Some(JobId(6432)) Some(JobId(6709))
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6733 6732 6731 6730 6729
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:09.129605837Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:09.891580704Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.38288444Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59902)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.383150957Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.3843447Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.384879792Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.384907875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(6432))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.384927301Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.384944237Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(6432))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.384960973Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(6432))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:10.385145519Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 6734 6733 6732 6731 6730
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 28 06:06:12.629 INFO request completed, latency_us: 370, response_code: 204, uri:[8830] Got stop action so:false kr:false
/stop/cid/2, method: POST, req_id: 2bf9ade2-578d-4171-a4ca-9b9eaf9a317e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:06:12.630 INFO request completed, latency_us: 131, response_code: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630711964Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
200, uri: /state/cid/2, method: GET, req_id: 9c69820d-91df-4f71-b616-2e7fa7a6bf34, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630795374Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
.0.1:63908, local_addr:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630835304Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:12.630890271Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:06:16.633 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/2, method: GET, req_id: 39f06c08-ea0a-45f7-b7de-6d207abf38a6, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 1072 len 20480 data: 7 5 5 5 4
002/300 Write block 1939 len 16384 data: 3 3 4 4
003/300 Read block 4069 len 40960
004/300 Read block 4749 len 40960
005/300 Read block 3436 len 24576
006/300 Write block 3803 len 8192 data: 3 5
007/300 Write block 4501 len 8192 data: 6 5
008/300 Write block 2833 len 32768 data: 3 2 3 3 3 3 3 3
009/300 Write block 1579 len 20480 data: 7 7 7 7 7
010/300 Write block 1326 len 28672 data: 6 6 6 6 5 6 5
011/300 Write block 849 len 28672 data: 3 2 2 2 3 4 4
012/300 Write block 4397 len 40960 data: 4 3 3 3 3 4 4 4 7 5
013/300 Read block 566 len 36864
014/300 Write block 3015 len 12288 data: 5 3 4
015/300 Read block 3837 len 12288
016/300 Flush
017/300 Read block 996 len 20480
018/300 Read block 1205 len 40960
019/300 Read block 3000 len 8192
020/300 Write block 4480 len 16384 data: 4 5 6 6
021/300 Write block 890 len 28672 data: 3 3 3 3 4 3 3
022/300 Read block 2768 len 12288
023/300 Read block 432 len 8192
024/300 Read block 2672 len 12288
025/300 Read block 454 len 20480
026/300 Flush
027/300 Read block 333 len 4096
028/300 Write block 3714 len 24576 data: 2 4 4 6 8 7
029/300 Read block 764 len 4096
030/300 Read block 766 len 24576
031/300 Write block 543 len 36864 data: 2 2 2 3 4 4 5 6 7
032/300 Write block 341 len 32768 data: 5 4 4 3 4 4 4 4
033/300 Write block 1059 len 16384 data: 5 5 4 4
034/300 Read block 2294 len 4096
035/300 Flush
036/300 Write block 3545 len 8192 data: 4 5
037/300 Read block 2307 len 4096
038/300 Write block 2604 len 24576 data: 5 4 3 2 2 2
039/300 Write block 2110 len 8192 data: 5 4
040/300 Read block 2958 len 4096
041/300 Read block 1017 len 4096
042/300 Write block 2528 len 8192 data: 5 6
043/300 Write block 1281 len 36864 data: 5 4 5 3 2 3 3 3 4
044/300 Read block 1615 len 8192
045/300 Write block 2500 len 4096 data: 4
046/300 Read block 4739 len 8192
047/300 Write block 3000 len 20480 data: 4 5 3 3 3
048/300 Read block 4186 len 36864
049/300 Write block 2348 len 24576 data: 4 4 3 2 2 3
050/300 Read block 2012 len 32768
051/300 Read block 4118 len 20480
052/300 Write block 553 len 28672 data: 8 4 4 3 4 5 5
053/300 Read block 2286 len 8192
054/300 Read block 2007 len 24576
055/300 Read block 1985 len 40960
056/300 Flush
057/300 Read block 1383 len 28672
058/300 Read block 2960 len 12288
059/300 Write block 2961 len 16384 data: 5 6 5 5
060/300 Read block 2099 len 16384
061/300 Read block 4427 len 20480
062/300 Write block 1226 len 12288 data: 4 5 4
063/300 Read block 3416 len 20480
064/300 Read block 2386 len 4096
065/300 Flush
066/300 Read block 1902 len 8192
067/300 Write block 271 len 4096 data: 5
068/300 Flush
069/300 Write block 1625 len 40960 data: 6 6 6 6 6 6 6 4 6 4
070/300 Flush
071/300 Read block 2297 len 32768
072/300 Read block 3850 len 8192
073/300 Read block 3500 len 12288
074/300 Read block 1852 len 4096
075/300 Read block 3608 len 28672
076/300 Read block 1601 len 24576
077/300 Read block 2458 len 8192
078/300 Write block 2085 len 32768 data: 5 7 7 7 8 8 8 4
079/300 Read block 1975 len 20480
080/300 Read block 2465 len 16384
081/300 Write block 3219 len 24576 data: 4 5 5 4 4 4
082/300 Write block 2908 len 28672 data: 2 2 2 2 2 4 4
083/300 Read block 2951 len 28672
084/300 Write block 1848 len 20480 data: 4 4 4 5 5
085/300 Write block 4847 len 32768 data: 3 2 3 3 4 3 4 4
086/300 Read block 3777 len 28672
087/300 Read block 2142 len 40960
088/300 Read block 3410 len 32768
089/300 Read block 747 len 36864
090/300 Read block 2189 len 28672
091/300 Read block 4792 len 32768
092/300 Write block 3360 len 8192 data: 3 3
093/300 Read block 1791 len 32768
094/300 Read block 3062 len 24576
095/300 Read block 171 len 4096
096/300 Write block 1465 len 32768 data: 3 4 5 5 5 4 5 5
097/300 Read block 2768 len 16384
098/300 Read block 2605 len 24576
099/300 Write block 1316 len 4096 data: 3
100/300 Read block 142 len 40960
101/300 Flush
102/300 Read block 4510 len 24576
103/300 Flush
104/300 Read block 851 len 32768
105/300 Read block 4301 len 32768
106/300 Flush
107/300 Flush
108/300 Write block 1011 len 36864 data: 4 4 4 4 5 5 6 6 8
109/300 Write block 3634 len 4096 data: 4
110/300 Write block 526 len 4096 data: 4
111/300 Write block 1722 len 16384 data: 6 7 7 6
112/300 Write block 601 len 32768 data: 5 5 5 3 2 2 3 3
113/300 Write block 4836 len 28672 data: 6 7 8 7 7 6 7
114/300 Read block 180 len 16384
115/300 Flush
116/300 Read block 3875 len 16384
117/300 Flush
118/300 Read block 3962 len 28672
119/300 Write block 2577 len 24576 data: 2 2 4 4 4 4
120/300 Read block 4724 len 32768
121/300 Read block 927 len 20480
122/300 Write block 4982 len 4096 data: 5
123/300 Write block 1173 len 28672 data: 7 8 7 7 8 9 9
124/300 Write block 1507 len 4096 data: 3
125/300 Read block 1400 len 8192
126/300 Read block 246 len 4096
127/300 Write block 4134 len 4096 data: 3
128/300 Read block 2244 len 32768
129/300 Flush
130/300 Read block 473 len 12288
131/300 Read block 1482 len 36864
132/300 Flush
133/300 Write block 2093 len 16384 data: 5 5 5 6
134/300 Write block 3076 len 20480 data: 3 3 3 3 2
135/300 Read block 913 len 4096
136/300 Write block 3670 len 16384 data: 3 2 2 2
137/300 Read block 2135 len 12288
138/300 Read block 4781 len 28672
139/300 Read block 3850 len 24576
140/300 Read block 4746 len 8192
141/300 Read block 506 len 12288
142/300 Read block 4198 len 40960
143/300 Write block 1212 len 28672 data: 3 3 3 3 3 3 2
144/300 Read block 4077 len 32768
145/300 Read block 1189 len 12288
146/300 Read block 1888 len 32768
147/300 Read block 323 len 16384
148/300 Read block 903 len 16384
149/300 Write block 2352 len 36864 data: 3 4 2 2 3 4 4 4 4
150/300 Read block 4580 len 12288
151/300 Write block 1929 len 16384 data: 4 3 3 4
152/300 Read block 4369 len 28672
153/300 Write block 1169 len 24576 data: 6 5 5 5 8 9
154/300 Read block 2669 len 20480
155/300 Read block 3000 len 32768
156/300 Read block 4553 len 24576
157/300 Write block 163 len 8192 data: 4 4
158/300 Write block 4485 len 24576 data: 6 6 6 5 4 4
159/300 Read block 4897 len 16384
160/300 Flush
161/300 Read block 1451 len 16384
162/300 Read block 3781 len 32768
163/300 Write block 3199 len 40960 data: 5 5 6 6 6 6 7 6 6 5
164/300 Read block 3987 len 20480
165/300 Read block 1256 len 24576
166/300 Read block 546 len 8192
167/300 Flush
168/300 Write block 4494 len 8192 data: 4 5
169/300 Read block 642 len 12288
170/300 Write block 494 len 20480 data: 4 5 6 5 6
171/300 Read block 708 len 16384
172/300 Flush
173/300 Read block 2941 len 4096
174/300 Read block 818 len 4096
175/300 Read block 226 len 24576
176/300 Flush
177/300 Read block 1814 len 20480
178/300 Read block 735 len 36864
179/300 Read block 2491 len 8192
180/300 Write block 3667 len 40960 data: 4 4 4 4 3 3 3 2 2 3
181/300 Read block 3785 len 24576
182/300 Write block 2894 len 20480 data: 2 3 3 4 5
183/300 Read block 3425 len 28672
184/300 Read block 3638 len 40960
185/300 Write block 2871 len 40960 data: 5 5 4 4 2 3 3 3 3 2
186/300 Read block 4605 len 24576
187/300 Write block 1586 len 32768 data: 5 5 4 3 3 2 2 3
188/300 Write block 4106 len 16384 data: 3 3 4 4
189/300 Read block 2959 len 12288
190/300 Read block 3828 len 20480
191/300 Read block 3654 len 16384
192/300 Flush
193/300 Write block 4159 len 40960 data: 10 9 7 7 5 5 5 5 5 5
194/300 Write block 1591 len 28672 data: 3 3 4 3 3 3 3
195/300 Read block 3952 len 36864
196/300 Write block 4725 len 40960 data: 5 5 5 5 6 7 7 7 6 6
197/300 Write block 3327 len 4096 data: 4
198/300 Read block 148 len 28672
199/300 Read block 3153 len 12288
200/300 Flush
201/300 Read block 10 len 16384
202/300 Write block 4277 len 20480 data: 4 4 4 4 4
203/300 Read block 1604 len 20480
204/300 Write block 4303 len 36864 data: 3 3 3 4 5 4 4 3 3
205/300 Read block 3548 len 4096
206/300 Read block 1252 len 32768
207/300 Read block 1609 len 36864
208/300 Read block 4314 len 20480
209/300 Write block 3845 len 20480 data: 6 5 6 8 7
210/300 Read block 2070 len 16384
211/300 Write block 3614 len 24576 data: 6 6 4 4 3 4
212/300 Read block 4892 len 8192
213/300 Write block 1516 len 24576 data: 4 4 4 3 3 4
214/300 Write block 3089 len 28672 data: 4 5 5 5 6 5 3
215/300 Read block 2401 len 40960
216/300 Write block 871 len 4096 data: 5
217/300 Write block 4 len 12288 data: 3 3 3
218/300 Read block 2583 len 36864
219/300 Write block 1311 len 36864 data: 5 6 6 3 3 4 3 3 4
220/300 Write block 290 len 8192 data: 4 4
221/300 Write block 4538 len 28672 data: 3 4 4 3 3 3 4
222/300 Flush
223/300 Write block 974 len 40960 data: 2 4 4 4 5 4 5 4 4 5
224/300 Read block 2548 len 4096
225/300 Write block 1665 len 32768 data: 7 7 5 6 6 5 5 4
226/300 Flush
227/300 Flush
228/300 Write block 1709 len 16384 data: 8 8 7 8
229/300 Read block 4047 len 16384
230/300 Read block 215 len 4096
231/300 Flush
232/300 Read block 2859 len 4096
233/300 Write block 4320 len 36864 data: 6 7 6 7 6 5 4 4 4
234/300 Write block 3762 len 36864 data: 5 4 3 3 3 3 3 3 2
235/300 Read block 2639 len 40960
236/300 Write block 813 len 16384 data: 9 7 5 5
237/300 Read block 2536 len 8192
238/300 Read block 4610 len 24576
239/300 Write block 648 len 36864 data: 5 4 4 4 4 4 4 4 3
240/300 Write block 2226 len 28672 data: 5 6 6 5 6 8 7
241/300 Read block 4690 len 12288
242/300 Write block 3074 len 16384 data: 4 3 4 4
243/300 Read block 864 len 24576
244/300 Read block 4815 len 8192
245/300 Write block 17 len 24576 data: 3 4 4 5 5 5
246/300 Write block 2748 len 20480 data: 5 5 6 5 3
247/300 Write block 1551 len 28672 data: 4 2 2 4 4 4 5
248/300 Write block 2818 len 40960 data: 6 7 6 5 4 3 5 5 5 3
249/300 Write block 1450 len 16384 data: 3 4 4 4
250/300 Read block 2131 len 20480
251/300 Write block 2642 len 40960 data: 5 5 3 3 3 5 5 5 5 5
252/300 Flush
253/300 Write block 2418 len 24576 data: 4 4 4 4 4 5
254/300 Read block 1516 len 4096
255/300 Write block 155 len 24576 data: 5 5 3 2 3 4
256/300 Read block 1717 len 16384
257/300 Write block 64 len 24576 data: 5 3 3 2 2 2
258/300 Flush
259/300 Write block 4755 len 32768 data: 4 8 8 8 8 6 5 6
260/300 Read block 4418 len 12288
261/300 Read block 1511 len 32768
262/300 Write block 4688 len 20480 data: 3 3 3 3 4
263/300 Read block 4843 len 4096
264/300 Write block 4547 len 28672 data: 4 3 3 4 4 3 3
265/300 Read block 226 len 40960
266/300 Read block 3064 len 36864
267/300 Flush
268/300 Read block 2437 len 12288
269/300 Read block 2517 len 16384
270/300 Write block 662 len 8192 data: 7 6
271/300 Write block 78 len 16384 data: 6 6 5 6
272/300 Flush
273/300 Read block 1327 len 32768
274/300 Write block 3080 len 24576 data: 3 2 3 4 4 3
275/300 Write block 4063 len 8192 data: 4 4
276/300 Write block 1566 len 28672 data: 5 4 5 5 4 4 4
277/300 Read block 1128 len 12288
278/300 Read block 1828 len 24576
279/300 Read block 2130 len 40960
280/300 Write block 1971 len 24576 data: 6 6 6 6 6 4
281/300 Read block 4762 len 28672
282/300 Write block 1809 len 24576 data: 4 5 5 4 4 4
283/300 Read block 2817 len 28672
284/300 Write block 367 len 12288 data: 3 3 4
285/300 Write block 479 len 20480 data: 5 6 6 6 8
286/300 Flush
287/300 Write block 3246 len 36864 data: 4 4 3 6 6 5 6 4 4
288/300 Write block 1379 len 16384 data: 5 5 5 5
289/300 Flush
290/300 Read block 3690 len 24576
291/300 Flush
292/300 Write block 1586 len 24576 data: 6 6 5 4 4 4
293/300 Read block 919 len 28672
294/300 Write block 1816 len 8192 data: 4 4
295/300 Write block 588 len 36864 data: 3 2 3 3 3 3 3 3 3
296/300 Read block 2109 len 8192
297/300 Write block 4832 len 20480 data: 8 5 5 6 7
298/300 Read block 4553 len 36864
299/300 Write block 522 len 20480 data: 6 6 6 4 5
300/300 Read block 1422 len 8192
301/300 Read block 3543 len 24576
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:06:16.872 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1ef05e60-d8bd-44aa-9041-1773891c3067, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6735 Acked Write 5 Done Done Sent false
6736 Acked Write 4 Done Done Sent false
6737 Acked Read 10 Done Done Sent false
6738 Acked Read 10 Done Done Sent false
6739 Acked Read 6 Done Done Sent false
6740 Acked Write 2 Done Done Sent false
6741 Acked Write 2 Done Done Sent false
6742 Acked Write 8 Done Done Sent false
6743 Acked Write 5 Done Done Sent false
6744 Acked Write 7 Done Done Sent false
6745 Acked Write 7 Done Done Sent false
6746 Acked Write 10 Done Done Sent false
6747 Acked Read 9 Done Done Sent false
6748 Acked Write 3 Done Done Sent false
6749 Acked Read 3 Done Done Sent false
6750 Acked Flush 0 Done Done Sent false
6751 Acked Read 5 Done Done Sent false
6752 Acked Read 10 Done Done Sent false
6753 Acked Read 2 Done Done Sent false
6754 Acked Write 4 Done Done Sent false
6755 Acked Write 7 Done Done Sent false
6756 Acked Read 3 Done Done Sent false
6757 Acked Read 2 Done Done Sent false
6758 Acked Read 3 Done Done Sent false
6759 Acked Read 5 Done Done Sent false
6760 Acked Flush 0 Done Done Sent false
6761 Acked Read 1 Done Done Sent false
6762 Acked Write 6 Done Done Sent false
6763 Acked Read 1 Done Done Sent false
6764 Acked Read 6 Done Done Sent false
6765 Acked Write 9 Done Done Sent false
6766 Acked Write 8 Done Done Sent false
6767 Acked Write 4 Done Done Sent false
6768 Acked Read 1 Done Done Sent false
6769 Acked Flush 0 Done Done Sent false
6770 Acked Write 2 Done Done Sent false
6771 Acked Read 1 Done Done Sent false
6772 Acked Write 6 Done Done Sent false
6773 Acked Write 2 Done Done Sent false
6774 Acked Read 1 Done Done Sent false
6775 Acked Read 1 Done Done Sent false
6776 Acked Write 2 Done Done Sent false
6777 Acked Write 9 Done Done Sent false
6778 Acked Read 2 Done Done Sent false
6779 Acked Write 1 Done Done Sent false
6780 Acked Read 2 Done Done Sent false
6781 Acked Write 5 Done Done Sent false
6782 Acked Read 9 Done Done Sent false
6783 Acked Write 6 Done Done Sent false
6784 Acked Read 8 Done Done Sent false
6785 Acked Read 5 Done Done Sent false
6786 Acked Write 7 Done Done Sent false
6787 Acked Read 2 Done Done Sent false
6788 Acked Read 6 Done Done Sent false
6789 Acked Read 10 Done Done Sent false
6790 Acked Flush 0 Done Done Sent false
6791 Acked Read 7 Done Done Sent false
6792 Acked Read 3 Done Done Sent false
6793 Acked Write 4 Done Done Sent false
6794 Acked Read 4 Done Done Sent false
6795 Acked Read 5 Done Done Sent false
6796 Acked Write 3 Done Done Sent false
6797 Acked Read 5 Done Done Sent false
6798 Acked Read 1 Done Done Sent false
6799 Acked Flush 0 Done Done Sent false
6800 Acked Read 2 Done Done Sent false
6801 Acked Write 1 Done Done Sent false
6802 Acked Flush 0 Done Done Sent false
6803 Acked Write 10 Done Done Sent false
6804 Acked Flush 0 Done Done Sent false
6805 Acked Read 8 Done Done Sent false
6806 Acked Read 2 Done Done Sent false
6807 Acked Read 3 Done Done Sent false
6808 Acked Read 1 Done Done Sent false
6809 Acked Read 7 Done Done Sent false
6810 Acked Read 6 Done Done Sent false
6811 Acked Read 2 Done Done Sent false
6812 Acked Write 8 Done Done Sent false
6813 Acked Read 5 Done Done Sent false
6814 Acked Read 4 Done Done Sent false
6815 Acked Write 6 Done Done Sent false
6816 Acked Write 7 Done Done Sent false
6817 Acked Read 7 Done Done Sent false
6818 Acked Write 5 Done Done Sent false
6819 Acked Write 8 Done Done Sent false
6820 Acked Read 7 Done Done Sent false
6821 Acked Read 10 Done Done Sent false
6822 Acked Read 8 Done Done Sent false
6823 Acked Read 9 Done Done Sent false
6824 Acked Read 7 Done Done Sent false
6825 Acked Read 8 Done Done Sent false
6826 Acked Write 2 Done Done Sent false
6827 Acked Read 8 Done Done Sent false
6828 Acked Read 6 Done Done Sent false
6829 Acked Read 1 Done Done Sent false
6830 Acked Write 8 Done Done Sent false
6831 Acked Read 4 Done Done Sent false
6832 Acked Read 6 Done Done Sent false
6833 Acked Write 1 Done Done Sent false
6834 Acked Read 10 Done Done Sent false
6835 Acked Flush 0 Done Done Sent false
6836 Acked Read 6 Done Done Sent false
6837 Acked Flush 0 Done Done Sent false
6838 Acked Read 8 Done Done Sent false
6839 Acked Read 8 Done Done Sent false
6840 Acked Flush 0 Done Done Sent false
6841 Acked Flush 0 Done Done Sent false
6842 Acked Write 9 Done Done Sent false
6843 Acked Write 1 Done Done Sent false
6844 Acked Write 1 Done Done Sent false
6845 Acked Write 4 Done Done Sent false
6846 Acked Write 8 Done Done Sent false
6847 Acked Write 7 Done Done Sent false
6848 Acked Read 4 Done Done Sent false
6849 Acked Flush 0 Done Done Sent false
6850 Acked Read 4 Done Done Sent false
6851 Acked Flush 0 Done Done Sent false
6852 Acked Read 7 Done Done Sent false
6853 Acked Write 6 Done Done Sent false
6854 Acked Read 8 Done Done Sent false
6855 Acked Read 5 Done Done Sent false
6856 Acked Write 1 Done Done Sent false
6857 Acked Write 7 Done Done Sent false
6858 Acked Write 1 Done Done Sent false
6859 Acked Read 2 Done Done Sent false
6860 Acked Read 1 Done Done Sent false
6861 Acked Write 1 Done Done Sent false
6862 Acked Read 8 Done Done Sent false
6863 Acked Flush 0 Done Done Sent false
6864 Acked Read 3 Done Done Sent false
6865 Acked Read 9 Done Done Sent false
6866 Acked Flush 0 Done Done Sent false
6867 Acked Write 4 Done Done Sent false
6868 Acked Write 5 Done Done Sent false
6869 Acked Read 1 Done Done Sent false
6870 Acked Write 4 Done Done Sent false
6871 Acked Read 3 Done Done Sent false
6872 Acked Read 7 Done Done Sent false
6873 Acked Read 6 Done Done Sent false
6874 Acked Read 2 Done Done Sent false
6875 Acked Read 3 Done Done Sent false
6876 Acked Read 10 Done Done Sent false
6877 Acked Write 7 Done Done Sent false
6878 Acked Read 8 Done Done Sent false
6879 Acked Read 3 Done Done Sent false
6880 Acked Read 8 Done Done Sent false
6881 Acked Read 4 Done Done Sent false
6882 Acked Read 4 Done Done Sent false
6883 Acked Write 9 Done Done Sent false
6884 Acked Read 3 Done Done Sent false
6885 Acked Write 4 Done Done Sent false
6886 Acked Read 7 Done Done Sent false
6887 Acked Write 6 Done Done Sent false
6888 Acked Read 5 Done Done Sent false
6889 Acked Read 8 Done Done Sent false
6890 Acked Read 6 Done Done Sent false
6891 Acked Write 2 Done Done Sent false
6892 Acked Write 6 Done Done Sent false
6893 Acked Read 4 Done Done Sent false
6894 Acked Flush 0 Done Done Sent false
6895 Acked Read 4 Done Done Sent false
6896 Acked Read 8 Done Done Sent false
6897 Acked Write 10 Done Done Sent false
6898 Acked Read 5 Done Done Sent false
6899 Acked Read 6 Done Done Sent false
6900 Acked Read 2 Done Done Sent false
6901 Acked Flush 0 Done Done Sent false
6902 Acked Write 2 Done Done Sent false
6903 Acked Read 3 Done Done Sent false
6904 Acked Write 5 Done Done Sent false
6905 Acked Read 4 Done Done Sent false
6906 Acked Flush 0 Done Done Sent false
6907 Acked Read 1 Done Done Sent false
6908 Acked Read 1 Done Done Sent false
6909 Acked Read 6 Done Done Sent false
6910 Acked Flush 0 Done Done Sent false
6911 Acked Read 5 Done Done Sent false
6912 Acked Read 9 Done Done Sent false
6913 Acked Read 2 Done Done Sent false
6914 Acked Write 10 Done Done Sent false
6915 Acked Read 6 Done Done Sent false
6916 Acked Write 5 Done Done Sent false
6917 Acked Read 7 Done Done Sent false
6918 Acked Read 10 Done Done Sent false
6919 Acked Write 10 Done Done Sent false
6920 Acked Read 6 Done Done Sent false
6921 Acked Write 8 Done Done Sent false
6922 Acked Write 4 Done Done Sent false
6923 Acked Read 3 Done Done Sent false
6924 Acked Read 5 Done Done Sent false
6925 Acked Read 4 Done Done Sent false
6926 Acked Flush 0 Done Done Sent false
6927 Acked Write 10 Done Done Sent false
6928 Acked Write 7 Done Done Sent false
6929 Acked Read 9 Done Done Sent false
6930 Acked Write 10 Done Done Sent false
6931 Acked Write 1 Done Done Sent false
6932 Acked Read 7 Done Done Sent false
6933 Acked Read 3 Done Done Sent false
6934 Acked Flush 0 Done Done Sent false
6935 Acked Read 4 Done Done Sent false
6936 Acked Write 5 Done Done Sent false
6937 Acked Read 5 Done Done Sent false
6938 Acked Write 9 Done Done Sent false
6939 Acked Read 1 Done Done Sent false
6940 Acked Read 8 Done Done Sent false
6941 Acked Read 9 Done Done Sent false
6942 Acked Read 5 Done Done Sent false
6943 Acked Write 5 Done Done Sent false
6944 Acked Read 4 Done Done Sent false
6945 Acked Write 6 Done Done Sent false
6946 Acked Read 2 Done Done Sent false
6947 Acked Write 6 Done Done Sent false
6948 Acked Write 7 Done Done Sent false
6949 Acked Read 10 Done Done Sent false
6950 Acked Write 1 Done Done Sent false
6951 Acked Write 3 Done Done Sent false
6952 Acked Read 9 Done Done Sent false
6953 Acked Write 9 Done Done Sent false
6954 Acked Write 2 Done Done Sent false
6955 Acked Write 7 Done Done Sent false
6956 Acked Flush 0 Done Done Sent false
6957 Acked Write 10 Done Done Sent false
6958 Acked Read 1 Done Done Sent false
6959 Acked Write 8 Done Done Sent false
6960 Acked Flush 0 Done Done Sent false
6961 Acked Flush 0 Done Done Sent false
6962 Acked Write 4 Done Done Sent false
6963 Acked Read 4 Done Done Sent false
6964 Acked Read 1 Done Done Sent false
6965 Acked Flush 0 Done Done Sent false
6966 Acked Read 1 Done Done Sent false
6967 Acked Write 9 Done Done Sent false
6968 Acked Write 9 Done Done Sent false
6969 Acked Read 10 Done Done Sent false
6970 Acked Write 4 Done Done Sent false
6971 Acked Read 2 Done Done Sent false
6972 Acked Read 6 Done Done Sent false
6973 Acked Write 9 Done Done Sent false
6974 Acked Write 7 Done Done Sent false
6975 Acked Read 3 Done Done Sent false
6976 Acked Write 4 Done Done Sent false
6977 Acked Read 6 Done Done Sent false
6978 Acked Read 2 Done Done Sent false
6979 Acked Write 6 Done Done Sent false
6980 Acked Write 5 Done Done Sent false
6981 Acked Write 7 Done Done Sent false
6982 Acked Write 10 Done Done Sent false
6983 Acked Write 4 Done Done Sent false
6984 Acked Read 5 Done Done Sent false
6985 Acked Write 10 Done Done Sent false
6986 Acked Flush 0 Done Done Sent false
6987 Acked Write 6 Done Done Sent false
6988 Acked Read 1 Done Done Sent false
6989 Acked Write 6 Done Done Sent false
6990 Acked Read 4 Done Done Sent false
6991 Acked Write 6 Done Done Sent false
6992 Acked Flush 0 Done Done Sent false
6993 Acked Write 8 Done Done Sent false
6994 Acked Read 3 Done Done Sent false
6995 Acked Read 8 Done Done Sent false
6996 Acked Write 5 Done Done Sent false
6997 Acked Read 1 Done Done Sent false
6998 Acked Write 7 Done Done Sent false
6999 Acked Read 10 Done Done Sent false
7000 Acked Read 9 Done Done Sent false
7001 Acked Flush 0 Done Done Sent false
7002 Acked Read 3 Done Done Sent false
7003 Acked Read 4 Done Done Sent false
7004 Acked Write 2 Done Done Sent false
7005 Acked Write 4 Done Done Sent false
7006 Acked Flush 0 Done Done Sent false
7007 Acked Read 8 Done Done Sent false
7008 Acked Write 6 Done Done Sent false
7009 Acked Write 2 Done Done Sent false
7010 Acked Write 7 Done Done Sent false
7011 Acked Read 3 Done Done Sent false
7012 Acked Read 6 Done Done Sent false
7013 Acked Read 10 Done Done Sent false
7014 Acked Write 6 Done Done Sent false
7015 Acked Read 7 Done Done Sent false
7016 Acked Write 6 Done Done Sent false
7017 Acked Read 7 Done Done Sent false
7018 Acked Write 3 Done Done Sent false
7019 Acked Write 5 Done Done Sent false
7020 Acked Flush 0 Done Done Sent false
7021 Acked Write 9 Done Done Sent false
7022 Acked Write 4 Done Done Sent false
7023 Acked Flush 0 Done Done Sent false
7024 Acked Read 6 Done Done Sent false
7025 Acked Flush 0 Done Done Sent false
7026 Acked Write 6 Done Done Sent false
7027 Acked Read 7 Done Done Sent false
7028 Acked Write 2 Done Done Sent false
7029 Acked Write 9 Done Done Sent false
7030 Acked Read 2 Done Done Sent false
7031 Acked Write 5 Done Done Sent false
7032 Acked Read 9 Done Done Sent false
7033 Acked Write 5 Done Done Sent false
7034 Acked Read 2 Done Done Sent false
7035 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7025)) Some(JobId(7025)) Some(JobId(6734))
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 7035 7034 7033 7032 7031
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:17.867373282Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:18.86939803Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:19.871324911Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6735 Acked Write 5 Done Done Sent false
6736 Acked Write 4 Done Done Sent false
6737 Acked Read 10 Done Done Sent false
6738 Acked Read 10 Done Done Sent false
6739 Acked Read 6 Done Done Sent false
6740 Acked Write 2 Done Done Sent false
6741 Acked Write 2 Done Done Sent false
6742 Acked Write 8 Done Done Sent false
6743 Acked Write 5 Done Done Sent false
6744 Acked Write 7 Done Done Sent false
6745 Acked Write 7 Done Done Sent false
6746 Acked Write 10 Done Done Sent false
6747 Acked Read 9 Done Done Sent false
6748 Acked Write 3 Done Done Sent false
6749 Acked Read 3 Done Done Sent false
6750 Acked Flush 0 Done Done Sent false
6751 Acked Read 5 Done Done Sent false
6752 Acked Read 10 Done Done Sent false
6753 Acked Read 2 Done Done Sent false
6754 Acked Write 4 Done Done Sent false
6755 Acked Write 7 Done Done Sent false
6756 Acked Read 3 Done Done Sent false
6757 Acked Read 2 Done Done Sent false
6758 Acked Read 3 Done Done Sent false
6759 Acked Read 5 Done Done Sent false
6760 Acked Flush 0 Done Done Sent false
6761 Acked Read 1 Done Done Sent false
6762 Acked Write 6 Done Done Sent false
6763 Acked Read 1 Done Done Sent false
6764 Acked Read 6 Done Done Sent false
6765 Acked Write 9 Done Done Sent false
6766 Acked Write 8 Done Done Sent false
6767 Acked Write 4 Done Done Sent false
6768 Acked Read 1 Done Done Sent false
6769 Acked Flush 0 Done Done Sent false
6770 Acked Write 2 Done Done Sent false
6771 Acked Read 1 Done Done Sent false
6772 Acked Write 6 Done Done Sent false
6773 Acked Write 2 Done Done Sent false
6774 Acked Read 1 Done Done Sent false
6775 Acked Read 1 Done Done Sent false
6776 Acked Write 2 Done Done Sent false
6777 Acked Write 9 Done Done Sent false
6778 Acked Read 2 Done Done Sent false
6779 Acked Write 1 Done Done Sent false
6780 Acked Read 2 Done Done Sent false
6781 Acked Write 5 Done Done Sent false
6782 Acked Read 9 Done Done Sent false
6783 Acked Write 6 Done Done Sent false
6784 Acked Read 8 Done Done Sent false
6785 Acked Read 5 Done Done Sent false
6786 Acked Write 7 Done Done Sent false
6787 Acked Read 2 Done Done Sent false
6788 Acked Read 6 Done Done Sent false
6789 Acked Read 10 Done Done Sent false
6790 Acked Flush 0 Done Done Sent false
6791 Acked Read 7 Done Done Sent false
6792 Acked Read 3 Done Done Sent false
6793 Acked Write 4 Done Done Sent false
6794 Acked Read 4 Done Done Sent false
6795 Acked Read 5 Done Done Sent false
6796 Acked Write 3 Done Done Sent false
6797 Acked Read 5 Done Done Sent false
6798 Acked Read 1 Done Done Sent false
6799 Acked Flush 0 Done Done Sent false
6800 Acked Read 2 Done Done Sent false
6801 Acked Write 1 Done Done Sent false
6802 Acked Flush 0 Done Done Sent false
6803 Acked Write 10 Done Done Sent false
6804 Acked Flush 0 Done Done Sent false
6805 Acked Read 8 Done Done Sent false
6806 Acked Read 2 Done Done Sent false
6807 Acked Read 3 Done Done Sent false
6808 Acked Read 1 Done Done Sent false
6809 Acked Read 7 Done Done Sent false
6810 Acked Read 6 Done Done Sent false
6811 Acked Read 2 Done Done Sent false
6812 Acked Write 8 Done Done Sent false
6813 Acked Read 5 Done Done Sent false
6814 Acked Read 4 Done Done Sent false
6815 Acked Write 6 Done Done Sent false
6816 Acked Write 7 Done Done Sent false
6817 Acked Read 7 Done Done Sent false
6818 Acked Write 5 Done Done Sent false
6819 Acked Write 8 Done Done Sent false
6820 Acked Read 7 Done Done Sent false
6821 Acked Read 10 Done Done Sent false
6822 Acked Read 8 Done Done Sent false
6823 Acked Read 9 Done Done Sent false
6824 Acked Read 7 Done Done Sent false
6825 Acked Read 8 Done Done Sent false
6826 Acked Write 2 Done Done Sent false
6827 Acked Read 8 Done Done Sent false
6828 Acked Read 6 Done Done Sent false
6829 Acked Read 1 Done Done Sent false
6830 Acked Write 8 Done Done Sent false
6831 Acked Read 4 Done Done Sent false
6832 Acked Read 6 Done Done Sent false
6833 Acked Write 1 Done Done Sent false
6834 Acked Read 10 Done Done Sent false
6835 Acked Flush 0 Done Done Sent false
6836 Acked Read 6 Done Done Sent false
6837 Acked Flush 0 Done Done Sent false
6838 Acked Read 8 Done Done Sent false
6839 Acked Read 8 Done Done Sent false
6840 Acked Flush 0 Done Done Sent false
6841 Acked Flush 0 Done Done Sent false
6842 Acked Write 9 Done Done Sent false
6843 Acked Write 1 Done Done Sent false
6844 Acked Write 1 Done Done Sent false
6845 Acked Write 4 Done Done Sent false
6846 Acked Write 8 Done Done Sent false
6847 Acked Write 7 Done Done Sent false
6848 Acked Read 4 Done Done Sent false
6849 Acked Flush 0 Done Done Sent false
6850 Acked Read 4 Done Done Sent false
6851 Acked Flush 0 Done Done Sent false
6852 Acked Read 7 Done Done Sent false
6853 Acked Write 6 Done Done Sent false
6854 Acked Read 8 Done Done Sent false
6855 Acked Read 5 Done Done Sent false
6856 Acked Write 1 Done Done Sent false
6857 Acked Write 7 Done Done Sent false
6858 Acked Write 1 Done Done Sent false
6859 Acked Read 2 Done Done Sent false
6860 Acked Read 1 Done Done Sent false
6861 Acked Write 1 Done Done Sent false
6862 Acked Read 8 Done Done Sent false
6863 Acked Flush 0 Done Done Sent false
6864 Acked Read 3 Done Done Sent false
6865 Acked Read 9 Done Done Sent false
6866 Acked Flush 0 Done Done Sent false
6867 Acked Write 4 Done Done Sent false
6868 Acked Write 5 Done Done Sent false
6869 Acked Read 1 Done Done Sent false
6870 Acked Write 4 Done Done Sent false
6871 Acked Read 3 Done Done Sent false
6872 Acked Read 7 Done Done Sent false
6873 Acked Read 6 Done Done Sent false
6874 Acked Read 2 Done Done Sent false
6875 Acked Read 3 Done Done Sent false
6876 Acked Read 10 Done Done Sent false
6877 Acked Write 7 Done Done Sent false
6878 Acked Read 8 Done Done Sent false
6879 Acked Read 3 Done Done Sent false
6880 Acked Read 8 Done Done Sent false
6881 Acked Read 4 Done Done Sent false
6882 Acked Read 4 Done Done Sent false
6883 Acked Write 9 Done Done Sent false
6884 Acked Read 3 Done Done Sent false
6885 Acked Write 4 Done Done Sent false
6886 Acked Read 7 Done Done Sent false
6887 Acked Write 6 Done Done Sent false
6888 Acked Read 5 Done Done Sent false
6889 Acked Read 8 Done Done Sent false
6890 Acked Read 6 Done Done Sent false
6891 Acked Write 2 Done Done Sent false
6892 Acked Write 6 Done Done Sent false
6893 Acked Read 4 Done Done Sent false
6894 Acked Flush 0 Done Done Sent false
6895 Acked Read 4 Done Done Sent false
6896 Acked Read 8 Done Done Sent false
6897 Acked Write 10 Done Done Sent false
6898 Acked Read 5 Done Done Sent false
6899 Acked Read 6 Done Done Sent false
6900 Acked Read 2 Done Done Sent false
6901 Acked Flush 0 Done Done Sent false
6902 Acked Write 2 Done Done Sent false
6903 Acked Read 3 Done Done Sent false
6904 Acked Write 5 Done Done Sent false
6905 Acked Read 4 Done Done Sent false
6906 Acked Flush 0 Done Done Sent false
6907 Acked Read 1 Done Done Sent false
6908 Acked Read 1 Done Done Sent false
6909 Acked Read 6 Done Done Sent false
6910 Acked Flush 0 Done Done Sent false
6911 Acked Read 5 Done Done Sent false
6912 Acked Read 9 Done Done Sent false
6913 Acked Read 2 Done Done Sent false
6914 Acked Write 10 Done Done Sent false
6915 Acked Read 6 Done Done Sent false
6916 Acked Write 5 Done Done Sent false
6917 Acked Read 7 Done Done Sent false
6918 Acked Read 10 Done Done Sent false
6919 Acked Write 10 Done Done Sent false
6920 Acked Read 6 Done Done Sent false
6921 Acked Write 8 Done Done Sent false
6922 Acked Write 4 Done Done Sent false
6923 Acked Read 3 Done Done Sent false
6924 Acked Read 5 Done Done Sent false
6925 Acked Read 4 Done Done Sent false
6926 Acked Flush 0 Done Done Sent false
6927 Acked Write 10 Done Done Sent false
6928 Acked Write 7 Done Done Sent false
6929 Acked Read 9 Done Done Sent false
6930 Acked Write 10 Done Done Sent false
6931 Acked Write 1 Done Done Sent false
6932 Acked Read 7 Done Done Sent false
6933 Acked Read 3 Done Done Sent false
6934 Acked Flush 0 Done Done Sent false
6935 Acked Read 4 Done Done Sent false
6936 Acked Write 5 Done Done Sent false
6937 Acked Read 5 Done Done Sent false
6938 Acked Write 9 Done Done Sent false
6939 Acked Read 1 Done Done Sent false
6940 Acked Read 8 Done Done Sent false
6941 Acked Read 9 Done Done Sent false
6942 Acked Read 5 Done Done Sent false
6943 Acked Write 5 Done Done Sent false
6944 Acked Read 4 Done Done Sent false
6945 Acked Write 6 Done Done Sent false
6946 Acked Read 2 Done Done Sent false
6947 Acked Write 6 Done Done Sent false
6948 Acked Write 7 Done Done Sent false
6949 Acked Read 10 Done Done Sent false
6950 Acked Write 1 Done Done Sent false
6951 Acked Write 3 Done Done Sent false
6952 Acked Read 9 Done Done Sent false
6953 Acked Write 9 Done Done Sent false
6954 Acked Write 2 Done Done Sent false
6955 Acked Write 7 Done Done Sent false
6956 Acked Flush 0 Done Done Sent false
6957 Acked Write 10 Done Done Sent false
6958 Acked Read 1 Done Done Sent false
6959 Acked Write 8 Done Done Sent false
6960 Acked Flush 0 Done Done Sent false
6961 Acked Flush 0 Done Done Sent false
6962 Acked Write 4 Done Done Sent false
6963 Acked Read 4 Done Done Sent false
6964 Acked Read 1 Done Done Sent false
6965 Acked Flush 0 Done Done Sent false
6966 Acked Read 1 Done Done Sent false
6967 Acked Write 9 Done Done Sent false
6968 Acked Write 9 Done Done Sent false
6969 Acked Read 10 Done Done Sent false
6970 Acked Write 4 Done Done Sent false
6971 Acked Read 2 Done Done Sent false
6972 Acked Read 6 Done Done Sent false
6973 Acked Write 9 Done Done Sent false
6974 Acked Write 7 Done Done Sent false
6975 Acked Read 3 Done Done Sent false
6976 Acked Write 4 Done Done Sent false
6977 Acked Read 6 Done Done Sent false
6978 Acked Read 2 Done Done Sent false
6979 Acked Write 6 Done Done Sent false
6980 Acked Write 5 Done Done Sent false
6981 Acked Write 7 Done Done Sent false
6982 Acked Write 10 Done Done Sent false
6983 Acked Write 4 Done Done Sent false
6984 Acked Read 5 Done Done Sent false
6985 Acked Write 10 Done Done Sent false
6986 Acked Flush 0 Done Done Sent false
6987 Acked Write 6 Done Done Sent false
6988 Acked Read 1 Done Done Sent false
6989 Acked Write 6 Done Done Sent false
6990 Acked Read 4 Done Done Sent false
6991 Acked Write 6 Done Done Sent false
6992 Acked Flush 0 Done Done Sent false
6993 Acked Write 8 Done Done Sent false
6994 Acked Read 3 Done Done Sent false
6995 Acked Read 8 Done Done Sent false
6996 Acked Write 5 Done Done Sent false
6997 Acked Read 1 Done Done Sent false
6998 Acked Write 7 Done Done Sent false
6999 Acked Read 10 Done Done Sent false
7000 Acked Read 9 Done Done Sent false
7001 Acked Flush 0 Done Done Sent false
7002 Acked Read 3 Done Done Sent false
7003 Acked Read 4 Done Done Sent false
7004 Acked Write 2 Done Done Sent false
7005 Acked Write 4 Done Done Sent false
7006 Acked Flush 0 Done Done Sent false
7007 Acked Read 8 Done Done Sent false
7008 Acked Write 6 Done Done Sent false
7009 Acked Write 2 Done Done Sent false
7010 Acked Write 7 Done Done Sent false
7011 Acked Read 3 Done Done Sent false
7012 Acked Read 6 Done Done Sent false
7013 Acked Read 10 Done Done Sent false
7014 Acked Write 6 Done Done Sent false
7015 Acked Read 7 Done Done Sent false
7016 Acked Write 6 Done Done Sent false
7017 Acked Read 7 Done Done Sent false
7018 Acked Write 3 Done Done Sent false
7019 Acked Write 5 Done Done Sent false
7020 Acked Flush 0 Done Done Sent false
7021 Acked Write 9 Done Done Sent false
7022 Acked Write 4 Done Done Sent false
7023 Acked Flush 0 Done Done Sent false
7024 Acked Read 6 Done Done Sent false
7025 Acked Flush 0 Done Done Sent false
7026 Acked Write 6 Done Done Sent false
7027 Acked Read 7 Done Done Sent false
7028 Acked Write 2 Done Done Sent false
7029 Acked Write 9 Done Done Sent false
7030 Acked Read 2 Done Done Sent false
7031 Acked Write 5 Done Done Sent false
7032 Acked Read 9 Done Done Sent false
7033 Acked Write 5 Done Done Sent false
7034 Acked Read 2 Done Done Sent false
7035 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7025)) Some(JobId(7025)) Some(JobId(6734))
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 7035 7034 7033 7032 7031
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:21.874425254Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.632607281Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35913)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.632799375Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.633936073Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634454479Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634478923Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(6734))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634492989Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634505846Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(6734))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634519343Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(6734))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:22.634733732Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7036 7035 7034 7033 7032
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:06:24.877[8820] Got stop action so:false kr:false
INFO request completed, latency_us: 328, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8db6f951-308f-4113-9945-44dc3d20bebc, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:06:24.878 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2a87e503-4aba-49ec-8fab-a6ef89ec9838, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:24.878679225Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:24.87874161Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:24.878765664Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:24.878859422Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:06:28.881 INFO request completed, latency_us: 312, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1c464c3a-1784-411b-beb8-00051d28d847, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 4028 len 40960
002/300 Flush
003/300 Read block 1692 len 8192
004/300 Write block 3319 len 24576 data: 2 4 4 5 5 5
005/300 Read block 2857 len 28672
006/300 Write block 1040 len 24576 data: 6 5 5 5 5 5
007/300 Read block 373 len 16384
008/300 Write block 3361 len 32768 data: 4 3 4 5 4 5 4 5
009/300 Read block 3657 len 16384
010/300 Read block 424 len 36864
011/300 Read block 3822 len 8192
012/300 Write block 2356 len 28672 data: 4 5 5 5 5 4 5
013/300 Read block 3689 len 32768
014/300 Read block 3145 len 16384
015/300 Read block 4274 len 20480
016/300 Flush
017/300 Write block 4354 len 24576 data: 6 8 6 7 7 6
018/300 Flush
019/300 Read block 2860 len 28672
020/300 Write block 1298 len 40960 data: 5 4 4 4 4 4 4 4 3 3
021/300 Read block 1065 len 28672
022/300 Write block 1120 len 32768 data: 4 3 4 4 5 7 7 7
023/300 Write block 2435 len 36864 data: 4 4 3 3 4 5 4 6 5
024/300 Write block 190 len 28672 data: 5 6 6 6 7 7 7
025/300 Read block 1622 len 40960
026/300 Read block 2146 len 20480
027/300 Read block 745 len 24576
028/300 Read block 368 len 12288
029/300 Flush
030/300 Read block 3193 len 8192
031/300 Read block 3558 len 12288
032/300 Read block 3083 len 8192
033/300 Write block 1049 len 4096 data: 6
034/300 Read block 2121 len 8192
035/300 Read block 359 len 36864
036/300 Write block 1942 len 32768 data: 5 4 4 3 4 5 4 2
037/300 Read block 4673 len 28672
038/300 Read block 744 len 40960
039/300 Write block 2626 len 40960 data: 5 4 4 4 5 5 4 2 3 3
040/300 Read block 2408 len 20480
041/300 Write block 2461 len 4096 data: 5
042/300 Write block 4496 len 24576 data: 6 5 5 6 6 7
043/300 Flush
044/300 Read block 248 len 40960
045/300 Read block 4105 len 24576
046/300 Read block 146 len 24576
047/300 Read block 3966 len 36864
048/300 Write block 4358 len 12288 data: 8 7 4
049/300 Read block 4752 len 28672
050/300 Read block 4691 len 40960
051/300 Read block 842 len 40960
052/300 Write block 2534 len 4096 data: 3
053/300 Read block 4328 len 20480
054/300 Read block 346 len 36864
055/300 Read block 2725 len 40960
056/300 Write block 1335 len 24576 data: 5 5 6 5 5 6
057/300 Write block 3209 len 32768 data: 5 4 4 4 4 4 5 5
058/300 Write block 3108 len 20480 data: 9 9 9 8 9
059/300 Write block 525 len 40960 data: 5 6 3 3 4 3 3 4 4 4
060/300 Write block 3633 len 16384 data: 5 5 4 4
061/300 Read block 3853 len 16384
062/300 Write block 4228 len 12288 data: 11 11 9
063/300 Read block 559 len 12288
064/300 Flush
065/300 Write block 1540 len 4096 data: 6
066/300 Read block 1040 len 24576
067/300 Read block 3126 len 36864
068/300 Read block 221 len 4096
069/300 Write block 1454 len 28672 data: 4 3 4 4 3 4 3
070/300 Read block 3406 len 32768
071/300 Write block 641 len 20480 data: 4 4 4 4 4
072/300 Write block 466 len 20480 data: 4 5 6 6 7
073/300 Read block 2433 len 8192
074/300 Write block 4306 len 28672 data: 5 6 5 5 4 4 4
075/300 Write block 2536 len 8192 data: 3 3
076/300 Read block 2622 len 40960
077/300 Read block 4205 len 36864
078/300 Read block 2433 len 28672
079/300 Write block 4204 len 32768 data: 6 5 6 4 4 4 4 4
080/300 Read block 1713 len 32768
081/300 Write block 4865 len 4096 data: 5
082/300 Write block 4596 len 28672 data: 8 7 6 4 4 4 4
083/300 Read block 3512 len 4096
084/300 Write block 2130 len 20480 data: 3 4 5 5 5
085/300 Write block 2721 len 40960 data: 4 3 2 3 3 3 3 3 4 7
086/300 Write block 3207 len 8192 data: 7 6
087/300 Read block 877 len 16384
088/300 Write block 561 len 28672 data: 5 5 5 5 5 3 3
089/300 Write block 2620 len 40960 data: 2 2 3 3 4 4 6 5 5 5
090/300 Write block 338 len 36864 data: 6 4 4 6 5 5 4 5 5
091/300 Write block 4973 len 24576 data: 5 5 4 4 3 5
092/300 Read block 98 len 24576
093/300 Read block 455 len 28672
094/300 Write block 2451 len 32768 data: 5 5 4 5 7 7 8 8
095/300 Read block 1614 len 32768
096/300 Read block 2979 len 16384
097/300 Write block 1546 len 40960 data: 5 4 4 4 4 5 3 3 5 5
098/300 Read block 826 len 12288
099/300 Read block 2055 len 12288
100/300 Read block 3772 len 12288
101/300 Write block 3148 len 24576 data: 3 2 2 3 3 3
102/300 Flush
103/300 Read block 3453 len 8192
104/300 Write block 345 len 4096 data: 6
105/300 Write block 3964 len 24576 data: 2 2 2 2 2 2
106/300 Read block 3696 len 32768
107/300 Flush
108/300 Flush
109/300 Write block 1118 len 16384 data: 3 3 5 4
110/300 Read block 3358 len 28672
111/300 Write block 3006 len 4096 data: 4
112/300 Read block 4745 len 8192
113/300 Read block 104 len 40960
114/300 Read block 499 len 8192
115/300 Write block 3703 len 24576 data: 2 3 4 5 7 5
116/300 Flush
117/300 Read block 1 len 16384
118/300 Read block 4952 len 12288
119/300 Write block 1190 len 4096 data: 4
120/300 Write block 4404 len 40960 data: 5 8 6 7 7 7 7 7 7 5
121/300 Write block 146 len 24576 data: 5 5 5 6 5 5
122/300 Write block 2905 len 24576 data: 2 2 2 3 3 3
123/300 Read block 28 len 16384
124/300 Read block 811 len 36864
125/300 Write block 994 len 24576 data: 4 3 3 4 4 4
126/300 Write block 994 len 4096 data: 5
127/300 Write block 4835 len 8192 data: 7 8
128/300 Read block 4787 len 12288
129/300 Write block 1716 len 28672 data: 5 5 4 3 4 5 7
130/300 Flush
131/300 Write block 4968 len 24576 data: 7 6 6 7 6 6
132/300 Write block 505 len 40960 data: 5 5 4 4 5 3 3 2 2 2
133/300 Write block 1747 len 20480 data: 6 5 5 5 3
134/300 Read block 3005 len 28672
135/300 Read block 2024 len 16384
136/300 Read block 1749 len 8192
137/300 Flush
138/300 Read block 3997 len 4096
139/300 Read block 635 len 28672
140/300 Write block 3616 len 36864 data: 5 5 4 5 3 4 4 5 5
141/300 Write block 1393 len 20480 data: 2 3 3 3 2
142/300 Read block 2894 len 20480
143/300 Write block 2751 len 4096 data: 6
144/300 Read block 3709 len 4096
145/300 Write block 4486 len 24576 data: 7 7 6 5 5 3
146/300 Write block 3538 len 20480 data: 6 4 4 4 4
147/300 Write block 1142 len 32768 data: 4 4 4 5 4 3 4 4
148/300 Read block 1716 len 16384
149/300 Write block 4747 len 20480 data: 3 3 2 3 3
150/300 Write block 742 len 20480 data: 4 4 5 4 4
151/300 Read block 4890 len 32768
152/300 Read block 1003 len 36864
153/300 Read block 4328 len 32768
154/300 Write block 1993 len 32768 data: 3 3 3 3 3 3 4 6
155/300 Read block 4361 len 16384
156/300 Flush
157/300 Write block 3354 len 28672 data: 5 5 5 4 2 2 4
158/300 Read block 3330 len 12288
159/300 Write block 256 len 24576 data: 5 5 4 4 5 4
160/300 Write block 1251 len 20480 data: 3 4 6 7 8
161/300 Read block 4779 len 12288
162/300 Read block 1672 len 40960
163/300 Write block 461 len 8192 data: 4 3
164/300 Read block 4090 len 24576
165/300 Read block 3268 len 12288
166/300 Write block 3266 len 16384 data: 3 3 3 4
167/300 Read block 4540 len 16384
168/300 Write block 2199 len 32768 data: 4 6 5 6 4 4 3 3
169/300 Read block 4049 len 20480
170/300 Read block 4551 len 40960
171/300 Read block 1478 len 8192
172/300 Flush
173/300 Read block 2419 len 16384
174/300 Read block 2429 len 8192
175/300 Read block 4814 len 20480
176/300 Write block 1433 len 20480 data: 4 4 4 5 6
177/300 Read block 269 len 16384
178/300 Flush
179/300 Read block 4743 len 12288
180/300 Write block 1082 len 8192 data: 5 4
181/300 Write block 1608 len 28672 data: 5 5 3 3 3 5 5
182/300 Write block 774 len 16384 data: 4 5 5 5
183/300 Write block 2911 len 24576 data: 3 3 5 5 4 3
184/300 Flush
185/300 Read block 619 len 32768
186/300 Read block 731 len 20480
187/300 Read block 1642 len 8192
188/300 Read block 4960 len 24576
189/300 Write block 2795 len 32768 data: 6 5 5 4 5 5 5 3
190/300 Read block 1018 len 24576
191/300 Flush
192/300 Read block 1080 len 8192
193/300 Read block 1151 len 40960
194/300 Read block 1736 len 20480
195/300 Write block 723 len 8192 data: 4 4
196/300 Write block 2603 len 4096 data: 5
197/300 Read block 475 len 12288
198/300 Flush
199/300 Flush
200/300 Read block 2908 len 4096
201/300 Write block 1420 len 12288 data: 5 6 5
202/300 Read block 2344 len 32768
203/300 Read block 2094 len 32768
204/300 Read block 2431 len 12288
205/300 Read block 2556 len 32768
206/300 Read block 17 len 40960
207/300 Write block 2931 len 8192 data: 5 4
208/300 Write block 1156 len 36864 data: 7 7 6 6 5 5 4 4 6
209/300 Write block 47 len 16384 data: 4 4 4 3
210/300 Read block 1896 len 20480
211/300 Read block 1249 len 20480
212/300 Write block 2905 len 24576 data: 3 3 3 4 4 4
213/300 Write block 515 len 40960 data: 4 4 4 4 4 5 6 7 7 7
214/300 Flush
215/300 Read block 4775 len 8192
216/300 Write block 2651 len 40960 data: 6 2 3 3 3 4 5 5 5 4
217/300 Read block 2912 len 20480
218/300 Write block 805 len 36864 data: 6 6 5 5 5 5 5 8 10
219/300 Write block 928 len 20480 data: 6 6 7 7 7
220/300 Write block 188 len 16384 data: 5 5 6 7
221/300 Read block 3013 len 36864
222/300 Read block 2050 len 28672
223/300 Read block 3817 len 8192
224/300 Read block 836 len 28672
225/300 Write block 3684 len 40960 data: 4 4 4 4 5 3 5 5 6 4
226/300 Flush
227/300 Write block 3675 len 24576 data: 3 4 3 3 3 3
228/300 Write block 3418 len 28672 data: 4 3 2 2 2 2 3
229/300 Read block 505 len 24576
230/300 Read block 3514 len 40960
231/300 Write block 2673 len 28672 data: 8 8 9 8 5 6 6
232/300 Write block 79 len 20480 data: 7 6 7 5 5
233/300 Write block 3588 len 16384 data: 4 5 4 4
234/300 Write block 2785 len 32768 data: 4 5 5 5 6 6 5 5
235/300 Read block 3018 len 16384
236/300 Read block 594 len 32768
237/300 Read block 4629 len 24576
238/300 Write block 4096 len 36864 data: 6 6 6 6 6 7 6 6 5
239/300 Read block 4293 len 8192
240/300 Flush
241/300 Read block 684 len 40960
242/300 Write block 3768 len 32768 data: 4 4 3 3 3 4 4 2
243/300 Write block 1261 len 32768 data: 7 7 6 7 5 4 3 3
244/300 Flush
245/300 Write block 432 len 40960 data: 5 5 5 6 8 7 8 8 8 8
246/300 Write block 1145 len 32768 data: 6 5 4 5 5 4 6 7
247/300 Read block 1246 len 8192
248/300 Write block 3325 len 40960 data: 5 4 5 4 4 4 4 5 4 5
249/300 Read block 195 len 16384
250/300 Read block 3272 len 8192
251/300 Flush
252/300 Write block 4045 len 16384 data: 7 6 4 3
253/300 Read block 3187 len 36864
254/300 Read block 4480 len 32768
255/300 Read block 139 len 16384
256/300 Read block 4577 len 20480
257/300 Read block 454 len 20480
258/300 Read block 942 len 40960
259/300 Write block 2 len 32768 data: 2 3 4 4 4 3 3 3
260/300 Read block 3529 len 4096
261/300 Read block 4940 len 16384
262/300 Read block 2082 len 16384
263/300 Write block 4887 len 8192 data: 11 10
264/300 Write block 1053 len 8192 data: 6 7
265/300 Read block 231 len 4096
266/300 Read block 1462 len 12288
267/300 Read block 1760 len 36864
268/300 Read block 1429 len 8192
269/300 Write block 1869 len 4096 data: 5
270/300 Write block 2878 len 28672 data: 4 4 3 2 3 4 6
271/300 Read block 2136 len 24576
272/300 Write block 1499 len 36864 data: 3 3 4 3 4 3 3 3 4
273/300 Read block 1747 len 36864
274/300 Write block 3474 len 20480 data: 5 6 5 5 4
275/300 Write block 2467 len 8192 data: 3 4
276/300 Write block 4888 len 40960 data: 11 11 10 9 7 8 8 8 7 7
277/300 Write block 623 len 12288 data: 4 4 5
278/300 Read block 4450 len 20480
279/300 Read block 1610 len 24576
280/300 Flush
281/300 Read block 755 len 16384
282/300 Write block 901 len 16384 data: 5 6 4 3
283/300 Write block 1309 len 12288 data: 5 5 6
284/300 Write block 2419 len 12288 data: 5 5 5
285/300 Read block 4834 len 36864
286/300 Read block 1862 len 16384
287/300 Write block 3742 len 24576 data: 7 8 8 8 6 6
288/300 Read block 3523 len 16384
289/300 Read block 1613 len 16384
290/300 Read block 999 len 8192
291/300 Flush
292/300 Read block 4139 len 24576
293/300 Flush
294/300 Read block 3208 len 4096
295/300 Flush
296/300 Write block 3515 len 32768 data: 3 3 2 2 2 3 3 6
297/300 Write block 2039 len 32768 data: 3 2 2 2 2 2 4 4
298/300 Write block 1037 len 16384 data: 4 4 5 7
299/300 Flush
300/300 Read block 3506 len 32768
301/300 Read block 4405 len 12288
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:06:29.125 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: bdd32c47-d06e-4b96-b21d-eb448149e30b, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 10 Done Sent Done false
7038 Acked Flush 0 Done Sent Done false
7039 Acked Read 2 Done Sent Done false
7040 Acked Write 6 Done Sent Done false
7041 Acked Read 7 Done Sent Done false
7042 Acked Write 6 Done Sent Done false
7043 Acked Read 4 Done Sent Done false
7044 Acked Write 8 Done Sent Done false
7045 Acked Read 4 Done Sent Done false
7046 Acked Read 9 Done Sent Done false
7047 Acked Read 2 Done Sent Done false
7048 Acked Write 7 Done Sent Done false
7049 Acked Read 8 Done Sent Done false
7050 Acked Read 4 Done Sent Done false
7051 Acked Read 5 Done Sent Done false
7052 Acked Flush 0 Done Sent Done false
7053 Acked Write 6 Done Sent Done false
7054 Acked Flush 0 Done Sent Done false
7055 Acked Read 7 Done Sent Done false
7056 Acked Write 10 Done Sent Done false
7057 Acked Read 7 Done Sent Done false
7058 Acked Write 8 Done Sent Done false
7059 Acked Write 9 Done Sent Done false
7060 Acked Write 7 Done Sent Done false
7061 Acked Read 10 Done Sent Done false
7062 Acked Read 5 Done Sent Done false
7063 Acked Read 6 Done Sent Done false
7064 Acked Read 3 Done Sent Done false
7065 Acked Flush 0 Done Sent Done false
7066 Acked Read 2 Done Sent Done false
7067 Acked Read 3 Done Sent Done false
7068 Acked Read 2 Done Sent Done false
7069 Acked Write 1 Done Sent Done false
7070 Acked Read 2 Done Sent Done false
7071 Acked Read 9 Done Sent Done false
7072 Acked Write 8 Done Sent Done false
7073 Acked Read 7 Done Sent Done false
7074 Acked Read 10 Done Sent Done false
7075 Acked Write 10 Done Sent Done false
7076 Acked Read 5 Done Sent Done false
7077 Acked Write 1 Done Sent Done false
7078 Acked Write 6 Done Sent Done false
7079 Acked Flush 0 Done Sent Done false
7080 Acked Read 10 Done Sent Done false
7081 Acked Read 6 Done Sent Done false
7082 Acked Read 6 Done Sent Done false
7083 Acked Read 9 Done Sent Done false
7084 Acked Write 3 Done Sent Done false
7085 Acked Read 7 Done Sent Done false
7086 Acked Read 10 Done Sent Done false
7087 Acked Read 10 Done Sent Done false
7088 Acked Write 1 Done Sent Done false
7089 Acked Read 5 Done Sent Done false
7090 Acked Read 9 Done Sent Done false
7091 Acked Read 10 Done Sent Done false
7092 Acked Write 6 Done Sent Done false
7093 Acked Write 8 Done Sent Done false
7094 Acked Write 5 Done Sent Done false
7095 Acked Write 10 Done Sent Done false
7096 Acked Write 4 Done Sent Done false
7097 Acked Read 4 Done Sent Done false
7098 Acked Write 3 Done Sent Done false
7099 Acked Read 3 Done Sent Done false
7100 Acked Flush 0 Done Sent Done false
7101 Acked Write 1 Done Sent Done false
7102 Acked Read 6 Done Sent Done false
7103 Acked Read 9 Done Sent Done false
7104 Acked Read 1 Done Sent Done false
7105 Acked Write 7 Done Sent Done false
7106 Acked Read 8 Done Sent Done false
7107 Acked Write 5 Done Sent Done false
7108 Acked Write 5 Done Sent Done false
7109 Acked Read 2 Done Sent Done false
7110 Acked Write 7 Done Sent Done false
7111 Acked Write 2 Done Sent Done false
7112 Acked Read 10 Done Sent Done false
7113 Acked Read 9 Done Sent Done false
7114 Acked Read 7 Done Sent Done false
7115 Acked Write 8 Done Sent Done false
7116 Acked Read 8 Done Sent Done false
7117 Acked Write 1 Done Sent Done false
7118 Acked Write 7 Done Sent Done false
7119 Acked Read 1 Done Sent Done false
7120 Acked Write 5 Done Sent Done false
7121 Acked Write 10 Done Sent Done false
7122 Acked Write 2 Done Sent Done false
7123 Acked Read 4 Done Sent Done false
7124 Acked Write 7 Done Sent Done false
7125 Acked Write 10 Done Sent Done false
7126 Acked Write 9 Done Sent Done false
7127 Acked Write 6 Done Sent Done false
7128 Acked Read 6 Done Sent Done false
7129 Acked Read 7 Done Sent Done false
7130 Acked Write 8 Done Sent Done false
7131 Acked Read 8 Done Sent Done false
7132 Acked Read 4 Done Sent Done false
7133 Acked Write 10 Done Sent Done false
7134 Acked Read 3 Done Sent Done false
7135 Acked Read 3 Done Sent Done false
7136 Acked Read 3 Done Sent Done false
7137 Acked Write 6 Done Sent Done false
7138 Acked Flush 0 Done Sent Done false
7139 Acked Read 2 Done Sent Done false
7140 Acked Write 1 Done Sent Done false
7141 Acked Write 6 Done Sent Done false
7142 Acked Read 8 Done Sent Done false
7143 Acked Flush 0 Done Sent Done false
7144 Acked Flush 0 Done Sent Done false
7145 Acked Write 4 Done Sent Done false
7146 Acked Read 7 Done Sent Done false
7147 Acked Write 1 Done Sent Done false
7148 Acked Read 2 Done Sent Done false
7149 Acked Read 10 Done Sent Done false
7150 Acked Read 2 Done Sent Done false
7151 Acked Write 6 Done Sent Done false
7152 Acked Flush 0 Done Sent Done false
7153 Acked Read 4 Done Sent Done false
7154 Acked Read 3 Done Sent Done false
7155 Acked Write 1 Done Sent Done false
7156 Acked Write 10 Done Sent Done false
7157 Acked Write 6 Done Sent Done false
7158 Acked Write 6 Done Sent Done false
7159 Acked Read 4 Done Sent Done false
7160 Acked Read 9 Done Sent Done false
7161 Acked Write 6 Done Sent Done false
7162 Acked Write 1 Done Sent Done false
7163 Acked Write 2 Done Sent Done false
7164 Acked Read 3 Done Sent Done false
7165 Acked Write 7 Done Sent Done false
7166 Acked Flush 0 Done Sent Done false
7167 Acked Write 6 Done Sent Done false
7168 Acked Write 10 Done Sent Done false
7169 Acked Write 5 Done Sent Done false
7170 Acked Read 7 Done Sent Done false
7171 Acked Read 4 Done Sent Done false
7172 Acked Read 2 Done Sent Done false
7173 Acked Flush 0 Done Sent Done false
7174 Acked Read 1 Done Sent Done false
7175 Acked Read 7 Done Sent Done false
7176 Acked Write 9 Done Sent Done false
7177 Acked Write 5 Done Sent Done false
7178 Acked Read 5 Done Sent Done false
7179 Acked Write 1 Done Sent Done false
7180 Acked Read 1 Done Sent Done false
7181 Acked Write 6 Done Sent Done false
7182 Acked Write 5 Done Sent Done false
7183 Acked Write 8 Done Sent Done false
7184 Acked Read 4 Done Sent Done false
7185 Acked Write 5 Done Sent Done false
7186 Acked Write 5 Done Sent Done false
7187 Acked Read 8 Done Sent Done false
7188 Acked Read 9 Done Sent Done false
7189 Acked Read 8 Done Sent Done false
7190 Acked Write 8 Done Sent Done false
7191 Acked Read 4 Done Sent Done false
7192 Acked Flush 0 Done Sent Done false
7193 Acked Write 7 Done Sent Done false
7194 Acked Read 3 Done Sent Done false
7195 Acked Write 6 Done Sent Done false
7196 Acked Write 5 Done Sent Done false
7197 Acked Read 3 Done Sent Done false
7198 Acked Read 10 Done Sent Done false
7199 Acked Write 2 Done Sent Done false
7200 Acked Read 6 Done Sent Done false
7201 Acked Read 3 Done Sent Done false
7202 Acked Write 4 Done Sent Done false
7203 Acked Read 4 Done Sent Done false
7204 Acked Write 8 Done Sent Done false
7205 Acked Read 5 Done Sent Done false
7206 Acked Read 10 Done Sent Done false
7207 Acked Read 2 Done Sent Done false
7208 Acked Flush 0 Done Sent Done false
7209 Acked Read 4 Done Sent Done false
7210 Acked Read 2 Done Sent Done false
7211 Acked Read 5 Done Sent Done false
7212 Acked Write 5 Done Sent Done false
7213 Acked Read 4 Done Sent Done false
7214 Acked Flush 0 Done Sent Done false
7215 Acked Read 3 Done Sent Done false
7216 Acked Write 2 Done Sent Done false
7217 Acked Write 7 Done Sent Done false
7218 Acked Write 4 Done Sent Done false
7219 Acked Write 6 Done Sent Done false
7220 Acked Flush 0 Done Sent Done false
7221 Acked Read 8 Done Sent Done false
7222 Acked Read 5 Done Sent Done false
7223 Acked Read 2 Done Sent Done false
7224 Acked Read 6 Done Sent Done false
7225 Acked Write 8 Done Sent Done false
7226 Acked Read 6 Done Sent Done false
7227 Acked Flush 0 Done Sent Done false
7228 Acked Read 2 Done Sent Done false
7229 Acked Read 10 Done Sent Done false
7230 Acked Read 5 Done Sent Done false
7231 Acked Write 2 Done Sent Done false
7232 Acked Write 1 Done Sent Done false
7233 Acked Read 3 Done Sent Done false
7234 Acked Flush 0 Done Sent Done false
7235 Acked Flush 0 Done Sent Done false
7236 Acked Read 1 Done Sent Done false
7237 Acked Write 3 Done Sent Done false
7238 Acked Read 8 Done Sent Done false
7239 Acked Read 8 Done Sent Done false
7240 Acked Read 3 Done Sent Done false
7241 Acked Read 8 Done Sent Done false
7242 Acked Read 10 Done Sent Done false
7243 Acked Write 2 Done Sent Done false
7244 Acked Write 9 Done Sent Done false
7245 Acked Write 4 Done Sent Done false
7246 Acked Read 5 Done Sent Done false
7247 Acked Read 5 Done Sent Done false
7248 Acked Write 6 Done Sent Done false
7249 Acked Write 10 Done Sent Done false
7250 Acked Flush 0 Done Sent Done false
7251 Acked Read 2 Done Sent Done false
7252 Acked Write 10 Done Sent Done false
7253 Acked Read 5 Done Sent Done false
7254 Acked Write 9 Done Sent Done false
7255 Acked Write 5 Done Sent Done false
7256 Acked Write 4 Done Sent Done false
7257 Acked Read 9 Done Sent Done false
7258 Acked Read 7 Done Sent Done false
7259 Acked Read 2 Done Sent Done false
7260 Acked Read 7 Done Sent Done false
7261 Acked Write 10 Done Sent Done false
7262 Acked Flush 0 Done Sent Done false
7263 Acked Write 6 Done Sent Done false
7264 Acked Write 7 Done Sent Done false
7265 Acked Read 6 Done Sent Done false
7266 Acked Read 10 Done Sent Done false
7267 Acked Write 7 Done Sent Done false
7268 Acked Write 5 Done Sent Done false
7269 Acked Write 4 Done Sent Done false
7270 Acked Write 8 Done Sent Done false
7271 Acked Read 4 Done Sent Done false
7272 Acked Read 8 Done Sent Done false
7273 Acked Read 6 Done Sent Done false
7274 Acked Write 9 Done Sent Done false
7275 Acked Read 2 Done Sent Done false
7276 Acked Flush 0 Done Sent Done false
7277 Acked Read 10 Done Sent Done false
7278 Acked Write 8 Done Sent Done false
7279 Acked Write 8 Done Sent Done false
7280 Acked Flush 0 Done Sent Done false
7281 Acked Write 10 Done Sent Done false
7282 Acked Write 8 Done Sent Done false
7283 Acked Read 2 Done Sent Done false
7284 Acked Write 10 Done Sent Done false
7285 Acked Read 4 Done Sent Done false
7286 Acked Read 2 Done Sent Done false
7287 Acked Flush 0 Done Sent Done false
7288 Acked Write 4 Done Sent Done false
7289 Acked Read 9 Done Sent Done false
7290 Acked Read 8 Done Sent Done false
7291 Acked Read 4 Done Sent Done false
7292 Acked Read 5 Done Sent Done false
7293 Acked Read 5 Done Sent Done false
7294 Acked Read 10 Done Sent Done false
7295 Acked Write 8 Done Sent Done false
7296 Acked Read 1 Done Sent Done false
7297 Acked Read 4 Done Sent Done false
7298 Acked Read 4 Done Sent Done false
7299 Acked Write 2 Done Sent Done false
7300 Acked Write 2 Done Sent Done false
7301 Acked Read 1 Done Sent Done false
7302 Acked Read 3 Done Sent Done false
7303 Acked Read 9 Done Sent Done false
7304 Acked Read 2 Done Sent Done false
7305 Acked Write 1 Done Sent Done false
7306 Acked Write 7 Done Sent Done false
7307 Acked Read 6 Done Sent Done false
7308 Acked Write 9 Done Sent Done false
7309 Acked Read 9 Done Sent Done false
7310 Acked Write 5 Done Sent Done false
7311 Acked Write 2 Done Sent Done false
7312 Acked Write 10 Done Sent Done false
7313 Acked Write 3 Done Sent Done false
7314 Acked Read 5 Done Sent Done false
7315 Acked Read 6 Done Sent Done false
7316 Acked Flush 0 Done Sent Done false
7317 Acked Read 4 Done Sent Done false
7318 Acked Write 4 Done Sent Done false
7319 Acked Write 3 Done Sent Done false
7320 Acked Write 3 Done Sent Done false
7321 Acked Read 9 Done Sent Done false
7322 Acked Read 4 Done Sent Done false
7323 Acked Write 6 Done Sent Done false
7324 Acked Read 4 Done Sent Done false
7325 Acked Read 4 Done Sent Done false
7326 Acked Read 2 Done Sent Done false
7327 Acked Flush 0 Done Sent Done false
7328 Acked Read 6 Done Sent Done false
7329 Acked Flush 0 Done Sent Done false
7330 Acked Read 1 Done Sent Done false
7331 Acked Flush 0 Done Sent Done false
7332 Acked Write 8 Done Sent Done false
7333 Acked Write 8 Done Sent Done false
7334 Acked Write 4 Done Sent Done false
7335 Acked Flush 0 Done Sent Done false
7336 Acked Read 8 Done Sent Done false
7337 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:29.888579808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:30.889770876Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:31.89110668Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 10 Done Sent Done false
7038 Acked Flush 0 Done Sent Done false
7039 Acked Read 2 Done Sent Done false
7040 Acked Write 6 Done Sent Done false
7041 Acked Read 7 Done Sent Done false
7042 Acked Write 6 Done Sent Done false
7043 Acked Read 4 Done Sent Done false
7044 Acked Write 8 Done Sent Done false
7045 Acked Read 4 Done Sent Done false
7046 Acked Read 9 Done Sent Done false
7047 Acked Read 2 Done Sent Done false
7048 Acked Write 7 Done Sent Done false
7049 Acked Read 8 Done Sent Done false
7050 Acked Read 4 Done Sent Done false
7051 Acked Read 5 Done Sent Done false
7052 Acked Flush 0 Done Sent Done false
7053 Acked Write 6 Done Sent Done false
7054 Acked Flush 0 Done Sent Done false
7055 Acked Read 7 Done Sent Done false
7056 Acked Write 10 Done Sent Done false
7057 Acked Read 7 Done Sent Done false
7058 Acked Write 8 Done Sent Done false
7059 Acked Write 9 Done Sent Done false
7060 Acked Write 7 Done Sent Done false
7061 Acked Read 10 Done Sent Done false
7062 Acked Read 5 Done Sent Done false
7063 Acked Read 6 Done Sent Done false
7064 Acked Read 3 Done Sent Done false
7065 Acked Flush 0 Done Sent Done false
7066 Acked Read 2 Done Sent Done false
7067 Acked Read 3 Done Sent Done false
7068 Acked Read 2 Done Sent Done false
7069 Acked Write 1 Done Sent Done false
7070 Acked Read 2 Done Sent Done false
7071 Acked Read 9 Done Sent Done false
7072 Acked Write 8 Done Sent Done false
7073 Acked Read 7 Done Sent Done false
7074 Acked Read 10 Done Sent Done false
7075 Acked Write 10 Done Sent Done false
7076 Acked Read 5 Done Sent Done false
7077 Acked Write 1 Done Sent Done false
7078 Acked Write 6 Done Sent Done false
7079 Acked Flush 0 Done Sent Done false
7080 Acked Read 10 Done Sent Done false
7081 Acked Read 6 Done Sent Done false
7082 Acked Read 6 Done Sent Done false
7083 Acked Read 9 Done Sent Done false
7084 Acked Write 3 Done Sent Done false
7085 Acked Read 7 Done Sent Done false
7086 Acked Read 10 Done Sent Done false
7087 Acked Read 10 Done Sent Done false
7088 Acked Write 1 Done Sent Done false
7089 Acked Read 5 Done Sent Done false
7090 Acked Read 9 Done Sent Done false
7091 Acked Read 10 Done Sent Done false
7092 Acked Write 6 Done Sent Done false
7093 Acked Write 8 Done Sent Done false
7094 Acked Write 5 Done Sent Done false
7095 Acked Write 10 Done Sent Done false
7096 Acked Write 4 Done Sent Done false
7097 Acked Read 4 Done Sent Done false
7098 Acked Write 3 Done Sent Done false
7099 Acked Read 3 Done Sent Done false
7100 Acked Flush 0 Done Sent Done false
7101 Acked Write 1 Done Sent Done false
7102 Acked Read 6 Done Sent Done false
7103 Acked Read 9 Done Sent Done false
7104 Acked Read 1 Done Sent Done false
7105 Acked Write 7 Done Sent Done false
7106 Acked Read 8 Done Sent Done false
7107 Acked Write 5 Done Sent Done false
7108 Acked Write 5 Done Sent Done false
7109 Acked Read 2 Done Sent Done false
7110 Acked Write 7 Done Sent Done false
7111 Acked Write 2 Done Sent Done false
7112 Acked Read 10 Done Sent Done false
7113 Acked Read 9 Done Sent Done false
7114 Acked Read 7 Done Sent Done false
7115 Acked Write 8 Done Sent Done false
7116 Acked Read 8 Done Sent Done false
7117 Acked Write 1 Done Sent Done false
7118 Acked Write 7 Done Sent Done false
7119 Acked Read 1 Done Sent Done false
7120 Acked Write 5 Done Sent Done false
7121 Acked Write 10 Done Sent Done false
7122 Acked Write 2 Done Sent Done false
7123 Acked Read 4 Done Sent Done false
7124 Acked Write 7 Done Sent Done false
7125 Acked Write 10 Done Sent Done false
7126 Acked Write 9 Done Sent Done false
7127 Acked Write 6 Done Sent Done false
7128 Acked Read 6 Done Sent Done false
7129 Acked Read 7 Done Sent Done false
7130 Acked Write 8 Done Sent Done false
7131 Acked Read 8 Done Sent Done false
7132 Acked Read 4 Done Sent Done false
7133 Acked Write 10 Done Sent Done false
7134 Acked Read 3 Done Sent Done false
7135 Acked Read 3 Done Sent Done false
7136 Acked Read 3 Done Sent Done false
7137 Acked Write 6 Done Sent Done false
7138 Acked Flush 0 Done Sent Done false
7139 Acked Read 2 Done Sent Done false
7140 Acked Write 1 Done Sent Done false
7141 Acked Write 6 Done Sent Done false
7142 Acked Read 8 Done Sent Done false
7143 Acked Flush 0 Done Sent Done false
7144 Acked Flush 0 Done Sent Done false
7145 Acked Write 4 Done Sent Done false
7146 Acked Read 7 Done Sent Done false
7147 Acked Write 1 Done Sent Done false
7148 Acked Read 2 Done Sent Done false
7149 Acked Read 10 Done Sent Done false
7150 Acked Read 2 Done Sent Done false
7151 Acked Write 6 Done Sent Done false
7152 Acked Flush 0 Done Sent Done false
7153 Acked Read 4 Done Sent Done false
7154 Acked Read 3 Done Sent Done false
7155 Acked Write 1 Done Sent Done false
7156 Acked Write 10 Done Sent Done false
7157 Acked Write 6 Done Sent Done false
7158 Acked Write 6 Done Sent Done false
7159 Acked Read 4 Done Sent Done false
7160 Acked Read 9 Done Sent Done false
7161 Acked Write 6 Done Sent Done false
7162 Acked Write 1 Done Sent Done false
7163 Acked Write 2 Done Sent Done false
7164 Acked Read 3 Done Sent Done false
7165 Acked Write 7 Done Sent Done false
7166 Acked Flush 0 Done Sent Done false
7167 Acked Write 6 Done Sent Done false
7168 Acked Write 10 Done Sent Done false
7169 Acked Write 5 Done Sent Done false
7170 Acked Read 7 Done Sent Done false
7171 Acked Read 4 Done Sent Done false
7172 Acked Read 2 Done Sent Done false
7173 Acked Flush 0 Done Sent Done false
7174 Acked Read 1 Done Sent Done false
7175 Acked Read 7 Done Sent Done false
7176 Acked Write 9 Done Sent Done false
7177 Acked Write 5 Done Sent Done false
7178 Acked Read 5 Done Sent Done false
7179 Acked Write 1 Done Sent Done false
7180 Acked Read 1 Done Sent Done false
7181 Acked Write 6 Done Sent Done false
7182 Acked Write 5 Done Sent Done false
7183 Acked Write 8 Done Sent Done false
7184 Acked Read 4 Done Sent Done false
7185 Acked Write 5 Done Sent Done false
7186 Acked Write 5 Done Sent Done false
7187 Acked Read 8 Done Sent Done false
7188 Acked Read 9 Done Sent Done false
7189 Acked Read 8 Done Sent Done false
7190 Acked Write 8 Done Sent Done false
7191 Acked Read 4 Done Sent Done false
7192 Acked Flush 0 Done Sent Done false
7193 Acked Write 7 Done Sent Done false
7194 Acked Read 3 Done Sent Done false
7195 Acked Write 6 Done Sent Done false
7196 Acked Write 5 Done Sent Done false
7197 Acked Read 3 Done Sent Done false
7198 Acked Read 10 Done Sent Done false
7199 Acked Write 2 Done Sent Done false
7200 Acked Read 6 Done Sent Done false
7201 Acked Read 3 Done Sent Done false
7202 Acked Write 4 Done Sent Done false
7203 Acked Read 4 Done Sent Done false
7204 Acked Write 8 Done Sent Done false
7205 Acked Read 5 Done Sent Done false
7206 Acked Read 10 Done Sent Done false
7207 Acked Read 2 Done Sent Done false
7208 Acked Flush 0 Done Sent Done false
7209 Acked Read 4 Done Sent Done false
7210 Acked Read 2 Done Sent Done false
7211 Acked Read 5 Done Sent Done false
7212 Acked Write 5 Done Sent Done false
7213 Acked Read 4 Done Sent Done false
7214 Acked Flush 0 Done Sent Done false
7215 Acked Read 3 Done Sent Done false
7216 Acked Write 2 Done Sent Done false
7217 Acked Write 7 Done Sent Done false
7218 Acked Write 4 Done Sent Done false
7219 Acked Write 6 Done Sent Done false
7220 Acked Flush 0 Done Sent Done false
7221 Acked Read 8 Done Sent Done false
7222 Acked Read 5 Done Sent Done false
7223 Acked Read 2 Done Sent Done false
7224 Acked Read 6 Done Sent Done false
7225 Acked Write 8 Done Sent Done false
7226 Acked Read 6 Done Sent Done false
7227 Acked Flush 0 Done Sent Done false
7228 Acked Read 2 Done Sent Done false
7229 Acked Read 10 Done Sent Done false
7230 Acked Read 5 Done Sent Done false
7231 Acked Write 2 Done Sent Done false
7232 Acked Write 1 Done Sent Done false
7233 Acked Read 3 Done Sent Done false
7234 Acked Flush 0 Done Sent Done false
7235 Acked Flush 0 Done Sent Done false
7236 Acked Read 1 Done Sent Done false
7237 Acked Write 3 Done Sent Done false
7238 Acked Read 8 Done Sent Done false
7239 Acked Read 8 Done Sent Done false
7240 Acked Read 3 Done Sent Done false
7241 Acked Read 8 Done Sent Done false
7242 Acked Read 10 Done Sent Done false
7243 Acked Write 2 Done Sent Done false
7244 Acked Write 9 Done Sent Done false
7245 Acked Write 4 Done Sent Done false
7246 Acked Read 5 Done Sent Done false
7247 Acked Read 5 Done Sent Done false
7248 Acked Write 6 Done Sent Done false
7249 Acked Write 10 Done Sent Done false
7250 Acked Flush 0 Done Sent Done false
7251 Acked Read 2 Done Sent Done false
7252 Acked Write 10 Done Sent Done false
7253 Acked Read 5 Done Sent Done false
7254 Acked Write 9 Done Sent Done false
7255 Acked Write 5 Done Sent Done false
7256 Acked Write 4 Done Sent Done false
7257 Acked Read 9 Done Sent Done false
7258 Acked Read 7 Done Sent Done false
7259 Acked Read 2 Done Sent Done false
7260 Acked Read 7 Done Sent Done false
7261 Acked Write 10 Done Sent Done false
7262 Acked Flush 0 Done Sent Done false
7263 Acked Write 6 Done Sent Done false
7264 Acked Write 7 Done Sent Done false
7265 Acked Read 6 Done Sent Done false
7266 Acked Read 10 Done Sent Done false
7267 Acked Write 7 Done Sent Done false
7268 Acked Write 5 Done Sent Done false
7269 Acked Write 4 Done Sent Done false
7270 Acked Write 8 Done Sent Done false
7271 Acked Read 4 Done Sent Done false
7272 Acked Read 8 Done Sent Done false
7273 Acked Read 6 Done Sent Done false
7274 Acked Write 9 Done Sent Done false
7275 Acked Read 2 Done Sent Done false
7276 Acked Flush 0 Done Sent Done false
7277 Acked Read 10 Done Sent Done false
7278 Acked Write 8 Done Sent Done false
7279 Acked Write 8 Done Sent Done false
7280 Acked Flush 0 Done Sent Done false
7281 Acked Write 10 Done Sent Done false
7282 Acked Write 8 Done Sent Done false
7283 Acked Read 2 Done Sent Done false
7284 Acked Write 10 Done Sent Done false
7285 Acked Read 4 Done Sent Done false
7286 Acked Read 2 Done Sent Done false
7287 Acked Flush 0 Done Sent Done false
7288 Acked Write 4 Done Sent Done false
7289 Acked Read 9 Done Sent Done false
7290 Acked Read 8 Done Sent Done false
7291 Acked Read 4 Done Sent Done false
7292 Acked Read 5 Done Sent Done false
7293 Acked Read 5 Done Sent Done false
7294 Acked Read 10 Done Sent Done false
7295 Acked Write 8 Done Sent Done false
7296 Acked Read 1 Done Sent Done false
7297 Acked Read 4 Done Sent Done false
7298 Acked Read 4 Done Sent Done false
7299 Acked Write 2 Done Sent Done false
7300 Acked Write 2 Done Sent Done false
7301 Acked Read 1 Done Sent Done false
7302 Acked Read 3 Done Sent Done false
7303 Acked Read 9 Done Sent Done false
7304 Acked Read 2 Done Sent Done false
7305 Acked Write 1 Done Sent Done false
7306 Acked Write 7 Done Sent Done false
7307 Acked Read 6 Done Sent Done false
7308 Acked Write 9 Done Sent Done false
7309 Acked Read 9 Done Sent Done false
7310 Acked Write 5 Done Sent Done false
7311 Acked Write 2 Done Sent Done false
7312 Acked Write 10 Done Sent Done false
7313 Acked Write 3 Done Sent Done false
7314 Acked Read 5 Done Sent Done false
7315 Acked Read 6 Done Sent Done false
7316 Acked Flush 0 Done Sent Done false
7317 Acked Read 4 Done Sent Done false
7318 Acked Write 4 Done Sent Done false
7319 Acked Write 3 Done Sent Done false
7320 Acked Write 3 Done Sent Done false
7321 Acked Read 9 Done Sent Done false
7322 Acked Read 4 Done Sent Done false
7323 Acked Write 6 Done Sent Done false
7324 Acked Read 4 Done Sent Done false
7325 Acked Read 4 Done Sent Done false
7326 Acked Read 2 Done Sent Done false
7327 Acked Flush 0 Done Sent Done false
7328 Acked Read 6 Done Sent Done false
7329 Acked Flush 0 Done Sent Done false
7330 Acked Read 1 Done Sent Done false
7331 Acked Flush 0 Done Sent Done false
7332 Acked Write 8 Done Sent Done false
7333 Acked Write 8 Done Sent Done false
7334 Acked Write 4 Done Sent Done false
7335 Acked Flush 0 Done Sent Done false
7336 Acked Read 8 Done Sent Done false
7337 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:33.894848769Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:34.879957751Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:34.880061426Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:34.880125101Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:34.880166571Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:34.880208661Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:35.381492705Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:35.897540181Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:36.397814407Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:36.898848371Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 10 Done Sent Done false
7038 Acked Flush 0 Done Sent Done false
7039 Acked Read 2 Done Sent Done false
7040 Acked Write 6 Done Sent Done false
7041 Acked Read 7 Done Sent Done false
7042 Acked Write 6 Done Sent Done false
7043 Acked Read 4 Done Sent Done false
7044 Acked Write 8 Done Sent Done false
7045 Acked Read 4 Done Sent Done false
7046 Acked Read 9 Done Sent Done false
7047 Acked Read 2 Done Sent Done false
7048 Acked Write 7 Done Sent Done false
7049 Acked Read 8 Done Sent Done false
7050 Acked Read 4 Done Sent Done false
7051 Acked Read 5 Done Sent Done false
7052 Acked Flush 0 Done Sent Done false
7053 Acked Write 6 Done Sent Done false
7054 Acked Flush 0 Done Sent Done false
7055 Acked Read 7 Done Sent Done false
7056 Acked Write 10 Done Sent Done false
7057 Acked Read 7 Done Sent Done false
7058 Acked Write 8 Done Sent Done false
7059 Acked Write 9 Done Sent Done false
7060 Acked Write 7 Done Sent Done false
7061 Acked Read 10 Done Sent Done false
7062 Acked Read 5 Done Sent Done false
7063 Acked Read 6 Done Sent Done false
7064 Acked Read 3 Done Sent Done false
7065 Acked Flush 0 Done Sent Done false
7066 Acked Read 2 Done Sent Done false
7067 Acked Read 3 Done Sent Done false
7068 Acked Read 2 Done Sent Done false
7069 Acked Write 1 Done Sent Done false
7070 Acked Read 2 Done Sent Done false
7071 Acked Read 9 Done Sent Done false
7072 Acked Write 8 Done Sent Done false
7073 Acked Read 7 Done Sent Done false
7074 Acked Read 10 Done Sent Done false
7075 Acked Write 10 Done Sent Done false
7076 Acked Read 5 Done Sent Done false
7077 Acked Write 1 Done Sent Done false
7078 Acked Write 6 Done Sent Done false
7079 Acked Flush 0 Done Sent Done false
7080 Acked Read 10 Done Sent Done false
7081 Acked Read 6 Done Sent Done false
7082 Acked Read 6 Done Sent Done false
7083 Acked Read 9 Done Sent Done false
7084 Acked Write 3 Done Sent Done false
7085 Acked Read 7 Done Sent Done false
7086 Acked Read 10 Done Sent Done false
7087 Acked Read 10 Done Sent Done false
7088 Acked Write 1 Done Sent Done false
7089 Acked Read 5 Done Sent Done false
7090 Acked Read 9 Done Sent Done false
7091 Acked Read 10 Done Sent Done false
7092 Acked Write 6 Done Sent Done false
7093 Acked Write 8 Done Sent Done false
7094 Acked Write 5 Done Sent Done false
7095 Acked Write 10 Done Sent Done false
7096 Acked Write 4 Done Sent Done false
7097 Acked Read 4 Done Sent Done false
7098 Acked Write 3 Done Sent Done false
7099 Acked Read 3 Done Sent Done false
7100 Acked Flush 0 Done Sent Done false
7101 Acked Write 1 Done Sent Done false
7102 Acked Read 6 Done Sent Done false
7103 Acked Read 9 Done Sent Done false
7104 Acked Read 1 Done Sent Done false
7105 Acked Write 7 Done Sent Done false
7106 Acked Read 8 Done Sent Done false
7107 Acked Write 5 Done Sent Done false
7108 Acked Write 5 Done Sent Done false
7109 Acked Read 2 Done Sent Done false
7110 Acked Write 7 Done Sent Done false
7111 Acked Write 2 Done Sent Done false
7112 Acked Read 10 Done Sent Done false
7113 Acked Read 9 Done Sent Done false
7114 Acked Read 7 Done Sent Done false
7115 Acked Write 8 Done Sent Done false
7116 Acked Read 8 Done Sent Done false
7117 Acked Write 1 Done Sent Done false
7118 Acked Write 7 Done Sent Done false
7119 Acked Read 1 Done Sent Done false
7120 Acked Write 5 Done Sent Done false
7121 Acked Write 10 Done Sent Done false
7122 Acked Write 2 Done Sent Done false
7123 Acked Read 4 Done Sent Done false
7124 Acked Write 7 Done Sent Done false
7125 Acked Write 10 Done Sent Done false
7126 Acked Write 9 Done Sent Done false
7127 Acked Write 6 Done Sent Done false
7128 Acked Read 6 Done Sent Done false
7129 Acked Read 7 Done Sent Done false
7130 Acked Write 8 Done Sent Done false
7131 Acked Read 8 Done Sent Done false
7132 Acked Read 4 Done Sent Done false
7133 Acked Write 10 Done Sent Done false
7134 Acked Read 3 Done Sent Done false
7135 Acked Read 3 Done Sent Done false
7136 Acked Read 3 Done Sent Done false
7137 Acked Write 6 Done Sent Done false
7138 Acked Flush 0 Done Sent Done false
7139 Acked Read 2 Done Sent Done false
7140 Acked Write 1 Done Sent Done false
7141 Acked Write 6 Done Sent Done false
7142 Acked Read 8 Done Sent Done false
7143 Acked Flush 0 Done Sent Done false
7144 Acked Flush 0 Done Sent Done false
7145 Acked Write 4 Done Sent Done false
7146 Acked Read 7 Done Sent Done false
7147 Acked Write 1 Done Sent Done false
7148 Acked Read 2 Done Sent Done false
7149 Acked Read 10 Done Sent Done false
7150 Acked Read 2 Done Sent Done false
7151 Acked Write 6 Done Sent Done false
7152 Acked Flush 0 Done Sent Done false
7153 Acked Read 4 Done Sent Done false
7154 Acked Read 3 Done Sent Done false
7155 Acked Write 1 Done Sent Done false
7156 Acked Write 10 Done Sent Done false
7157 Acked Write 6 Done Sent Done false
7158 Acked Write 6 Done Sent Done false
7159 Acked Read 4 Done Sent Done false
7160 Acked Read 9 Done Sent Done false
7161 Acked Write 6 Done Sent Done false
7162 Acked Write 1 Done Sent Done false
7163 Acked Write 2 Done Sent Done false
7164 Acked Read 3 Done Sent Done false
7165 Acked Write 7 Done Sent Done false
7166 Acked Flush 0 Done Sent Done false
7167 Acked Write 6 Done Sent Done false
7168 Acked Write 10 Done Sent Done false
7169 Acked Write 5 Done Sent Done false
7170 Acked Read 7 Done Sent Done false
7171 Acked Read 4 Done Sent Done false
7172 Acked Read 2 Done Sent Done false
7173 Acked Flush 0 Done Sent Done false
7174 Acked Read 1 Done Sent Done false
7175 Acked Read 7 Done Sent Done false
7176 Acked Write 9 Done Sent Done false
7177 Acked Write 5 Done Sent Done false
7178 Acked Read 5 Done Sent Done false
7179 Acked Write 1 Done Sent Done false
7180 Acked Read 1 Done Sent Done false
7181 Acked Write 6 Done Sent Done false
7182 Acked Write 5 Done Sent Done false
7183 Acked Write 8 Done Sent Done false
7184 Acked Read 4 Done Sent Done false
7185 Acked Write 5 Done Sent Done false
7186 Acked Write 5 Done Sent Done false
7187 Acked Read 8 Done Sent Done false
7188 Acked Read 9 Done Sent Done false
7189 Acked Read 8 Done Sent Done false
7190 Acked Write 8 Done Sent Done false
7191 Acked Read 4 Done Sent Done false
7192 Acked Flush 0 Done Sent Done false
7193 Acked Write 7 Done Sent Done false
7194 Acked Read 3 Done Sent Done false
7195 Acked Write 6 Done Sent Done false
7196 Acked Write 5 Done Sent Done false
7197 Acked Read 3 Done Sent Done false
7198 Acked Read 10 Done Sent Done false
7199 Acked Write 2 Done Sent Done false
7200 Acked Read 6 Done Sent Done false
7201 Acked Read 3 Done Sent Done false
7202 Acked Write 4 Done Sent Done false
7203 Acked Read 4 Done Sent Done false
7204 Acked Write 8 Done Sent Done false
7205 Acked Read 5 Done Sent Done false
7206 Acked Read 10 Done Sent Done false
7207 Acked Read 2 Done Sent Done false
7208 Acked Flush 0 Done Sent Done false
7209 Acked Read 4 Done Sent Done false
7210 Acked Read 2 Done Sent Done false
7211 Acked Read 5 Done Sent Done false
7212 Acked Write 5 Done Sent Done false
7213 Acked Read 4 Done Sent Done false
7214 Acked Flush 0 Done Sent Done false
7215 Acked Read 3 Done Sent Done false
7216 Acked Write 2 Done Sent Done false
7217 Acked Write 7 Done Sent Done false
7218 Acked Write 4 Done Sent Done false
7219 Acked Write 6 Done Sent Done false
7220 Acked Flush 0 Done Sent Done false
7221 Acked Read 8 Done Sent Done false
7222 Acked Read 5 Done Sent Done false
7223 Acked Read 2 Done Sent Done false
7224 Acked Read 6 Done Sent Done false
7225 Acked Write 8 Done Sent Done false
7226 Acked Read 6 Done Sent Done false
7227 Acked Flush 0 Done Sent Done false
7228 Acked Read 2 Done Sent Done false
7229 Acked Read 10 Done Sent Done false
7230 Acked Read 5 Done Sent Done false
7231 Acked Write 2 Done Sent Done false
7232 Acked Write 1 Done Sent Done false
7233 Acked Read 3 Done Sent Done false
7234 Acked Flush 0 Done Sent Done false
7235 Acked Flush 0 Done Sent Done false
7236 Acked Read 1 Done Sent Done false
7237 Acked Write 3 Done Sent Done false
7238 Acked Read 8 Done Sent Done false
7239 Acked Read 8 Done Sent Done false
7240 Acked Read 3 Done Sent Done false
7241 Acked Read 8 Done Sent Done false
7242 Acked Read 10 Done Sent Done false
7243 Acked Write 2 Done Sent Done false
7244 Acked Write 9 Done Sent Done false
7245 Acked Write 4 Done Sent Done false
7246 Acked Read 5 Done Sent Done false
7247 Acked Read 5 Done Sent Done false
7248 Acked Write 6 Done Sent Done false
7249 Acked Write 10 Done Sent Done false
7250 Acked Flush 0 Done Sent Done false
7251 Acked Read 2 Done Sent Done false
7252 Acked Write 10 Done Sent Done false
7253 Acked Read 5 Done Sent Done false
7254 Acked Write 9 Done Sent Done false
7255 Acked Write 5 Done Sent Done false
7256 Acked Write 4 Done Sent Done false
7257 Acked Read 9 Done Sent Done false
7258 Acked Read 7 Done Sent Done false
7259 Acked Read 2 Done Sent Done false
7260 Acked Read 7 Done Sent Done false
7261 Acked Write 10 Done Sent Done false
7262 Acked Flush 0 Done Sent Done false
7263 Acked Write 6 Done Sent Done false
7264 Acked Write 7 Done Sent Done false
7265 Acked Read 6 Done Sent Done false
7266 Acked Read 10 Done Sent Done false
7267 Acked Write 7 Done Sent Done false
7268 Acked Write 5 Done Sent Done false
7269 Acked Write 4 Done Sent Done false
7270 Acked Write 8 Done Sent Done false
7271 Acked Read 4 Done Sent Done false
7272 Acked Read 8 Done Sent Done false
7273 Acked Read 6 Done Sent Done false
7274 Acked Write 9 Done Sent Done false
7275 Acked Read 2 Done Sent Done false
7276 Acked Flush 0 Done Sent Done false
7277 Acked Read 10 Done Sent Done false
7278 Acked Write 8 Done Sent Done false
7279 Acked Write 8 Done Sent Done false
7280 Acked Flush 0 Done Sent Done false
7281 Acked Write 10 Done Sent Done false
7282 Acked Write 8 Done Sent Done false
7283 Acked Read 2 Done Sent Done false
7284 Acked Write 10 Done Sent Done false
7285 Acked Read 4 Done Sent Done false
7286 Acked Read 2 Done Sent Done false
7287 Acked Flush 0 Done Sent Done false
7288 Acked Write 4 Done Sent Done false
7289 Acked Read 9 Done Sent Done false
7290 Acked Read 8 Done Sent Done false
7291 Acked Read 4 Done Sent Done false
7292 Acked Read 5 Done Sent Done false
7293 Acked Read 5 Done Sent Done false
7294 Acked Read 10 Done Sent Done false
7295 Acked Write 8 Done Sent Done false
7296 Acked Read 1 Done Sent Done false
7297 Acked Read 4 Done Sent Done false
7298 Acked Read 4 Done Sent Done false
7299 Acked Write 2 Done Sent Done false
7300 Acked Write 2 Done Sent Done false
7301 Acked Read 1 Done Sent Done false
7302 Acked Read 3 Done Sent Done false
7303 Acked Read 9 Done Sent Done false
7304 Acked Read 2 Done Sent Done false
7305 Acked Write 1 Done Sent Done false
7306 Acked Write 7 Done Sent Done false
7307 Acked Read 6 Done Sent Done false
7308 Acked Write 9 Done Sent Done false
7309 Acked Read 9 Done Sent Done false
7310 Acked Write 5 Done Sent Done false
7311 Acked Write 2 Done Sent Done false
7312 Acked Write 10 Done Sent Done false
7313 Acked Write 3 Done Sent Done false
7314 Acked Read 5 Done Sent Done false
7315 Acked Read 6 Done Sent Done false
7316 Acked Flush 0 Done Sent Done false
7317 Acked Read 4 Done Sent Done false
7318 Acked Write 4 Done Sent Done false
7319 Acked Write 3 Done Sent Done false
7320 Acked Write 3 Done Sent Done false
7321 Acked Read 9 Done Sent Done false
7322 Acked Read 4 Done Sent Done false
7323 Acked Write 6 Done Sent Done false
7324 Acked Read 4 Done Sent Done false
7325 Acked Read 4 Done Sent Done false
7326 Acked Read 2 Done Sent Done false
7327 Acked Flush 0 Done Sent Done false
7328 Acked Read 6 Done Sent Done false
7329 Acked Flush 0 Done Sent Done false
7330 Acked Read 1 Done Sent Done false
7331 Acked Flush 0 Done Sent Done false
7332 Acked Write 8 Done Sent Done false
7333 Acked Write 8 Done Sent Done false
7334 Acked Write 4 Done Sent Done false
7335 Acked Flush 0 Done Sent Done false
7336 Acked Read 8 Done Sent Done false
7337 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:38.137476026Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:38.902046451Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:39.90423407Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:40.906241012Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 10 Done Sent Done false
7038 Acked Flush 0 Done Sent Done false
7039 Acked Read 2 Done Sent Done false
7040 Acked Write 6 Done Sent Done false
7041 Acked Read 7 Done Sent Done false
7042 Acked Write 6 Done Sent Done false
7043 Acked Read 4 Done Sent Done false
7044 Acked Write 8 Done Sent Done false
7045 Acked Read 4 Done Sent Done false
7046 Acked Read 9 Done Sent Done false
7047 Acked Read 2 Done Sent Done false
7048 Acked Write 7 Done Sent Done false
7049 Acked Read 8 Done Sent Done false
7050 Acked Read 4 Done Sent Done false
7051 Acked Read 5 Done Sent Done false
7052 Acked Flush 0 Done Sent Done false
7053 Acked Write 6 Done Sent Done false
7054 Acked Flush 0 Done Sent Done false
7055 Acked Read 7 Done Sent Done false
7056 Acked Write 10 Done Sent Done false
7057 Acked Read 7 Done Sent Done false
7058 Acked Write 8 Done Sent Done false
7059 Acked Write 9 Done Sent Done false
7060 Acked Write 7 Done Sent Done false
7061 Acked Read 10 Done Sent Done false
7062 Acked Read 5 Done Sent Done false
7063 Acked Read 6 Done Sent Done false
7064 Acked Read 3 Done Sent Done false
7065 Acked Flush 0 Done Sent Done false
7066 Acked Read 2 Done Sent Done false
7067 Acked Read 3 Done Sent Done false
7068 Acked Read 2 Done Sent Done false
7069 Acked Write 1 Done Sent Done false
7070 Acked Read 2 Done Sent Done false
7071 Acked Read 9 Done Sent Done false
7072 Acked Write 8 Done Sent Done false
7073 Acked Read 7 Done Sent Done false
7074 Acked Read 10 Done Sent Done false
7075 Acked Write 10 Done Sent Done false
7076 Acked Read 5 Done Sent Done false
7077 Acked Write 1 Done Sent Done false
7078 Acked Write 6 Done Sent Done false
7079 Acked Flush 0 Done Sent Done false
7080 Acked Read 10 Done Sent Done false
7081 Acked Read 6 Done Sent Done false
7082 Acked Read 6 Done Sent Done false
7083 Acked Read 9 Done Sent Done false
7084 Acked Write 3 Done Sent Done false
7085 Acked Read 7 Done Sent Done false
7086 Acked Read 10 Done Sent Done false
7087 Acked Read 10 Done Sent Done false
7088 Acked Write 1 Done Sent Done false
7089 Acked Read 5 Done Sent Done false
7090 Acked Read 9 Done Sent Done false
7091 Acked Read 10 Done Sent Done false
7092 Acked Write 6 Done Sent Done false
7093 Acked Write 8 Done Sent Done false
7094 Acked Write 5 Done Sent Done false
7095 Acked Write 10 Done Sent Done false
7096 Acked Write 4 Done Sent Done false
7097 Acked Read 4 Done Sent Done false
7098 Acked Write 3 Done Sent Done false
7099 Acked Read 3 Done Sent Done false
7100 Acked Flush 0 Done Sent Done false
7101 Acked Write 1 Done Sent Done false
7102 Acked Read 6 Done Sent Done false
7103 Acked Read 9 Done Sent Done false
7104 Acked Read 1 Done Sent Done false
7105 Acked Write 7 Done Sent Done false
7106 Acked Read 8 Done Sent Done false
7107 Acked Write 5 Done Sent Done false
7108 Acked Write 5 Done Sent Done false
7109 Acked Read 2 Done Sent Done false
7110 Acked Write 7 Done Sent Done false
7111 Acked Write 2 Done Sent Done false
7112 Acked Read 10 Done Sent Done false
7113 Acked Read 9 Done Sent Done false
7114 Acked Read 7 Done Sent Done false
7115 Acked Write 8 Done Sent Done false
7116 Acked Read 8 Done Sent Done false
7117 Acked Write 1 Done Sent Done false
7118 Acked Write 7 Done Sent Done false
7119 Acked Read 1 Done Sent Done false
7120 Acked Write 5 Done Sent Done false
7121 Acked Write 10 Done Sent Done false
7122 Acked Write 2 Done Sent Done false
7123 Acked Read 4 Done Sent Done false
7124 Acked Write 7 Done Sent Done false
7125 Acked Write 10 Done Sent Done false
7126 Acked Write 9 Done Sent Done false
7127 Acked Write 6 Done Sent Done false
7128 Acked Read 6 Done Sent Done false
7129 Acked Read 7 Done Sent Done false
7130 Acked Write 8 Done Sent Done false
7131 Acked Read 8 Done Sent Done false
7132 Acked Read 4 Done Sent Done false
7133 Acked Write 10 Done Sent Done false
7134 Acked Read 3 Done Sent Done false
7135 Acked Read 3 Done Sent Done false
7136 Acked Read 3 Done Sent Done false
7137 Acked Write 6 Done Sent Done false
7138 Acked Flush 0 Done Sent Done false
7139 Acked Read 2 Done Sent Done false
7140 Acked Write 1 Done Sent Done false
7141 Acked Write 6 Done Sent Done false
7142 Acked Read 8 Done Sent Done false
7143 Acked Flush 0 Done Sent Done false
7144 Acked Flush 0 Done Sent Done false
7145 Acked Write 4 Done Sent Done false
7146 Acked Read 7 Done Sent Done false
7147 Acked Write 1 Done Sent Done false
7148 Acked Read 2 Done Sent Done false
7149 Acked Read 10 Done Sent Done false
7150 Acked Read 2 Done Sent Done false
7151 Acked Write 6 Done Sent Done false
7152 Acked Flush 0 Done Sent Done false
7153 Acked Read 4 Done Sent Done false
7154 Acked Read 3 Done Sent Done false
7155 Acked Write 1 Done Sent Done false
7156 Acked Write 10 Done Sent Done false
7157 Acked Write 6 Done Sent Done false
7158 Acked Write 6 Done Sent Done false
7159 Acked Read 4 Done Sent Done false
7160 Acked Read 9 Done Sent Done false
7161 Acked Write 6 Done Sent Done false
7162 Acked Write 1 Done Sent Done false
7163 Acked Write 2 Done Sent Done false
7164 Acked Read 3 Done Sent Done false
7165 Acked Write 7 Done Sent Done false
7166 Acked Flush 0 Done Sent Done false
7167 Acked Write 6 Done Sent Done false
7168 Acked Write 10 Done Sent Done false
7169 Acked Write 5 Done Sent Done false
7170 Acked Read 7 Done Sent Done false
7171 Acked Read 4 Done Sent Done false
7172 Acked Read 2 Done Sent Done false
7173 Acked Flush 0 Done Sent Done false
7174 Acked Read 1 Done Sent Done false
7175 Acked Read 7 Done Sent Done false
7176 Acked Write 9 Done Sent Done false
7177 Acked Write 5 Done Sent Done false
7178 Acked Read 5 Done Sent Done false
7179 Acked Write 1 Done Sent Done false
7180 Acked Read 1 Done Sent Done false
7181 Acked Write 6 Done Sent Done false
7182 Acked Write 5 Done Sent Done false
7183 Acked Write 8 Done Sent Done false
7184 Acked Read 4 Done Sent Done false
7185 Acked Write 5 Done Sent Done false
7186 Acked Write 5 Done Sent Done false
7187 Acked Read 8 Done Sent Done false
7188 Acked Read 9 Done Sent Done false
7189 Acked Read 8 Done Sent Done false
7190 Acked Write 8 Done Sent Done false
7191 Acked Read 4 Done Sent Done false
7192 Acked Flush 0 Done Sent Done false
7193 Acked Write 7 Done Sent Done false
7194 Acked Read 3 Done Sent Done false
7195 Acked Write 6 Done Sent Done false
7196 Acked Write 5 Done Sent Done false
7197 Acked Read 3 Done Sent Done false
7198 Acked Read 10 Done Sent Done false
7199 Acked Write 2 Done Sent Done false
7200 Acked Read 6 Done Sent Done false
7201 Acked Read 3 Done Sent Done false
7202 Acked Write 4 Done Sent Done false
7203 Acked Read 4 Done Sent Done false
7204 Acked Write 8 Done Sent Done false
7205 Acked Read 5 Done Sent Done false
7206 Acked Read 10 Done Sent Done false
7207 Acked Read 2 Done Sent Done false
7208 Acked Flush 0 Done Sent Done false
7209 Acked Read 4 Done Sent Done false
7210 Acked Read 2 Done Sent Done false
7211 Acked Read 5 Done Sent Done false
7212 Acked Write 5 Done Sent Done false
7213 Acked Read 4 Done Sent Done false
7214 Acked Flush 0 Done Sent Done false
7215 Acked Read 3 Done Sent Done false
7216 Acked Write 2 Done Sent Done false
7217 Acked Write 7 Done Sent Done false
7218 Acked Write 4 Done Sent Done false
7219 Acked Write 6 Done Sent Done false
7220 Acked Flush 0 Done Sent Done false
7221 Acked Read 8 Done Sent Done false
7222 Acked Read 5 Done Sent Done false
7223 Acked Read 2 Done Sent Done false
7224 Acked Read 6 Done Sent Done false
7225 Acked Write 8 Done Sent Done false
7226 Acked Read 6 Done Sent Done false
7227 Acked Flush 0 Done Sent Done false
7228 Acked Read 2 Done Sent Done false
7229 Acked Read 10 Done Sent Done false
7230 Acked Read 5 Done Sent Done false
7231 Acked Write 2 Done Sent Done false
7232 Acked Write 1 Done Sent Done false
7233 Acked Read 3 Done Sent Done false
7234 Acked Flush 0 Done Sent Done false
7235 Acked Flush 0 Done Sent Done false
7236 Acked Read 1 Done Sent Done false
7237 Acked Write 3 Done Sent Done false
7238 Acked Read 8 Done Sent Done false
7239 Acked Read 8 Done Sent Done false
7240 Acked Read 3 Done Sent Done false
7241 Acked Read 8 Done Sent Done false
7242 Acked Read 10 Done Sent Done false
7243 Acked Write 2 Done Sent Done false
7244 Acked Write 9 Done Sent Done false
7245 Acked Write 4 Done Sent Done false
7246 Acked Read 5 Done Sent Done false
7247 Acked Read 5 Done Sent Done false
7248 Acked Write 6 Done Sent Done false
7249 Acked Write 10 Done Sent Done false
7250 Acked Flush 0 Done Sent Done false
7251 Acked Read 2 Done Sent Done false
7252 Acked Write 10 Done Sent Done false
7253 Acked Read 5 Done Sent Done false
7254 Acked Write 9 Done Sent Done false
7255 Acked Write 5 Done Sent Done false
7256 Acked Write 4 Done Sent Done false
7257 Acked Read 9 Done Sent Done false
7258 Acked Read 7 Done Sent Done false
7259 Acked Read 2 Done Sent Done false
7260 Acked Read 7 Done Sent Done false
7261 Acked Write 10 Done Sent Done false
7262 Acked Flush 0 Done Sent Done false
7263 Acked Write 6 Done Sent Done false
7264 Acked Write 7 Done Sent Done false
7265 Acked Read 6 Done Sent Done false
7266 Acked Read 10 Done Sent Done false
7267 Acked Write 7 Done Sent Done false
7268 Acked Write 5 Done Sent Done false
7269 Acked Write 4 Done Sent Done false
7270 Acked Write 8 Done Sent Done false
7271 Acked Read 4 Done Sent Done false
7272 Acked Read 8 Done Sent Done false
7273 Acked Read 6 Done Sent Done false
7274 Acked Write 9 Done Sent Done false
7275 Acked Read 2 Done Sent Done false
7276 Acked Flush 0 Done Sent Done false
7277 Acked Read 10 Done Sent Done false
7278 Acked Write 8 Done Sent Done false
7279 Acked Write 8 Done Sent Done false
7280 Acked Flush 0 Done Sent Done false
7281 Acked Write 10 Done Sent Done false
7282 Acked Write 8 Done Sent Done false
7283 Acked Read 2 Done Sent Done false
7284 Acked Write 10 Done Sent Done false
7285 Acked Read 4 Done Sent Done false
7286 Acked Read 2 Done Sent Done false
7287 Acked Flush 0 Done Sent Done false
7288 Acked Write 4 Done Sent Done false
7289 Acked Read 9 Done Sent Done false
7290 Acked Read 8 Done Sent Done false
7291 Acked Read 4 Done Sent Done false
7292 Acked Read 5 Done Sent Done false
7293 Acked Read 5 Done Sent Done false
7294 Acked Read 10 Done Sent Done false
7295 Acked Write 8 Done Sent Done false
7296 Acked Read 1 Done Sent Done false
7297 Acked Read 4 Done Sent Done false
7298 Acked Read 4 Done Sent Done false
7299 Acked Write 2 Done Sent Done false
7300 Acked Write 2 Done Sent Done false
7301 Acked Read 1 Done Sent Done false
7302 Acked Read 3 Done Sent Done false
7303 Acked Read 9 Done Sent Done false
7304 Acked Read 2 Done Sent Done false
7305 Acked Write 1 Done Sent Done false
7306 Acked Write 7 Done Sent Done false
7307 Acked Read 6 Done Sent Done false
7308 Acked Write 9 Done Sent Done false
7309 Acked Read 9 Done Sent Done false
7310 Acked Write 5 Done Sent Done false
7311 Acked Write 2 Done Sent Done false
7312 Acked Write 10 Done Sent Done false
7313 Acked Write 3 Done Sent Done false
7314 Acked Read 5 Done Sent Done false
7315 Acked Read 6 Done Sent Done false
7316 Acked Flush 0 Done Sent Done false
7317 Acked Read 4 Done Sent Done false
7318 Acked Write 4 Done Sent Done false
7319 Acked Write 3 Done Sent Done false
7320 Acked Write 3 Done Sent Done false
7321 Acked Read 9 Done Sent Done false
7322 Acked Read 4 Done Sent Done false
7323 Acked Write 6 Done Sent Done false
7324 Acked Read 4 Done Sent Done false
7325 Acked Read 4 Done Sent Done false
7326 Acked Read 2 Done Sent Done false
7327 Acked Flush 0 Done Sent Done false
7328 Acked Read 6 Done Sent Done false
7329 Acked Flush 0 Done Sent Done false
7330 Acked Read 1 Done Sent Done false
7331 Acked Flush 0 Done Sent Done false
7332 Acked Write 8 Done Sent Done false
7333 Acked Write 8 Done Sent Done false
7334 Acked Write 4 Done Sent Done false
7335 Acked Flush 0 Done Sent Done false
7336 Acked Read 8 Done Sent Done false
7337 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:41.908330995Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:43.139134094Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:43.912693036Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.881585754Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54743)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.881762961Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.882741767Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.883225161Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.883244466Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(7036))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.883258623Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.88327194Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(7036))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.883285547Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(7036))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:44.88348032Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7336 Acked Read 8 Done Done Done true
7337 Acked Read 3 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 2 2 2 6
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7335)) Some(JobId(7335)) Some(JobId(7335))
Downstairs last five completed: 7335 7334 7333 7332 7331
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:2 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7338 7337 7336 7335 7334
Upstairs last five completed: 7338 7337 7336 7335 7334
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:06:49.136 INFO request completed, latency_us: 495[8820] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6dd990bf-b99c-4948-848e-753885f0867e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:06:49.137 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/1, method: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137181493Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
GET, req_id: 58f73262-0c46-4b79-b299-f2802a12dc64, remote_addr: 127.0.0.1:63908, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137264533Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
local_addr: 127.0.0.1:9998{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137302584Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:49.137380695Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:06:53.139 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 38235a08-4405-4a0e-8913-e9310ffacbf1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Flush
003/300 Flush
004/300 Write block 2607 len 20480 data: 3 3 3 2 4
005/300 Read block 4059 len 4096
006/300 Read block 2440 len 20480
007/300 Write block 2751 len 24576 data: 7 4 3 3 4 4
008/300 Read block 7 len 36864
009/300 Write block 2274 len 32768 data: 6 6 7 7 8 8 8 7
010/300 Read block 2058 len 4096
011/300 Read block 1949 len 20480
012/300 Write block 4329 len 20480 data: 3 3 2 3 4
013/300 Read block 140 len 28672
014/300 Read block 2326 len 16384
015/300 Read block 1811 len 40960
016/300 Write block 1137 len 40960 data: 5 5 4 4 3 5 5 5 7 6
017/300 Write block 4598 len 16384 data: 7 5 5 5
018/300 Write block 1930 len 20480 data: 4 4 5 4 4
019/300 Read block 3548 len 36864
020/300 Read block 4689 len 24576
021/300 Flush
022/300 Write block 2065 len 16384 data: 5 4 4 5
023/300 Write block 3424 len 16384 data: 4 3 3 3
024/300 Read block 4125 len 4096
025/300 Read block 285 len 32768
026/300 Read block 4132 len 8192
027/300 Write block 3323 len 20480 data: 6 6 6 5 6
028/300 Write block 1218 len 28672 data: 3 4 4 4 5 5 5
029/300 Read block 3828 len 16384
030/300 Read block 2139 len 24576
031/300 Read block 2697 len 40960
032/300 Write block 2892 len 32768 data: 3 2 3 4 4 5 6 4
033/300 Write block 4646 len 12288 data: 4 7 6
034/300 Read block 1790 len 32768
035/300 Read block 3597 len 36864
036/300 Write block 4501 len 8192 data: 8 6
037/300 Read block 2045 len 36864
038/300 Read block 2321 len 20480
039/300 Flush
040/300 Read block 2178 len 4096
041/300 Read block 4439 len 28672
042/300 Read block 556 len 32768
043/300 Flush
044/300 Read block 1739 len 28672
045/300 Read block 4026 len 28672
046/300 Read block 4736 len 28672
047/300 Write block 740 len 28672 data: 5 4 5 5 6 5 5
048/300 Read block 1238 len 24576
049/300 Flush
050/300 Read block 2710 len 4096
051/300 Write block 3213 len 36864 data: 5 5 6 6 4 4 5 6 6
052/300 Read block 2021 len 24576
053/300 Write block 4729 len 8192 data: 7 8
054/300 Write block 2508 len 16384 data: 3 4 4 6
055/300 Write block 2994 len 40960 data: 5 4 5 5 5 5 5 6 4 4
056/300 Read block 1924 len 4096
057/300 Write block 1497 len 8192 data: 2 2
058/300 Read block 4616 len 24576
059/300 Read block 2399 len 24576
060/300 Read block 931 len 16384
061/300 Read block 1478 len 40960
062/300 Read block 2949 len 24576
063/300 Flush
064/300 Read block 4383 len 36864
065/300 Read block 51 len 32768
066/300 Write block 3462 len 16384 data: 3 3 3 5
067/300 Write block 3943 len 12288 data: 5 6 4
068/300 Read block 157 len 16384
069/300 Read block 42 len 12288
070/300 Read block 4256 len 8192
071/300 Flush
072/300 Read block 3636 len 16384
073/300 Write block 4476 len 4096 data: 5
074/300 Read block 3619 len 24576
075/300 Write block 440 len 20480 data: 9 9 10 10 11
076/300 Write block 3865 len 16384 data: 6 6 7 6
077/300 Read block 4732 len 40960
078/300 Write block 53 len 24576 data: 5 5 5 5 5 5
079/300 Write block 668 len 12288 data: 6 6 7
080/300 Flush
081/300 Read block 3189 len 40960
082/300 Write block 4419 len 4096 data: 4
083/300 Write block 2111 len 36864 data: 5 4 4 5 4 5 6 6 5
084/300 Read block 3404 len 4096
085/300 Read block 1051 len 8192
086/300 Write block 4192 len 28672 data: 5 5 4 4 4 5 5
087/300 Read block 2696 len 28672
088/300 Read block 4307 len 4096
089/300 Flush
090/300 Flush
091/300 Read block 2700 len 12288
092/300 Write block 2948 len 28672 data: 4 4 5 5 5 5 5
093/300 Read block 3534 len 36864
094/300 Write block 2120 len 28672 data: 5 6 6 5 6 5 5
095/300 Read block 3900 len 24576
096/300 Read block 1880 len 8192
097/300 Write block 1424 len 20480 data: 5 6 5 5 3
098/300 Read block 4859 len 24576
099/300 Read block 3996 len 20480
100/300 Read block 2924 len 24576
101/300 Flush
102/300 Flush
103/300 Read block 3507 len 36864
104/300 Write block 1934 len 12288 data: 5 3 4
105/300 Read block 3445 len 16384
106/300 Read block 4961 len 32768
107/300 Read block 3353 len 40960
108/300 Read block 4552 len 8192
109/300 Write block 4086 len 8192 data: 4 4
110/300 Read block 3230 len 32768
111/300 Read block 4287 len 40960
112/300 Write block 743 len 32768 data: 6 7 6 6 5 6 7 7
113/300 Read block 3982 len 20480
114/300 Flush
115/300 Read block 541 len 16384
116/300 Write block 4693 len 32768 data: 4 4 5 6 6 5 7 5
117/300 Read block 3059 len 36864
118/300 Flush
119/300 Read block 4574 len 12288
120/300 Read block 3381 len 32768
121/300 Read block 92 len 32768
122/300 Read block 1165 len 32768
123/300 Read block 1519 len 8192
124/300 Read block 4081 len 36864
125/300 Write block 1055 len 8192 data: 7 6
126/300 Write block 4949 len 40960 data: 2 4 6 5 6 7 6 6 7 8
127/300 Read block 683 len 20480
128/300 Read block 3941 len 4096
129/300 Flush
130/300 Write block 2452 len 16384 data: 6 5 6 8
131/300 Write block 1162 len 28672 data: 5 5 7 6 6 6 5
132/300 Write block 3383 len 28672 data: 3 3 3 2 2 2 3
133/300 Read block 2934 len 20480
134/300 Read block 481 len 16384
135/300 Write block 2080 len 28672 data: 2 2 2 4 4 6 8
136/300 Read block 737 len 40960
137/300 Write block 3690 len 40960 data: 6 6 7 5 3 2 2 2 2 3
138/300 Write block 2584 len 20480 data: 5 5 4 4 3
139/300 Flush
140/300 Write block 1758 len 24576 data: 5 5 5 5 5 5
141/300 Write block 768 len 24576 data: 3 5 5 4 5 4
142/300 Read block 1154 len 8192
143/300 Read block 4519 len 28672
144/300 Read block 441 len 36864
145/300 Write block 4593 len 24576 data: 6 6 8 9 8 8
146/300 Read block 2942 len 40960
147/300 Read block 4833 len 28672
148/300 Read block 3216 len 12288
149/300 Read block 2480 len 36864
150/300 Read block 3381 len 20480
151/300 Read block 2569 len 24576
152/300 Read block 1992 len 40960
153/300 Read block 3607 len 16384
154/300 Read block 4919 len 20480
155/300 Read block 428 len 28672
156/300 Read block 392 len 12288
157/300 Read block 4157 len 12288
158/300 Read block 1090 len 40960
159/300 Read block 3039 len 8192
160/300 Flush
161/300 Write block 273 len 24576 data: 5 6 5 5 7 8
162/300 Read block 3172 len 28672
163/300 Read block 3826 len 20480
164/300 Read block 4182 len 24576
165/300 Read block 981 len 28672
166/300 Write block 586 len 20480 data: 5 3 4 3 4
167/300 Read block 466 len 28672
168/300 Read block 4899 len 28672
169/300 Read block 4683 len 4096
170/300 Read block 3692 len 20480
171/300 Read block 1860 len 24576
172/300 Write block 424 len 4096 data: 6
173/300 Read block 4521 len 36864
174/300 Read block 872 len 20480
175/300 Write block 4534 len 12288 data: 4 5 5
176/300 Write block 4794 len 20480 data: 4 4 4 4 4
177/300 Write block 4977 len 32768 data: 4 6 5 4 4 6 5 6
178/300 Flush
179/300 Read block 1549 len 32768
180/300 Read block 955 len 16384
181/300 Flush
182/300 Write block 3931 len 12288 data: 3 3 5
183/300 Read block 4411 len 8192
184/300 Read block 2393 len 8192
185/300 Read block 331 len 16384
186/300 Read block 890 len 20480
187/300 Read block 381 len 40960
188/300 Read block 981 len 4096
189/300 Read block 2771 len 4096
190/300 Read block 4 len 36864
191/300 Read block 2644 len 16384
192/300 Flush
193/300 Read block 2201 len 24576
194/300 Write block 3766 len 40960 data: 4 4 5 5 4 4 4 5 5 3
195/300 Write block 1170 len 40960 data: 6 6 6 9 10 8 8 9 10 10
196/300 Write block 170 len 36864 data: 2 2 3 3 4 4 4 4 4
197/300 Write block 4231 len 16384 data: 9 5 5 4
198/300 Write block 1470 len 8192 data: 5 6
199/300 Read block 1048 len 20480
200/300 Read block 2800 len 8192
201/300 Write block 1869 len 8192 data: 6 4
202/300 Write block 1287 len 40960 data: 4 4 5 5 6 5 5 4 3 3
203/300 Read block 2146 len 12288
204/300 Read block 2801 len 40960
205/300 Read block 4442 len 16384
206/300 Write block 1490 len 12288 data: 8 8 8
207/300 Read block 546 len 16384
208/300 Write block 4379 len 12288 data: 8 8 7
209/300 Flush
210/300 Read block 602 len 40960
211/300 Read block 1109 len 16384
212/300 Read block 1932 len 36864
213/300 Read block 3952 len 20480
214/300 Write block 4925 len 36864 data: 6 7 6 4 2 3 3 3 3
215/300 Write block 2371 len 24576 data: 4 3 5 5 6 7
216/300 Flush
217/300 Write block 2441 len 20480 data: 5 7 6 6 6
218/300 Read block 2256 len 36864
219/300 Read block 2188 len 40960
220/300 Read block 862 len 28672
221/300 Write block 412 len 4096 data: 4
222/300 Flush
223/300 Write block 3249 len 12288 data: 7 7 6
224/300 Write block 3778 len 36864 data: 4 4 5 5 6 6 4 3 3
225/300 Write block 812 len 16384 data: 9 11 8 6
226/300 Write block 2702 len 40960 data: 5 5 5 6 6 5 5 5 4 4
227/300 Write block 4512 len 36864 data: 5 6 7 6 3 4 4 4 6
228/300 Write block 2128 len 40960 data: 5 3 4 5 6 6 6 5 4 6
229/300 Read block 4221 len 32768
230/300 Write block 4215 len 8192 data: 5 5
231/300 Write block 1644 len 32768 data: 3 3 3 4 4 4 4 4
232/300 Read block 1288 len 32768
233/300 Write block 4449 len 12288 data: 2 2 2
234/300 Read block 4677 len 28672
235/300 Write block 1529 len 24576 data: 5 4 3 3 3 5
236/300 Read block 2092 len 16384
237/300 Read block 199 len 8192
238/300 Read block 4570 len 32768
239/300 Write block 2601 len 32768 data: 6 5 6 6 5 4 4 4
240/300 Read block 4149 len 32768
241/300 Write block 213 len 32768 data: 4 6 7 5 6 7 9 6
242/300 Write block 4312 len 28672 data: 5 4 4 3 3 3 4
243/300 Read block 4943 len 28672
244/300 Write block 4579 len 20480 data: 2 2 2 2 2
245/300 Write block 1752 len 24576 data: 4 4 5 5 4 4
246/300 Read block 3865 len 36864
247/300 Write block 524 len 8192 data: 8 6
248/300 Write block 2613 len 32768 data: 3 2 3 3 3 3 3 3
249/300 Read block 1328 len 20480
250/300 Read block 84 len 40960
251/300 Write block 3419 len 20480 data: 4 3 3 3 3
252/300 Write block 1868 len 24576 data: 4 7 5 4 5 6
253/300 Write block 73 len 20480 data: 4 4 7 6 6
254/300 Write block 3528 len 24576 data: 8 7 5 5 5 5
255/300 Write block 912 len 36864 data: 5 5 4 5 3 3 4 4 4
256/300 Read block 3526 len 20480
257/300 Write block 387 len 36864 data: 3 2 2 2 2 3 2 3 3
258/300 Write block 1308 len 24576 data: 5 6 6 7 7 7
259/300 Read block 4679 len 28672
260/300 Write block 4863 len 8192 data: 4 5
261/300 Read block 841 len 8192
262/300 Read block 1985 len 32768
263/300 Write block 4101 len 24576 data: 8 7 7 6 3 4
264/300 Read block 1769 len 28672
265/300 Write block 582 len 28672 data: 7 7 5 5 6 4 5
266/300 Write block 2607 len 24576 data: 5 5 4 3 5 3
267/300 Read block 4916 len 20480
268/300 Read block 4338 len 12288
269/300 Read block 4287 len 32768
270/300 Read block 3547 len 40960
271/300 Read block 54 len 40960
272/300 Flush
273/300 Write block 406 len 16384 data: 4 4 5 4
274/300 Write block 4336 len 36864 data: 6 6 5 6 5 5 8 7 7
275/300 Flush
276/300 Write block 1139 len 20480 data: 5 5 4 6 6
277/300 Write block 2772 len 12288 data: 3 3 3
278/300 Read block 4609 len 8192
279/300 Write block 1662 len 36864 data: 5 6 6 8 8 6 7 7 6
280/300 Read block 1342 len 8192
281/300 Read block 3349 len 20480
282/300 Write block 4684 len 16384 data: 5 4 4 5
283/300 Write block 3743 len 12288 data: 9 9 9
284/300 Write block 4510 len 4096 data: 6
285/300 Read block 3098 len 28672
286/300 Write block 4784 len 4096 data: 5
287/300 Write block 2252 len 20480 data: 2 3 4 3 4
288/300 Read block 2913 len 24576
289/300 Flush
290/300 Write block 4646 len 36864 data: 5 8 7 7 6 6 8 8 8
291/300 Flush
292/300 Read block 1122 len 36864
293/300 Write block 4554 len 40960 data: 3 3 2 3 3 3 3 3 3 5
294/300 Read block 4175 len 8192
295/300 Write block 3816 len 36864 data: 3 3 3 3 2 2 3 4 3
296/300 Read block 1254 len 12288
297/300 Flush
298/300 Write block 4752 len 12288 data: 3 3 3
299/300 Write block 3287 len 20480 data: 4 4 4 4 3
300/300 Read block 878 len 20480
301/300 Read block 2821 len 36864
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:06:53.390 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/1, method: POST, req_id: 377e2997-78d2-4fff-a5f6-7fe8f7c0f798, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7339 Acked Flush 0 Done Sent Done false
7340 Acked Flush 0 Done Sent Done false
7341 Acked Flush 0 Done Sent Done false
7342 Acked Write 5 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Read 5 Done Sent Done false
7345 Acked Write 6 Done Sent Done false
7346 Acked Read 9 Done Sent Done false
7347 Acked Write 8 Done Sent Done false
7348 Acked Read 1 Done Sent Done false
7349 Acked Read 5 Done Sent Done false
7350 Acked Write 5 Done Sent Done false
7351 Acked Read 7 Done Sent Done false
7352 Acked Read 4 Done Sent Done false
7353 Acked Read 10 Done Sent Done false
7354 Acked Write 10 Done Sent Done false
7355 Acked Write 4 Done Sent Done false
7356 Acked Write 5 Done Sent Done false
7357 Acked Read 9 Done Sent Done false
7358 Acked Read 6 Done Sent Done false
7359 Acked Flush 0 Done Sent Done false
7360 Acked Write 4 Done Sent Done false
7361 Acked Write 4 Done Sent Done false
7362 Acked Read 1 Done Sent Done false
7363 Acked Read 8 Done Sent Done false
7364 Acked Read 2 Done Sent Done false
7365 Acked Write 5 Done Sent Done false
7366 Acked Write 7 Done Sent Done false
7367 Acked Read 4 Done Sent Done false
7368 Acked Read 6 Done Sent Done false
7369 Acked Read 10 Done Sent Done false
7370 Acked Write 8 Done Sent Done false
7371 Acked Write 3 Done Sent Done false
7372 Acked Read 8 Done Sent Done false
7373 Acked Read 9 Done Sent Done false
7374 Acked Write 2 Done Sent Done false
7375 Acked Read 9 Done Sent Done false
7376 Acked Read 5 Done Sent Done false
7377 Acked Flush 0 Done Sent Done false
7378 Acked Read 1 Done Sent Done false
7379 Acked Read 7 Done Sent Done false
7380 Acked Read 8 Done Sent Done false
7381 Acked Flush 0 Done Sent Done false
7382 Acked Read 7 Done Sent Done false
7383 Acked Read 7 Done Sent Done false
7384 Acked Read 7 Done Sent Done false
7385 Acked Write 7 Done Sent Done false
7386 Acked Read 6 Done Sent Done false
7387 Acked Flush 0 Done Sent Done false
7388 Acked Read 1 Done Sent Done false
7389 Acked Write 9 Done Sent Done false
7390 Acked Read 6 Done Sent Done false
7391 Acked Write 2 Done Sent Done false
7392 Acked Write 4 Done Sent Done false
7393 Acked Write 10 Done Sent Done false
7394 Acked Read 1 Done Sent Done false
7395 Acked Write 2 Done Sent Done false
7396 Acked Read 6 Done Sent Done false
7397 Acked Read 6 Done Sent Done false
7398 Acked Read 4 Done Sent Done false
7399 Acked Read 10 Done Sent Done false
7400 Acked Read 6 Done Sent Done false
7401 Acked Flush 0 Done Sent Done false
7402 Acked Read 9 Done Sent Done false
7403 Acked Read 8 Done Sent Done false
7404 Acked Write 4 Done Sent Done false
7405 Acked Write 3 Done Sent Done false
7406 Acked Read 4 Done Sent Done false
7407 Acked Read 3 Done Sent Done false
7408 Acked Read 2 Done Sent Done false
7409 Acked Flush 0 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 1 Done Sent Done false
7412 Acked Read 6 Done Sent Done false
7413 Acked Write 5 Done Sent Done false
7414 Acked Write 4 Done Sent Done false
7415 Acked Read 10 Done Sent Done false
7416 Acked Write 6 Done Sent Done false
7417 Acked Write 3 Done Sent Done false
7418 Acked Flush 0 Done Sent Done false
7419 Acked Read 10 Done Sent Done false
7420 Acked Write 1 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 1 Done Sent Done false
7423 Acked Read 2 Done Sent Done false
7424 Acked Write 7 Done Sent Done false
7425 Acked Read 7 Done Sent Done false
7426 Acked Read 1 Done Sent Done false
7427 Acked Flush 0 Done Sent Done false
7428 Acked Flush 0 Done Sent Done false
7429 Acked Read 3 Done Sent Done false
7430 Acked Write 7 Done Sent Done false
7431 Acked Read 9 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 6 Done Sent Done false
7434 Acked Read 2 Done Sent Done false
7435 Acked Write 5 Done Sent Done false
7436 Acked Read 6 Done Sent Done false
7437 Acked Read 5 Done Sent Done false
7438 Acked Read 6 Done Sent Done false
7439 Acked Flush 0 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Read 9 Done Sent Done false
7442 Acked Write 3 Done Sent Done false
7443 Acked Read 4 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 10 Done Sent Done false
7446 Acked Read 2 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 8 Done Sent Done false
7449 Acked Read 10 Done Sent Done false
7450 Acked Write 8 Done Sent Done false
7451 Acked Read 5 Done Sent Done false
7452 Acked Flush 0 Done Sent Done false
7453 Acked Read 4 Done Sent Done false
7454 Acked Write 8 Done Sent Done false
7455 Acked Read 9 Done Sent Done false
7456 Acked Flush 0 Done Sent Done false
7457 Acked Read 3 Done Sent Done false
7458 Acked Read 8 Done Sent Done false
7459 Acked Read 8 Done Sent Done false
7460 Acked Read 8 Done Sent Done false
7461 Acked Read 2 Done Sent Done false
7462 Acked Read 9 Done Sent Done false
7463 Acked Write 2 Done Sent Done false
7464 Acked Write 10 Done Sent Done false
7465 Acked Read 5 Done Sent Done false
7466 Acked Read 1 Done Sent Done false
7467 Acked Flush 0 Done Sent Done false
7468 Acked Write 4 Done Sent Done false
7469 Acked Write 7 Done Sent Done false
7470 Acked Write 7 Done Sent Done false
7471 Acked Read 5 Done Sent Done false
7472 Acked Read 4 Done Sent Done false
7473 Acked Write 7 Done Sent Done false
7474 Acked Read 10 Done Sent Done false
7475 Acked Write 10 Done Sent Done false
7476 Acked Write 5 Done Sent Done false
7477 Acked Flush 0 Done Sent Done false
7478 Acked Write 6 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 2 Done Sent Done false
7481 Acked Read 7 Done Sent Done false
7482 Acked Read 9 Done Sent Done false
7483 Acked Write 6 Done Sent Done false
7484 Acked Read 10 Done Sent Done false
7485 Acked Read 7 Done Sent Done false
7486 Acked Read 3 Done Sent Done false
7487 Acked Read 9 Done Sent Done false
7488 Acked Read 5 Done Sent Done false
7489 Acked Read 6 Done Sent Done false
7490 Acked Read 10 Done Sent Done false
7491 Acked Read 4 Done Sent Done false
7492 Acked Read 5 Done Sent Done false
7493 Acked Read 7 Done Sent Done false
7494 Acked Read 3 Done Sent Done false
7495 Acked Read 3 Done Sent Done false
7496 Acked Read 10 Done Sent Done false
7497 Acked Read 2 Done Sent Done false
7498 Acked Flush 0 Done Sent Done false
7499 Acked Write 6 Done Sent Done false
7500 Acked Read 7 Done Sent Done false
7501 Acked Read 5 Done Sent Done false
7502 Acked Read 6 Done Sent Done false
7503 Acked Read 7 Done Sent Done false
7504 Acked Write 5 Done Sent Done false
7505 Acked Read 7 Done Sent Done false
7506 Acked Read 7 Done Sent Done false
7507 Acked Read 1 Done Sent Done false
7508 Acked Read 5 Done Sent Done false
7509 Acked Read 6 Done Sent Done false
7510 Acked Write 1 Done Sent Done false
7511 Acked Read 9 Done Sent Done false
7512 Acked Read 5 Done Sent Done false
7513 Acked Write 3 Done Sent Done false
7514 Acked Write 5 Done Sent Done false
7515 Acked Write 8 Done Sent Done false
7516 Acked Flush 0 Done Sent Done false
7517 Acked Read 8 Done Sent Done false
7518 Acked Read 4 Done Sent Done false
7519 Acked Flush 0 Done Sent Done false
7520 Acked Write 3 Done Sent Done false
7521 Acked Read 2 Done Sent Done false
7522 Acked Read 2 Done Sent Done false
7523 Acked Read 4 Done Sent Done false
7524 Acked Read 5 Done Sent Done false
7525 Acked Read 10 Done Sent Done false
7526 Acked Read 1 Done Sent Done false
7527 Acked Read 1 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Read 4 Done Sent Done false
7530 Acked Flush 0 Done Sent Done false
7531 Acked Read 6 Done Sent Done false
7532 Acked Write 10 Done Sent Done false
7533 Acked Write 10 Done Sent Done false
7534 Acked Write 9 Done Sent Done false
7535 Acked Write 4 Done Sent Done false
7536 Acked Write 2 Done Sent Done false
7537 Acked Read 5 Done Sent Done false
7538 Acked Read 2 Done Sent Done false
7539 Acked Write 2 Done Sent Done false
7540 Acked Write 10 Done Sent Done false
7541 Acked Read 3 Done Sent Done false
7542 Acked Read 10 Done Sent Done false
7543 Acked Read 4 Done Sent Done false
7544 Acked Write 3 Done Sent Done false
7545 Acked Read 4 Done Sent Done false
7546 Acked Write 3 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Read 10 Done Sent Done false
7549 Acked Read 4 Done Sent Done false
7550 Acked Read 9 Done Sent Done false
7551 Acked Read 5 Done Sent Done false
7552 Acked Write 9 Done Sent Done false
7553 Acked Write 6 Done Sent Done false
7554 Acked Flush 0 Done Sent Done false
7555 Acked Write 5 Done Sent Done false
7556 Acked Read 9 Done Sent Done false
7557 Acked Read 10 Done Sent Done false
7558 Acked Read 7 Done Sent Done false
7559 Acked Write 1 Done Sent Done false
7560 Acked Flush 0 Done Sent Done false
7561 Acked Write 3 Done Sent Done false
7562 Acked Write 9 Done Sent Done false
7563 Acked Write 4 Done Sent Done false
7564 Acked Write 10 Done Sent Done false
7565 Acked Write 9 Done Sent Done false
7566 Acked Write 10 Done Sent Done false
7567 Acked Read 8 Done Sent Done false
7568 Acked Write 2 Done Sent Done false
7569 Acked Write 8 Done Sent Done false
7570 Acked Read 8 Done Sent Done false
7571 Acked Write 3 Done Sent Done false
7572 Acked Read 7 Done Sent Done false
7573 Acked Write 6 Done Sent Done false
7574 Acked Read 4 Done Sent Done false
7575 Acked Read 2 Done Sent Done false
7576 Acked Read 8 Done Sent Done false
7577 Acked Write 8 Done Sent Done false
7578 Acked Read 8 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 7 Done Sent Done false
7581 Acked Read 7 Done Sent Done false
7582 Acked Write 5 Done Sent Done false
7583 Acked Write 6 Done Sent Done false
7584 Acked Read 9 Done Sent Done false
7585 Acked Write 2 Done Sent Done false
7586 Acked Write 8 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 5 Done Sent Done false
7590 Acked Write 6 Done Sent Done false
7591 Acked Write 5 Done Sent Done false
7592 Acked Write 6 Done Sent Done false
7593 Acked Write 9 Done Sent Done false
7594 Acked Read 5 Done Sent Done false
7595 Acked Write 9 Done Sent Done false
7596 Acked Write 6 Done Sent Done false
7597 Acked Read 7 Done Sent Done false
7598 Acked Write 2 Done Sent Done false
7599 Acked Read 2 Done Sent Done false
7600 Acked Read 8 Done Sent Done false
7601 Acked Write 6 Done Sent Done false
7602 Acked Read 7 Done Sent Done false
7603 Acked Write 7 Done Sent Done false
7604 Acked Write 6 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 3 Done Sent Done false
7607 Acked Read 8 Done Sent Done false
7608 Acked Read 10 Done Sent Done false
7609 Acked Read 10 Done Sent Done false
7610 Acked Flush 0 Done Sent Done false
7611 Acked Write 4 Done Sent Done false
7612 Acked Write 9 Done Sent Done false
7613 Acked Flush 0 Done Sent Done false
7614 Acked Write 5 Done Sent Done false
7615 Acked Write 3 Done Sent Done false
7616 Acked Read 2 Done Sent Done false
7617 Acked Write 9 Done Sent Done false
7618 Acked Read 2 Done Sent Done false
7619 Acked Read 5 Done Sent Done false
7620 Acked Write 4 Done Sent Done false
7621 Acked Write 3 Done Sent Done false
7622 Acked Write 1 Done Sent Done false
7623 Acked Read 7 Done Sent Done false
7624 Acked Write 1 Done Sent Done false
7625 Acked Write 5 Done Sent Done false
7626 Acked Read 6 Done Sent Done false
7627 Acked Flush 0 Done Sent Done false
7628 Acked Write 9 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Read 9 Done Sent Done false
7631 Acked Write 10 Done Sent Done false
7632 Acked Read 2 Done Sent Done false
7633 Acked Write 9 Done Sent Done false
7634 Acked Read 3 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Write 3 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 5 Done Sent Done false
7639 Acked Read 9 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7635)) Some(JobId(7338)) Some(JobId(7635))
Downstairs last five completed: 7338 7337 7336 7335 7334
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:53.928355986Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:54.930506604Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:55.932741782Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:56.932891589Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7339 Acked Flush 0 Done Sent Done false
7340 Acked Flush 0 Done Sent Done false
7341 Acked Flush 0 Done Sent Done false
7342 Acked Write 5 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Read 5 Done Sent Done false
7345 Acked Write 6 Done Sent Done false
7346 Acked Read 9 Done Sent Done false
7347 Acked Write 8 Done Sent Done false
7348 Acked Read 1 Done Sent Done false
7349 Acked Read 5 Done Sent Done false
7350 Acked Write 5 Done Sent Done false
7351 Acked Read 7 Done Sent Done false
7352 Acked Read 4 Done Sent Done false
7353 Acked Read 10 Done Sent Done false
7354 Acked Write 10 Done Sent Done false
7355 Acked Write 4 Done Sent Done false
7356 Acked Write 5 Done Sent Done false
7357 Acked Read 9 Done Sent Done false
7358 Acked Read 6 Done Sent Done false
7359 Acked Flush 0 Done Sent Done false
7360 Acked Write 4 Done Sent Done false
7361 Acked Write 4 Done Sent Done false
7362 Acked Read 1 Done Sent Done false
7363 Acked Read 8 Done Sent Done false
7364 Acked Read 2 Done Sent Done false
7365 Acked Write 5 Done Sent Done false
7366 Acked Write 7 Done Sent Done false
7367 Acked Read 4 Done Sent Done false
7368 Acked Read 6 Done Sent Done false
7369 Acked Read 10 Done Sent Done false
7370 Acked Write 8 Done Sent Done false
7371 Acked Write 3 Done Sent Done false
7372 Acked Read 8 Done Sent Done false
7373 Acked Read 9 Done Sent Done false
7374 Acked Write 2 Done Sent Done false
7375 Acked Read 9 Done Sent Done false
7376 Acked Read 5 Done Sent Done false
7377 Acked Flush 0 Done Sent Done false
7378 Acked Read 1 Done Sent Done false
7379 Acked Read 7 Done Sent Done false
7380 Acked Read 8 Done Sent Done false
7381 Acked Flush 0 Done Sent Done false
7382 Acked Read 7 Done Sent Done false
7383 Acked Read 7 Done Sent Done false
7384 Acked Read 7 Done Sent Done false
7385 Acked Write 7 Done Sent Done false
7386 Acked Read 6 Done Sent Done false
7387 Acked Flush 0 Done Sent Done false
7388 Acked Read 1 Done Sent Done false
7389 Acked Write 9 Done Sent Done false
7390 Acked Read 6 Done Sent Done false
7391 Acked Write 2 Done Sent Done false
7392 Acked Write 4 Done Sent Done false
7393 Acked Write 10 Done Sent Done false
7394 Acked Read 1 Done Sent Done false
7395 Acked Write 2 Done Sent Done false
7396 Acked Read 6 Done Sent Done false
7397 Acked Read 6 Done Sent Done false
7398 Acked Read 4 Done Sent Done false
7399 Acked Read 10 Done Sent Done false
7400 Acked Read 6 Done Sent Done false
7401 Acked Flush 0 Done Sent Done false
7402 Acked Read 9 Done Sent Done false
7403 Acked Read 8 Done Sent Done false
7404 Acked Write 4 Done Sent Done false
7405 Acked Write 3 Done Sent Done false
7406 Acked Read 4 Done Sent Done false
7407 Acked Read 3 Done Sent Done false
7408 Acked Read 2 Done Sent Done false
7409 Acked Flush 0 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 1 Done Sent Done false
7412 Acked Read 6 Done Sent Done false
7413 Acked Write 5 Done Sent Done false
7414 Acked Write 4 Done Sent Done false
7415 Acked Read 10 Done Sent Done false
7416 Acked Write 6 Done Sent Done false
7417 Acked Write 3 Done Sent Done false
7418 Acked Flush 0 Done Sent Done false
7419 Acked Read 10 Done Sent Done false
7420 Acked Write 1 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 1 Done Sent Done false
7423 Acked Read 2 Done Sent Done false
7424 Acked Write 7 Done Sent Done false
7425 Acked Read 7 Done Sent Done false
7426 Acked Read 1 Done Sent Done false
7427 Acked Flush 0 Done Sent Done false
7428 Acked Flush 0 Done Sent Done false
7429 Acked Read 3 Done Sent Done false
7430 Acked Write 7 Done Sent Done false
7431 Acked Read 9 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 6 Done Sent Done false
7434 Acked Read 2 Done Sent Done false
7435 Acked Write 5 Done Sent Done false
7436 Acked Read 6 Done Sent Done false
7437 Acked Read 5 Done Sent Done false
7438 Acked Read 6 Done Sent Done false
7439 Acked Flush 0 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Read 9 Done Sent Done false
7442 Acked Write 3 Done Sent Done false
7443 Acked Read 4 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 10 Done Sent Done false
7446 Acked Read 2 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 8 Done Sent Done false
7449 Acked Read 10 Done Sent Done false
7450 Acked Write 8 Done Sent Done false
7451 Acked Read 5 Done Sent Done false
7452 Acked Flush 0 Done Sent Done false
7453 Acked Read 4 Done Sent Done false
7454 Acked Write 8 Done Sent Done false
7455 Acked Read 9 Done Sent Done false
7456 Acked Flush 0 Done Sent Done false
7457 Acked Read 3 Done Sent Done false
7458 Acked Read 8 Done Sent Done false
7459 Acked Read 8 Done Sent Done false
7460 Acked Read 8 Done Sent Done false
7461 Acked Read 2 Done Sent Done false
7462 Acked Read 9 Done Sent Done false
7463 Acked Write 2 Done Sent Done false
7464 Acked Write 10 Done Sent Done false
7465 Acked Read 5 Done Sent Done false
7466 Acked Read 1 Done Sent Done false
7467 Acked Flush 0 Done Sent Done false
7468 Acked Write 4 Done Sent Done false
7469 Acked Write 7 Done Sent Done false
7470 Acked Write 7 Done Sent Done false
7471 Acked Read 5 Done Sent Done false
7472 Acked Read 4 Done Sent Done false
7473 Acked Write 7 Done Sent Done false
7474 Acked Read 10 Done Sent Done false
7475 Acked Write 10 Done Sent Done false
7476 Acked Write 5 Done Sent Done false
7477 Acked Flush 0 Done Sent Done false
7478 Acked Write 6 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 2 Done Sent Done false
7481 Acked Read 7 Done Sent Done false
7482 Acked Read 9 Done Sent Done false
7483 Acked Write 6 Done Sent Done false
7484 Acked Read 10 Done Sent Done false
7485 Acked Read 7 Done Sent Done false
7486 Acked Read 3 Done Sent Done false
7487 Acked Read 9 Done Sent Done false
7488 Acked Read 5 Done Sent Done false
7489 Acked Read 6 Done Sent Done false
7490 Acked Read 10 Done Sent Done false
7491 Acked Read 4 Done Sent Done false
7492 Acked Read 5 Done Sent Done false
7493 Acked Read 7 Done Sent Done false
7494 Acked Read 3 Done Sent Done false
7495 Acked Read 3 Done Sent Done false
7496 Acked Read 10 Done Sent Done false
7497 Acked Read 2 Done Sent Done false
7498 Acked Flush 0 Done Sent Done false
7499 Acked Write 6 Done Sent Done false
7500 Acked Read 7 Done Sent Done false
7501 Acked Read 5 Done Sent Done false
7502 Acked Read 6 Done Sent Done false
7503 Acked Read 7 Done Sent Done false
7504 Acked Write 5 Done Sent Done false
7505 Acked Read 7 Done Sent Done false
7506 Acked Read 7 Done Sent Done false
7507 Acked Read 1 Done Sent Done false
7508 Acked Read 5 Done Sent Done false
7509 Acked Read 6 Done Sent Done false
7510 Acked Write 1 Done Sent Done false
7511 Acked Read 9 Done Sent Done false
7512 Acked Read 5 Done Sent Done false
7513 Acked Write 3 Done Sent Done false
7514 Acked Write 5 Done Sent Done false
7515 Acked Write 8 Done Sent Done false
7516 Acked Flush 0 Done Sent Done false
7517 Acked Read 8 Done Sent Done false
7518 Acked Read 4 Done Sent Done false
7519 Acked Flush 0 Done Sent Done false
7520 Acked Write 3 Done Sent Done false
7521 Acked Read 2 Done Sent Done false
7522 Acked Read 2 Done Sent Done false
7523 Acked Read 4 Done Sent Done false
7524 Acked Read 5 Done Sent Done false
7525 Acked Read 10 Done Sent Done false
7526 Acked Read 1 Done Sent Done false
7527 Acked Read 1 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Read 4 Done Sent Done false
7530 Acked Flush 0 Done Sent Done false
7531 Acked Read 6 Done Sent Done false
7532 Acked Write 10 Done Sent Done false
7533 Acked Write 10 Done Sent Done false
7534 Acked Write 9 Done Sent Done false
7535 Acked Write 4 Done Sent Done false
7536 Acked Write 2 Done Sent Done false
7537 Acked Read 5 Done Sent Done false
7538 Acked Read 2 Done Sent Done false
7539 Acked Write 2 Done Sent Done false
7540 Acked Write 10 Done Sent Done false
7541 Acked Read 3 Done Sent Done false
7542 Acked Read 10 Done Sent Done false
7543 Acked Read 4 Done Sent Done false
7544 Acked Write 3 Done Sent Done false
7545 Acked Read 4 Done Sent Done false
7546 Acked Write 3 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Read 10 Done Sent Done false
7549 Acked Read 4 Done Sent Done false
7550 Acked Read 9 Done Sent Done false
7551 Acked Read 5 Done Sent Done false
7552 Acked Write 9 Done Sent Done false
7553 Acked Write 6 Done Sent Done false
7554 Acked Flush 0 Done Sent Done false
7555 Acked Write 5 Done Sent Done false
7556 Acked Read 9 Done Sent Done false
7557 Acked Read 10 Done Sent Done false
7558 Acked Read 7 Done Sent Done false
7559 Acked Write 1 Done Sent Done false
7560 Acked Flush 0 Done Sent Done false
7561 Acked Write 3 Done Sent Done false
7562 Acked Write 9 Done Sent Done false
7563 Acked Write 4 Done Sent Done false
7564 Acked Write 10 Done Sent Done false
7565 Acked Write 9 Done Sent Done false
7566 Acked Write 10 Done Sent Done false
7567 Acked Read 8 Done Sent Done false
7568 Acked Write 2 Done Sent Done false
7569 Acked Write 8 Done Sent Done false
7570 Acked Read 8 Done Sent Done false
7571 Acked Write 3 Done Sent Done false
7572 Acked Read 7 Done Sent Done false
7573 Acked Write 6 Done Sent Done false
7574 Acked Read 4 Done Sent Done false
7575 Acked Read 2 Done Sent Done false
7576 Acked Read 8 Done Sent Done false
7577 Acked Write 8 Done Sent Done false
7578 Acked Read 8 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 7 Done Sent Done false
7581 Acked Read 7 Done Sent Done false
7582 Acked Write 5 Done Sent Done false
7583 Acked Write 6 Done Sent Done false
7584 Acked Read 9 Done Sent Done false
7585 Acked Write 2 Done Sent Done false
7586 Acked Write 8 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 5 Done Sent Done false
7590 Acked Write 6 Done Sent Done false
7591 Acked Write 5 Done Sent Done false
7592 Acked Write 6 Done Sent Done false
7593 Acked Write 9 Done Sent Done false
7594 Acked Read 5 Done Sent Done false
7595 Acked Write 9 Done Sent Done false
7596 Acked Write 6 Done Sent Done false
7597 Acked Read 7 Done Sent Done false
7598 Acked Write 2 Done Sent Done false
7599 Acked Read 2 Done Sent Done false
7600 Acked Read 8 Done Sent Done false
7601 Acked Write 6 Done Sent Done false
7602 Acked Read 7 Done Sent Done false
7603 Acked Write 7 Done Sent Done false
7604 Acked Write 6 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 3 Done Sent Done false
7607 Acked Read 8 Done Sent Done false
7608 Acked Read 10 Done Sent Done false
7609 Acked Read 10 Done Sent Done false
7610 Acked Flush 0 Done Sent Done false
7611 Acked Write 4 Done Sent Done false
7612 Acked Write 9 Done Sent Done false
7613 Acked Flush 0 Done Sent Done false
7614 Acked Write 5 Done Sent Done false
7615 Acked Write 3 Done Sent Done false
7616 Acked Read 2 Done Sent Done false
7617 Acked Write 9 Done Sent Done false
7618 Acked Read 2 Done Sent Done false
7619 Acked Read 5 Done Sent Done false
7620 Acked Write 4 Done Sent Done false
7621 Acked Write 3 Done Sent Done false
7622 Acked Write 1 Done Sent Done false
7623 Acked Read 7 Done Sent Done false
7624 Acked Write 1 Done Sent Done false
7625 Acked Write 5 Done Sent Done false
7626 Acked Read 6 Done Sent Done false
7627 Acked Flush 0 Done Sent Done false
7628 Acked Write 9 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Read 9 Done Sent Done false
7631 Acked Write 10 Done Sent Done false
7632 Acked Read 2 Done Sent Done false
7633 Acked Write 9 Done Sent Done false
7634 Acked Read 3 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Write 3 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 5 Done Sent Done false
7639 Acked Read 9 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7635)) Some(JobId(7338)) Some(JobId(7635))
Downstairs last five completed: 7338 7337 7336 7335 7334
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:58.142211128Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:58.937099617Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.138410812Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44062)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.138639926Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.139937855Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.140486954Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.140519936Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(7338))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.140537722Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.140554058Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(7338))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.140570604Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(7338))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:06:59.14079345Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7640 7639 7638 7637 7636
Upstairs last five completed: 7640 7639 7638 7637 7636
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Feb 28 06:07:01.395 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5bfaf8e0-4f20-4368-8c7c-d18f2c185a5e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:01.395818189Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:01.395875905Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:01.39589853Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:07:01.396 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:01.396007914Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
154, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7d1082fa-4b70-46a3-b6e2-7472c6f38a54, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:07:05.397 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: ac63af56-12cb-4e13-baa9-5424b23f22a5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 2575 len 40960 data: 2 2 3 3 5 5 5 5 5 6
002/300 Read block 609 len 36864
003/300 Read block 2856 len 32768
004/300 Write block 1785 len 12288 data: 4 3 3
005/300 Read block 3235 len 16384
006/300 Flush
007/300 Write block 547 len 16384 data: 5 5 6 7
008/300 Write block 4184 len 40960 data: 4 4 6 7 5 5 7 7 6 6
009/300 Read block 4417 len 4096
010/300 Write block 3219 len 16384 data: 6 7 7 5
011/300 Read block 2952 len 40960
012/300 Read block 3475 len 40960
013/300 Read block 2098 len 36864
014/300 Read block 1329 len 12288
015/300 Write block 3739 len 24576 data: 4 4 5 8 10 10
016/300 Read block 1633 len 8192
017/300 Read block 2779 len 28672
018/300 Read block 663 len 36864
019/300 Write block 3169 len 32768 data: 5 5 3 3 2 2 2 2
020/300 Read block 4765 len 32768
021/300 Flush
022/300 Flush
023/300 Write block 3996 len 4096 data: 4
024/300 Flush
025/300 Write block 4865 len 4096 data: 6
026/300 Flush
027/300 Write block 4264 len 20480 data: 3 3 5 6 6
028/300 Read block 1709 len 32768
029/300 Read block 811 len 36864
030/300 Read block 2581 len 4096
031/300 Read block 2662 len 12288
032/300 Write block 561 len 20480 data: 6 6 6 6 6
033/300 Write block 2837 len 40960 data: 4 4 4 4 5 5 6 6 5 6
034/300 Read block 2475 len 8192
035/300 Flush
036/300 Read block 3931 len 8192
037/300 Write block 1051 len 8192 data: 5 6
038/300 Read block 4595 len 40960
039/300 Write block 4442 len 4096 data: 6
040/300 Read block 3608 len 16384
041/300 Read block 1903 len 20480
042/300 Write block 3414 len 40960 data: 3 2 3 5 5 5 4 4 4 4
043/300 Write block 4472 len 32768 data: 4 4 4 5 6 4 4 5
044/300 Write block 4481 len 4096 data: 6
045/300 Write block 203 len 16384 data: 5 5 4 4
046/300 Write block 1068 len 8192 data: 5 6
047/300 Write block 2081 len 12288 data: 3 3 5
048/300 Read block 155 len 4096
049/300 Read block 2750 len 20480
050/300 Read block 1695 len 40960
051/300 Write block 4879 len 4096 data: 8
052/300 Flush
053/300 Read block 2223 len 20480
054/300 Read block 2853 len 32768
055/300 Read block 3416 len 36864
056/300 Read block 2307 len 40960
057/300 Write block 4543 len 16384 data: 4 5 4 4
058/300 Read block 494 len 24576
059/300 Read block 2441 len 16384
060/300 Read block 4166 len 28672
061/300 Write block 4594 len 20480 data: 7 9 10 9 9
062/300 Write block 1728 len 4096 data: 5
063/300 Flush
064/300 Read block 741 len 28672
065/300 Write block 2657 len 4096 data: 6
066/300 Read block 4168 len 4096
067/300 Write block 2672 len 36864 data: 9 9 9 10 9 6 7 7 6
068/300 Flush
069/300 Read block 2701 len 4096
070/300 Write block 4964 len 24576 data: 5 6 5 6 8 7
071/300 Read block 4058 len 12288
072/300 Read block 2129 len 16384
073/300 Write block 3461 len 28672 data: 4 4 4 4 6 5 5
074/300 Read block 33 len 12288
075/300 Write block 2657 len 8192 data: 7 6
076/300 Write block 4405 len 16384 data: 9 7 8 8
077/300 Write block 4133 len 28672 data: 5 4 4 4 4 5 5
078/300 Read block 3487 len 36864
079/300 Read block 1386 len 8192
080/300 Read block 1359 len 16384
081/300 Write block 4596 len 20480 data: 11 10 10 6 6
082/300 Read block 2960 len 24576
083/300 Write block 1702 len 16384 data: 5 7 5 4
084/300 Write block 713 len 24576 data: 5 4 5 6 6 7
085/300 Write block 1441 len 8192 data: 7 7
086/300 Write block 3734 len 20480 data: 7 7 5 5 4
087/300 Read block 4116 len 28672
088/300 Read block 1847 len 8192
089/300 Write block 2468 len 8192 data: 5 3
090/300 Read block 2933 len 36864
091/300 Read block 4820 len 20480
092/300 Read block 1858 len 16384
093/300 Write block 2216 len 20480 data: 9 9 9 7 5
094/300 Read block 3178 len 32768
095/300 Write block 4267 len 20480 data: 7 7 6 6 8
096/300 Read block 2434 len 20480
097/300 Read block 2576 len 16384
098/300 Read block 686 len 4096
099/300 Read block 1217 len 24576
100/300 Read block 4066 len 28672
101/300 Read block 2082 len 4096
102/300 Read block 2092 len 12288
103/300 Read block 2146 len 28672
104/300 Read block 618 len 36864
105/300 Write block 1665 len 28672 data: 9 9 7 8 8 7 6
106/300 Read block 3567 len 40960
107/300 Write block 4599 len 12288 data: 7 7 6
108/300 Write block 3407 len 32768 data: 4 4 3 4 4 3 3 4
109/300 Read block 3903 len 32768
110/300 Read block 604 len 20480
111/300 Write block 3771 len 12288 data: 5 5 6
112/300 Read block 3591 len 8192
113/300 Flush
114/300 Read block 3997 len 20480
115/300 Read block 1132 len 4096
116/300 Write block 1385 len 36864 data: 5 5 5 3 3 3 4 4 3
117/300 Write block 4886 len 28672 data: 8 12 12 12 11 10 8
118/300 Write block 3410 len 16384 data: 5 5 4 4
119/300 Flush
120/300 Read block 2807 len 40960
121/300 Write block 2564 len 28672 data: 6 5 4 3 4 5 6
122/300 Write block 1130 len 32768 data: 5 6 5 4 5 5 6 6
123/300 Flush
124/300 Read block 859 len 32768
125/300 Write block 644 len 28672 data: 5 5 4 5 6 5 5
126/300 Read block 4161 len 20480
127/300 Read block 2738 len 32768
128/300 Write block 4829 len 8192 data: 8 10
129/300 Read block 2403 len 8192
130/300 Read block 2805 len 12288
131/300 Write block 2812 len 20480 data: 5 6 7 7 7
132/300 Read block 1877 len 36864
133/300 Read block 71 len 8192
134/300 Read block 2524 len 4096
135/300 Read block 1859 len 40960
136/300 Read block 2288 len 32768
137/300 Write block 2477 len 20480 data: 6 5 5 5 4
138/300 Write block 1092 len 24576 data: 2 2 2 2 2 3
139/300 Read block 443 len 12288
140/300 Read block 2916 len 12288
141/300 Read block 2948 len 4096
142/300 Write block 4262 len 20480 data: 4 4 4 4 6
143/300 Write block 1134 len 32768 data: 6 6 7 7 6 6 6 5
144/300 Read block 2620 len 4096
145/300 Write block 103 len 24576 data: 4 4 4 4 4 3
146/300 Write block 3767 len 8192 data: 5 6
147/300 Write block 1531 len 32768 data: 4 4 4 6 5 5 5 6
148/300 Write block 1357 len 20480 data: 5 6 7 7 6
149/300 Read block 262 len 16384
150/300 Read block 2586 len 36864
151/300 Read block 3294 len 16384
152/300 Read block 1890 len 20480
153/300 Read block 2752 len 28672
154/300 Flush
155/300 Write block 501 len 36864 data: 5 5 5 5 6 6 5 5 6
156/300 Read block 2127 len 40960
157/300 Read block 2001 len 12288
158/300 Read block 2528 len 4096
159/300 Read block 897 len 40960
160/300 Write block 1640 len 20480 data: 2 4 5 4 4
161/300 Read block 2441 len 4096
162/300 Flush
163/300 Read block 1855 len 36864
164/300 Write block 2889 len 24576 data: 5 3 3 4 3 4
165/300 Read block 4185 len 4096
166/300 Read block 1029 len 12288
167/300 Write block 3989 len 24576 data: 4 4 4 4 5 5
168/300 Write block 4673 len 32768 data: 3 3 3 3 4 5 5 5
169/300 Read block 343 len 28672
170/300 Write block 3641 len 4096 data: 2
171/300 Read block 1733 len 28672
172/300 Read block 213 len 40960
173/300 Read block 1025 len 28672
174/300 Read block 3849 len 4096
175/300 Write block 2637 len 28672 data: 5 6 6 6 6 6 6
176/300 Write block 1318 len 24576 data: 4 5 5 5 6 6
177/300 Write block 3984 len 32768 data: 3 4 7 6 4 5 5 5
178/300 Read block 4056 len 12288
179/300 Write block 1999 len 40960 data: 5 7 5 6 8 6 5 5 4 3
180/300 Write block 1355 len 40960 data: 6 5 6 7 8 8 7 4 4 5
181/300 Read block 2292 len 24576
182/300 Write block 1044 len 40960 data: 6 6 5 4 4 7 6 6 7 7
183/300 Write block 2189 len 4096 data: 8
184/300 Flush
185/300 Read block 755 len 28672
186/300 Read block 2081 len 28672
187/300 Write block 4801 len 40960 data: 3 3 2 3 4 4 5 5 4 5
188/300 Flush
189/300 Read block 1277 len 36864
190/300 Write block 1456 len 36864 data: 5 5 4 5 4 3 3 3 3
191/300 Write block 2901 len 36864 data: 3 3 3 3 4 4 4 5 5
192/300 Write block 535 len 40960 data: 3 2 3 3 3 3 2 2 3 3
193/300 Write block 534 len 16384 data: 5 4 3 4
194/300 Read block 4416 len 16384
195/300 Read block 4615 len 16384
196/300 Write block 94 len 12288 data: 10 11 7
197/300 Read block 679 len 40960
198/300 Write block 390 len 16384 data: 3 3 4 3
199/300 Read block 1568 len 4096
200/300 Write block 3702 len 4096 data: 3
201/300 Write block 4507 len 16384 data: 6 4 4 7
202/300 Write block 3390 len 24576 data: 3 3 2 2 2 3
203/300 Write block 4733 len 16384 data: 7 7 5 5
204/300 Write block 2087 len 32768 data: 8 8 9 9 9 5 6 6
205/300 Write block 2648 len 40960 data: 6 6 6 7 3 4 4 4 5 8
206/300 Read block 599 len 36864
207/300 Write block 1869 len 24576 data: 8 6 5 6 7 7
208/300 Read block 1218 len 28672
209/300 Read block 3612 len 24576
210/300 Flush
211/300 Read block 3805 len 28672
212/300 Write block 4327 len 8192 data: 5 5
213/300 Read block 1066 len 8192
214/300 Write block 620 len 32768 data: 4 4 4 5 5 6 4 4
215/300 Write block 3018 len 36864 data: 4 4 4 4 4 4 4 3 3
216/300 Write block 3995 len 8192 data: 4 5
217/300 Read block 1776 len 36864
218/300 Write block 2825 len 28672 data: 6 6 4 3 2 2 2
219/300 Write block 4832 len 20480 data: 9 6 6 8 9
220/300 Flush
221/300 Read block 1007 len 12288
222/300 Flush
223/300 Write block 3604 len 28672 data: 3 4 4 5 5 4 6
224/300 Write block 1864 len 36864 data: 4 4 4 4 5 9 7 6 7
225/300 Write block 671 len 12288 data: 7 8 7
226/300 Flush
227/300 Read block 779 len 20480
228/300 Write block 1034 len 40960 data: 3 3 3 5 5 6 8 6 6 6
229/300 Write block 229 len 12288 data: 3 5 5
230/300 Read block 2800 len 4096
231/300 Read block 43 len 36864
232/300 Read block 2720 len 40960
233/300 Read block 3190 len 4096
234/300 Write block 1169 len 4096 data: 7
235/300 Write block 3869 len 28672 data: 8 7 4 4 6 6 6
236/300 Read block 3722 len 32768
237/300 Write block 4507 len 36864 data: 7 5 5 8 6 6 7 8 7
238/300 Read block 582 len 20480
239/300 Write block 1972 len 16384 data: 7 7 7 7
240/300 Write block 2129 len 16384 data: 4 5 6 7
241/300 Write block 4675 len 4096 data: 4
242/300 Read block 1290 len 36864
243/300 Flush
244/300 Write block 488 len 8192 data: 5 4
245/300 Write block 2427 len 40960 data: 5 4 3 4 5 5 4 4 5 5
246/300 Read block 4548 len 28672
247/300 Write block 748 len 16384 data: 7 8 8 5
248/300 Flush
249/300 Read block 2619 len 40960
250/300 Read block 583 len 40960
251/300 Write block 706 len 28672 data: 6 6 5 7 7 6 5
252/300 Write block 1365 len 40960 data: 6 5 5 6 4 4 5 5 5 5
253/300 Write block 1264 len 24576 data: 8 6 5 4 4 3
254/300 Write block 313 len 12288 data: 2 2 3
255/300 Write block 1396 len 4096 data: 4
256/300 Read block 1043 len 20480
257/300 Read block 383 len 28672
258/300 Flush
259/300 Read block 3730 len 40960
260/300 Write block 1700 len 4096 data: 7
261/300 Write block 2890 len 32768 data: 4 4 5 4 5 5 5 6
262/300 Read block 2059 len 12288
263/300 Write block 4407 len 16384 data: 9 9 8 8
264/300 Read block 1356 len 32768
265/300 Write block 3094 len 16384 data: 6 4 5 5
266/300 Read block 883 len 28672
267/300 Read block 503 len 20480
268/300 Write block 3749 len 4096 data: 5
269/300 Write block 2419 len 4096 data: 6
270/300 Write block 413 len 40960 data: 7 9 7 7 8 10 11 9 10 11
271/300 Write block 351 len 8192 data: 4 5
272/300 Write block 1767 len 4096 data: 3
273/300 Write block 3148 len 36864 data: 4 3 3 4 4 4 5 5 8
274/300 Read block 4510 len 24576
275/300 Read block 1134 len 20480
276/300 Write block 3526 len 36864 data: 7 8 9 8 6 6 6 6 5
277/300 Write block 1758 len 8192 data: 6 6
278/300 Read block 2856 len 8192
279/300 Write block 1925 len 12288 data: 2 3 4
280/300 Write block 3103 len 20480 data: 7 6 7 6 8
281/300 Write block 3864 len 40960 data: 7 7 7 8 7 9 8 5 5 7
282/300 Read block 1498 len 16384
283/300 Write block 3515 len 12288 data: 4 4 3
284/300 Read block 1253 len 20480
285/300 Read block 649 len 4096
286/300 Write block 4871 len 16384 data: 4 4 4 4
287/300 Write block 2267 len 32768 data: 6 6 5 5 4 4 6 7
288/300 Write block 3443 len 20480 data: 4 4 3 3 3
289/300 Read block 504 len 16384
290/300 Read block 4405 len 24576
291/300 Write block 2958 len 24576 data: 5 4 4 6 7 6
292/300 Read block 150 len 20480
293/300 Write block 4298 len 8192 data: 5 3
294/300 Write block 4235 len 12288 data: 4 4 4
295/300 Read block 4931 len 32768
296/300 Write block 310 len 24576 data: 2 3 2 3 3 4
297/300 Flush
298/300 Write block 3032 len 40960 data: 5 5 5 5 5 5 4 4 3 3
299/300 Read block 497 len 28672
300/300 Read block 1829 len 36864
301/300 Read block 166 len 12288
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:07:05.635 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/0, method: POST, req_id: ab8db44f-a524-4572-af33-6cdbe37f5f46, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7641 Acked Write 10 Sent Done Done false
7642 Acked Read 9 Sent Done Done false
7643 Acked Read 8 Sent Done Done false
7644 Acked Write 3 Sent Done Done false
7645 Acked Read 4 Sent Done Done false
7646 Acked Flush 0 Sent Done Done false
7647 Acked Write 4 Sent Done Done false
7648 Acked Write 10 Sent Done Done false
7649 Acked Read 1 Sent Done Done false
7650 Acked Write 4 Sent Done Done false
7651 Acked Read 10 Sent Done Done false
7652 Acked Read 10 Sent Done Done false
7653 Acked Read 9 Sent Done Done false
7654 Acked Read 3 Sent Done Done false
7655 Acked Write 6 Sent Done Done false
7656 Acked Read 2 Sent Done Done false
7657 Acked Read 7 Sent Done Done false
7658 Acked Read 9 Sent Done Done false
7659 Acked Write 8 Sent Done Done false
7660 Acked Read 8 Sent Done Done false
7661 Acked Flush 0 Sent Done Done false
7662 Acked Flush 0 Sent Done Done false
7663 Acked Write 1 Sent Done Done false
7664 Acked Flush 0 Sent Done Done false
7665 Acked Write 1 Sent Done Done false
7666 Acked Flush 0 Sent Done Done false
7667 Acked Write 5 Sent Done Done false
7668 Acked Read 8 Sent Done Done false
7669 Acked Read 9 Sent Done Done false
7670 Acked Read 1 Sent Done Done false
7671 Acked Read 3 Sent Done Done false
7672 Acked Write 5 Sent Done Done false
7673 Acked Write 10 Sent Done Done false
7674 Acked Read 2 Sent Done Done false
7675 Acked Flush 0 Sent Done Done false
7676 Acked Read 2 Sent Done Done false
7677 Acked Write 2 Sent Done Done false
7678 Acked Read 10 Sent Done Done false
7679 Acked Write 1 Sent Done Done false
7680 Acked Read 4 Sent Done Done false
7681 Acked Read 5 Sent Done Done false
7682 Acked Write 10 Sent Done Done false
7683 Acked Write 8 Sent Done Done false
7684 Acked Write 1 Sent Done Done false
7685 Acked Write 4 Sent Done Done false
7686 Acked Write 2 Sent Done Done false
7687 Acked Write 3 Sent Done Done false
7688 Acked Read 1 Sent Done Done false
7689 Acked Read 5 Sent Done Done false
7690 Acked Read 10 Sent Done Done false
7691 Acked Write 1 Sent Done Done false
7692 Acked Flush 0 Sent Done Done false
7693 Acked Read 5 Sent Done Done false
7694 Acked Read 8 Sent Done Done false
7695 Acked Read 9 Sent Done Done false
7696 Acked Read 10 Sent Done Done false
7697 Acked Write 4 Sent Done Done false
7698 Acked Read 6 Sent Done Done false
7699 Acked Read 4 Sent Done Done false
7700 Acked Read 7 Sent Done Done false
7701 Acked Write 5 Sent Done Done false
7702 Acked Write 1 Sent Done Done false
7703 Acked Flush 0 Sent Done Done false
7704 Acked Read 7 Sent Done Done false
7705 Acked Write 1 Sent Done Done false
7706 Acked Read 1 Sent Done Done false
7707 Acked Write 9 Sent Done Done false
7708 Acked Flush 0 Sent Done Done false
7709 Acked Read 1 Sent Done Done false
7710 Acked Write 6 Sent Done Done false
7711 Acked Read 3 Sent Done Done false
7712 Acked Read 4 Sent Done Done false
7713 Acked Write 7 Sent Done Done false
7714 Acked Read 3 Sent Done Done false
7715 Acked Write 2 Sent Done Done false
7716 Acked Write 4 Sent Done Done false
7717 Acked Write 7 Sent Done Done false
7718 Acked Read 9 Sent Done Done false
7719 Acked Read 2 Sent Done Done false
7720 Acked Read 4 Sent Done Done false
7721 Acked Write 5 Sent Done Done false
7722 Acked Read 6 Sent Done Done false
7723 Acked Write 4 Sent Done Done false
7724 Acked Write 6 Sent Done Done false
7725 Acked Write 2 Sent Done Done false
7726 Acked Write 5 Sent Done Done false
7727 Acked Read 7 Sent Done Done false
7728 Acked Read 2 Sent Done Done false
7729 Acked Write 2 Sent Done Done false
7730 Acked Read 9 Sent Done Done false
7731 Acked Read 5 Sent Done Done false
7732 Acked Read 4 Sent Done Done false
7733 Acked Write 5 Sent Done Done false
7734 Acked Read 8 Sent Done Done false
7735 Acked Write 5 Sent Done Done false
7736 Acked Read 5 Sent Done Done false
7737 Acked Read 4 Sent Done Done false
7738 Acked Read 1 Sent Done Done false
7739 Acked Read 6 Sent Done Done false
7740 Acked Read 7 Sent Done Done false
7741 Acked Read 1 Sent Done Done false
7742 Acked Read 3 Sent Done Done false
7743 Acked Read 7 Sent Done Done false
7744 Acked Read 9 Sent Done Done false
7745 Acked Write 7 Sent Done Done false
7746 Acked Read 10 Sent Done Done false
7747 Acked Write 3 Sent Done Done false
7748 Acked Write 8 Sent Done Done false
7749 Acked Read 8 Sent Done Done false
7750 Acked Read 5 Sent Done Done false
7751 Acked Write 3 Sent Done Done false
7752 Acked Read 2 Sent Done Done false
7753 Acked Flush 0 Sent Done Done false
7754 Acked Read 5 Sent Done Done false
7755 Acked Read 1 Sent Done Done false
7756 Acked Write 9 Sent Done Done false
7757 Acked Write 7 Sent Done Done false
7758 Acked Write 4 Sent Done Done false
7759 Acked Flush 0 Sent Done Done false
7760 Acked Read 10 Sent Done Done false
7761 Acked Write 7 Sent Done Done false
7762 Acked Write 8 Sent Done Done false
7763 Acked Flush 0 Sent Done Done false
7764 Acked Read 8 Sent Done Done false
7765 Acked Write 7 Sent Done Done false
7766 Acked Read 5 Sent Done Done false
7767 Acked Read 8 Sent Done Done false
7768 Acked Write 2 Sent Done Done false
7769 Acked Read 2 Sent Done Done false
7770 Acked Read 3 Sent Done Done false
7771 Acked Write 5 Sent Done Done false
7772 Acked Read 9 Sent Done Done false
7773 Acked Read 2 Sent Done Done false
7774 Acked Read 1 Sent Done Done false
7775 Acked Read 10 Sent Done Done false
7776 Acked Read 8 Sent Done Done false
7777 Acked Write 5 Sent Done Done false
7778 Acked Write 6 Sent Done Done false
7779 Acked Read 3 Sent Done Done false
7780 Acked Read 3 Sent Done Done false
7781 Acked Read 1 Sent Done Done false
7782 Acked Write 5 Sent Done Done false
7783 Acked Write 8 Sent Done Done false
7784 Acked Read 1 Sent Done Done false
7785 Acked Write 6 Sent Done Done false
7786 Acked Write 2 Sent Done Done false
7787 Acked Write 8 Sent Done Done false
7788 Acked Write 5 Sent Done Done false
7789 Acked Read 4 Sent Done Done false
7790 Acked Read 9 Sent Done Done false
7791 Acked Read 4 Sent Done Done false
7792 Acked Read 5 Sent Done Done false
7793 Acked Read 7 Sent Done Done false
7794 Acked Flush 0 Sent Done Done false
7795 Acked Write 9 Sent Done Done false
7796 Acked Read 10 Sent Done Done false
7797 Acked Read 3 Sent Done Done false
7798 Acked Read 1 Sent Done Done false
7799 Acked Read 10 Sent Done Done false
7800 Acked Write 5 Sent Done Done false
7801 Acked Read 1 Sent Done Done false
7802 Acked Flush 0 Sent Done Done false
7803 Acked Read 9 Sent Done Done false
7804 Acked Write 6 Sent Done Done false
7805 Acked Read 1 Sent Done Done false
7806 Acked Read 3 Sent Done Done false
7807 Acked Write 6 Sent Done Done false
7808 Acked Write 8 Sent Done Done false
7809 Acked Read 7 Sent Done Done false
7810 Acked Write 1 Sent Done Done false
7811 Acked Read 7 Sent Done Done false
7812 Acked Read 10 Sent Done Done false
7813 Acked Read 7 Sent Done Done false
7814 Acked Read 1 Sent Done Done false
7815 Acked Write 7 Sent Done Done false
7816 Acked Write 6 Sent Done Done false
7817 Acked Write 8 Sent Done Done false
7818 Acked Read 3 Sent Done Done false
7819 Acked Write 10 Sent Done Done false
7820 Acked Write 10 Sent Done Done false
7821 Acked Read 6 Sent Done Done false
7822 Acked Write 10 Sent Done Done false
7823 Acked Write 1 Sent Done Done false
7824 Acked Flush 0 Sent Done Done false
7825 Acked Read 7 Sent Done Done false
7826 Acked Read 7 Sent Done Done false
7827 Acked Write 10 Sent Done Done false
7828 Acked Flush 0 Sent Done Done false
7829 Acked Read 9 Sent Done Done false
7830 Acked Write 9 Sent Done Done false
7831 Acked Write 9 Sent Done Done false
7832 Acked Write 10 Sent Done Done false
7833 Acked Write 4 Sent Done Done false
7834 Acked Read 4 Sent Done Done false
7835 Acked Read 4 Sent Done Done false
7836 Acked Write 3 Sent Done Done false
7837 Acked Read 10 Sent Done Done false
7838 Acked Write 4 Sent Done Done false
7839 Acked Read 1 Sent Done Done false
7840 Acked Write 1 Sent Done Done false
7841 Acked Write 4 Sent Done Done false
7842 Acked Write 6 Sent Done Done false
7843 Acked Write 4 Sent Done Done false
7844 Acked Write 8 Sent Done Done false
7845 Acked Write 10 Sent Done Done false
7846 Acked Read 9 Sent Done Done false
7847 Acked Write 6 Sent Done Done false
7848 Acked Read 7 Sent Done Done false
7849 Acked Read 6 Sent Done Done false
7850 Acked Flush 0 Sent Done Done false
7851 Acked Read 7 Sent Done Done false
7852 Acked Write 2 Sent Done Done false
7853 Acked Read 2 Sent Done Done false
7854 Acked Write 8 Sent Done Done false
7855 Acked Write 9 Sent Done Done false
7856 Acked Write 2 Sent Done Done false
7857 Acked Read 9 Sent Done Done false
7858 Acked Write 7 Sent Done Done false
7859 Acked Write 5 Sent Done Done false
7860 Acked Flush 0 Sent Done Done false
7861 Acked Read 3 Sent Done Done false
7862 Acked Flush 0 Sent Done Done false
7863 Acked Write 7 Sent Done Done false
7864 Acked Write 9 Sent Done Done false
7865 Acked Write 3 Sent Done Done false
7866 Acked Flush 0 Sent Done Done false
7867 Acked Read 5 Sent Done Done false
7868 Acked Write 10 Sent Done Done false
7869 Acked Write 3 Sent Done Done false
7870 Acked Read 1 Sent Done Done false
7871 Acked Read 9 Sent Done Done false
7872 Acked Read 10 Sent Done Done false
7873 Acked Read 1 Sent Done Done false
7874 Acked Write 1 Sent Done Done false
7875 Acked Write 7 Sent Done Done false
7876 Acked Read 8 Sent Done Done false
7877 Acked Write 9 Sent Done Done false
7878 Acked Read 5 Sent Done Done false
7879 Acked Write 4 Sent Done Done false
7880 Acked Write 4 Sent Done Done false
7881 Acked Write 1 Sent Done Done false
7882 Acked Read 9 Sent Done Done false
7883 Acked Flush 0 Sent Done Done false
7884 Acked Write 2 Sent Done Done false
7885 Acked Write 10 Sent Done Done false
7886 Acked Read 7 Sent Done Done false
7887 Acked Write 4 Sent Done Done false
7888 Acked Flush 0 Sent Done Done false
7889 Acked Read 10 Sent Done Done false
7890 Acked Read 10 Sent Done Done false
7891 Acked Write 7 Sent Done Done false
7892 Acked Write 10 Sent Done Done false
7893 Acked Write 6 Sent Done Done false
7894 Acked Write 3 Sent Done Done false
7895 Acked Write 1 Sent Done Done false
7896 Acked Read 5 Sent Done Done false
7897 Acked Read 7 Sent Done Done false
7898 Acked Flush 0 Sent Done Done false
7899 Acked Read 10 Sent Done Done false
7900 Acked Write 1 Sent Done Done false
7901 Acked Write 8 Sent Done Done false
7902 Acked Read 3 Sent Done Done false
7903 Acked Write 4 Sent Done Done false
7904 Acked Read 8 Sent Done Done false
7905 Acked Write 4 Sent Done Done false
7906 Acked Read 7 Sent Done Done false
7907 Acked Read 5 Sent Done Done false
7908 Acked Write 1 Sent Done Done false
7909 Acked Write 1 Sent Done Done false
7910 Acked Write 10 Sent Done Done false
7911 Acked Write 2 Sent Done Done false
7912 Acked Write 1 Sent Done Done false
7913 Acked Write 9 Sent Done Done false
7914 Acked Read 6 Sent Done Done false
7915 Acked Read 5 Sent Done Done false
7916 Acked Write 9 Sent Done Done false
7917 Acked Write 2 Sent Done Done false
7918 Acked Read 2 Sent Done Done false
7919 Acked Write 3 Sent Done Done false
7920 Acked Write 5 Sent Done Done false
7921 Acked Write 10 Sent Done Done false
7922 Acked Read 4 Sent Done Done false
7923 Acked Write 3 Sent Done Done false
7924 Acked Read 5 Sent Done Done false
7925 Acked Read 1 Sent Done Done false
7926 Acked Write 4 Sent Done Done false
7927 Acked Write 8 Sent Done Done false
7928 Acked Write 5 Sent Done Done false
7929 Acked Read 4 Sent Done Done false
7930 Acked Read 6 Sent Done Done false
7931 Acked Write 6 Sent Done Done false
7932 Acked Read 5 Sent Done Done false
7933 Acked Write 2 Sent Done Done false
7934 Acked Write 3 Sent Done Done false
7935 Acked Read 8 Sent Done Done false
7936 Acked Write 6 Sent Done Done false
7937 Acked Flush 0 Sent Done Done false
7938 Acked Write 10 Sent Done Done false
7939 Acked Read 7 Sent Done Done false
7940 Acked Read 9 Sent Done Done false
7941 Acked Read 3 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
Downstairs last five completed: 7640 7639 7638 7637 7636
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:06.13802009Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:06.951196787Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:08.143049872Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:08.955003581Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7641 Acked Write 10 Sent Done Done false
7642 Acked Read 9 Sent Done Done false
7643 Acked Read 8 Sent Done Done false
7644 Acked Write 3 Sent Done Done false
7645 Acked Read 4 Sent Done Done false
7646 Acked Flush 0 Sent Done Done false
7647 Acked Write 4 Sent Done Done false
7648 Acked Write 10 Sent Done Done false
7649 Acked Read 1 Sent Done Done false
7650 Acked Write 4 Sent Done Done false
7651 Acked Read 10 Sent Done Done false
7652 Acked Read 10 Sent Done Done false
7653 Acked Read 9 Sent Done Done false
7654 Acked Read 3 Sent Done Done false
7655 Acked Write 6 Sent Done Done false
7656 Acked Read 2 Sent Done Done false
7657 Acked Read 7 Sent Done Done false
7658 Acked Read 9 Sent Done Done false
7659 Acked Write 8 Sent Done Done false
7660 Acked Read 8 Sent Done Done false
7661 Acked Flush 0 Sent Done Done false
7662 Acked Flush 0 Sent Done Done false
7663 Acked Write 1 Sent Done Done false
7664 Acked Flush 0 Sent Done Done false
7665 Acked Write 1 Sent Done Done false
7666 Acked Flush 0 Sent Done Done false
7667 Acked Write 5 Sent Done Done false
7668 Acked Read 8 Sent Done Done false
7669 Acked Read 9 Sent Done Done false
7670 Acked Read 1 Sent Done Done false
7671 Acked Read 3 Sent Done Done false
7672 Acked Write 5 Sent Done Done false
7673 Acked Write 10 Sent Done Done false
7674 Acked Read 2 Sent Done Done false
7675 Acked Flush 0 Sent Done Done false
7676 Acked Read 2 Sent Done Done false
7677 Acked Write 2 Sent Done Done false
7678 Acked Read 10 Sent Done Done false
7679 Acked Write 1 Sent Done Done false
7680 Acked Read 4 Sent Done Done false
7681 Acked Read 5 Sent Done Done false
7682 Acked Write 10 Sent Done Done false
7683 Acked Write 8 Sent Done Done false
7684 Acked Write 1 Sent Done Done false
7685 Acked Write 4 Sent Done Done false
7686 Acked Write 2 Sent Done Done false
7687 Acked Write 3 Sent Done Done false
7688 Acked Read 1 Sent Done Done false
7689 Acked Read 5 Sent Done Done false
7690 Acked Read 10 Sent Done Done false
7691 Acked Write 1 Sent Done Done false
7692 Acked Flush 0 Sent Done Done false
7693 Acked Read 5 Sent Done Done false
7694 Acked Read 8 Sent Done Done false
7695 Acked Read 9 Sent Done Done false
7696 Acked Read 10 Sent Done Done false
7697 Acked Write 4 Sent Done Done false
7698 Acked Read 6 Sent Done Done false
7699 Acked Read 4 Sent Done Done false
7700 Acked Read 7 Sent Done Done false
7701 Acked Write 5 Sent Done Done false
7702 Acked Write 1 Sent Done Done false
7703 Acked Flush 0 Sent Done Done false
7704 Acked Read 7 Sent Done Done false
7705 Acked Write 1 Sent Done Done false
7706 Acked Read 1 Sent Done Done false
7707 Acked Write 9 Sent Done Done false
7708 Acked Flush 0 Sent Done Done false
7709 Acked Read 1 Sent Done Done false
7710 Acked Write 6 Sent Done Done false
7711 Acked Read 3 Sent Done Done false
7712 Acked Read 4 Sent Done Done false
7713 Acked Write 7 Sent Done Done false
7714 Acked Read 3 Sent Done Done false
7715 Acked Write 2 Sent Done Done false
7716 Acked Write 4 Sent Done Done false
7717 Acked Write 7 Sent Done Done false
7718 Acked Read 9 Sent Done Done false
7719 Acked Read 2 Sent Done Done false
7720 Acked Read 4 Sent Done Done false
7721 Acked Write 5 Sent Done Done false
7722 Acked Read 6 Sent Done Done false
7723 Acked Write 4 Sent Done Done false
7724 Acked Write 6 Sent Done Done false
7725 Acked Write 2 Sent Done Done false
7726 Acked Write 5 Sent Done Done false
7727 Acked Read 7 Sent Done Done false
7728 Acked Read 2 Sent Done Done false
7729 Acked Write 2 Sent Done Done false
7730 Acked Read 9 Sent Done Done false
7731 Acked Read 5 Sent Done Done false
7732 Acked Read 4 Sent Done Done false
7733 Acked Write 5 Sent Done Done false
7734 Acked Read 8 Sent Done Done false
7735 Acked Write 5 Sent Done Done false
7736 Acked Read 5 Sent Done Done false
7737 Acked Read 4 Sent Done Done false
7738 Acked Read 1 Sent Done Done false
7739 Acked Read 6 Sent Done Done false
7740 Acked Read 7 Sent Done Done false
7741 Acked Read 1 Sent Done Done false
7742 Acked Read 3 Sent Done Done false
7743 Acked Read 7 Sent Done Done false
7744 Acked Read 9 Sent Done Done false
7745 Acked Write 7 Sent Done Done false
7746 Acked Read 10 Sent Done Done false
7747 Acked Write 3 Sent Done Done false
7748 Acked Write 8 Sent Done Done false
7749 Acked Read 8 Sent Done Done false
7750 Acked Read 5 Sent Done Done false
7751 Acked Write 3 Sent Done Done false
7752 Acked Read 2 Sent Done Done false
7753 Acked Flush 0 Sent Done Done false
7754 Acked Read 5 Sent Done Done false
7755 Acked Read 1 Sent Done Done false
7756 Acked Write 9 Sent Done Done false
7757 Acked Write 7 Sent Done Done false
7758 Acked Write 4 Sent Done Done false
7759 Acked Flush 0 Sent Done Done false
7760 Acked Read 10 Sent Done Done false
7761 Acked Write 7 Sent Done Done false
7762 Acked Write 8 Sent Done Done false
7763 Acked Flush 0 Sent Done Done false
7764 Acked Read 8 Sent Done Done false
7765 Acked Write 7 Sent Done Done false
7766 Acked Read 5 Sent Done Done false
7767 Acked Read 8 Sent Done Done false
7768 Acked Write 2 Sent Done Done false
7769 Acked Read 2 Sent Done Done false
7770 Acked Read 3 Sent Done Done false
7771 Acked Write 5 Sent Done Done false
7772 Acked Read 9 Sent Done Done false
7773 Acked Read 2 Sent Done Done false
7774 Acked Read 1 Sent Done Done false
7775 Acked Read 10 Sent Done Done false
7776 Acked Read 8 Sent Done Done false
7777 Acked Write 5 Sent Done Done false
7778 Acked Write 6 Sent Done Done false
7779 Acked Read 3 Sent Done Done false
7780 Acked Read 3 Sent Done Done false
7781 Acked Read 1 Sent Done Done false
7782 Acked Write 5 Sent Done Done false
7783 Acked Write 8 Sent Done Done false
7784 Acked Read 1 Sent Done Done false
7785 Acked Write 6 Sent Done Done false
7786 Acked Write 2 Sent Done Done false
7787 Acked Write 8 Sent Done Done false
7788 Acked Write 5 Sent Done Done false
7789 Acked Read 4 Sent Done Done false
7790 Acked Read 9 Sent Done Done false
7791 Acked Read 4 Sent Done Done false
7792 Acked Read 5 Sent Done Done false
7793 Acked Read 7 Sent Done Done false
7794 Acked Flush 0 Sent Done Done false
7795 Acked Write 9 Sent Done Done false
7796 Acked Read 10 Sent Done Done false
7797 Acked Read 3 Sent Done Done false
7798 Acked Read 1 Sent Done Done false
7799 Acked Read 10 Sent Done Done false
7800 Acked Write 5 Sent Done Done false
7801 Acked Read 1 Sent Done Done false
7802 Acked Flush 0 Sent Done Done false
7803 Acked Read 9 Sent Done Done false
7804 Acked Write 6 Sent Done Done false
7805 Acked Read 1 Sent Done Done false
7806 Acked Read 3 Sent Done Done false
7807 Acked Write 6 Sent Done Done false
7808 Acked Write 8 Sent Done Done false
7809 Acked Read 7 Sent Done Done false
7810 Acked Write 1 Sent Done Done false
7811 Acked Read 7 Sent Done Done false
7812 Acked Read 10 Sent Done Done false
7813 Acked Read 7 Sent Done Done false
7814 Acked Read 1 Sent Done Done false
7815 Acked Write 7 Sent Done Done false
7816 Acked Write 6 Sent Done Done false
7817 Acked Write 8 Sent Done Done false
7818 Acked Read 3 Sent Done Done false
7819 Acked Write 10 Sent Done Done false
7820 Acked Write 10 Sent Done Done false
7821 Acked Read 6 Sent Done Done false
7822 Acked Write 10 Sent Done Done false
7823 Acked Write 1 Sent Done Done false
7824 Acked Flush 0 Sent Done Done false
7825 Acked Read 7 Sent Done Done false
7826 Acked Read 7 Sent Done Done false
7827 Acked Write 10 Sent Done Done false
7828 Acked Flush 0 Sent Done Done false
7829 Acked Read 9 Sent Done Done false
7830 Acked Write 9 Sent Done Done false
7831 Acked Write 9 Sent Done Done false
7832 Acked Write 10 Sent Done Done false
7833 Acked Write 4 Sent Done Done false
7834 Acked Read 4 Sent Done Done false
7835 Acked Read 4 Sent Done Done false
7836 Acked Write 3 Sent Done Done false
7837 Acked Read 10 Sent Done Done false
7838 Acked Write 4 Sent Done Done false
7839 Acked Read 1 Sent Done Done false
7840 Acked Write 1 Sent Done Done false
7841 Acked Write 4 Sent Done Done false
7842 Acked Write 6 Sent Done Done false
7843 Acked Write 4 Sent Done Done false
7844 Acked Write 8 Sent Done Done false
7845 Acked Write 10 Sent Done Done false
7846 Acked Read 9 Sent Done Done false
7847 Acked Write 6 Sent Done Done false
7848 Acked Read 7 Sent Done Done false
7849 Acked Read 6 Sent Done Done false
7850 Acked Flush 0 Sent Done Done false
7851 Acked Read 7 Sent Done Done false
7852 Acked Write 2 Sent Done Done false
7853 Acked Read 2 Sent Done Done false
7854 Acked Write 8 Sent Done Done false
7855 Acked Write 9 Sent Done Done false
7856 Acked Write 2 Sent Done Done false
7857 Acked Read 9 Sent Done Done false
7858 Acked Write 7 Sent Done Done false
7859 Acked Write 5 Sent Done Done false
7860 Acked Flush 0 Sent Done Done false
7861 Acked Read 3 Sent Done Done false
7862 Acked Flush 0 Sent Done Done false
7863 Acked Write 7 Sent Done Done false
7864 Acked Write 9 Sent Done Done false
7865 Acked Write 3 Sent Done Done false
7866 Acked Flush 0 Sent Done Done false
7867 Acked Read 5 Sent Done Done false
7868 Acked Write 10 Sent Done Done false
7869 Acked Write 3 Sent Done Done false
7870 Acked Read 1 Sent Done Done false
7871 Acked Read 9 Sent Done Done false
7872 Acked Read 10 Sent Done Done false
7873 Acked Read 1 Sent Done Done false
7874 Acked Write 1 Sent Done Done false
7875 Acked Write 7 Sent Done Done false
7876 Acked Read 8 Sent Done Done false
7877 Acked Write 9 Sent Done Done false
7878 Acked Read 5 Sent Done Done false
7879 Acked Write 4 Sent Done Done false
7880 Acked Write 4 Sent Done Done false
7881 Acked Write 1 Sent Done Done false
7882 Acked Read 9 Sent Done Done false
7883 Acked Flush 0 Sent Done Done false
7884 Acked Write 2 Sent Done Done false
7885 Acked Write 10 Sent Done Done false
7886 Acked Read 7 Sent Done Done false
7887 Acked Write 4 Sent Done Done false
7888 Acked Flush 0 Sent Done Done false
7889 Acked Read 10 Sent Done Done false
7890 Acked Read 10 Sent Done Done false
7891 Acked Write 7 Sent Done Done false
7892 Acked Write 10 Sent Done Done false
7893 Acked Write 6 Sent Done Done false
7894 Acked Write 3 Sent Done Done false
7895 Acked Write 1 Sent Done Done false
7896 Acked Read 5 Sent Done Done false
7897 Acked Read 7 Sent Done Done false
7898 Acked Flush 0 Sent Done Done false
7899 Acked Read 10 Sent Done Done false
7900 Acked Write 1 Sent Done Done false
7901 Acked Write 8 Sent Done Done false
7902 Acked Read 3 Sent Done Done false
7903 Acked Write 4 Sent Done Done false
7904 Acked Read 8 Sent Done Done false
7905 Acked Write 4 Sent Done Done false
7906 Acked Read 7 Sent Done Done false
7907 Acked Read 5 Sent Done Done false
7908 Acked Write 1 Sent Done Done false
7909 Acked Write 1 Sent Done Done false
7910 Acked Write 10 Sent Done Done false
7911 Acked Write 2 Sent Done Done false
7912 Acked Write 1 Sent Done Done false
7913 Acked Write 9 Sent Done Done false
7914 Acked Read 6 Sent Done Done false
7915 Acked Read 5 Sent Done Done false
7916 Acked Write 9 Sent Done Done false
7917 Acked Write 2 Sent Done Done false
7918 Acked Read 2 Sent Done Done false
7919 Acked Write 3 Sent Done Done false
7920 Acked Write 5 Sent Done Done false
7921 Acked Write 10 Sent Done Done false
7922 Acked Read 4 Sent Done Done false
7923 Acked Write 3 Sent Done Done false
7924 Acked Read 5 Sent Done Done false
7925 Acked Read 1 Sent Done Done false
7926 Acked Write 4 Sent Done Done false
7927 Acked Write 8 Sent Done Done false
7928 Acked Write 5 Sent Done Done false
7929 Acked Read 4 Sent Done Done false
7930 Acked Read 6 Sent Done Done false
7931 Acked Write 6 Sent Done Done false
7932 Acked Read 5 Sent Done Done false
7933 Acked Write 2 Sent Done Done false
7934 Acked Write 3 Sent Done Done false
7935 Acked Read 8 Sent Done Done false
7936 Acked Write 6 Sent Done Done false
7937 Acked Flush 0 Sent Done Done false
7938 Acked Write 10 Sent Done Done false
7939 Acked Read 7 Sent Done Done false
7940 Acked Read 9 Sent Done Done false
7941 Acked Read 3 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
Downstairs last five completed: 7640 7639 7638 7637 7636
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:10.139919098Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:10.95812186Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:11.39741827Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:11.397495781Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:11.397536511Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:11.397592118Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:11.397779183Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:11.960064507Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:13.144004488Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7641 Acked Write 10 Sent Done Done false
7642 Acked Read 9 Sent Done Done false
7643 Acked Read 8 Sent Done Done false
7644 Acked Write 3 Sent Done Done false
7645 Acked Read 4 Sent Done Done false
7646 Acked Flush 0 Sent Done Done false
7647 Acked Write 4 Sent Done Done false
7648 Acked Write 10 Sent Done Done false
7649 Acked Read 1 Sent Done Done false
7650 Acked Write 4 Sent Done Done false
7651 Acked Read 10 Sent Done Done false
7652 Acked Read 10 Sent Done Done false
7653 Acked Read 9 Sent Done Done false
7654 Acked Read 3 Sent Done Done false
7655 Acked Write 6 Sent Done Done false
7656 Acked Read 2 Sent Done Done false
7657 Acked Read 7 Sent Done Done false
7658 Acked Read 9 Sent Done Done false
7659 Acked Write 8 Sent Done Done false
7660 Acked Read 8 Sent Done Done false
7661 Acked Flush 0 Sent Done Done false
7662 Acked Flush 0 Sent Done Done false
7663 Acked Write 1 Sent Done Done false
7664 Acked Flush 0 Sent Done Done false
7665 Acked Write 1 Sent Done Done false
7666 Acked Flush 0 Sent Done Done false
7667 Acked Write 5 Sent Done Done false
7668 Acked Read 8 Sent Done Done false
7669 Acked Read 9 Sent Done Done false
7670 Acked Read 1 Sent Done Done false
7671 Acked Read 3 Sent Done Done false
7672 Acked Write 5 Sent Done Done false
7673 Acked Write 10 Sent Done Done false
7674 Acked Read 2 Sent Done Done false
7675 Acked Flush 0 Sent Done Done false
7676 Acked Read 2 Sent Done Done false
7677 Acked Write 2 Sent Done Done false
7678 Acked Read 10 Sent Done Done false
7679 Acked Write 1 Sent Done Done false
7680 Acked Read 4 Sent Done Done false
7681 Acked Read 5 Sent Done Done false
7682 Acked Write 10 Sent Done Done false
7683 Acked Write 8 Sent Done Done false
7684 Acked Write 1 Sent Done Done false
7685 Acked Write 4 Sent Done Done false
7686 Acked Write 2 Sent Done Done false
7687 Acked Write 3 Sent Done Done false
7688 Acked Read 1 Sent Done Done false
7689 Acked Read 5 Sent Done Done false
7690 Acked Read 10 Sent Done Done false
7691 Acked Write 1 Sent Done Done false
7692 Acked Flush 0 Sent Done Done false
7693 Acked Read 5 Sent Done Done false
7694 Acked Read 8 Sent Done Done false
7695 Acked Read 9 Sent Done Done false
7696 Acked Read 10 Sent Done Done false
7697 Acked Write 4 Sent Done Done false
7698 Acked Read 6 Sent Done Done false
7699 Acked Read 4 Sent Done Done false
7700 Acked Read 7 Sent Done Done false
7701 Acked Write 5 Sent Done Done false
7702 Acked Write 1 Sent Done Done false
7703 Acked Flush 0 Sent Done Done false
7704 Acked Read 7 Sent Done Done false
7705 Acked Write 1 Sent Done Done false
7706 Acked Read 1 Sent Done Done false
7707 Acked Write 9 Sent Done Done false
7708 Acked Flush 0 Sent Done Done false
7709 Acked Read 1 Sent Done Done false
7710 Acked Write 6 Sent Done Done false
7711 Acked Read 3 Sent Done Done false
7712 Acked Read 4 Sent Done Done false
7713 Acked Write 7 Sent Done Done false
7714 Acked Read 3 Sent Done Done false
7715 Acked Write 2 Sent Done Done false
7716 Acked Write 4 Sent Done Done false
7717 Acked Write 7 Sent Done Done false
7718 Acked Read 9 Sent Done Done false
7719 Acked Read 2 Sent Done Done false
7720 Acked Read 4 Sent Done Done false
7721 Acked Write 5 Sent Done Done false
7722 Acked Read 6 Sent Done Done false
7723 Acked Write 4 Sent Done Done false
7724 Acked Write 6 Sent Done Done false
7725 Acked Write 2 Sent Done Done false
7726 Acked Write 5 Sent Done Done false
7727 Acked Read 7 Sent Done Done false
7728 Acked Read 2 Sent Done Done false
7729 Acked Write 2 Sent Done Done false
7730 Acked Read 9 Sent Done Done false
7731 Acked Read 5 Sent Done Done false
7732 Acked Read 4 Sent Done Done false
7733 Acked Write 5 Sent Done Done false
7734 Acked Read 8 Sent Done Done false
7735 Acked Write 5 Sent Done Done false
7736 Acked Read 5 Sent Done Done false
7737 Acked Read 4 Sent Done Done false
7738 Acked Read 1 Sent Done Done false
7739 Acked Read 6 Sent Done Done false
7740 Acked Read 7 Sent Done Done false
7741 Acked Read 1 Sent Done Done false
7742 Acked Read 3 Sent Done Done false
7743 Acked Read 7 Sent Done Done false
7744 Acked Read 9 Sent Done Done false
7745 Acked Write 7 Sent Done Done false
7746 Acked Read 10 Sent Done Done false
7747 Acked Write 3 Sent Done Done false
7748 Acked Write 8 Sent Done Done false
7749 Acked Read 8 Sent Done Done false
7750 Acked Read 5 Sent Done Done false
7751 Acked Write 3 Sent Done Done false
7752 Acked Read 2 Sent Done Done false
7753 Acked Flush 0 Sent Done Done false
7754 Acked Read 5 Sent Done Done false
7755 Acked Read 1 Sent Done Done false
7756 Acked Write 9 Sent Done Done false
7757 Acked Write 7 Sent Done Done false
7758 Acked Write 4 Sent Done Done false
7759 Acked Flush 0 Sent Done Done false
7760 Acked Read 10 Sent Done Done false
7761 Acked Write 7 Sent Done Done false
7762 Acked Write 8 Sent Done Done false
7763 Acked Flush 0 Sent Done Done false
7764 Acked Read 8 Sent Done Done false
7765 Acked Write 7 Sent Done Done false
7766 Acked Read 5 Sent Done Done false
7767 Acked Read 8 Sent Done Done false
7768 Acked Write 2 Sent Done Done false
7769 Acked Read 2 Sent Done Done false
7770 Acked Read 3 Sent Done Done false
7771 Acked Write 5 Sent Done Done false
7772 Acked Read 9 Sent Done Done false
7773 Acked Read 2 Sent Done Done false
7774 Acked Read 1 Sent Done Done false
7775 Acked Read 10 Sent Done Done false
7776 Acked Read 8 Sent Done Done false
7777 Acked Write 5 Sent Done Done false
7778 Acked Write 6 Sent Done Done false
7779 Acked Read 3 Sent Done Done false
7780 Acked Read 3 Sent Done Done false
7781 Acked Read 1 Sent Done Done false
7782 Acked Write 5 Sent Done Done false
7783 Acked Write 8 Sent Done Done false
7784 Acked Read 1 Sent Done Done false
7785 Acked Write 6 Sent Done Done false
7786 Acked Write 2 Sent Done Done false
7787 Acked Write 8 Sent Done Done false
7788 Acked Write 5 Sent Done Done false
7789 Acked Read 4 Sent Done Done false
7790 Acked Read 9 Sent Done Done false
7791 Acked Read 4 Sent Done Done false
7792 Acked Read 5 Sent Done Done false
7793 Acked Read 7 Sent Done Done false
7794 Acked Flush 0 Sent Done Done false
7795 Acked Write 9 Sent Done Done false
7796 Acked Read 10 Sent Done Done false
7797 Acked Read 3 Sent Done Done false
7798 Acked Read 1 Sent Done Done false
7799 Acked Read 10 Sent Done Done false
7800 Acked Write 5 Sent Done Done false
7801 Acked Read 1 Sent Done Done false
7802 Acked Flush 0 Sent Done Done false
7803 Acked Read 9 Sent Done Done false
7804 Acked Write 6 Sent Done Done false
7805 Acked Read 1 Sent Done Done false
7806 Acked Read 3 Sent Done Done false
7807 Acked Write 6 Sent Done Done false
7808 Acked Write 8 Sent Done Done false
7809 Acked Read 7 Sent Done Done false
7810 Acked Write 1 Sent Done Done false
7811 Acked Read 7 Sent Done Done false
7812 Acked Read 10 Sent Done Done false
7813 Acked Read 7 Sent Done Done false
7814 Acked Read 1 Sent Done Done false
7815 Acked Write 7 Sent Done Done false
7816 Acked Write 6 Sent Done Done false
7817 Acked Write 8 Sent Done Done false
7818 Acked Read 3 Sent Done Done false
7819 Acked Write 10 Sent Done Done false
7820 Acked Write 10 Sent Done Done false
7821 Acked Read 6 Sent Done Done false
7822 Acked Write 10 Sent Done Done false
7823 Acked Write 1 Sent Done Done false
7824 Acked Flush 0 Sent Done Done false
7825 Acked Read 7 Sent Done Done false
7826 Acked Read 7 Sent Done Done false
7827 Acked Write 10 Sent Done Done false
7828 Acked Flush 0 Sent Done Done false
7829 Acked Read 9 Sent Done Done false
7830 Acked Write 9 Sent Done Done false
7831 Acked Write 9 Sent Done Done false
7832 Acked Write 10 Sent Done Done false
7833 Acked Write 4 Sent Done Done false
7834 Acked Read 4 Sent Done Done false
7835 Acked Read 4 Sent Done Done false
7836 Acked Write 3 Sent Done Done false
7837 Acked Read 10 Sent Done Done false
7838 Acked Write 4 Sent Done Done false
7839 Acked Read 1 Sent Done Done false
7840 Acked Write 1 Sent Done Done false
7841 Acked Write 4 Sent Done Done false
7842 Acked Write 6 Sent Done Done false
7843 Acked Write 4 Sent Done Done false
7844 Acked Write 8 Sent Done Done false
7845 Acked Write 10 Sent Done Done false
7846 Acked Read 9 Sent Done Done false
7847 Acked Write 6 Sent Done Done false
7848 Acked Read 7 Sent Done Done false
7849 Acked Read 6 Sent Done Done false
7850 Acked Flush 0 Sent Done Done false
7851 Acked Read 7 Sent Done Done false
7852 Acked Write 2 Sent Done Done false
7853 Acked Read 2 Sent Done Done false
7854 Acked Write 8 Sent Done Done false
7855 Acked Write 9 Sent Done Done false
7856 Acked Write 2 Sent Done Done false
7857 Acked Read 9 Sent Done Done false
7858 Acked Write 7 Sent Done Done false
7859 Acked Write 5 Sent Done Done false
7860 Acked Flush 0 Sent Done Done false
7861 Acked Read 3 Sent Done Done false
7862 Acked Flush 0 Sent Done Done false
7863 Acked Write 7 Sent Done Done false
7864 Acked Write 9 Sent Done Done false
7865 Acked Write 3 Sent Done Done false
7866 Acked Flush 0 Sent Done Done false
7867 Acked Read 5 Sent Done Done false
7868 Acked Write 10 Sent Done Done false
7869 Acked Write 3 Sent Done Done false
7870 Acked Read 1 Sent Done Done false
7871 Acked Read 9 Sent Done Done false
7872 Acked Read 10 Sent Done Done false
7873 Acked Read 1 Sent Done Done false
7874 Acked Write 1 Sent Done Done false
7875 Acked Write 7 Sent Done Done false
7876 Acked Read 8 Sent Done Done false
7877 Acked Write 9 Sent Done Done false
7878 Acked Read 5 Sent Done Done false
7879 Acked Write 4 Sent Done Done false
7880 Acked Write 4 Sent Done Done false
7881 Acked Write 1 Sent Done Done false
7882 Acked Read 9 Sent Done Done false
7883 Acked Flush 0 Sent Done Done false
7884 Acked Write 2 Sent Done Done false
7885 Acked Write 10 Sent Done Done false
7886 Acked Read 7 Sent Done Done false
7887 Acked Write 4 Sent Done Done false
7888 Acked Flush 0 Sent Done Done false
7889 Acked Read 10 Sent Done Done false
7890 Acked Read 10 Sent Done Done false
7891 Acked Write 7 Sent Done Done false
7892 Acked Write 10 Sent Done Done false
7893 Acked Write 6 Sent Done Done false
7894 Acked Write 3 Sent Done Done false
7895 Acked Write 1 Sent Done Done false
7896 Acked Read 5 Sent Done Done false
7897 Acked Read 7 Sent Done Done false
7898 Acked Flush 0 Sent Done Done false
7899 Acked Read 10 Sent Done Done false
7900 Acked Write 1 Sent Done Done false
7901 Acked Write 8 Sent Done Done false
7902 Acked Read 3 Sent Done Done false
7903 Acked Write 4 Sent Done Done false
7904 Acked Read 8 Sent Done Done false
7905 Acked Write 4 Sent Done Done false
7906 Acked Read 7 Sent Done Done false
7907 Acked Read 5 Sent Done Done false
7908 Acked Write 1 Sent Done Done false
7909 Acked Write 1 Sent Done Done false
7910 Acked Write 10 Sent Done Done false
7911 Acked Write 2 Sent Done Done false
7912 Acked Write 1 Sent Done Done false
7913 Acked Write 9 Sent Done Done false
7914 Acked Read 6 Sent Done Done false
7915 Acked Read 5 Sent Done Done false
7916 Acked Write 9 Sent Done Done false
7917 Acked Write 2 Sent Done Done false
7918 Acked Read 2 Sent Done Done false
7919 Acked Write 3 Sent Done Done false
7920 Acked Write 5 Sent Done Done false
7921 Acked Write 10 Sent Done Done false
7922 Acked Read 4 Sent Done Done false
7923 Acked Write 3 Sent Done Done false
7924 Acked Read 5 Sent Done Done false
7925 Acked Read 1 Sent Done Done false
7926 Acked Write 4 Sent Done Done false
7927 Acked Write 8 Sent Done Done false
7928 Acked Write 5 Sent Done Done false
7929 Acked Read 4 Sent Done Done false
7930 Acked Read 6 Sent Done Done false
7931 Acked Write 6 Sent Done Done false
7932 Acked Read 5 Sent Done Done false
7933 Acked Write 2 Sent Done Done false
7934 Acked Write 3 Sent Done Done false
7935 Acked Read 8 Sent Done Done false
7936 Acked Write 6 Sent Done Done false
7937 Acked Flush 0 Sent Done Done false
7938 Acked Write 10 Sent Done Done false
7939 Acked Read 7 Sent Done Done false
7940 Acked Read 9 Sent Done Done false
7941 Acked Read 3 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
Downstairs last five completed: 7640 7639 7638 7637 7636
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:14.142368114Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:14.965090403Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:15.965849474Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:16.968094129Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7641 Acked Write 10 Sent Done Done false
7642 Acked Read 9 Sent Done Done false
7643 Acked Read 8 Sent Done Done false
7644 Acked Write 3 Sent Done Done false
7645 Acked Read 4 Sent Done Done false
7646 Acked Flush 0 Sent Done Done false
7647 Acked Write 4 Sent Done Done false
7648 Acked Write 10 Sent Done Done false
7649 Acked Read 1 Sent Done Done false
7650 Acked Write 4 Sent Done Done false
7651 Acked Read 10 Sent Done Done false
7652 Acked Read 10 Sent Done Done false
7653 Acked Read 9 Sent Done Done false
7654 Acked Read 3 Sent Done Done false
7655 Acked Write 6 Sent Done Done false
7656 Acked Read 2 Sent Done Done false
7657 Acked Read 7 Sent Done Done false
7658 Acked Read 9 Sent Done Done false
7659 Acked Write 8 Sent Done Done false
7660 Acked Read 8 Sent Done Done false
7661 Acked Flush 0 Sent Done Done false
7662 Acked Flush 0 Sent Done Done false
7663 Acked Write 1 Sent Done Done false
7664 Acked Flush 0 Sent Done Done false
7665 Acked Write 1 Sent Done Done false
7666 Acked Flush 0 Sent Done Done false
7667 Acked Write 5 Sent Done Done false
7668 Acked Read 8 Sent Done Done false
7669 Acked Read 9 Sent Done Done false
7670 Acked Read 1 Sent Done Done false
7671 Acked Read 3 Sent Done Done false
7672 Acked Write 5 Sent Done Done false
7673 Acked Write 10 Sent Done Done false
7674 Acked Read 2 Sent Done Done false
7675 Acked Flush 0 Sent Done Done false
7676 Acked Read 2 Sent Done Done false
7677 Acked Write 2 Sent Done Done false
7678 Acked Read 10 Sent Done Done false
7679 Acked Write 1 Sent Done Done false
7680 Acked Read 4 Sent Done Done false
7681 Acked Read 5 Sent Done Done false
7682 Acked Write 10 Sent Done Done false
7683 Acked Write 8 Sent Done Done false
7684 Acked Write 1 Sent Done Done false
7685 Acked Write 4 Sent Done Done false
7686 Acked Write 2 Sent Done Done false
7687 Acked Write 3 Sent Done Done false
7688 Acked Read 1 Sent Done Done false
7689 Acked Read 5 Sent Done Done false
7690 Acked Read 10 Sent Done Done false
7691 Acked Write 1 Sent Done Done false
7692 Acked Flush 0 Sent Done Done false
7693 Acked Read 5 Sent Done Done false
7694 Acked Read 8 Sent Done Done false
7695 Acked Read 9 Sent Done Done false
7696 Acked Read 10 Sent Done Done false
7697 Acked Write 4 Sent Done Done false
7698 Acked Read 6 Sent Done Done false
7699 Acked Read 4 Sent Done Done false
7700 Acked Read 7 Sent Done Done false
7701 Acked Write 5 Sent Done Done false
7702 Acked Write 1 Sent Done Done false
7703 Acked Flush 0 Sent Done Done false
7704 Acked Read 7 Sent Done Done false
7705 Acked Write 1 Sent Done Done false
7706 Acked Read 1 Sent Done Done false
7707 Acked Write 9 Sent Done Done false
7708 Acked Flush 0 Sent Done Done false
7709 Acked Read 1 Sent Done Done false
7710 Acked Write 6 Sent Done Done false
7711 Acked Read 3 Sent Done Done false
7712 Acked Read 4 Sent Done Done false
7713 Acked Write 7 Sent Done Done false
7714 Acked Read 3 Sent Done Done false
7715 Acked Write 2 Sent Done Done false
7716 Acked Write 4 Sent Done Done false
7717 Acked Write 7 Sent Done Done false
7718 Acked Read 9 Sent Done Done false
7719 Acked Read 2 Sent Done Done false
7720 Acked Read 4 Sent Done Done false
7721 Acked Write 5 Sent Done Done false
7722 Acked Read 6 Sent Done Done false
7723 Acked Write 4 Sent Done Done false
7724 Acked Write 6 Sent Done Done false
7725 Acked Write 2 Sent Done Done false
7726 Acked Write 5 Sent Done Done false
7727 Acked Read 7 Sent Done Done false
7728 Acked Read 2 Sent Done Done false
7729 Acked Write 2 Sent Done Done false
7730 Acked Read 9 Sent Done Done false
7731 Acked Read 5 Sent Done Done false
7732 Acked Read 4 Sent Done Done false
7733 Acked Write 5 Sent Done Done false
7734 Acked Read 8 Sent Done Done false
7735 Acked Write 5 Sent Done Done false
7736 Acked Read 5 Sent Done Done false
7737 Acked Read 4 Sent Done Done false
7738 Acked Read 1 Sent Done Done false
7739 Acked Read 6 Sent Done Done false
7740 Acked Read 7 Sent Done Done false
7741 Acked Read 1 Sent Done Done false
7742 Acked Read 3 Sent Done Done false
7743 Acked Read 7 Sent Done Done false
7744 Acked Read 9 Sent Done Done false
7745 Acked Write 7 Sent Done Done false
7746 Acked Read 10 Sent Done Done false
7747 Acked Write 3 Sent Done Done false
7748 Acked Write 8 Sent Done Done false
7749 Acked Read 8 Sent Done Done false
7750 Acked Read 5 Sent Done Done false
7751 Acked Write 3 Sent Done Done false
7752 Acked Read 2 Sent Done Done false
7753 Acked Flush 0 Sent Done Done false
7754 Acked Read 5 Sent Done Done false
7755 Acked Read 1 Sent Done Done false
7756 Acked Write 9 Sent Done Done false
7757 Acked Write 7 Sent Done Done false
7758 Acked Write 4 Sent Done Done false
7759 Acked Flush 0 Sent Done Done false
7760 Acked Read 10 Sent Done Done false
7761 Acked Write 7 Sent Done Done false
7762 Acked Write 8 Sent Done Done false
7763 Acked Flush 0 Sent Done Done false
7764 Acked Read 8 Sent Done Done false
7765 Acked Write 7 Sent Done Done false
7766 Acked Read 5 Sent Done Done false
7767 Acked Read 8 Sent Done Done false
7768 Acked Write 2 Sent Done Done false
7769 Acked Read 2 Sent Done Done false
7770 Acked Read 3 Sent Done Done false
7771 Acked Write 5 Sent Done Done false
7772 Acked Read 9 Sent Done Done false
7773 Acked Read 2 Sent Done Done false
7774 Acked Read 1 Sent Done Done false
7775 Acked Read 10 Sent Done Done false
7776 Acked Read 8 Sent Done Done false
7777 Acked Write 5 Sent Done Done false
7778 Acked Write 6 Sent Done Done false
7779 Acked Read 3 Sent Done Done false
7780 Acked Read 3 Sent Done Done false
7781 Acked Read 1 Sent Done Done false
7782 Acked Write 5 Sent Done Done false
7783 Acked Write 8 Sent Done Done false
7784 Acked Read 1 Sent Done Done false
7785 Acked Write 6 Sent Done Done false
7786 Acked Write 2 Sent Done Done false
7787 Acked Write 8 Sent Done Done false
7788 Acked Write 5 Sent Done Done false
7789 Acked Read 4 Sent Done Done false
7790 Acked Read 9 Sent Done Done false
7791 Acked Read 4 Sent Done Done false
7792 Acked Read 5 Sent Done Done false
7793 Acked Read 7 Sent Done Done false
7794 Acked Flush 0 Sent Done Done false
7795 Acked Write 9 Sent Done Done false
7796 Acked Read 10 Sent Done Done false
7797 Acked Read 3 Sent Done Done false
7798 Acked Read 1 Sent Done Done false
7799 Acked Read 10 Sent Done Done false
7800 Acked Write 5 Sent Done Done false
7801 Acked Read 1 Sent Done Done false
7802 Acked Flush 0 Sent Done Done false
7803 Acked Read 9 Sent Done Done false
7804 Acked Write 6 Sent Done Done false
7805 Acked Read 1 Sent Done Done false
7806 Acked Read 3 Sent Done Done false
7807 Acked Write 6 Sent Done Done false
7808 Acked Write 8 Sent Done Done false
7809 Acked Read 7 Sent Done Done false
7810 Acked Write 1 Sent Done Done false
7811 Acked Read 7 Sent Done Done false
7812 Acked Read 10 Sent Done Done false
7813 Acked Read 7 Sent Done Done false
7814 Acked Read 1 Sent Done Done false
7815 Acked Write 7 Sent Done Done false
7816 Acked Write 6 Sent Done Done false
7817 Acked Write 8 Sent Done Done false
7818 Acked Read 3 Sent Done Done false
7819 Acked Write 10 Sent Done Done false
7820 Acked Write 10 Sent Done Done false
7821 Acked Read 6 Sent Done Done false
7822 Acked Write 10 Sent Done Done false
7823 Acked Write 1 Sent Done Done false
7824 Acked Flush 0 Sent Done Done false
7825 Acked Read 7 Sent Done Done false
7826 Acked Read 7 Sent Done Done false
7827 Acked Write 10 Sent Done Done false
7828 Acked Flush 0 Sent Done Done false
7829 Acked Read 9 Sent Done Done false
7830 Acked Write 9 Sent Done Done false
7831 Acked Write 9 Sent Done Done false
7832 Acked Write 10 Sent Done Done false
7833 Acked Write 4 Sent Done Done false
7834 Acked Read 4 Sent Done Done false
7835 Acked Read 4 Sent Done Done false
7836 Acked Write 3 Sent Done Done false
7837 Acked Read 10 Sent Done Done false
7838 Acked Write 4 Sent Done Done false
7839 Acked Read 1 Sent Done Done false
7840 Acked Write 1 Sent Done Done false
7841 Acked Write 4 Sent Done Done false
7842 Acked Write 6 Sent Done Done false
7843 Acked Write 4 Sent Done Done false
7844 Acked Write 8 Sent Done Done false
7845 Acked Write 10 Sent Done Done false
7846 Acked Read 9 Sent Done Done false
7847 Acked Write 6 Sent Done Done false
7848 Acked Read 7 Sent Done Done false
7849 Acked Read 6 Sent Done Done false
7850 Acked Flush 0 Sent Done Done false
7851 Acked Read 7 Sent Done Done false
7852 Acked Write 2 Sent Done Done false
7853 Acked Read 2 Sent Done Done false
7854 Acked Write 8 Sent Done Done false
7855 Acked Write 9 Sent Done Done false
7856 Acked Write 2 Sent Done Done false
7857 Acked Read 9 Sent Done Done false
7858 Acked Write 7 Sent Done Done false
7859 Acked Write 5 Sent Done Done false
7860 Acked Flush 0 Sent Done Done false
7861 Acked Read 3 Sent Done Done false
7862 Acked Flush 0 Sent Done Done false
7863 Acked Write 7 Sent Done Done false
7864 Acked Write 9 Sent Done Done false
7865 Acked Write 3 Sent Done Done false
7866 Acked Flush 0 Sent Done Done false
7867 Acked Read 5 Sent Done Done false
7868 Acked Write 10 Sent Done Done false
7869 Acked Write 3 Sent Done Done false
7870 Acked Read 1 Sent Done Done false
7871 Acked Read 9 Sent Done Done false
7872 Acked Read 10 Sent Done Done false
7873 Acked Read 1 Sent Done Done false
7874 Acked Write 1 Sent Done Done false
7875 Acked Write 7 Sent Done Done false
7876 Acked Read 8 Sent Done Done false
7877 Acked Write 9 Sent Done Done false
7878 Acked Read 5 Sent Done Done false
7879 Acked Write 4 Sent Done Done false
7880 Acked Write 4 Sent Done Done false
7881 Acked Write 1 Sent Done Done false
7882 Acked Read 9 Sent Done Done false
7883 Acked Flush 0 Sent Done Done false
7884 Acked Write 2 Sent Done Done false
7885 Acked Write 10 Sent Done Done false
7886 Acked Read 7 Sent Done Done false
7887 Acked Write 4 Sent Done Done false
7888 Acked Flush 0 Sent Done Done false
7889 Acked Read 10 Sent Done Done false
7890 Acked Read 10 Sent Done Done false
7891 Acked Write 7 Sent Done Done false
7892 Acked Write 10 Sent Done Done false
7893 Acked Write 6 Sent Done Done false
7894 Acked Write 3 Sent Done Done false
7895 Acked Write 1 Sent Done Done false
7896 Acked Read 5 Sent Done Done false
7897 Acked Read 7 Sent Done Done false
7898 Acked Flush 0 Sent Done Done false
7899 Acked Read 10 Sent Done Done false
7900 Acked Write 1 Sent Done Done false
7901 Acked Write 8 Sent Done Done false
7902 Acked Read 3 Sent Done Done false
7903 Acked Write 4 Sent Done Done false
7904 Acked Read 8 Sent Done Done false
7905 Acked Write 4 Sent Done Done false
7906 Acked Read 7 Sent Done Done false
7907 Acked Read 5 Sent Done Done false
7908 Acked Write 1 Sent Done Done false
7909 Acked Write 1 Sent Done Done false
7910 Acked Write 10 Sent Done Done false
7911 Acked Write 2 Sent Done Done false
7912 Acked Write 1 Sent Done Done false
7913 Acked Write 9 Sent Done Done false
7914 Acked Read 6 Sent Done Done false
7915 Acked Read 5 Sent Done Done false
7916 Acked Write 9 Sent Done Done false
7917 Acked Write 2 Sent Done Done false
7918 Acked Read 2 Sent Done Done false
7919 Acked Write 3 Sent Done Done false
7920 Acked Write 5 Sent Done Done false
7921 Acked Write 10 Sent Done Done false
7922 Acked Read 4 Sent Done Done false
7923 Acked Write 3 Sent Done Done false
7924 Acked Read 5 Sent Done Done false
7925 Acked Read 1 Sent Done Done false
7926 Acked Write 4 Sent Done Done false
7927 Acked Write 8 Sent Done Done false
7928 Acked Write 5 Sent Done Done false
7929 Acked Read 4 Sent Done Done false
7930 Acked Read 6 Sent Done Done false
7931 Acked Write 6 Sent Done Done false
7932 Acked Read 5 Sent Done Done false
7933 Acked Write 2 Sent Done Done false
7934 Acked Write 3 Sent Done Done false
7935 Acked Read 8 Sent Done Done false
7936 Acked Write 6 Sent Done Done false
7937 Acked Flush 0 Sent Done Done false
7938 Acked Write 10 Sent Done Done false
7939 Acked Read 7 Sent Done Done false
7940 Acked Read 9 Sent Done Done false
7941 Acked Read 3 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
Downstairs last five completed: 7640 7639 7638 7637 7636
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:18.146446799Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:18.972560366Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:19.974261921Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:20.976269863Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.399517159Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35367)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.399759021Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401120855Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401702975Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401727379Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(7640))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401744545Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401760541Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(7640))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.401776937Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(7640))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:21.402031786Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7938 Acked Write 10 Done Done Done true
7939 Acked Read 7 Done Done Done true
7940 Acked Read 9 Done Done Done true
7941 Acked Read 3 Done Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7937)) Some(JobId(7937)) Some(JobId(7937))
Downstairs last five completed: 7937 7936 7935 7934 7933
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:4 act:3
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7942 7941 7940 7939 7938
Upstairs last five completed: 7942 7941 7940 7939 7938
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:07:25.646 [8810] Got stop action so:false kr:false
INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9847fb62-9fa5-4b89-a07c-68c92e504a40, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:25.647049789Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:07:25.647 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7b57a94b-041b-4b65-9a79-85d454478e06, remote_addr: 127.0.0.1:63908, local_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:25.647165041Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
0{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:25.647202792Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:25.647308127Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:07:29.648 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: acc55c25-cd21-453a-84f8-4d3fa8f64a8d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Write block 3375 len 20480 data: 3 4 3 4 4
003/300 Write block 2352 len 24576 data: 4 5 3 3 5 6
004/300 Read block 793 len 24576
005/300 Write block 4511 len 16384 data: 7 7 8 9
006/300 Read block 1506 len 20480
007/300 Write block 584 len 28672 data: 6 6 7 5 6 4 5
008/300 Write block 1340 len 28672 data: 7 6 6 5 3 3 2
009/300 Read block 2989 len 20480
010/300 Read block 4512 len 16384
011/300 Read block 1434 len 36864
012/300 Read block 1747 len 20480
013/300 Read block 2478 len 4096
014/300 Read block 79 len 24576
015/300 Read block 1796 len 40960
016/300 Read block 1206 len 24576
017/300 Write block 2401 len 16384 data: 2 2 2 2
018/300 Flush
019/300 Write block 3769 len 36864 data: 6 5 6 6 7 6 4 3 3
020/300 Read block 2836 len 28672
021/300 Read block 2578 len 4096
022/300 Flush
023/300 Write block 1894 len 4096 data: 8
024/300 Read block 422 len 32768
025/300 Write block 3154 len 28672 data: 6 6 9 9 9 10 8
026/300 Read block 3444 len 20480
027/300 Read block 2482 len 24576
028/300 Read block 4018 len 32768
029/300 Flush
030/300 Write block 4328 len 12288 data: 6 4 4
031/300 Read block 997 len 28672
032/300 Read block 3983 len 16384
033/300 Read block 1546 len 40960
034/300 Read block 4333 len 20480
035/300 Write block 208 len 16384 data: 6 6 6 6
036/300 Write block 2939 len 12288 data: 6 6 5
037/300 Read block 1373 len 16384
038/300 Flush
039/300 Write block 2601 len 12288 data: 7 6 7
040/300 Write block 465 len 24576 data: 3 5 6 7 7 8
041/300 Write block 3034 len 24576 data: 6 6 6 6 5 5
042/300 Read block 3461 len 12288
043/300 Read block 16 len 40960
044/300 Read block 225 len 28672
045/300 Write block 4171 len 4096 data: 4
046/300 Flush
047/300 Read block 3160 len 4096
048/300 Read block 3221 len 4096
049/300 Read block 3960 len 28672
050/300 Write block 1415 len 8192 data: 5 4
051/300 Read block 4693 len 12288
052/300 Read block 2384 len 36864
053/300 Flush
054/300 Flush
055/300 Read block 120 len 36864
056/300 Read block 3023 len 12288
057/300 Read block 772 len 40960
058/300 Write block 4087 len 8192 data: 5 4
059/300 Flush
060/300 Read block 2236 len 12288
061/300 Read block 4727 len 4096
062/300 Read block 1130 len 4096
063/300 Read block 3644 len 28672
064/300 Flush
065/300 Read block 4532 len 32768
066/300 Read block 936 len 32768
067/300 Read block 1499 len 32768
068/300 Write block 1319 len 28672 data: 6 6 6 7 7 5 5
069/300 Write block 2063 len 28672 data: 4 5 6 5 5 6 4
070/300 Read block 2174 len 4096
071/300 Read block 177 len 36864
072/300 Read block 1802 len 4096
073/300 Read block 3119 len 36864
074/300 Read block 1074 len 40960
075/300 Write block 3662 len 4096 data: 4
076/300 Read block 2997 len 20480
077/300 Read block 4383 len 36864
078/300 Write block 4471 len 16384 data: 4 5 5 5
079/300 Read block 3094 len 40960
080/300 Read block 4150 len 28672
081/300 Write block 2854 len 16384 data: 5 5 5 6
082/300 Write block 4340 len 20480 data: 6 6 9 8 8
083/300 Write block 1520 len 20480 data: 4 5 4 3 3
084/300 Read block 1407 len 28672
085/300 Read block 329 len 20480
086/300 Read block 1480 len 12288
087/300 Write block 4336 len 20480 data: 7 7 6 7 7
088/300 Write block 3445 len 28672 data: 4 4 4 3 2 3 3
089/300 Read block 123 len 40960
090/300 Read block 4160 len 28672
091/300 Write block 1522 len 32768 data: 5 4 4 4 4 5 5 6
092/300 Read block 3118 len 12288
093/300 Read block 18 len 20480
094/300 Write block 58 len 24576 data: 6 4 4 4 4 5
095/300 Read block 2426 len 28672
096/300 Read block 72 len 28672
097/300 Write block 3280 len 20480 data: 3 2 3 2 2
098/300 Read block 3470 len 24576
099/300 Read block 2833 len 8192
100/300 Read block 2228 len 36864
101/300 Flush
102/300 Read block 845 len 8192
103/300 Write block 4734 len 32768 data: 8 6 6 4 4 4 4 4
104/300 Write block 3480 len 4096 data: 8
105/300 Write block 3267 len 36864 data: 4 4 5 4 5 5 4 5 5
106/300 Read block 2474 len 8192
107/300 Write block 471 len 28672 data: 5 5 4 6 7 8 7
108/300 Read block 698 len 24576
109/300 Read block 1015 len 24576
110/300 Flush
111/300 Write block 1151 len 20480 data: 7 8 8 7 8
112/300 Write block 2795 len 40960 data: 7 6 6 5 6 6 6 4 5 5
113/300 Flush
114/300 Read block 4252 len 40960
115/300 Write block 1005 len 24576 data: 4 3 3 4 4 4
116/300 Write block 2297 len 40960 data: 3 4 3 3 3 3 3 4 4 4
117/300 Read block 3559 len 8192
118/300 Write block 860 len 24576 data: 5 5 6 6 6 5
119/300 Read block 3968 len 36864
120/300 Read block 1509 len 28672
121/300 Write block 4704 len 32768 data: 3 3 3 3 3 4 4 5
122/300 Write block 2230 len 16384 data: 7 9 8 7
123/300 Write block 4272 len 36864 data: 8 6 6 5 4 5 5 5 5
124/300 Write block 2302 len 4096 data: 4
125/300 Read block 884 len 40960
126/300 Read block 2182 len 20480
127/300 Write block 918 len 40960 data: 5 5 5 5 5 4 4 4 5 5
128/300 Read block 3115 len 20480
129/300 Write block 3592 len 24576 data: 4 5 4 4 4 5
130/300 Flush
131/300 Write block 2946 len 12288 data: 3 4 5
132/300 Read block 4059 len 4096
133/300 Flush
134/300 Read block 454 len 36864
135/300 Write block 999 len 24576 data: 5 5 6 6 5 4
136/300 Write block 2042 len 8192 data: 3 3
137/300 Flush
138/300 Read block 1220 len 20480
139/300 Read block 4963 len 32768
140/300 Read block 2765 len 24576
141/300 Flush
142/300 Read block 784 len 8192
143/300 Write block 3866 len 20480 data: 8 9 8 10 9
144/300 Read block 2780 len 8192
145/300 Read block 4923 len 20480
146/300 Read block 167 len 28672
147/300 Write block 1628 len 4096 data: 7
148/300 Read block 2297 len 16384
149/300 Read block 1645 len 12288
150/300 Read block 4764 len 36864
151/300 Write block 4513 len 8192 data: 9 10
152/300 Read block 3288 len 12288
153/300 Read block 2709 len 16384
154/300 Write block 71 len 28672 data: 3 3 5 5 8 7 7
155/300 Read block 1584 len 32768
156/300 Read block 1522 len 16384
157/300 Read block 1047 len 4096
158/300 Read block 2109 len 40960
159/300 Read block 3029 len 12288
160/300 Flush
161/300 Read block 4248 len 28672
162/300 Read block 689 len 8192
163/300 Write block 1964 len 4096 data: 4
164/300 Read block 398 len 8192
165/300 Read block 3740 len 4096
166/300 Write block 3417 len 24576 data: 6 6 6 5 5 5
167/300 Read block 3895 len 16384
168/300 Read block 2574 len 40960
169/300 Read block 1364 len 32768
170/300 Flush
171/300 Write block 1963 len 20480 data: 3 5 4 4 5
172/300 Read block 1536 len 20480
173/300 Write block 2765 len 16384 data: 3 3 3 3
174/300 Write block 4088 len 4096 data: 5
175/300 Read block 3710 len 24576
176/300 Write block 2758 len 16384 data: 5 7 6 5
177/300 Read block 1148 len 40960
178/300 Read block 2160 len 36864
179/300 Write block 1 len 20480 data: 2 3 4 5 5
180/300 Read block 584 len 20480
181/300 Read block 4217 len 8192
182/300 Flush
183/300 Write block 2714 len 12288 data: 8 8 9
184/300 Flush
185/300 Read block 4890 len 16384
186/300 Write block 1293 len 36864 data: 6 5 4 4 5 6 5 5 5
187/300 Write block 3072 len 28672 data: 4 5 5 4 5 5 4
188/300 Read block 404 len 36864
189/300 Write block 3135 len 24576 data: 3 3 3 3 2 2
190/300 Read block 2546 len 20480
191/300 Write block 4188 len 28672 data: 6 6 8 8 7 7 5
192/300 Flush
193/300 Write block 4363 len 8192 data: 3 4
194/300 Read block 2296 len 28672
195/300 Write block 4006 len 16384 data: 6 6 6 5
196/300 Write block 4561 len 12288 data: 4 4 6
197/300 Flush
198/300 Read block 4171 len 8192
199/300 Read block 1540 len 16384
200/300 Write block 2259 len 24576 data: 6 6 4 5 5 6
201/300 Read block 686 len 24576
202/300 Read block 4857 len 8192
203/300 Read block 406 len 12288
204/300 Write block 4175 len 8192 data: 4 4
205/300 Flush
206/300 Read block 3854 len 12288
207/300 Write block 244 len 36864 data: 5 5 4 3 3 4 4 5 4
208/300 Write block 949 len 16384 data: 7 7 7 4
209/300 Flush
210/300 Write block 4733 len 28672 data: 8 9 7 7 5 5 5
211/300 Read block 3963 len 24576
212/300 Read block 3415 len 32768
213/300 Write block 3559 len 4096 data: 5
214/300 Read block 2586 len 4096
215/300 Read block 4175 len 28672
216/300 Write block 4050 len 8192 data: 3 3
217/300 Read block 1049 len 8192
218/300 Write block 1379 len 4096 data: 6
219/300 Read block 3775 len 40960
220/300 Read block 3830 len 20480
221/300 Read block 3685 len 16384
222/300 Write block 1743 len 32768 data: 5 4 4 4 7 6 6 6
223/300 Write block 322 len 8192 data: 6 6
224/300 Read block 176 len 36864
225/300 Write block 567 len 40960 data: 4 3 3 4 4 5 5 5 5 4
226/300 Write block 4979 len 20480 data: 6 5 5 7 6
227/300 Write block 1469 len 28672 data: 6 6 7 6 6 6 4
228/300 Write block 3898 len 4096 data: 3
229/300 Write block 3640 len 32768 data: 2 3 2 2 2 4 4 4
230/300 Read block 2286 len 8192
231/300 Write block 4213 len 24576 data: 7 5 6 6 5 2
232/300 Flush
233/300 Read block 4609 len 8192
234/300 Read block 781 len 12288
235/300 Write block 2670 len 8192 data: 10 7
236/300 Flush
237/300 Flush
238/300 Read block 4386 len 20480
239/300 Read block 1497 len 36864
240/300 Read block 2589 len 16384
241/300 Write block 453 len 16384 data: 7 7 6 6
242/300 Flush
243/300 Read block 3273 len 40960
244/300 Write block 3909 len 40960 data: 4 3 3 3 3 4 4 4 3 4
245/300 Read block 2161 len 24576
246/300 Read block 2373 len 20480
247/300 Read block 1927 len 4096
248/300 Write block 2858 len 32768 data: 6 4 3 5 5 6 5 3
249/300 Read block 3743 len 4096
250/300 Read block 802 len 36864
251/300 Read block 3420 len 16384
252/300 Write block 3081 len 20480 data: 3 4 5 5 4
253/300 Write block 742 len 36864 data: 6 7 8 7 7 6 8 9 9
254/300 Write block 4546 len 32768 data: 5 5 4 4 5 5 4 4
255/300 Read block 3888 len 8192
256/300 Write block 1230 len 4096 data: 5
257/300 Read block 2839 len 40960
258/300 Write block 871 len 40960 data: 6 4 5 4 3 4 4 3 3 2
259/300 Write block 1217 len 4096 data: 4
260/300 Flush
261/300 Read block 3726 len 4096
262/300 Write block 3526 len 36864 data: 8 9 10 9 7 7 7 7 6
263/300 Read block 2960 len 12288
264/300 Read block 788 len 12288
265/300 Write block 3290 len 32768 data: 5 4 3 7 5 5 6 7
266/300 Read block 1598 len 12288
267/300 Flush
268/300 Write block 1467 len 4096 data: 6
269/300 Write block 2512 len 8192 data: 7 6
270/300 Read block 4983 len 36864
271/300 Read block 3450 len 40960
272/300 Write block 2536 len 32768 data: 4 4 3 3 3 3 4 4
273/300 Write block 4685 len 32768 data: 5 5 6 4 4 4 4 5
274/300 Write block 4937 len 28672 data: 3 3 3 2 3 3 3
275/300 Write block 1459 len 8192 data: 6 5
276/300 Write block 585 len 20480 data: 7 8 6 7 5
277/300 Write block 2274 len 8192 data: 8 7
278/300 Write block 3071 len 40960 data: 4 5 6 6 5 6 6 5 4 4
279/300 Write block 1676 len 4096 data: 6
280/300 Read block 4925 len 4096
281/300 Read block 497 len 4096
282/300 Read block 262 len 24576
283/300 Read block 4462 len 40960
284/300 Read block 1632 len 24576
285/300 Write block 4757 len 4096 data: 9
286/300 Read block 2978 len 40960
287/300 Flush
288/300 Read block 4138 len 8192
289/300 Read block 3104 len 28672
290/300 Flush
291/300 Flush
292/300 Read block 1312 len 40960
293/300 Read block 4947 len 16384
294/300 Read block 126 len 36864
295/300 Write block 724 len 4096 data: 5
296/300 Flush
297/300 Flush
298/300 Read block 1198 len 8192
299/300 Flush
300/300 Read block 4158 len 16384
301/300 Write block 905 len 16384 data: 4 4 4 5
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:07:29.889 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9818b1f0-5206-4c4b-8102-63f4d6293b16, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7943 Acked Flush 0 Sent Done Done false
7944 Acked Write 5 Sent Done Done false
7945 Acked Write 6 Sent Done Done false
7946 Acked Read 6 Sent Done Done false
7947 Acked Write 4 Sent Done Done false
7948 Acked Read 5 Sent Done Done false
7949 Acked Write 7 Sent Done Done false
7950 Acked Write 7 Sent Done Done false
7951 Acked Read 5 Sent Done Done false
7952 Acked Read 4 Sent Done Done false
7953 Acked Read 9 Sent Done Done false
7954 Acked Read 5 Sent Done Done false
7955 Acked Read 1 Sent Done Done false
7956 Acked Read 6 Sent Done Done false
7957 Acked Read 10 Sent Done Done false
7958 Acked Read 6 Sent Done Done false
7959 Acked Write 4 Sent Done Done false
7960 Acked Flush 0 Sent Done Done false
7961 Acked Write 9 Sent Done Done false
7962 Acked Read 7 Sent Done Done false
7963 Acked Read 1 Sent Done Done false
7964 Acked Flush 0 Sent Done Done false
7965 Acked Write 1 Sent Done Done false
7966 Acked Read 8 Sent Done Done false
7967 Acked Write 7 Sent Done Done false
7968 Acked Read 5 Sent Done Done false
7969 Acked Read 6 Sent Done Done false
7970 Acked Read 8 Sent Done Done false
7971 Acked Flush 0 Sent Done Done false
7972 Acked Write 3 Sent Done Done false
7973 Acked Read 7 Sent Done Done false
7974 Acked Read 4 Sent Done Done false
7975 Acked Read 10 Sent Done Done false
7976 Acked Read 5 Sent Done Done false
7977 Acked Write 4 Sent Done Done false
7978 Acked Write 3 Sent Done Done false
7979 Acked Read 4 Sent Done Done false
7980 Acked Flush 0 Sent Done Done false
7981 Acked Write 3 Sent Done Done false
7982 Acked Write 6 Sent Done Done false
7983 Acked Write 6 Sent Done Done false
7984 Acked Read 3 Sent Done Done false
7985 Acked Read 10 Sent Done Done false
7986 Acked Read 7 Sent Done Done false
7987 Acked Write 1 Sent Done Done false
7988 Acked Flush 0 Sent Done Done false
7989 Acked Read 1 Sent Done Done false
7990 Acked Read 1 Sent Done Done false
7991 Acked Read 7 Sent Done Done false
7992 Acked Write 2 Sent Done Done false
7993 Acked Read 3 Sent Done Done false
7994 Acked Read 9 Sent Done Done false
7995 Acked Flush 0 Sent Done Done false
7996 Acked Flush 0 Sent Done Done false
7997 Acked Read 9 Sent Done Done false
7998 Acked Read 3 Sent Done Done false
7999 Acked Read 10 Sent Done Done false
8000 Acked Write 2 Sent Done Done false
8001 Acked Flush 0 Sent Done Done false
8002 Acked Read 3 Sent Done Done false
8003 Acked Read 1 Sent Done Done false
8004 Acked Read 1 Sent Done Done false
8005 Acked Read 7 Sent Done Done false
8006 Acked Flush 0 Sent Done Done false
8007 Acked Read 8 Sent Done Done false
8008 Acked Read 8 Sent Done Done false
8009 Acked Read 8 Sent Done Done false
8010 Acked Write 7 Sent Done Done false
8011 Acked Write 7 Sent Done Done false
8012 Acked Read 1 Sent Done Done false
8013 Acked Read 9 Sent Done Done false
8014 Acked Read 1 Sent Done Done false
8015 Acked Read 9 Sent Done Done false
8016 Acked Read 10 Sent Done Done false
8017 Acked Write 1 Sent Done Done false
8018 Acked Read 5 Sent Done Done false
8019 Acked Read 9 Sent Done Done false
8020 Acked Write 4 Sent Done Done false
8021 Acked Read 10 Sent Done Done false
8022 Acked Read 7 Sent Done Done false
8023 Acked Write 4 Sent Done Done false
8024 Acked Write 5 Sent Done Done false
8025 Acked Write 5 Sent Done Done false
8026 Acked Read 7 Sent Done Done false
8027 Acked Read 5 Sent Done Done false
8028 Acked Read 3 Sent Done Done false
8029 Acked Write 5 Sent Done Done false
8030 Acked Write 7 Sent Done Done false
8031 Acked Read 10 Sent Done Done false
8032 Acked Read 7 Sent Done Done false
8033 Acked Write 8 Sent Done Done false
8034 Acked Read 3 Sent Done Done false
8035 Acked Read 5 Sent Done Done false
8036 Acked Write 6 Sent Done Done false
8037 Acked Read 7 Sent Done Done false
8038 Acked Read 7 Sent Done Done false
8039 Acked Write 5 Sent Done Done false
8040 Acked Read 6 Sent Done Done false
8041 Acked Read 2 Sent Done Done false
8042 Acked Read 9 Sent Done Done false
8043 Acked Flush 0 Sent Done Done false
8044 Acked Read 2 Sent Done Done false
8045 Acked Write 8 Sent Done Done false
8046 Acked Write 1 Sent Done Done false
8047 Acked Write 9 Sent Done Done false
8048 Acked Read 2 Sent Done Done false
8049 Acked Write 7 Sent Done Done false
8050 Acked Read 6 Sent Done Done false
8051 Acked Read 6 Sent Done Done false
8052 Acked Flush 0 Sent Done Done false
8053 Acked Write 5 Sent Done Done false
8054 Acked Write 10 Sent Done Done false
8055 Acked Flush 0 Sent Done Done false
8056 Acked Read 10 Sent Done Done false
8057 Acked Write 6 Sent Done Done false
8058 Acked Write 10 Sent Done Done false
8059 Acked Read 2 Sent Done Done false
8060 Acked Write 6 Sent Done Done false
8061 Acked Read 9 Sent Done Done false
8062 Acked Read 7 Sent Done Done false
8063 Acked Write 8 Sent Done Done false
8064 Acked Write 4 Sent Done Done false
8065 Acked Write 9 Sent Done Done false
8066 Acked Write 1 Sent Done Done false
8067 Acked Read 10 Sent Done Done false
8068 Acked Read 5 Sent Done Done false
8069 Acked Write 10 Sent Done Done false
8070 Acked Read 5 Sent Done Done false
8071 Acked Write 6 Sent Done Done false
8072 Acked Flush 0 Sent Done Done false
8073 Acked Write 3 Sent Done Done false
8074 Acked Read 1 Sent Done Done false
8075 Acked Flush 0 Sent Done Done false
8076 Acked Read 9 Sent Done Done false
8077 Acked Write 6 Sent Done Done false
8078 Acked Write 2 Sent Done Done false
8079 Acked Flush 0 Sent Done Done false
8080 Acked Read 5 Sent Done Done false
8081 Acked Read 8 Sent Done Done false
8082 Acked Read 6 Sent Done Done false
8083 Acked Flush 0 Sent Done Done false
8084 Acked Read 2 Sent Done Done false
8085 Acked Write 5 Sent Done Done false
8086 Acked Read 2 Sent Done Done false
8087 Acked Read 5 Sent Done Done false
8088 Acked Read 7 Sent Done Done false
8089 Acked Write 1 Sent Done Done false
8090 Acked Read 4 Sent Done Done false
8091 Acked Read 3 Sent Done Done false
8092 Acked Read 9 Sent Done Done false
8093 Acked Write 2 Sent Done Done false
8094 Acked Read 3 Sent Done Done false
8095 Acked Read 4 Sent Done Done false
8096 Acked Write 7 Sent Done Done false
8097 Acked Read 8 Sent Done Done false
8098 Acked Read 4 Sent Done Done false
8099 Acked Read 1 Sent Done Done false
8100 Acked Read 10 Sent Done Done false
8101 Acked Read 3 Sent Done Done false
8102 Acked Flush 0 Sent Done Done false
8103 Acked Read 7 Sent Done Done false
8104 Acked Read 2 Sent Done Done false
8105 Acked Write 1 Sent Done Done false
8106 Acked Read 2 Sent Done Done false
8107 Acked Read 1 Sent Done Done false
8108 Acked Write 6 Sent Done Done false
8109 Acked Read 4 Sent Done Done false
8110 Acked Read 10 Sent Done Done false
8111 Acked Read 8 Sent Done Done false
8112 Acked Flush 0 Sent Done Done false
8113 Acked Write 5 Sent Done Done false
8114 Acked Read 5 Sent Done Done false
8115 Acked Write 4 Sent Done Done false
8116 Acked Write 1 Sent Done Done false
8117 Acked Read 6 Sent Done Done false
8118 Acked Write 4 Sent Done Done false
8119 Acked Read 10 Sent Done Done false
8120 Acked Read 9 Sent Done Done false
8121 Acked Write 5 Sent Done Done false
8122 Acked Read 5 Sent Done Done false
8123 Acked Read 2 Sent Done Done false
8124 Acked Flush 0 Sent Done Done false
8125 Acked Write 3 Sent Done Done false
8126 Acked Flush 0 Sent Done Done false
8127 Acked Read 4 Sent Done Done false
8128 Acked Write 9 Sent Done Done false
8129 Acked Write 7 Sent Done Done false
8130 Acked Read 9 Sent Done Done false
8131 Acked Write 6 Sent Done Done false
8132 Acked Read 5 Sent Done Done false
8133 Acked Write 7 Sent Done Done false
8134 Acked Flush 0 Sent Done Done false
8135 Acked Write 2 Sent Done Done false
8136 Acked Read 7 Sent Done Done false
8137 Acked Write 4 Sent Done Done false
8138 Acked Write 3 Sent Done Done false
8139 Acked Flush 0 Sent Done Done false
8140 Acked Read 2 Sent Done Done false
8141 Acked Read 4 Sent Done Done false
8142 Acked Write 6 Sent Done Done false
8143 Acked Read 6 Sent Done Done false
8144 Acked Read 2 Sent Done Done false
8145 Acked Read 3 Sent Done Done false
8146 Acked Write 2 Sent Done Done false
8147 Acked Flush 0 Sent Done Done false
8148 Acked Read 3 Sent Done Done false
8149 Acked Write 9 Sent Done Done false
8150 Acked Write 4 Sent Done Done false
8151 Acked Flush 0 Sent Done Done false
8152 Acked Write 7 Sent Done Done false
8153 Acked Read 6 Sent Done Done false
8154 Acked Read 8 Sent Done Done false
8155 Acked Write 1 Sent Done Done false
8156 Acked Read 1 Sent Done Done false
8157 Acked Read 7 Sent Done Done false
8158 Acked Write 2 Sent Done Done false
8159 Acked Read 2 Sent Done Done false
8160 Acked Write 1 Sent Done Done false
8161 Acked Read 10 Sent Done Done false
8162 Acked Read 5 Sent Done Done false
8163 Acked Read 4 Sent Done Done false
8164 Acked Write 8 Sent Done Done false
8165 Acked Write 2 Sent Done Done false
8166 Acked Read 9 Sent Done Done false
8167 Acked Write 10 Sent Done Done false
8168 Acked Write 5 Sent Done Done false
8169 Acked Write 7 Sent Done Done false
8170 Acked Write 1 Sent Done Done false
8171 Acked Write 8 Sent Done Done false
8172 Acked Read 2 Sent Done Done false
8173 Acked Write 6 Sent Done Done false
8174 Acked Flush 0 Sent Done Done false
8175 Acked Read 2 Sent Done Done false
8176 Acked Read 3 Sent Done Done false
8177 Acked Write 2 Sent Done Done false
8178 Acked Flush 0 Sent Done Done false
8179 Acked Flush 0 Sent Done Done false
8180 Acked Read 5 Sent Done Done false
8181 Acked Read 9 Sent Done Done false
8182 Acked Read 4 Sent Done Done false
8183 Acked Write 4 Sent Done Done false
8184 Acked Flush 0 Sent Done Done false
8185 Acked Read 10 Sent Done Done false
8186 Acked Write 10 Sent Done Done false
8187 Acked Read 6 Sent Done Done false
8188 Acked Read 5 Sent Done Done false
8189 Acked Read 1 Sent Done Done false
8190 Acked Write 8 Sent Done Done false
8191 Acked Read 1 Sent Done Done false
8192 Acked Read 9 Sent Done Done false
8193 Acked Read 4 Sent Done Done false
8194 Acked Write 5 Sent Done Done false
8195 Acked Write 9 Sent Done Done false
8196 Acked Write 8 Sent Done Done false
8197 Acked Read 2 Sent Done Done false
8198 Acked Write 1 Sent Done Done false
8199 Acked Read 10 Sent Done Done false
8200 Acked Write 10 Sent Done Done false
8201 Acked Write 1 Sent Done Done false
8202 Acked Flush 0 Sent Done Done false
8203 Acked Read 1 Sent Done Done false
8204 Acked Write 9 Sent Done Done false
8205 Acked Read 3 Sent Done Done false
8206 Acked Read 3 Sent Done Done false
8207 Acked Write 8 Sent Done Done false
8208 Acked Read 3 Sent Done Done false
8209 Acked Flush 0 Sent Done Done false
8210 Acked Write 1 Sent Done Done false
8211 Acked Write 2 Sent Done Done false
8212 Acked Read 9 Sent Done Done false
8213 Acked Read 10 Sent Done Done false
8214 Acked Write 8 Sent Done Done false
8215 Acked Write 8 Sent Done Done false
8216 Acked Write 7 Sent Done Done false
8217 Acked Write 2 Sent Done Done false
8218 Acked Write 5 Sent Done Done false
8219 Acked Write 2 Sent Done Done false
8220 Acked Write 10 Sent Done Done false
8221 Acked Write 1 Sent Done Done false
8222 Acked Read 1 Sent Done Done false
8223 Acked Read 1 Sent Done Done false
8224 Acked Read 6 Sent Done Done false
8225 Acked Read 10 Sent Done Done false
8226 Acked Read 6 Sent Done Done false
8227 Acked Write 1 Sent Done Done false
8228 Acked Read 10 Sent Done Done false
8229 Acked Flush 0 Sent Done Done false
8230 Acked Read 2 Sent Done Done false
8231 Acked Read 7 Sent Done Done false
8232 Acked Flush 0 Sent Done Done false
8233 Acked Flush 0 Sent Done Done false
8234 Acked Read 10 Sent Done Done false
8235 Acked Read 4 Sent Done Done false
8236 Acked Read 9 Sent Done Done false
8237 Acked Write 1 Sent Done Done false
8238 Acked Flush 0 Sent Done Done false
8239 Acked Flush 0 Sent Done Done false
8240 Acked Read 2 Sent Done Done false
8241 Acked Flush 0 Sent Done Done false
8242 Acked Read 4 Sent Done Done false
8243 Acked Write 4 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
Downstairs last five completed: 7942 7941 7940 7939 7938
Upstairs last five completed: 8243 8242 8241 8240 8239
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:30.39103878Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:30.992920875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1235)
[8810][0] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:31.994800128Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:33.147927103Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7943 Acked Flush 0 Sent Done Done false
7944 Acked Write 5 Sent Done Done false
7945 Acked Write 6 Sent Done Done false
7946 Acked Read 6 Sent Done Done false
7947 Acked Write 4 Sent Done Done false
7948 Acked Read 5 Sent Done Done false
7949 Acked Write 7 Sent Done Done false
7950 Acked Write 7 Sent Done Done false
7951 Acked Read 5 Sent Done Done false
7952 Acked Read 4 Sent Done Done false
7953 Acked Read 9 Sent Done Done false
7954 Acked Read 5 Sent Done Done false
7955 Acked Read 1 Sent Done Done false
7956 Acked Read 6 Sent Done Done false
7957 Acked Read 10 Sent Done Done false
7958 Acked Read 6 Sent Done Done false
7959 Acked Write 4 Sent Done Done false
7960 Acked Flush 0 Sent Done Done false
7961 Acked Write 9 Sent Done Done false
7962 Acked Read 7 Sent Done Done false
7963 Acked Read 1 Sent Done Done false
7964 Acked Flush 0 Sent Done Done false
7965 Acked Write 1 Sent Done Done false
7966 Acked Read 8 Sent Done Done false
7967 Acked Write 7 Sent Done Done false
7968 Acked Read 5 Sent Done Done false
7969 Acked Read 6 Sent Done Done false
7970 Acked Read 8 Sent Done Done false
7971 Acked Flush 0 Sent Done Done false
7972 Acked Write 3 Sent Done Done false
7973 Acked Read 7 Sent Done Done false
7974 Acked Read 4 Sent Done Done false
7975 Acked Read 10 Sent Done Done false
7976 Acked Read 5 Sent Done Done false
7977 Acked Write 4 Sent Done Done false
7978 Acked Write 3 Sent Done Done false
7979 Acked Read 4 Sent Done Done false
7980 Acked Flush 0 Sent Done Done false
7981 Acked Write 3 Sent Done Done false
7982 Acked Write 6 Sent Done Done false
7983 Acked Write 6 Sent Done Done false
7984 Acked Read 3 Sent Done Done false
7985 Acked Read 10 Sent Done Done false
7986 Acked Read 7 Sent Done Done false
7987 Acked Write 1 Sent Done Done false
7988 Acked Flush 0 Sent Done Done false
7989 Acked Read 1 Sent Done Done false
7990 Acked Read 1 Sent Done Done false
7991 Acked Read 7 Sent Done Done false
7992 Acked Write 2 Sent Done Done false
7993 Acked Read 3 Sent Done Done false
7994 Acked Read 9 Sent Done Done false
7995 Acked Flush 0 Sent Done Done false
7996 Acked Flush 0 Sent Done Done false
7997 Acked Read 9 Sent Done Done false
7998 Acked Read 3 Sent Done Done false
7999 Acked Read 10 Sent Done Done false
8000 Acked Write 2 Sent Done Done false
8001 Acked Flush 0 Sent Done Done false
8002 Acked Read 3 Sent Done Done false
8003 Acked Read 1 Sent Done Done false
8004 Acked Read 1 Sent Done Done false
8005 Acked Read 7 Sent Done Done false
8006 Acked Flush 0 Sent Done Done false
8007 Acked Read 8 Sent Done Done false
8008 Acked Read 8 Sent Done Done false
8009 Acked Read 8 Sent Done Done false
8010 Acked Write 7 Sent Done Done false
8011 Acked Write 7 Sent Done Done false
8012 Acked Read 1 Sent Done Done false
8013 Acked Read 9 Sent Done Done false
8014 Acked Read 1 Sent Done Done false
8015 Acked Read 9 Sent Done Done false
8016 Acked Read 10 Sent Done Done false
8017 Acked Write 1 Sent Done Done false
8018 Acked Read 5 Sent Done Done false
8019 Acked Read 9 Sent Done Done false
8020 Acked Write 4 Sent Done Done false
8021 Acked Read 10 Sent Done Done false
8022 Acked Read 7 Sent Done Done false
8023 Acked Write 4 Sent Done Done false
8024 Acked Write 5 Sent Done Done false
8025 Acked Write 5 Sent Done Done false
8026 Acked Read 7 Sent Done Done false
8027 Acked Read 5 Sent Done Done false
8028 Acked Read 3 Sent Done Done false
8029 Acked Write 5 Sent Done Done false
8030 Acked Write 7 Sent Done Done false
8031 Acked Read 10 Sent Done Done false
8032 Acked Read 7 Sent Done Done false
8033 Acked Write 8 Sent Done Done false
8034 Acked Read 3 Sent Done Done false
8035 Acked Read 5 Sent Done Done false
8036 Acked Write 6 Sent Done Done false
8037 Acked Read 7 Sent Done Done false
8038 Acked Read 7 Sent Done Done false
8039 Acked Write 5 Sent Done Done false
8040 Acked Read 6 Sent Done Done false
8041 Acked Read 2 Sent Done Done false
8042 Acked Read 9 Sent Done Done false
8043 Acked Flush 0 Sent Done Done false
8044 Acked Read 2 Sent Done Done false
8045 Acked Write 8 Sent Done Done false
8046 Acked Write 1 Sent Done Done false
8047 Acked Write 9 Sent Done Done false
8048 Acked Read 2 Sent Done Done false
8049 Acked Write 7 Sent Done Done false
8050 Acked Read 6 Sent Done Done false
8051 Acked Read 6 Sent Done Done false
8052 Acked Flush 0 Sent Done Done false
8053 Acked Write 5 Sent Done Done false
8054 Acked Write 10 Sent Done Done false
8055 Acked Flush 0 Sent Done Done false
8056 Acked Read 10 Sent Done Done false
8057 Acked Write 6 Sent Done Done false
8058 Acked Write 10 Sent Done Done false
8059 Acked Read 2 Sent Done Done false
8060 Acked Write 6 Sent Done Done false
8061 Acked Read 9 Sent Done Done false
8062 Acked Read 7 Sent Done Done false
8063 Acked Write 8 Sent Done Done false
8064 Acked Write 4 Sent Done Done false
8065 Acked Write 9 Sent Done Done false
8066 Acked Write 1 Sent Done Done false
8067 Acked Read 10 Sent Done Done false
8068 Acked Read 5 Sent Done Done false
8069 Acked Write 10 Sent Done Done false
8070 Acked Read 5 Sent Done Done false
8071 Acked Write 6 Sent Done Done false
8072 Acked Flush 0 Sent Done Done false
8073 Acked Write 3 Sent Done Done false
8074 Acked Read 1 Sent Done Done false
8075 Acked Flush 0 Sent Done Done false
8076 Acked Read 9 Sent Done Done false
8077 Acked Write 6 Sent Done Done false
8078 Acked Write 2 Sent Done Done false
8079 Acked Flush 0 Sent Done Done false
8080 Acked Read 5 Sent Done Done false
8081 Acked Read 8 Sent Done Done false
8082 Acked Read 6 Sent Done Done false
8083 Acked Flush 0 Sent Done Done false
8084 Acked Read 2 Sent Done Done false
8085 Acked Write 5 Sent Done Done false
8086 Acked Read 2 Sent Done Done false
8087 Acked Read 5 Sent Done Done false
8088 Acked Read 7 Sent Done Done false
8089 Acked Write 1 Sent Done Done false
8090 Acked Read 4 Sent Done Done false
8091 Acked Read 3 Sent Done Done false
8092 Acked Read 9 Sent Done Done false
8093 Acked Write 2 Sent Done Done false
8094 Acked Read 3 Sent Done Done false
8095 Acked Read 4 Sent Done Done false
8096 Acked Write 7 Sent Done Done false
8097 Acked Read 8 Sent Done Done false
8098 Acked Read 4 Sent Done Done false
8099 Acked Read 1 Sent Done Done false
8100 Acked Read 10 Sent Done Done false
8101 Acked Read 3 Sent Done Done false
8102 Acked Flush 0 Sent Done Done false
8103 Acked Read 7 Sent Done Done false
8104 Acked Read 2 Sent Done Done false
8105 Acked Write 1 Sent Done Done false
8106 Acked Read 2 Sent Done Done false
8107 Acked Read 1 Sent Done Done false
8108 Acked Write 6 Sent Done Done false
8109 Acked Read 4 Sent Done Done false
8110 Acked Read 10 Sent Done Done false
8111 Acked Read 8 Sent Done Done false
8112 Acked Flush 0 Sent Done Done false
8113 Acked Write 5 Sent Done Done false
8114 Acked Read 5 Sent Done Done false
8115 Acked Write 4 Sent Done Done false
8116 Acked Write 1 Sent Done Done false
8117 Acked Read 6 Sent Done Done false
8118 Acked Write 4 Sent Done Done false
8119 Acked Read 10 Sent Done Done false
8120 Acked Read 9 Sent Done Done false
8121 Acked Write 5 Sent Done Done false
8122 Acked Read 5 Sent Done Done false
8123 Acked Read 2 Sent Done Done false
8124 Acked Flush 0 Sent Done Done false
8125 Acked Write 3 Sent Done Done false
8126 Acked Flush 0 Sent Done Done false
8127 Acked Read 4 Sent Done Done false
8128 Acked Write 9 Sent Done Done false
8129 Acked Write 7 Sent Done Done false
8130 Acked Read 9 Sent Done Done false
8131 Acked Write 6 Sent Done Done false
8132 Acked Read 5 Sent Done Done false
8133 Acked Write 7 Sent Done Done false
8134 Acked Flush 0 Sent Done Done false
8135 Acked Write 2 Sent Done Done false
8136 Acked Read 7 Sent Done Done false
8137 Acked Write 4 Sent Done Done false
8138 Acked Write 3 Sent Done Done false
8139 Acked Flush 0 Sent Done Done false
8140 Acked Read 2 Sent Done Done false
8141 Acked Read 4 Sent Done Done false
8142 Acked Write 6 Sent Done Done false
8143 Acked Read 6 Sent Done Done false
8144 Acked Read 2 Sent Done Done false
8145 Acked Read 3 Sent Done Done false
8146 Acked Write 2 Sent Done Done false
8147 Acked Flush 0 Sent Done Done false
8148 Acked Read 3 Sent Done Done false
8149 Acked Write 9 Sent Done Done false
8150 Acked Write 4 Sent Done Done false
8151 Acked Flush 0 Sent Done Done false
8152 Acked Write 7 Sent Done Done false
8153 Acked Read 6 Sent Done Done false
8154 Acked Read 8 Sent Done Done false
8155 Acked Write 1 Sent Done Done false
8156 Acked Read 1 Sent Done Done false
8157 Acked Read 7 Sent Done Done false
8158 Acked Write 2 Sent Done Done false
8159 Acked Read 2 Sent Done Done false
8160 Acked Write 1 Sent Done Done false
8161 Acked Read 10 Sent Done Done false
8162 Acked Read 5 Sent Done Done false
8163 Acked Read 4 Sent Done Done false
8164 Acked Write 8 Sent Done Done false
8165 Acked Write 2 Sent Done Done false
8166 Acked Read 9 Sent Done Done false
8167 Acked Write 10 Sent Done Done false
8168 Acked Write 5 Sent Done Done false
8169 Acked Write 7 Sent Done Done false
8170 Acked Write 1 Sent Done Done false
8171 Acked Write 8 Sent Done Done false
8172 Acked Read 2 Sent Done Done false
8173 Acked Write 6 Sent Done Done false
8174 Acked Flush 0 Sent Done Done false
8175 Acked Read 2 Sent Done Done false
8176 Acked Read 3 Sent Done Done false
8177 Acked Write 2 Sent Done Done false
8178 Acked Flush 0 Sent Done Done false
8179 Acked Flush 0 Sent Done Done false
8180 Acked Read 5 Sent Done Done false
8181 Acked Read 9 Sent Done Done false
8182 Acked Read 4 Sent Done Done false
8183 Acked Write 4 Sent Done Done false
8184 Acked Flush 0 Sent Done Done false
8185 Acked Read 10 Sent Done Done false
8186 Acked Write 10 Sent Done Done false
8187 Acked Read 6 Sent Done Done false
8188 Acked Read 5 Sent Done Done false
8189 Acked Read 1 Sent Done Done false
8190 Acked Write 8 Sent Done Done false
8191 Acked Read 1 Sent Done Done false
8192 Acked Read 9 Sent Done Done false
8193 Acked Read 4 Sent Done Done false
8194 Acked Write 5 Sent Done Done false
8195 Acked Write 9 Sent Done Done false
8196 Acked Write 8 Sent Done Done false
8197 Acked Read 2 Sent Done Done false
8198 Acked Write 1 Sent Done Done false
8199 Acked Read 10 Sent Done Done false
8200 Acked Write 10 Sent Done Done false
8201 Acked Write 1 Sent Done Done false
8202 Acked Flush 0 Sent Done Done false
8203 Acked Read 1 Sent Done Done false
8204 Acked Write 9 Sent Done Done false
8205 Acked Read 3 Sent Done Done false
8206 Acked Read 3 Sent Done Done false
8207 Acked Write 8 Sent Done Done false
8208 Acked Read 3 Sent Done Done false
8209 Acked Flush 0 Sent Done Done false
8210 Acked Write 1 Sent Done Done false
8211 Acked Write 2 Sent Done Done false
8212 Acked Read 9 Sent Done Done false
8213 Acked Read 10 Sent Done Done false
8214 Acked Write 8 Sent Done Done false
8215 Acked Write 8 Sent Done Done false
8216 Acked Write 7 Sent Done Done false
8217 Acked Write 2 Sent Done Done false
8218 Acked Write 5 Sent Done Done false
8219 Acked Write 2 Sent Done Done false
8220 Acked Write 10 Sent Done Done false
8221 Acked Write 1 Sent Done Done false
8222 Acked Read 1 Sent Done Done false
8223 Acked Read 1 Sent Done Done false
8224 Acked Read 6 Sent Done Done false
8225 Acked Read 10 Sent Done Done false
8226 Acked Read 6 Sent Done Done false
8227 Acked Write 1 Sent Done Done false
8228 Acked Read 10 Sent Done Done false
8229 Acked Flush 0 Sent Done Done false
8230 Acked Read 2 Sent Done Done false
8231 Acked Read 7 Sent Done Done false
8232 Acked Flush 0 Sent Done Done false
8233 Acked Flush 0 Sent Done Done false
8234 Acked Read 10 Sent Done Done false
8235 Acked Read 4 Sent Done Done false
8236 Acked Read 9 Sent Done Done false
8237 Acked Write 1 Sent Done Done false
8238 Acked Flush 0 Sent Done Done false
8239 Acked Flush 0 Sent Done Done false
8240 Acked Read 2 Sent Done Done false
8241 Acked Flush 0 Sent Done Done false
8242 Acked Read 4 Sent Done Done false
8243 Acked Write 4 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
Downstairs last five completed: 7942 7941 7940 7939 7938
Upstairs last five completed: 8243 8242 8241 8240 8239
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:34.999979456Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.648210814Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46837)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.648336984Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.649284566Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.649822508Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.649841283Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(7942))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.64985479Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.649867797Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(7942))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.649880563Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(7942))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:35.650123515Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8244 8243 8242 8241 8240
Upstairs last five completed: 8244 8243 8242 8241 8240
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 28 06:07:37.894[8830] Got stop action so:false kr:false
INFO request completed, latency_us: 331, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 84abda63-d63c-452d-ae90-b9bfdb3ec74f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:37.895249996Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:37.895303813Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:37.895323518Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:07:37.895 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/2, method: GET, req_id{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:37.895419695Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
: ac171e59-e9a4-420d-92f3-682344d867a7, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:07:41.897 INFO request completed, latency_us: 315, response_code: 200, uri: /state/cid/2, method: GET, req_id: f9ca2c7e-c992-4c2a-b04c-ac39c4901a7f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 3220 len 28672
002/300 Read block 3935 len 8192
003/300 Write block 3657 len 36864 data: 3 2 2 2 2 5 4 4 4
004/300 Flush
005/300 Read block 1581 len 24576
006/300 Read block 3288 len 24576
007/300 Read block 321 len 32768
008/300 Flush
009/300 Write block 4738 len 12288 data: 6 6 5
010/300 Write block 4639 len 8192 data: 5 4
011/300 Flush
012/300 Read block 4677 len 36864
013/300 Write block 5 len 28672 data: 6 5 4 4 4 5 4
014/300 Read block 2394 len 40960
015/300 Write block 2537 len 12288 data: 5 4 4
016/300 Write block 2259 len 12288 data: 7 7 5
017/300 Read block 1334 len 28672
018/300 Read block 3031 len 32768
019/300 Read block 592 len 40960
020/300 Read block 4350 len 12288
021/300 Write block 4740 len 20480 data: 6 5 4 3 4
022/300 Read block 1269 len 8192
023/300 Write block 2066 len 28672 data: 6 6 7 5 4 5 5
024/300 Write block 1974 len 16384 data: 8 8 5 3
025/300 Read block 1070 len 20480
026/300 Write block 3903 len 24576 data: 6 6 5 4 5 4
027/300 Read block 2706 len 28672
028/300 Write block 4958 len 4096 data: 9
029/300 Write block 171 len 16384 data: 3 4 4 5
030/300 Write block 2123 len 16384 data: 6 7 6 6
031/300 Read block 4565 len 8192
032/300 Write block 3486 len 4096 data: 6
033/300 Read block 308 len 24576
034/300 Write block 479 len 16384 data: 6 7 7 7
035/300 Read block 3175 len 40960
036/300 Write block 2303 len 4096 data: 4
037/300 Read block 2781 len 20480
038/300 Read block 3358 len 24576
039/300 Write block 1481 len 40960 data: 4 4 5 5 6 6 6 7 8 9
040/300 Read block 4372 len 20480
041/300 Write block 283 len 4096 data: 5
042/300 Write block 1563 len 36864 data: 2 3 4 6 5 6 6 5 5
043/300 Read block 1057 len 16384
044/300 Write block 4767 len 40960 data: 4 5 6 9 8 7 6 5 6 7
045/300 Read block 2983 len 8192
046/300 Read block 1969 len 8192
047/300 Read block 3832 len 24576
048/300 Flush
049/300 Read block 4484 len 4096
050/300 Read block 1909 len 12288
051/300 Flush
052/300 Write block 3593 len 12288 data: 6 5 5
053/300 Write block 4637 len 24576 data: 5 5 6 5 3 4
054/300 Read block 41 len 12288
055/300 Read block 3571 len 40960
056/300 Read block 1523 len 36864
057/300 Read block 4189 len 24576
058/300 Write block 2261 len 28672 data: 6 6 6 7 5 5 7
059/300 Write block 4058 len 12288 data: 4 4 4
060/300 Read block 1239 len 32768
061/300 Write block 4542 len 24576 data: 4 5 6 5 6 6
062/300 Read block 4478 len 16384
063/300 Read block 115 len 36864
064/300 Flush
065/300 Write block 4968 len 20480 data: 9 8 7 8 7
066/300 Write block 3026 len 24576 data: 4 5 5 5 5 5
067/300 Write block 2285 len 16384 data: 5 4 2 3
068/300 Read block 4315 len 16384
069/300 Write block 627 len 32768 data: 5 3 3 3 2 3 3 3
070/300 Flush
071/300 Write block 2101 len 20480 data: 5 5 2 2 2
072/300 Write block 2901 len 20480 data: 4 4 4 4 5
073/300 Read block 3904 len 20480
074/300 Write block 3233 len 4096 data: 2
075/300 Read block 3396 len 40960
076/300 Flush
077/300 Read block 3283 len 28672
078/300 Write block 2986 len 32768 data: 6 6 6 6 8 8 9 6
079/300 Read block 881 len 16384
080/300 Write block 1524 len 16384 data: 5 5 5 6
081/300 Read block 3646 len 12288
082/300 Read block 3103 len 32768
083/300 Read block 2160 len 16384
084/300 Read block 1152 len 12288
085/300 Write block 1802 len 20480 data: 6 4 3 3 3
086/300 Write block 4785 len 16384 data: 5 5 4 5
087/300 Read block 814 len 32768
088/300 Read block 1580 len 12288
089/300 Write block 405 len 12288 data: 4 5 5
090/300 Write block 1560 len 28672 data: 4 5 3 3 4 5 7
091/300 Read block 4080 len 36864
092/300 Read block 2326 len 36864
093/300 Read block 4120 len 8192
094/300 Flush
095/300 Read block 4324 len 36864
096/300 Read block 4326 len 36864
097/300 Write block 2139 len 4096 data: 5
098/300 Write block 4680 len 28672 data: 6 5 4 5 6 6 6
099/300 Read block 1976 len 32768
100/300 Write block 108 len 24576 data: 4 3 3 4 4 3
101/300 Write block 1164 len 40960 data: 8 7 7 7 6 8 7 7 7 10
102/300 Read block 2183 len 40960
103/300 Write block 3678 len 36864 data: 4 4 4 4 5 5 5 5 5
104/300 Write block 3532 len 12288 data: 8 8 7
105/300 Read block 458 len 36864
106/300 Write block 1832 len 20480 data: 4 3 3 4 6
107/300 Flush
108/300 Write block 2058 len 20480 data: 5 4 4 5 4
109/300 Read block 1815 len 28672
110/300 Read block 4569 len 32768
111/300 Write block 2870 len 36864 data: 5 6 6 5 5 3 4 4 5
112/300 Read block 4881 len 32768
113/300 Read block 1530 len 40960
114/300 Write block 918 len 40960 data: 6 6 6 6 6 5 5 5 6 6
115/300 Read block 2551 len 24576
116/300 Read block 54 len 8192
117/300 Read block 1656 len 28672
118/300 Write block 3755 len 36864 data: 3 3 3 3 3 4 4 6 5
119/300 Read block 31 len 40960
120/300 Read block 1545 len 4096
121/300 Read block 2723 len 20480
122/300 Read block 1512 len 8192
123/300 Write block 3321 len 8192 data: 5 6
124/300 Write block 208 len 40960 data: 7 7 7 7 5 5 7 8 6 7
125/300 Write block 4064 len 32768 data: 5 3 6 5 5 5 5 5
126/300 Read block 3675 len 4096
127/300 Read block 1249 len 4096
128/300 Write block 2536 len 12288 data: 5 6 5
129/300 Read block 4973 len 24576
130/300 Read block 2749 len 28672
131/300 Read block 565 len 16384
132/300 Flush
133/300 Read block 2063 len 32768
134/300 Read block 1259 len 20480
135/300 Read block 1564 len 40960
136/300 Write block 866 len 24576 data: 4 4 3 3 6 7
137/300 Read block 2075 len 8192
138/300 Read block 3315 len 12288
139/300 Write block 4624 len 40960 data: 5 5 6 8 8 8 8 9 9 9
140/300 Write block 3512 len 16384 data: 3 3 3 5
141/300 Flush
142/300 Read block 2332 len 32768
143/300 Read block 3895 len 20480
144/300 Write block 3994 len 28672 data: 6 5 6 5 5 5 4
145/300 Read block 2116 len 36864
146/300 Read block 682 len 16384
147/300 Read block 1020 len 40960
148/300 Flush
149/300 Write block 3359 len 4096 data: 3
150/300 Write block 3325 len 4096 data: 7
151/300 Read block 4472 len 16384
152/300 Write block 2899 len 4096 data: 5
153/300 Read block 2446 len 32768
154/300 Write block 3697 len 24576 data: 3 3 4 3 3 4
155/300 Write block 1942 len 36864 data: 6 5 5 4 5 6 5 3 3
156/300 Write block 4015 len 36864 data: 2 2 2 4 4 4 4 4 4
157/300 Write block 843 len 8192 data: 4 6
158/300 Read block 1024 len 40960
159/300 Flush
160/300 Write block 907 len 32768 data: 5 6 3 4 5 6 6 5
161/300 Read block 2481 len 32768
162/300 Read block 2998 len 8192
163/300 Write block 4793 len 28672 data: 4 5 5 5 5 5 3
164/300 Read block 2066 len 24576
165/300 Read block 2991 len 36864
166/300 Read block 3766 len 40960
167/300 Read block 2745 len 40960
168/300 Write block 3550 len 40960 data: 5 5 3 3 3 2 3 4 4 6
169/300 Write block 1622 len 4096 data: 3
170/300 Read block 227 len 24576
171/300 Write block 3263 len 8192 data: 2 2
172/300 Read block 4669 len 4096
173/300 Read block 849 len 36864
174/300 Read block 416 len 24576
175/300 Write block 2805 len 4096 data: 6
176/300 Write block 516 len 24576 data: 5 5 5 5 6 7
177/300 Read block 4259 len 16384
178/300 Read block 3695 len 32768
179/300 Read block 3163 len 12288
180/300 Write block 9 len 16384 data: 5 6 5 4
181/300 Write block 4070 len 24576 data: 6 6 4 3 3 4
182/300 Read block 4824 len 40960
183/300 Write block 4320 len 32768 data: 7 8 7 8 7 6 5 6
184/300 Read block 4185 len 8192
185/300 Write block 37 len 40960 data: 4 4 4 4 5 5 3 3 3 3
186/300 Write block 1286 len 32768 data: 4 5 5 6 6 7 6 7
187/300 Read block 4644 len 24576
188/300 Write block 1954 len 20480 data: 4 5 4 4 4
189/300 Read block 3790 len 12288
190/300 Read block 1380 len 12288
191/300 Flush
192/300 Read block 1674 len 20480
193/300 Read block 2436 len 32768
194/300 Flush
195/300 Write block 3197 len 12288 data: 4 4 6
196/300 Flush
197/300 Write block 594 len 20480 data: 4 4 4 4 5
198/300 Read block 2549 len 8192
199/300 Read block 1227 len 24576
200/300 Flush
201/300 Read block 4909 len 16384
202/300 Write block 1426 len 8192 data: 6 6
203/300 Read block 257 len 36864
204/300 Read block 597 len 28672
205/300 Flush
206/300 Read block 4460 len 28672
207/300 Flush
208/300 Write block 1130 len 36864 data: 6 7 6 5 7 7 8 8 7
209/300 Write block 2580 len 28672 data: 6 6 6 6 7 6 5
210/300 Read block 1913 len 8192
211/300 Flush
212/300 Read block 275 len 8192
213/300 Read block 3723 len 28672
214/300 Flush
215/300 Read block 1497 len 20480
216/300 Read block 4035 len 12288
217/300 Write block 620 len 28672 data: 5 5 5 6 6 7 5
218/300 Read block 4890 len 40960
219/300 Read block 685 len 32768
220/300 Read block 156 len 12288
221/300 Write block 640 len 36864 data: 2 5 5 5 6 6 5 6 7
222/300 Read block 1580 len 12288
223/300 Read block 4858 len 32768
224/300 Read block 4931 len 4096
225/300 Write block 3771 len 20480 data: 7 7 8 7 5
226/300 Write block 3746 len 36864 data: 7 7 5 6 4 5 5 4 3
227/300 Read block 715 len 4096
228/300 Read block 1148 len 12288
229/300 Write block 2931 len 32768 data: 6 5 5 5 3 3 5 6
230/300 Write block 3035 len 36864 data: 7 7 7 6 6 4 4 3 3
231/300 Flush
232/300 Read block 577 len 24576
233/300 Write block 4507 len 8192 data: 8 6
234/300 Read block 823 len 32768
235/300 Write block 2751 len 24576 data: 8 5 4 4 5 5
236/300 Read block 1894 len 12288
237/300 Write block 744 len 20480 data: 9 8 8 7 9
238/300 Write block 1226 len 32768 data: 5 6 5 5 6 3 2 2
239/300 Flush
240/300 Read block 818 len 20480
241/300 Read block 4471 len 12288
242/300 Read block 3375 len 36864
243/300 Read block 3296 len 16384
244/300 Write block 2248 len 8192 data: 2 2
245/300 Read block 1535 len 20480
246/300 Write block 4754 len 32768 data: 4 5 9 10 9 9 7 6
247/300 Read block 661 len 28672
248/300 Read block 818 len 8192
249/300 Write block 3279 len 4096 data: 4
250/300 Write block 4232 len 36864 data: 6 6 5 5 5 5 3 3 3
251/300 Write block 1147 len 24576 data: 5 6 6 5 8 9
252/300 Flush
253/300 Write block 3611 len 16384 data: 7 7 6 7
254/300 Read block 2200 len 12288
255/300 Read block 4580 len 8192
256/300 Read block 3138 len 40960
257/300 Write block 4887 len 40960 data: 13 13 13 12 11 9 9 9 9 8
258/300 Read block 2224 len 4096
259/300 Write block 4672 len 32768 data: 3 4 4 5 4 5 6 6
260/300 Write block 2663 len 36864 data: 6 7 7 7 8 8 10 11 8
261/300 Read block 3291 len 20480
262/300 Read block 969 len 36864
263/300 Flush
264/300 Read block 2187 len 4096
265/300 Write block 4996 len 12288 data: 4 4 4
266/300 Write block 583 len 40960 data: 8 7 8 9 7 8 6 6 4 4
267/300 Read block 1782 len 20480
268/300 Read block 2682 len 20480
269/300 Read block 4660 len 28672
270/300 Read block 2366 len 20480
271/300 Write block 2306 len 12288 data: 5 5 5
272/300 Read block 520 len 24576
273/300 Read block 3052 len 8192
274/300 Read block 2267 len 40960
275/300 Write block 1155 len 12288 data: 9 8 8
276/300 Read block 1235 len 28672
277/300 Read block 1414 len 32768
278/300 Flush
279/300 Read block 4744 len 28672
280/300 Read block 2123 len 8192
281/300 Read block 4337 len 4096
282/300 Flush
283/300 Read block 1086 len 40960
284/300 Read block 2978 len 24576
285/300 Flush
286/300 Read block 3851 len 36864
287/300 Read block 2651 len 20480
288/300 Write block 947 len 4096 data: 5
289/300 Read block 4978 len 16384
290/300 Write block 79 len 24576 data: 8 7 8 6 6 4
291/300 Read block 4437 len 24576
292/300 Write block 20 len 40960 data: 6 6 6 5 5 7 5 3 3 2
293/300 Read block 4317 len 28672
294/300 Read block 4761 len 4096
295/300 Read block 2384 len 32768
296/300 Write block 288 len 32768 data: 4 4 5 5 6 5 4 6
297/300 Write block 3759 len 24576 data: 4 5 5 7 6 4
298/300 Flush
299/300 Write block 2392 len 28672 data: 4 5 4 3 2 2 2
300/300 Write block 26 len 4096 data: 6
301/300 Read block 2772 len 20480
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:07:42.151 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 667a4671-cae3-41ff-bc0b-6b75440a99fb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8245 Acked Read 7 Done Done Sent false
8246 Acked Read 2 Done Done Sent false
8247 Acked Write 9 Done Done Sent false
8248 Acked Flush 0 Done Done Sent false
8249 Acked Read 6 Done Done Sent false
8250 Acked Read 6 Done Done Sent false
8251 Acked Read 8 Done Done Sent false
8252 Acked Flush 0 Done Done Sent false
8253 Acked Write 3 Done Done Sent false
8254 Acked Write 2 Done Done Sent false
8255 Acked Flush 0 Done Done Sent false
8256 Acked Read 9 Done Done Sent false
8257 Acked Write 7 Done Done Sent false
8258 Acked Read 10 Done Done Sent false
8259 Acked Write 3 Done Done Sent false
8260 Acked Write 3 Done Done Sent false
8261 Acked Read 7 Done Done Sent false
8262 Acked Read 8 Done Done Sent false
8263 Acked Read 10 Done Done Sent false
8264 Acked Read 3 Done Done Sent false
8265 Acked Write 5 Done Done Sent false
8266 Acked Read 2 Done Done Sent false
8267 Acked Write 7 Done Done Sent false
8268 Acked Write 4 Done Done Sent false
8269 Acked Read 5 Done Done Sent false
8270 Acked Write 6 Done Done Sent false
8271 Acked Read 7 Done Done Sent false
8272 Acked Write 1 Done Done Sent false
8273 Acked Write 4 Done Done Sent false
8274 Acked Write 4 Done Done Sent false
8275 Acked Read 2 Done Done Sent false
8276 Acked Write 1 Done Done Sent false
8277 Acked Read 6 Done Done Sent false
8278 Acked Write 4 Done Done Sent false
8279 Acked Read 10 Done Done Sent false
8280 Acked Write 1 Done Done Sent false
8281 Acked Read 5 Done Done Sent false
8282 Acked Read 6 Done Done Sent false
8283 Acked Write 10 Done Done Sent false
8284 Acked Read 5 Done Done Sent false
8285 Acked Write 1 Done Done Sent false
8286 Acked Write 9 Done Done Sent false
8287 Acked Read 4 Done Done Sent false
8288 Acked Write 10 Done Done Sent false
8289 Acked Read 2 Done Done Sent false
8290 Acked Read 2 Done Done Sent false
8291 Acked Read 6 Done Done Sent false
8292 Acked Flush 0 Done Done Sent false
8293 Acked Read 1 Done Done Sent false
8294 Acked Read 3 Done Done Sent false
8295 Acked Flush 0 Done Done Sent false
8296 Acked Write 3 Done Done Sent false
8297 Acked Write 6 Done Done Sent false
8298 Acked Read 3 Done Done Sent false
8299 Acked Read 10 Done Done Sent false
8300 Acked Read 9 Done Done Sent false
8301 Acked Read 6 Done Done Sent false
8302 Acked Write 7 Done Done Sent false
8303 Acked Write 3 Done Done Sent false
8304 Acked Read 8 Done Done Sent false
8305 Acked Write 6 Done Done Sent false
8306 Acked Read 4 Done Done Sent false
8307 Acked Read 9 Done Done Sent false
8308 Acked Flush 0 Done Done Sent false
8309 Acked Write 5 Done Done Sent false
8310 Acked Write 6 Done Done Sent false
8311 Acked Write 4 Done Done Sent false
8312 Acked Read 4 Done Done Sent false
8313 Acked Write 8 Done Done Sent false
8314 Acked Flush 0 Done Done Sent false
8315 Acked Write 5 Done Done Sent false
8316 Acked Write 5 Done Done Sent false
8317 Acked Read 5 Done Done Sent false
8318 Acked Write 1 Done Done Sent false
8319 Acked Read 10 Done Done Sent false
8320 Acked Flush 0 Done Done Sent false
8321 Acked Read 7 Done Done Sent false
8322 Acked Write 8 Done Done Sent false
8323 Acked Read 4 Done Done Sent false
8324 Acked Write 4 Done Done Sent false
8325 Acked Read 3 Done Done Sent false
8326 Acked Read 8 Done Done Sent false
8327 Acked Read 4 Done Done Sent false
8328 Acked Read 3 Done Done Sent false
8329 Acked Write 5 Done Done Sent false
8330 Acked Write 4 Done Done Sent false
8331 Acked Read 8 Done Done Sent false
8332 Acked Read 3 Done Done Sent false
8333 Acked Write 3 Done Done Sent false
8334 Acked Write 7 Done Done Sent false
8335 Acked Read 9 Done Done Sent false
8336 Acked Read 9 Done Done Sent false
8337 Acked Read 2 Done Done Sent false
8338 Acked Flush 0 Done Done Sent false
8339 Acked Read 9 Done Done Sent false
8340 Acked Read 9 Done Done Sent false
8341 Acked Write 1 Done Done Sent false
8342 Acked Write 7 Done Done Sent false
8343 Acked Read 8 Done Done Sent false
8344 Acked Write 6 Done Done Sent false
8345 Acked Write 10 Done Done Sent false
8346 Acked Read 10 Done Done Sent false
8347 Acked Write 9 Done Done Sent false
8348 Acked Write 3 Done Done Sent false
8349 Acked Read 9 Done Done Sent false
8350 Acked Write 5 Done Done Sent false
8351 Acked Flush 0 Done Done Sent false
8352 Acked Write 5 Done Done Sent false
8353 Acked Read 7 Done Done Sent false
8354 Acked Read 8 Done Done Sent false
8355 Acked Write 9 Done Done Sent false
8356 Acked Read 8 Done Done Sent false
8357 Acked Read 10 Done Done Sent false
8358 Acked Write 10 Done Done Sent false
8359 Acked Read 6 Done Done Sent false
8360 Acked Read 2 Done Done Sent false
8361 Acked Read 7 Done Done Sent false
8362 Acked Write 9 Done Done Sent false
8363 Acked Read 10 Done Done Sent false
8364 Acked Read 1 Done Done Sent false
8365 Acked Read 5 Done Done Sent false
8366 Acked Read 2 Done Done Sent false
8367 Acked Write 2 Done Done Sent false
8368 Acked Write 10 Done Done Sent false
8369 Acked Write 8 Done Done Sent false
8370 Acked Read 1 Done Done Sent false
8371 Acked Read 1 Done Done Sent false
8372 Acked Write 3 Done Done Sent false
8373 Acked Read 6 Done Done Sent false
8374 Acked Read 7 Done Done Sent false
8375 Acked Read 4 Done Done Sent false
8376 Acked Flush 0 Done Done Sent false
8377 Acked Read 8 Done Done Sent false
8378 Acked Read 5 Done Done Sent false
8379 Acked Read 10 Done Done Sent false
8380 Acked Write 6 Done Done Sent false
8381 Acked Read 2 Done Done Sent false
8382 Acked Read 3 Done Done Sent false
8383 Acked Write 10 Done Done Sent false
8384 Acked Write 4 Done Done Sent false
8385 Acked Flush 0 Done Done Sent false
8386 Acked Read 8 Done Done Sent false
8387 Acked Read 5 Done Done Sent false
8388 Acked Write 7 Done Done Sent false
8389 Acked Read 9 Done Done Sent false
8390 Acked Read 4 Done Done Sent false
8391 Acked Read 10 Done Done Sent false
8392 Acked Flush 0 Done Done Sent false
8393 Acked Write 1 Done Done Sent false
8394 Acked Write 1 Done Done Sent false
8395 Acked Read 4 Done Done Sent false
8396 Acked Write 1 Done Done Sent false
8397 Acked Read 8 Done Done Sent false
8398 Acked Write 6 Done Done Sent false
8399 Acked Write 9 Done Done Sent false
8400 Acked Write 9 Done Done Sent false
8401 Acked Write 2 Done Done Sent false
8402 Acked Read 10 Done Done Sent false
8403 Acked Flush 0 Done Done Sent false
8404 Acked Write 8 Done Done Sent false
8405 Acked Read 8 Done Done Sent false
8406 Acked Read 2 Done Done Sent false
8407 Acked Write 7 Done Done Sent false
8408 Acked Read 6 Done Done Sent false
8409 Acked Read 9 Done Done Sent false
8410 Acked Read 10 Done Done Sent false
8411 Acked Read 10 Done Done Sent false
8412 Acked Write 10 Done Done Sent false
8413 Acked Write 1 Done Done Sent false
8414 Acked Read 6 Done Done Sent false
8415 Acked Write 2 Done Done Sent false
8416 Acked Read 1 Done Done Sent false
8417 Acked Read 9 Done Done Sent false
8418 Acked Read 6 Done Done Sent false
8419 Acked Write 1 Done Done Sent false
8420 Acked Write 6 Done Done Sent false
8421 Acked Read 4 Done Done Sent false
8422 Acked Read 8 Done Done Sent false
8423 Acked Read 3 Done Done Sent false
8424 Acked Write 4 Done Done Sent false
8425 Acked Write 6 Done Done Sent false
8426 Acked Read 10 Done Done Sent false
8427 Acked Write 8 Done Done Sent false
8428 Acked Read 2 Done Done Sent false
8429 Acked Write 10 Done Done Sent false
8430 Acked Write 8 Done Done Sent false
8431 Acked Read 6 Done Done Sent false
8432 Acked Write 5 Done Done Sent false
8433 Acked Read 3 Done Done Sent false
8434 Acked Read 3 Done Done Sent false
8435 Acked Flush 0 Done Done Sent false
8436 Acked Read 5 Done Done Sent false
8437 Acked Read 8 Done Done Sent false
8438 Acked Flush 0 Done Done Sent false
8439 Acked Write 3 Done Done Sent false
8440 Acked Flush 0 Done Done Sent false
8441 Acked Write 5 Done Done Sent false
8442 Acked Read 2 Done Done Sent false
8443 Acked Read 6 Done Done Sent false
8444 Acked Flush 0 Done Done Sent false
8445 Acked Read 4 Done Done Sent false
8446 Acked Write 2 Done Done Sent false
8447 Acked Read 9 Done Done Sent false
8448 Acked Read 7 Done Done Sent false
8449 Acked Flush 0 Done Done Sent false
8450 Acked Read 7 Done Done Sent false
8451 Acked Flush 0 Done Done Sent false
8452 Acked Write 9 Done Done Sent false
8453 Acked Write 7 Done Done Sent false
8454 Acked Read 2 Done Done Sent false
8455 Acked Flush 0 Done Done Sent false
8456 Acked Read 2 Done Done Sent false
8457 Acked Read 7 Done Done Sent false
8458 Acked Flush 0 Done Done Sent false
8459 Acked Read 5 Done Done Sent false
8460 Acked Read 3 Done Done Sent false
8461 Acked Write 7 Done Done Sent false
8462 Acked Read 10 Done Done Sent false
8463 Acked Read 8 Done Done Sent false
8464 Acked Read 3 Done Done Sent false
8465 Acked Write 9 Done Done Sent false
8466 Acked Read 3 Done Done Sent false
8467 Acked Read 8 Done Done Sent false
8468 Acked Read 1 Done Done Sent false
8469 Acked Write 5 Done Done Sent false
8470 Acked Write 9 Done Done Sent false
8471 Acked Read 1 Done Done Sent false
8472 Acked Read 3 Done Done Sent false
8473 Acked Write 8 Done Done Sent false
8474 Acked Write 9 Done Done Sent false
8475 Acked Flush 0 Done Done Sent false
8476 Acked Read 6 Done Done Sent false
8477 Acked Write 2 Done Done Sent false
8478 Acked Read 8 Done Done Sent false
8479 Acked Write 6 Done Done Sent false
8480 Acked Read 3 Done Done Sent false
8481 Acked Write 5 Done Done Sent false
8482 Acked Write 8 Done Done Sent false
8483 Acked Flush 0 Done Done Sent false
8484 Acked Read 5 Done Done Sent false
8485 Acked Read 3 Done Done Sent false
8486 Acked Read 9 Done Done Sent false
8487 Acked Read 4 Done Done Sent false
8488 Acked Write 2 Done Done Sent false
8489 Acked Read 5 Done Done Sent false
8490 Acked Write 8 Done Done Sent false
8491 Acked Read 7 Done Done Sent false
8492 Acked Read 2 Done Done Sent false
8493 Acked Write 1 Done Done Sent false
8494 Acked Write 9 Done Done Sent false
8495 Acked Write 6 Done Done Sent false
8496 Acked Flush 0 Done Done Sent false
8497 Acked Write 4 Done Done Sent false
8498 Acked Read 3 Done Done Sent false
8499 Acked Read 2 Done Done Sent false
8500 Acked Read 10 Done Done Sent false
8501 Acked Write 10 Done Done Sent false
8502 Acked Read 1 Done Done Sent false
8503 Acked Write 8 Done Done Sent false
8504 Acked Write 9 Done Done Sent false
8505 Acked Read 5 Done Done Sent false
8506 Acked Read 9 Done Done Sent false
8507 Acked Flush 0 Done Done Sent false
8508 Acked Read 1 Done Done Sent false
8509 Acked Write 3 Done Done Sent false
8510 Acked Write 10 Done Done Sent false
8511 Acked Read 5 Done Done Sent false
8512 Acked Read 5 Done Done Sent false
8513 Acked Read 7 Done Done Sent false
8514 Acked Read 5 Done Done Sent false
8515 Acked Write 3 Done Done Sent false
8516 Acked Read 6 Done Done Sent false
8517 Acked Read 2 Done Done Sent false
8518 Acked Read 10 Done Done Sent false
8519 Acked Write 3 Done Done Sent false
8520 Acked Read 7 Done Done Sent false
8521 Acked Read 8 Done Done Sent false
8522 Acked Flush 0 Done Done Sent false
8523 Acked Read 7 Done Done Sent false
8524 Acked Read 2 Done Done Sent false
8525 Acked Read 1 Done Done Sent false
8526 Acked Flush 0 Done Done Sent false
8527 Acked Read 10 Done Done Sent false
8528 Acked Read 6 Done Done Sent false
8529 Acked Flush 0 Done Done Sent false
8530 Acked Read 9 Done Done Sent false
8531 Acked Read 5 Done Done Sent false
8532 Acked Write 1 Done Done Sent false
8533 Acked Read 4 Done Done Sent false
8534 Acked Write 6 Done Done Sent false
8535 Acked Read 6 Done Done Sent false
8536 Acked Write 10 Done Done Sent false
8537 Acked Read 7 Done Done Sent false
8538 Acked Read 1 Done Done Sent false
8539 Acked Read 8 Done Done Sent false
8540 Acked Write 8 Done Done Sent false
8541 Acked Write 6 Done Done Sent false
8542 Acked Flush 0 Done Done Sent false
8543 Acked Write 7 Done Done Sent false
8544 Acked Write 1 Done Done Sent false
8545 Acked Read 5 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
Downstairs last five completed: 8244 8243 8242 8241 8240
Upstairs last five completed: 8545 8544 8543 8542 8541
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:43.014796272Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1236)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:44.015864299Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:45.018491893Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:46.151727375Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8245 Acked Read 7 Done Done Sent false
8246 Acked Read 2 Done Done Sent false
8247 Acked Write 9 Done Done Sent false
8248 Acked Flush 0 Done Done Sent false
8249 Acked Read 6 Done Done Sent false
8250 Acked Read 6 Done Done Sent false
8251 Acked Read 8 Done Done Sent false
8252 Acked Flush 0 Done Done Sent false
8253 Acked Write 3 Done Done Sent false
8254 Acked Write 2 Done Done Sent false
8255 Acked Flush 0 Done Done Sent false
8256 Acked Read 9 Done Done Sent false
8257 Acked Write 7 Done Done Sent false
8258 Acked Read 10 Done Done Sent false
8259 Acked Write 3 Done Done Sent false
8260 Acked Write 3 Done Done Sent false
8261 Acked Read 7 Done Done Sent false
8262 Acked Read 8 Done Done Sent false
8263 Acked Read 10 Done Done Sent false
8264 Acked Read 3 Done Done Sent false
8265 Acked Write 5 Done Done Sent false
8266 Acked Read 2 Done Done Sent false
8267 Acked Write 7 Done Done Sent false
8268 Acked Write 4 Done Done Sent false
8269 Acked Read 5 Done Done Sent false
8270 Acked Write 6 Done Done Sent false
8271 Acked Read 7 Done Done Sent false
8272 Acked Write 1 Done Done Sent false
8273 Acked Write 4 Done Done Sent false
8274 Acked Write 4 Done Done Sent false
8275 Acked Read 2 Done Done Sent false
8276 Acked Write 1 Done Done Sent false
8277 Acked Read 6 Done Done Sent false
8278 Acked Write 4 Done Done Sent false
8279 Acked Read 10 Done Done Sent false
8280 Acked Write 1 Done Done Sent false
8281 Acked Read 5 Done Done Sent false
8282 Acked Read 6 Done Done Sent false
8283 Acked Write 10 Done Done Sent false
8284 Acked Read 5 Done Done Sent false
8285 Acked Write 1 Done Done Sent false
8286 Acked Write 9 Done Done Sent false
8287 Acked Read 4 Done Done Sent false
8288 Acked Write 10 Done Done Sent false
8289 Acked Read 2 Done Done Sent false
8290 Acked Read 2 Done Done Sent false
8291 Acked Read 6 Done Done Sent false
8292 Acked Flush 0 Done Done Sent false
8293 Acked Read 1 Done Done Sent false
8294 Acked Read 3 Done Done Sent false
8295 Acked Flush 0 Done Done Sent false
8296 Acked Write 3 Done Done Sent false
8297 Acked Write 6 Done Done Sent false
8298 Acked Read 3 Done Done Sent false
8299 Acked Read 10 Done Done Sent false
8300 Acked Read 9 Done Done Sent false
8301 Acked Read 6 Done Done Sent false
8302 Acked Write 7 Done Done Sent false
8303 Acked Write 3 Done Done Sent false
8304 Acked Read 8 Done Done Sent false
8305 Acked Write 6 Done Done Sent false
8306 Acked Read 4 Done Done Sent false
8307 Acked Read 9 Done Done Sent false
8308 Acked Flush 0 Done Done Sent false
8309 Acked Write 5 Done Done Sent false
8310 Acked Write 6 Done Done Sent false
8311 Acked Write 4 Done Done Sent false
8312 Acked Read 4 Done Done Sent false
8313 Acked Write 8 Done Done Sent false
8314 Acked Flush 0 Done Done Sent false
8315 Acked Write 5 Done Done Sent false
8316 Acked Write 5 Done Done Sent false
8317 Acked Read 5 Done Done Sent false
8318 Acked Write 1 Done Done Sent false
8319 Acked Read 10 Done Done Sent false
8320 Acked Flush 0 Done Done Sent false
8321 Acked Read 7 Done Done Sent false
8322 Acked Write 8 Done Done Sent false
8323 Acked Read 4 Done Done Sent false
8324 Acked Write 4 Done Done Sent false
8325 Acked Read 3 Done Done Sent false
8326 Acked Read 8 Done Done Sent false
8327 Acked Read 4 Done Done Sent false
8328 Acked Read 3 Done Done Sent false
8329 Acked Write 5 Done Done Sent false
8330 Acked Write 4 Done Done Sent false
8331 Acked Read 8 Done Done Sent false
8332 Acked Read 3 Done Done Sent false
8333 Acked Write 3 Done Done Sent false
8334 Acked Write 7 Done Done Sent false
8335 Acked Read 9 Done Done Sent false
8336 Acked Read 9 Done Done Sent false
8337 Acked Read 2 Done Done Sent false
8338 Acked Flush 0 Done Done Sent false
8339 Acked Read 9 Done Done Sent false
8340 Acked Read 9 Done Done Sent false
8341 Acked Write 1 Done Done Sent false
8342 Acked Write 7 Done Done Sent false
8343 Acked Read 8 Done Done Sent false
8344 Acked Write 6 Done Done Sent false
8345 Acked Write 10 Done Done Sent false
8346 Acked Read 10 Done Done Sent false
8347 Acked Write 9 Done Done Sent false
8348 Acked Write 3 Done Done Sent false
8349 Acked Read 9 Done Done Sent false
8350 Acked Write 5 Done Done Sent false
8351 Acked Flush 0 Done Done Sent false
8352 Acked Write 5 Done Done Sent false
8353 Acked Read 7 Done Done Sent false
8354 Acked Read 8 Done Done Sent false
8355 Acked Write 9 Done Done Sent false
8356 Acked Read 8 Done Done Sent false
8357 Acked Read 10 Done Done Sent false
8358 Acked Write 10 Done Done Sent false
8359 Acked Read 6 Done Done Sent false
8360 Acked Read 2 Done Done Sent false
8361 Acked Read 7 Done Done Sent false
8362 Acked Write 9 Done Done Sent false
8363 Acked Read 10 Done Done Sent false
8364 Acked Read 1 Done Done Sent false
8365 Acked Read 5 Done Done Sent false
8366 Acked Read 2 Done Done Sent false
8367 Acked Write 2 Done Done Sent false
8368 Acked Write 10 Done Done Sent false
8369 Acked Write 8 Done Done Sent false
8370 Acked Read 1 Done Done Sent false
8371 Acked Read 1 Done Done Sent false
8372 Acked Write 3 Done Done Sent false
8373 Acked Read 6 Done Done Sent false
8374 Acked Read 7 Done Done Sent false
8375 Acked Read 4 Done Done Sent false
8376 Acked Flush 0 Done Done Sent false
8377 Acked Read 8 Done Done Sent false
8378 Acked Read 5 Done Done Sent false
8379 Acked Read 10 Done Done Sent false
8380 Acked Write 6 Done Done Sent false
8381 Acked Read 2 Done Done Sent false
8382 Acked Read 3 Done Done Sent false
8383 Acked Write 10 Done Done Sent false
8384 Acked Write 4 Done Done Sent false
8385 Acked Flush 0 Done Done Sent false
8386 Acked Read 8 Done Done Sent false
8387 Acked Read 5 Done Done Sent false
8388 Acked Write 7 Done Done Sent false
8389 Acked Read 9 Done Done Sent false
8390 Acked Read 4 Done Done Sent false
8391 Acked Read 10 Done Done Sent false
8392 Acked Flush 0 Done Done Sent false
8393 Acked Write 1 Done Done Sent false
8394 Acked Write 1 Done Done Sent false
8395 Acked Read 4 Done Done Sent false
8396 Acked Write 1 Done Done Sent false
8397 Acked Read 8 Done Done Sent false
8398 Acked Write 6 Done Done Sent false
8399 Acked Write 9 Done Done Sent false
8400 Acked Write 9 Done Done Sent false
8401 Acked Write 2 Done Done Sent false
8402 Acked Read 10 Done Done Sent false
8403 Acked Flush 0 Done Done Sent false
8404 Acked Write 8 Done Done Sent false
8405 Acked Read 8 Done Done Sent false
8406 Acked Read 2 Done Done Sent false
8407 Acked Write 7 Done Done Sent false
8408 Acked Read 6 Done Done Sent false
8409 Acked Read 9 Done Done Sent false
8410 Acked Read 10 Done Done Sent false
8411 Acked Read 10 Done Done Sent false
8412 Acked Write 10 Done Done Sent false
8413 Acked Write 1 Done Done Sent false
8414 Acked Read 6 Done Done Sent false
8415 Acked Write 2 Done Done Sent false
8416 Acked Read 1 Done Done Sent false
8417 Acked Read 9 Done Done Sent false
8418 Acked Read 6 Done Done Sent false
8419 Acked Write 1 Done Done Sent false
8420 Acked Write 6 Done Done Sent false
8421 Acked Read 4 Done Done Sent false
8422 Acked Read 8 Done Done Sent false
8423 Acked Read 3 Done Done Sent false
8424 Acked Write 4 Done Done Sent false
8425 Acked Write 6 Done Done Sent false
8426 Acked Read 10 Done Done Sent false
8427 Acked Write 8 Done Done Sent false
8428 Acked Read 2 Done Done Sent false
8429 Acked Write 10 Done Done Sent false
8430 Acked Write 8 Done Done Sent false
8431 Acked Read 6 Done Done Sent false
8432 Acked Write 5 Done Done Sent false
8433 Acked Read 3 Done Done Sent false
8434 Acked Read 3 Done Done Sent false
8435 Acked Flush 0 Done Done Sent false
8436 Acked Read 5 Done Done Sent false
8437 Acked Read 8 Done Done Sent false
8438 Acked Flush 0 Done Done Sent false
8439 Acked Write 3 Done Done Sent false
8440 Acked Flush 0 Done Done Sent false
8441 Acked Write 5 Done Done Sent false
8442 Acked Read 2 Done Done Sent false
8443 Acked Read 6 Done Done Sent false
8444 Acked Flush 0 Done Done Sent false
8445 Acked Read 4 Done Done Sent false
8446 Acked Write 2 Done Done Sent false
8447 Acked Read 9 Done Done Sent false
8448 Acked Read 7 Done Done Sent false
8449 Acked Flush 0 Done Done Sent false
8450 Acked Read 7 Done Done Sent false
8451 Acked Flush 0 Done Done Sent false
8452 Acked Write 9 Done Done Sent false
8453 Acked Write 7 Done Done Sent false
8454 Acked Read 2 Done Done Sent false
8455 Acked Flush 0 Done Done Sent false
8456 Acked Read 2 Done Done Sent false
8457 Acked Read 7 Done Done Sent false
8458 Acked Flush 0 Done Done Sent false
8459 Acked Read 5 Done Done Sent false
8460 Acked Read 3 Done Done Sent false
8461 Acked Write 7 Done Done Sent false
8462 Acked Read 10 Done Done Sent false
8463 Acked Read 8 Done Done Sent false
8464 Acked Read 3 Done Done Sent false
8465 Acked Write 9 Done Done Sent false
8466 Acked Read 3 Done Done Sent false
8467 Acked Read 8 Done Done Sent false
8468 Acked Read 1 Done Done Sent false
8469 Acked Write 5 Done Done Sent false
8470 Acked Write 9 Done Done Sent false
8471 Acked Read 1 Done Done Sent false
8472 Acked Read 3 Done Done Sent false
8473 Acked Write 8 Done Done Sent false
8474 Acked Write 9 Done Done Sent false
8475 Acked Flush 0 Done Done Sent false
8476 Acked Read 6 Done Done Sent false
8477 Acked Write 2 Done Done Sent false
8478 Acked Read 8 Done Done Sent false
8479 Acked Write 6 Done Done Sent false
8480 Acked Read 3 Done Done Sent false
8481 Acked Write 5 Done Done Sent false
8482 Acked Write 8 Done Done Sent false
8483 Acked Flush 0 Done Done Sent false
8484 Acked Read 5 Done Done Sent false
8485 Acked Read 3 Done Done Sent false
8486 Acked Read 9 Done Done Sent false
8487 Acked Read 4 Done Done Sent false
8488 Acked Write 2 Done Done Sent false
8489 Acked Read 5 Done Done Sent false
8490 Acked Write 8 Done Done Sent false
8491 Acked Read 7 Done Done Sent false
8492 Acked Read 2 Done Done Sent false
8493 Acked Write 1 Done Done Sent false
8494 Acked Write 9 Done Done Sent false
8495 Acked Write 6 Done Done Sent false
8496 Acked Flush 0 Done Done Sent false
8497 Acked Write 4 Done Done Sent false
8498 Acked Read 3 Done Done Sent false
8499 Acked Read 2 Done Done Sent false
8500 Acked Read 10 Done Done Sent false
8501 Acked Write 10 Done Done Sent false
8502 Acked Read 1 Done Done Sent false
8503 Acked Write 8 Done Done Sent false
8504 Acked Write 9 Done Done Sent false
8505 Acked Read 5 Done Done Sent false
8506 Acked Read 9 Done Done Sent false
8507 Acked Flush 0 Done Done Sent false
8508 Acked Read 1 Done Done Sent false
8509 Acked Write 3 Done Done Sent false
8510 Acked Write 10 Done Done Sent false
8511 Acked Read 5 Done Done Sent false
8512 Acked Read 5 Done Done Sent false
8513 Acked Read 7 Done Done Sent false
8514 Acked Read 5 Done Done Sent false
8515 Acked Write 3 Done Done Sent false
8516 Acked Read 6 Done Done Sent false
8517 Acked Read 2 Done Done Sent false
8518 Acked Read 10 Done Done Sent false
8519 Acked Write 3 Done Done Sent false
8520 Acked Read 7 Done Done Sent false
8521 Acked Read 8 Done Done Sent false
8522 Acked Flush 0 Done Done Sent false
8523 Acked Read 7 Done Done Sent false
8524 Acked Read 2 Done Done Sent false
8525 Acked Read 1 Done Done Sent false
8526 Acked Flush 0 Done Done Sent false
8527 Acked Read 10 Done Done Sent false
8528 Acked Read 6 Done Done Sent false
8529 Acked Flush 0 Done Done Sent false
8530 Acked Read 9 Done Done Sent false
8531 Acked Read 5 Done Done Sent false
8532 Acked Write 1 Done Done Sent false
8533 Acked Read 4 Done Done Sent false
8534 Acked Write 6 Done Done Sent false
8535 Acked Read 6 Done Done Sent false
8536 Acked Write 10 Done Done Sent false
8537 Acked Read 7 Done Done Sent false
8538 Acked Read 1 Done Done Sent false
8539 Acked Read 8 Done Done Sent false
8540 Acked Write 8 Done Done Sent false
8541 Acked Write 6 Done Done Sent false
8542 Acked Flush 0 Done Done Sent false
8543 Acked Write 7 Done Done Sent false
8544 Acked Write 1 Done Done Sent false
8545 Acked Read 5 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
Downstairs last five completed: 8244 8243 8242 8241 8240
Upstairs last five completed: 8545 8544 8543 8542 8541
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:47.020975264Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.895813492Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49653)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.895944881Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.896915818Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897455709Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897482822Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(8244))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897501878Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897515544Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(8244))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897529131Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(8244))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:47.897697921Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8546 8545 8544 8543 8542
Upstairs last five completed: 8546 8545 8544 8543 8542
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 28 06:07:50.156[8820] Got stop action so:false kr:false
INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/1, method: POST, req_id: d63e5b64-b763-497b-8931-99fedc8932cb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:50.157541814Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:50.157606488Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:50.157628233Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:07:50.157 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: b9b6841f-e691-4373-b38c-2bc8151cd52e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:50.157743946Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Feb 28 06:07:54.159 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2a93ffde-7239-4bea-889d-3759fe7914f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 4010 len 24576 data: 5 5 4 3 2 3
002/300 Write block 2517 len 12288 data: 5 5 6
003/300 Write block 823 len 12288 data: 6 3 2
004/300 Read block 777 len 32768
005/300 Write block 972 len 24576 data: 2 2 3 5 5 5
006/300 Flush
007/300 Read block 140 len 12288
008/300 Write block 356 len 8192 data: 6 3
009/300 Write block 2675 len 8192 data: 11 10
010/300 Write block 1510 len 36864 data: 4 4 5 4 5 4 5 5 5
011/300 Write block 2323 len 20480 data: 5 4 4 5 4
012/300 Write block 4099 len 12288 data: 7 7 9
013/300 Flush
014/300 Write block 4948 len 16384 data: 3 3 5 7
015/300 Write block 1066 len 24576 data: 7 5 6 7 6 6
016/300 Read block 409 len 28672
017/300 Write block 2996 len 8192 data: 6 6
018/300 Flush
019/300 Read block 2119 len 20480
020/300 Read block 3229 len 20480
021/300 Write block 4187 len 8192 data: 8 7
022/300 Write block 766 len 4096 data: 2
023/300 Write block 1773 len 20480 data: 3 4 4 2 2
024/300 Read block 3336 len 40960
025/300 Flush
026/300 Write block 3415 len 20480 data: 3 4 7 7 7
027/300 Read block 3662 len 20480
028/300 Read block 2829 len 40960
029/300 Read block 3097 len 36864
030/300 Write block 2140 len 36864 data: 6 5 5 5 4 4 4 4 5
031/300 Read block 1693 len 24576
032/300 Read block 71 len 12288
033/300 Read block 455 len 32768
034/300 Read block 3764 len 32768
035/300 Read block 4443 len 36864
036/300 Read block 1220 len 12288
037/300 Read block 2869 len 40960
038/300 Write block 1358 len 4096 data: 8
039/300 Read block 275 len 16384
040/300 Read block 2509 len 16384
041/300 Flush
042/300 Write block 2245 len 28672 data: 3 3 3 3 3 2 2
043/300 Read block 3059 len 32768
044/300 Read block 477 len 8192
045/300 Flush
046/300 Read block 2051 len 20480
047/300 Read block 1833 len 12288
048/300 Read block 588 len 24576
049/300 Read block 1501 len 4096
050/300 Read block 2412 len 4096
051/300 Write block 2122 len 16384 data: 7 7 8 7
052/300 Flush
053/300 Read block 1601 len 36864
054/300 Read block 139 len 32768
055/300 Read block 4666 len 8192
056/300 Write block 4163 len 8192 data: 6 6
057/300 Write block 2825 len 16384 data: 7 7 5 4
058/300 Read block 970 len 4096
059/300 Write block 4545 len 20480 data: 6 7 7 5 5
060/300 Write block 1725 len 12288 data: 7 5 6
061/300 Write block 667 len 8192 data: 6 7
062/300 Write block 172 len 20480 data: 5 5 6 5 5
063/300 Flush
064/300 Read block 3904 len 36864
065/300 Read block 137 len 40960
066/300 Read block 1640 len 4096
067/300 Read block 2634 len 32768
068/300 Read block 1286 len 40960
069/300 Write block 109 len 40960 data: 4 4 5 5 4 3 4 2 2 2
070/300 Read block 3939 len 20480
071/300 Read block 2344 len 20480
072/300 Read block 1315 len 12288
073/300 Read block 2042 len 16384
074/300 Read block 4432 len 32768
075/300 Read block 3341 len 4096
076/300 Read block 2459 len 32768
077/300 Read block 3775 len 4096
078/300 Flush
079/300 Write block 3922 len 12288 data: 4 2 2
080/300 Write block 3200 len 40960 data: 6 7 7 7 7 8 7 8 7 6
081/300 Write block 3032 len 4096 data: 6
082/300 Write block 2888 len 4096 data: 5
083/300 Read block 4542 len 20480
084/300 Read block 2757 len 36864
085/300 Read block 3097 len 8192
086/300 Read block 4830 len 12288
087/300 Write block 1927 len 36864 data: 5 4 5 5 5 6 5 6 4
088/300 Write block 4886 len 32768 data: 9 14 14 14 13 12 10 10
089/300 Read block 3139 len 32768
090/300 Write block 770 len 20480 data: 6 5 6 5 5
091/300 Write block 215 len 32768 data: 9 7 8 8 10 7 7 6
092/300 Read block 1872 len 12288
093/300 Write block 2713 len 28672 data: 9 9 9 10 9 7 5
094/300 Write block 4387 len 16384 data: 6 7 6 8
095/300 Write block 106 len 4096 data: 5
096/300 Write block 336 len 12288 data: 5 6 7
097/300 Write block 3560 len 20480 data: 4 4 3 4 5
098/300 Read block 1887 len 32768
099/300 Read block 3612 len 28672
100/300 Read block 3343 len 24576
101/300 Write block 4274 len 4096 data: 7
102/300 Write block 4488 len 36864 data: 7 6 6 4 3 4 5 6 7
103/300 Write block 1148 len 16384 data: 7 7 6 9
104/300 Write block 867 len 16384 data: 5 4 4 7
105/300 Read block 3165 len 8192
106/300 Write block 3206 len 36864 data: 8 9 8 7 5 5 5 6 6
107/300 Read block 2097 len 24576
108/300 Read block 2786 len 20480
109/300 Write block 1357 len 16384 data: 7 9 9 9
110/300 Flush
111/300 Read block 2663 len 32768
112/300 Write block 3578 len 4096 data: 2
113/300 Write block 1913 len 36864 data: 4 4 4 4 4 3 3 3 3
114/300 Write block 1146 len 36864 data: 7 6 8 8 7 10 10 9 8
115/300 Read block 344 len 36864
116/300 Read block 4940 len 24576
117/300 Read block 3732 len 8192
118/300 Read block 2045 len 32768
119/300 Write block 3303 len 40960 data: 6 6 7 7 8 8 7 5 5 6
120/300 Read block 210 len 24576
121/300 Write block 3600 len 32768 data: 3 2 3 3 4 5 5 6
122/300 Write block 2379 len 40960 data: 3 3 3 3 3 4 4 3 3 4
123/300 Write block 345 len 8192 data: 7 6
124/300 Write block 1887 len 8192 data: 4 4
125/300 Read block 428 len 36864
126/300 Write block 3131 len 20480 data: 4 4 4 4 4
127/300 Read block 1637 len 20480
128/300 Flush
129/300 Flush
130/300 Read block 2832 len 16384
131/300 Write block 877 len 8192 data: 5 4
132/300 Read block 4483 len 40960
133/300 Read block 3733 len 28672
134/300 Write block 1918 len 4096 data: 4
135/300 Read block 91 len 4096
136/300 Flush
137/300 Read block 862 len 20480
138/300 Read block 3425 len 28672
139/300 Write block 656 len 40960 data: 4 5 6 7 7 7 8 7 7 7
140/300 Read block 1311 len 20480
141/300 Read block 3067 len 40960
142/300 Flush
143/300 Read block 3684 len 4096
144/300 Write block 2722 len 8192 data: 4 3
145/300 Read block 967 len 32768
146/300 Read block 906 len 4096
147/300 Read block 2268 len 4096
148/300 Read block 3996 len 20480
149/300 Read block 2559 len 32768
150/300 Read block 457 len 12288
151/300 Read block 4543 len 20480
152/300 Write block 2460 len 4096 data: 6
153/300 Write block 2229 len 12288 data: 6 8 10
154/300 Write block 388 len 16384 data: 3 3 4 4
155/300 Read block 202 len 32768
156/300 Write block 451 len 16384 data: 9 7 8 8
157/300 Read block 1388 len 8192
158/300 Read block 3214 len 20480
159/300 Write block 1720 len 32768 data: 5 6 8 8 8 8 6 7
160/300 Read block 1338 len 16384
161/300 Read block 1416 len 16384
162/300 Read block 1688 len 20480
163/300 Read block 3892 len 4096
164/300 Write block 1661 len 4096 data: 6
165/300 Flush
166/300 Read block 4091 len 8192
167/300 Read block 2550 len 16384
168/300 Write block 4187 len 20480 data: 9 8 7 9 9
169/300 Read block 2027 len 4096
170/300 Read block 264 len 12288
171/300 Write block 1042 len 24576 data: 7 7 7 7 6 5
172/300 Read block 287 len 28672
173/300 Read block 163 len 36864
174/300 Read block 4003 len 4096
175/300 Read block 4817 len 28672
176/300 Read block 2402 len 20480
177/300 Read block 358 len 20480
178/300 Read block 866 len 36864
179/300 Write block 1222 len 28672 data: 6 6 6 5 6 7 6
180/300 Flush
181/300 Read block 2775 len 24576
182/300 Read block 2023 len 12288
183/300 Write block 2729 len 28672 data: 5 8 6 6 7 5 6
184/300 Read block 4203 len 24576
185/300 Read block 905 len 16384
186/300 Read block 4339 len 20480
187/300 Write block 1462 len 32768 data: 4 4 4 4 5 7 6 7
188/300 Read block 195 len 40960
189/300 Read block 3993 len 8192
190/300 Write block 173 len 36864 data: 6 7 6 6 5 5 2 2 2
191/300 Write block 4685 len 8192 data: 7 7
192/300 Write block 4500 len 12288 data: 7 9 7
193/300 Read block 4551 len 20480
194/300 Read block 3386 len 20480
195/300 Write block 4963 len 12288 data: 5 6 7
196/300 Flush
197/300 Read block 4256 len 8192
198/300 Write block 3103 len 16384 data: 8 7 8 7
199/300 Flush
200/300 Write block 1983 len 8192 data: 7 7
201/300 Write block 32 len 4096 data: 3
202/300 Read block 2509 len 24576
203/300 Read block 2057 len 4096
204/300 Read block 871 len 8192
205/300 Read block 3191 len 20480
206/300 Write block 1458 len 28672 data: 5 7 6 4 5 5 5
207/300 Write block 3680 len 12288 data: 5 5 6
208/300 Read block 3663 len 4096
209/300 Read block 952 len 36864
210/300 Write block 4375 len 12288 data: 6 6 7
211/300 Read block 658 len 16384
212/300 Write block 1865 len 16384 data: 5 5 5 6
213/300 Read block 564 len 40960
214/300 Write block 3854 len 32768 data: 6 5 5 3 2 3 4 5
215/300 Write block 2772 len 12288 data: 4 4 4
216/300 Flush
217/300 Read block 3472 len 8192
218/300 Read block 2610 len 28672
219/300 Write block 1410 len 12288 data: 6 6 5
220/300 Write block 1876 len 8192 data: 7 9
221/300 Write block 4880 len 32768 data: 6 6 7 8 7 7 10 15
222/300 Read block 1990 len 8192
223/300 Read block 2411 len 24576
224/300 Read block 2904 len 36864
225/300 Flush
226/300 Read block 4573 len 4096
227/300 Read block 4858 len 28672
228/300 Read block 498 len 16384
229/300 Read block 705 len 28672
230/300 Write block 2526 len 32768 data: 7 7 6 7 5 3 3 3
231/300 Write block 4612 len 4096 data: 5
232/300 Write block 4189 len 16384 data: 8 10 10 8
233/300 Write block 1184 len 36864 data: 5 5 5 5 4 4 5 4 4
234/300 Write block 901 len 20480 data: 6 7 5 4 5
235/300 Read block 176 len 12288
236/300 Write block 1522 len 36864 data: 6 5 6 6 6 7 6 7 5
237/300 Read block 2254 len 24576
238/300 Read block 538 len 40960
239/300 Write block 4606 len 4096 data: 5
240/300 Read block 3215 len 36864
241/300 Write block 395 len 8192 data: 4 3
242/300 Read block 2326 len 4096
243/300 Read block 2789 len 28672
244/300 Read block 3119 len 32768
245/300 Read block 2510 len 8192
246/300 Write block 2650 len 16384 data: 7 8 4 5
247/300 Flush
248/300 Read block 4447 len 28672
249/300 Write block 3715 len 28672 data: 5 5 7 9 8 7 7
250/300 Write block 429 len 36864 data: 5 6 6 6 6 6 7 9 8
251/300 Read block 1076 len 40960
252/300 Read block 1854 len 12288
253/300 Write block 3747 len 8192 data: 8 6
254/300 Write block 2156 len 20480 data: 4 3 3 3 3
255/300 Flush
256/300 Flush
257/300 Write block 673 len 16384 data: 8 6 6 7
258/300 Read block 4221 len 40960
259/300 Read block 1959 len 36864
260/300 Flush
261/300 Write block 1682 len 40960 data: 7 8 8 6 7 6 5 4 4 5
262/300 Write block 3909 len 36864 data: 5 4 4 4 4 5 5 5 4
263/300 Read block 1831 len 24576
264/300 Read block 3429 len 32768
265/300 Read block 1970 len 16384
266/300 Read block 4317 len 20480
267/300 Read block 152 len 28672
268/300 Read block 1561 len 32768
269/300 Read block 1294 len 28672
270/300 Write block 1978 len 36864 data: 3 3 3 4 6 8 8 6 6
271/300 Read block 544 len 32768
272/300 Read block 4346 len 4096
273/300 Write block 3051 len 24576 data: 3 2 3 3 3 5
274/300 Write block 2580 len 4096 data: 7
275/300 Read block 192 len 16384
276/300 Write block 4491 len 8192 data: 5 4
277/300 Read block 3806 len 36864
278/300 Read block 3718 len 32768
279/300 Read block 2979 len 36864
280/300 Read block 4460 len 8192
281/300 Write block 1483 len 16384 data: 6 6 7 7
282/300 Read block 398 len 40960
283/300 Read block 506 len 40960
284/300 Write block 4532 len 8192 data: 4 4
285/300 Write block 3214 len 12288 data: 7 7 7
286/300 Read block 4819 len 8192
287/300 Read block 1178 len 28672
288/300 Write block 1113 len 16384 data: 2 2 2 3
289/300 Read block 266 len 16384
290/300 Read block 373 len 40960
291/300 Read block 839 len 8192
292/300 Read block 3036 len 40960
293/300 Read block 2282 len 8192
294/300 Write block 4826 len 32768 data: 6 6 9 9 11 9 10 7
295/300 Read block 3310 len 12288
296/300 Read block 2143 len 20480
297/300 Read block 1761 len 40960
298/300 Write block 3899 len 32768 data: 3 3 3 3 7 7 6 5
299/300 Write block 18 len 32768 data: 5 5 7 7 7 6 6 8
300/300 Write block 4518 len 32768 data: 5 5 7 7 6 6 8 7
301/300 Read block 4134 len 36864
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 28 06:07:54.405 INFO request completed, latency_us: 177, response_code: 204, uri: /start/cid/1, method: POST, req_id: a526873a-9e03-48c2-875e-2cc1b43fcf85, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8547 Acked Write 6 Done Sent Done false
8548 Acked Write 3 Done Sent Done false
8549 Acked Write 3 Done Sent Done false
8550 Acked Read 8 Done Sent Done false
8551 Acked Write 6 Done Sent Done false
8552 Acked Flush 0 Done Sent Done false
8553 Acked Read 3 Done Sent Done false
8554 Acked Write 2 Done Sent Done false
8555 Acked Write 2 Done Sent Done false
8556 Acked Write 9 Done Sent Done false
8557 Acked Write 5 Done Sent Done false
8558 Acked Write 3 Done Sent Done false
8559 Acked Flush 0 Done Sent Done false
8560 Acked Write 4 Done Sent Done false
8561 Acked Write 6 Done Sent Done false
8562 Acked Read 7 Done Sent Done false
8563 Acked Write 2 Done Sent Done false
8564 Acked Flush 0 Done Sent Done false
8565 Acked Read 5 Done Sent Done false
8566 Acked Read 5 Done Sent Done false
8567 Acked Write 2 Done Sent Done false
8568 Acked Write 1 Done Sent Done false
8569 Acked Write 5 Done Sent Done false
8570 Acked Read 10 Done Sent Done false
8571 Acked Flush 0 Done Sent Done false
8572 Acked Write 5 Done Sent Done false
8573 Acked Read 5 Done Sent Done false
8574 Acked Read 10 Done Sent Done false
8575 Acked Read 9 Done Sent Done false
8576 Acked Write 9 Done Sent Done false
8577 Acked Read 6 Done Sent Done false
8578 Acked Read 3 Done Sent Done false
8579 Acked Read 8 Done Sent Done false
8580 Acked Read 8 Done Sent Done false
8581 Acked Read 9 Done Sent Done false
8582 Acked Read 3 Done Sent Done false
8583 Acked Read 10 Done Sent Done false
8584 Acked Write 1 Done Sent Done false
8585 Acked Read 4 Done Sent Done false
8586 Acked Read 4 Done Sent Done false
8587 Acked Flush 0 Done Sent Done false
8588 Acked Write 7 Done Sent Done false
8589 Acked Read 8 Done Sent Done false
8590 Acked Read 2 Done Sent Done false
8591 Acked Flush 0 Done Sent Done false
8592 Acked Read 5 Done Sent Done false
8593 Acked Read 3 Done Sent Done false
8594 Acked Read 6 Done Sent Done false
8595 Acked Read 1 Done Sent Done false
8596 Acked Read 1 Done Sent Done false
8597 Acked Write 4 Done Sent Done false
8598 Acked Flush 0 Done Sent Done false
8599 Acked Read 9 Done Sent Done false
8600 Acked Read 8 Done Sent Done false
8601 Acked Read 2 Done Sent Done false
8602 Acked Write 2 Done Sent Done false
8603 Acked Write 4 Done Sent Done false
8604 Acked Read 1 Done Sent Done false
8605 Acked Write 5 Done Sent Done false
8606 Acked Write 3 Done Sent Done false
8607 Acked Write 2 Done Sent Done false
8608 Acked Write 5 Done Sent Done false
8609 Acked Flush 0 Done Sent Done false
8610 Acked Read 9 Done Sent Done false
8611 Acked Read 10 Done Sent Done false
8612 Acked Read 1 Done Sent Done false
8613 Acked Read 8 Done Sent Done false
8614 Acked Read 10 Done Sent Done false
8615 Acked Write 10 Done Sent Done false
8616 Acked Read 5 Done Sent Done false
8617 Acked Read 5 Done Sent Done false
8618 Acked Read 3 Done Sent Done false
8619 Acked Read 4 Done Sent Done false
8620 Acked Read 8 Done Sent Done false
8621 Acked Read 1 Done Sent Done false
8622 Acked Read 8 Done Sent Done false
8623 Acked Read 1 Done Sent Done false
8624 Acked Flush 0 Done Sent Done false
8625 Acked Write 3 Done Sent Done false
8626 Acked Write 10 Done Sent Done false
8627 Acked Write 1 Done Sent Done false
8628 Acked Write 1 Done Sent Done false
8629 Acked Read 5 Done Sent Done false
8630 Acked Read 9 Done Sent Done false
8631 Acked Read 2 Done Sent Done false
8632 Acked Read 3 Done Sent Done false
8633 Acked Write 9 Done Sent Done false
8634 Acked Write 8 Done Sent Done false
8635 Acked Read 8 Done Sent Done false
8636 Acked Write 5 Done Sent Done false
8637 Acked Write 8 Done Sent Done false
8638 Acked Read 3 Done Sent Done false
8639 Acked Write 7 Done Sent Done false
8640 Acked Write 4 Done Sent Done false
8641 Acked Write 1 Done Sent Done false
8642 Acked Write 3 Done Sent Done false
8643 Acked Write 5 Done Sent Done false
8644 Acked Read 8 Done Sent Done false
8645 Acked Read 7 Done Sent Done false
8646 Acked Read 6 Done Sent Done false
8647 Acked Write 1 Done Sent Done false
8648 Acked Write 9 Done Sent Done false
8649 Acked Write 4 Done Sent Done false
8650 Acked Write 4 Done Sent Done false
8651 Acked Read 2 Done Sent Done false
8652 Acked Write 9 Done Sent Done false
8653 Acked Read 6 Done Sent Done false
8654 Acked Read 5 Done Sent Done false
8655 Acked Write 4 Done Sent Done false
8656 Acked Flush 0 Done Sent Done false
8657 Acked Read 8 Done Sent Done false
8658 Acked Write 1 Done Sent Done false
8659 Acked Write 9 Done Sent Done false
8660 Acked Write 9 Done Sent Done false
8661 Acked Read 9 Done Sent Done false
8662 Acked Read 6 Done Sent Done false
8663 Acked Read 2 Done Sent Done false
8664 Acked Read 8 Done Sent Done false
8665 Acked Write 10 Done Sent Done false
8666 Acked Read 6 Done Sent Done false
8667 Acked Write 8 Done Sent Done false
8668 Acked Write 10 Done Sent Done false
8669 Acked Write 2 Done Sent Done false
8670 Acked Write 2 Done Sent Done false
8671 Acked Read 9 Done Sent Done false
8672 Acked Write 5 Done Sent Done false
8673 Acked Read 5 Done Sent Done false
8674 Acked Flush 0 Done Sent Done false
8675 Acked Flush 0 Done Sent Done false
8676 Acked Read 4 Done Sent Done false
8677 Acked Write 2 Done Sent Done false
8678 Acked Read 10 Done Sent Done false
8679 Acked Read 7 Done Sent Done false
8680 Acked Write 1 Done Sent Done false
8681 Acked Read 1 Done Sent Done false
8682 Acked Flush 0 Done Sent Done false
8683 Acked Read 5 Done Sent Done false
8684 Acked Read 7 Done Sent Done false
8685 Acked Write 10 Done Sent Done false
8686 Acked Read 5 Done Sent Done false
8687 Acked Read 10 Done Sent Done false
8688 Acked Flush 0 Done Sent Done false
8689 Acked Read 1 Done Sent Done false
8690 Acked Write 2 Done Sent Done false
8691 Acked Read 8 Done Sent Done false
8692 Acked Read 1 Done Sent Done false
8693 Acked Read 1 Done Sent Done false
8694 Acked Read 5 Done Sent Done false
8695 Acked Read 8 Done Sent Done false
8696 Acked Read 3 Done Sent Done false
8697 Acked Read 5 Done Sent Done false
8698 Acked Write 1 Done Sent Done false
8699 Acked Write 3 Done Sent Done false
8700 Acked Write 4 Done Sent Done false
8701 Acked Read 8 Done Sent Done false
8702 Acked Write 4 Done Sent Done false
8703 Acked Read 2 Done Sent Done false
8704 Acked Read 5 Done Sent Done false
8705 Acked Write 8 Done Sent Done false
8706 Acked Read 4 Done Sent Done false
8707 Acked Read 4 Done Sent Done false
8708 Acked Read 5 Done Sent Done false
8709 Acked Read 1 Done Sent Done false
8710 Acked Write 1 Done Sent Done false
8711 Acked Flush 0 Done Sent Done false
8712 Acked Read 2 Done Sent Done false
8713 Acked Read 4 Done Sent Done false
8714 Acked Write 5 Done Sent Done false
8715 Acked Read 1 Done Sent Done false
8716 Acked Read 3 Done Sent Done false
8717 Acked Write 6 Done Sent Done false
8718 Acked Read 7 Done Sent Done false
8719 Acked Read 9 Done Sent Done false
8720 Acked Read 1 Done Sent Done false
8721 Acked Read 7 Done Sent Done false
8722 Acked Read 5 Done Sent Done false
8723 Acked Read 5 Done Sent Done false
8724 Acked Read 9 Done Sent Done false
8725 Acked Write 7 Done Sent Done false
8726 Acked Flush 0 Done Sent Done false
8727 Acked Read 6 Done Sent Done false
8728 Acked Read 3 Done Sent Done false
8729 Acked Write 7 Done Sent Done false
8730 Acked Read 6 Done Sent Done false
8731 Acked Read 4 Done Sent Done false
8732 Acked Read 5 Done Sent Done false
8733 Acked Write 8 Done Sent Done false
8734 Acked Read 10 Done Sent Done false
8735 Acked Read 2 Done Sent Done false
8736 Acked Write 9 Done Sent Done false
8737 Acked Write 2 Done Sent Done false
8738 Acked Write 3 Done Sent Done false
8739 Acked Read 5 Done Sent Done false
8740 Acked Read 5 Done Sent Done false
8741 Acked Write 3 Done Sent Done false
8742 Acked Flush 0 Done Sent Done false
8743 Acked Read 2 Done Sent Done false
8744 Acked Write 4 Done Sent Done false
8745 Acked Flush 0 Done Sent Done false
8746 Acked Write 2 Done Sent Done false
8747 Acked Write 1 Done Sent Done false
8748 Acked Read 6 Done Sent Done false
8749 Acked Read 1 Done Sent Done false
8750 Acked Read 2 Done Sent Done false
8751 Acked Read 5 Done Sent Done false
8752 Acked Write 7 Done Sent Done false
8753 Acked Write 3 Done Sent Done false
8754 Acked Read 1 Done Sent Done false
8755 Acked Read 9 Done Sent Done false
8756 Acked Write 3 Done Sent Done false
8757 Acked Read 4 Done Sent Done false
8758 Acked Write 4 Done Sent Done false
8759 Acked Read 10 Done Sent Done false
8760 Acked Write 8 Done Sent Done false
8761 Acked Write 3 Done Sent Done false
8762 Acked Flush 0 Done Sent Done false
8763 Acked Read 2 Done Sent Done false
8764 Acked Read 7 Done Sent Done false
8765 Acked Write 3 Done Sent Done false
8766 Acked Write 2 Done Sent Done false
8767 Acked Write 8 Done Sent Done false
8768 Acked Read 2 Done Sent Done false
8769 Acked Read 6 Done Sent Done false
8770 Acked Read 9 Done Sent Done false
8771 Acked Flush 0 Done Sent Done false
8772 Acked Read 1 Done Sent Done false
8773 Acked Read 7 Done Sent Done false
8774 Acked Read 4 Done Sent Done false
8775 Acked Read 7 Done Sent Done false
8776 Acked Write 8 Done Sent Done false
8777 Acked Write 1 Done Sent Done false
8778 Acked Write 4 Done Sent Done false
8779 Acked Write 9 Done Sent Done false
8780 Acked Write 5 Done Sent Done false
8781 Acked Read 3 Done Sent Done false
8782 Acked Write 9 Done Sent Done false
8783 Acked Read 6 Done Sent Done false
8784 Acked Read 10 Done Sent Done false
8785 Acked Write 1 Done Sent Done false
8786 Acked Read 9 Done Sent Done false
8787 Acked Write 2 Done Sent Done false
8788 Acked Read 1 Done Sent Done false
8789 Acked Read 7 Done Sent Done false
8790 Acked Read 8 Done Sent Done false
8791 Acked Read 2 Done Sent Done false
8792 Acked Write 4 Done Sent Done false
8793 Acked Flush 0 Done Sent Done false
8794 Acked Read 7 Done Sent Done false
8795 Acked Write 7 Done Sent Done false
8796 Acked Write 9 Done Sent Done false
8797 Acked Read 10 Done Sent Done false
8798 Acked Read 3 Done Sent Done false
8799 Acked Write 2 Done Sent Done false
8800 Acked Write 5 Done Sent Done false
8801 Acked Flush 0 Done Sent Done false
8802 Acked Flush 0 Done Sent Done false
8803 Acked Write 4 Done Sent Done false
8804 Acked Read 10 Done Sent Done false
8805 Acked Read 9 Done Sent Done false
8806 Acked Flush 0 Done Sent Done false
8807 Acked Write 10 Done Sent Done false
8808 Acked Write 9 Done Sent Done false
8809 Acked Read 6 Done Sent Done false
8810 Acked Read 8 Done Sent Done false
8811 Acked Read 4 Done Sent Done false
8812 Acked Read 5 Done Sent Done false
8813 Acked Read 7 Done Sent Done false
8814 Acked Read 8 Done Sent Done false
8815 Acked Read 7 Done Sent Done false
8816 Acked Write 9 Done Sent Done false
8817 Acked Read 8 Done Sent Done false
8818 Acked Read 1 Done Sent Done false
8819 Acked Write 6 Done Sent Done false
8820 Acked Write 1 Done Sent Done false
8821 Acked Read 4 Done Sent Done false
8822 Acked Write 2 Done Sent Done false
8823 Acked Read 9 Done Sent Done false
8824 Acked Read 8 Done Sent Done false
8825 Acked Read 9 Done Sent Done false
8826 Acked Read 2 Done Sent Done false
8827 Acked Write 4 Done Sent Done false
8828 Acked Read 10 Done Sent Done false
8829 Acked Read 10 Done Sent Done false
8830 Acked Write 2 Done Sent Done false
8831 Acked Write 3 Done Sent Done false
8832 Acked Read 2 Done Sent Done false
8833 Acked Read 7 Done Sent Done false
8834 Acked Write 4 Done Sent Done false
8835 Acked Read 4 Done Sent Done false
8836 Acked Read 10 Done Sent Done false
8837 Acked Read 2 Done Sent Done false
8838 Acked Read 10 Done Sent Done false
8839 Acked Read 2 Done Sent Done false
8840 Acked Write 8 Done Sent Done false
8841 Acked Read 3 Done Sent Done false
8842 Acked Read 5 Done Sent Done false
8843 Acked Read 10 Done Sent Done false
8844 Acked Write 8 Done Sent Done false
8845 Acked Write 8 Done Sent Done false
8846 Acked Write 8 Done Sent Done false
8847 Acked Read 9 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8806)) Some(JobId(8546)) Some(JobId(8806))
Downstairs last five completed: 8546 8545 8544 8543 8542
Upstairs last five completed: 8847 8846 8845 8844 8843
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:55.035207899Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:56.153972251Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:57.037904431Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1237)
[8820][1] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:58.400099042Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8547 Acked Write 6 Done Sent Done false
8548 Acked Write 3 Done Sent Done false
8549 Acked Write 3 Done Sent Done false
8550 Acked Read 8 Done Sent Done false
8551 Acked Write 6 Done Sent Done false
8552 Acked Flush 0 Done Sent Done false
8553 Acked Read 3 Done Sent Done false
8554 Acked Write 2 Done Sent Done false
8555 Acked Write 2 Done Sent Done false
8556 Acked Write 9 Done Sent Done false
8557 Acked Write 5 Done Sent Done false
8558 Acked Write 3 Done Sent Done false
8559 Acked Flush 0 Done Sent Done false
8560 Acked Write 4 Done Sent Done false
8561 Acked Write 6 Done Sent Done false
8562 Acked Read 7 Done Sent Done false
8563 Acked Write 2 Done Sent Done false
8564 Acked Flush 0 Done Sent Done false
8565 Acked Read 5 Done Sent Done false
8566 Acked Read 5 Done Sent Done false
8567 Acked Write 2 Done Sent Done false
8568 Acked Write 1 Done Sent Done false
8569 Acked Write 5 Done Sent Done false
8570 Acked Read 10 Done Sent Done false
8571 Acked Flush 0 Done Sent Done false
8572 Acked Write 5 Done Sent Done false
8573 Acked Read 5 Done Sent Done false
8574 Acked Read 10 Done Sent Done false
8575 Acked Read 9 Done Sent Done false
8576 Acked Write 9 Done Sent Done false
8577 Acked Read 6 Done Sent Done false
8578 Acked Read 3 Done Sent Done false
8579 Acked Read 8 Done Sent Done false
8580 Acked Read 8 Done Sent Done false
8581 Acked Read 9 Done Sent Done false
8582 Acked Read 3 Done Sent Done false
8583 Acked Read 10 Done Sent Done false
8584 Acked Write 1 Done Sent Done false
8585 Acked Read 4 Done Sent Done false
8586 Acked Read 4 Done Sent Done false
8587 Acked Flush 0 Done Sent Done false
8588 Acked Write 7 Done Sent Done false
8589 Acked Read 8 Done Sent Done false
8590 Acked Read 2 Done Sent Done false
8591 Acked Flush 0 Done Sent Done false
8592 Acked Read 5 Done Sent Done false
8593 Acked Read 3 Done Sent Done false
8594 Acked Read 6 Done Sent Done false
8595 Acked Read 1 Done Sent Done false
8596 Acked Read 1 Done Sent Done false
8597 Acked Write 4 Done Sent Done false
8598 Acked Flush 0 Done Sent Done false
8599 Acked Read 9 Done Sent Done false
8600 Acked Read 8 Done Sent Done false
8601 Acked Read 2 Done Sent Done false
8602 Acked Write 2 Done Sent Done false
8603 Acked Write 4 Done Sent Done false
8604 Acked Read 1 Done Sent Done false
8605 Acked Write 5 Done Sent Done false
8606 Acked Write 3 Done Sent Done false
8607 Acked Write 2 Done Sent Done false
8608 Acked Write 5 Done Sent Done false
8609 Acked Flush 0 Done Sent Done false
8610 Acked Read 9 Done Sent Done false
8611 Acked Read 10 Done Sent Done false
8612 Acked Read 1 Done Sent Done false
8613 Acked Read 8 Done Sent Done false
8614 Acked Read 10 Done Sent Done false
8615 Acked Write 10 Done Sent Done false
8616 Acked Read 5 Done Sent Done false
8617 Acked Read 5 Done Sent Done false
8618 Acked Read 3 Done Sent Done false
8619 Acked Read 4 Done Sent Done false
8620 Acked Read 8 Done Sent Done false
8621 Acked Read 1 Done Sent Done false
8622 Acked Read 8 Done Sent Done false
8623 Acked Read 1 Done Sent Done false
8624 Acked Flush 0 Done Sent Done false
8625 Acked Write 3 Done Sent Done false
8626 Acked Write 10 Done Sent Done false
8627 Acked Write 1 Done Sent Done false
8628 Acked Write 1 Done Sent Done false
8629 Acked Read 5 Done Sent Done false
8630 Acked Read 9 Done Sent Done false
8631 Acked Read 2 Done Sent Done false
8632 Acked Read 3 Done Sent Done false
8633 Acked Write 9 Done Sent Done false
8634 Acked Write 8 Done Sent Done false
8635 Acked Read 8 Done Sent Done false
8636 Acked Write 5 Done Sent Done false
8637 Acked Write 8 Done Sent Done false
8638 Acked Read 3 Done Sent Done false
8639 Acked Write 7 Done Sent Done false
8640 Acked Write 4 Done Sent Done false
8641 Acked Write 1 Done Sent Done false
8642 Acked Write 3 Done Sent Done false
8643 Acked Write 5 Done Sent Done false
8644 Acked Read 8 Done Sent Done false
8645 Acked Read 7 Done Sent Done false
8646 Acked Read 6 Done Sent Done false
8647 Acked Write 1 Done Sent Done false
8648 Acked Write 9 Done Sent Done false
8649 Acked Write 4 Done Sent Done false
8650 Acked Write 4 Done Sent Done false
8651 Acked Read 2 Done Sent Done false
8652 Acked Write 9 Done Sent Done false
8653 Acked Read 6 Done Sent Done false
8654 Acked Read 5 Done Sent Done false
8655 Acked Write 4 Done Sent Done false
8656 Acked Flush 0 Done Sent Done false
8657 Acked Read 8 Done Sent Done false
8658 Acked Write 1 Done Sent Done false
8659 Acked Write 9 Done Sent Done false
8660 Acked Write 9 Done Sent Done false
8661 Acked Read 9 Done Sent Done false
8662 Acked Read 6 Done Sent Done false
8663 Acked Read 2 Done Sent Done false
8664 Acked Read 8 Done Sent Done false
8665 Acked Write 10 Done Sent Done false
8666 Acked Read 6 Done Sent Done false
8667 Acked Write 8 Done Sent Done false
8668 Acked Write 10 Done Sent Done false
8669 Acked Write 2 Done Sent Done false
8670 Acked Write 2 Done Sent Done false
8671 Acked Read 9 Done Sent Done false
8672 Acked Write 5 Done Sent Done false
8673 Acked Read 5 Done Sent Done false
8674 Acked Flush 0 Done Sent Done false
8675 Acked Flush 0 Done Sent Done false
8676 Acked Read 4 Done Sent Done false
8677 Acked Write 2 Done Sent Done false
8678 Acked Read 10 Done Sent Done false
8679 Acked Read 7 Done Sent Done false
8680 Acked Write 1 Done Sent Done false
8681 Acked Read 1 Done Sent Done false
8682 Acked Flush 0 Done Sent Done false
8683 Acked Read 5 Done Sent Done false
8684 Acked Read 7 Done Sent Done false
8685 Acked Write 10 Done Sent Done false
8686 Acked Read 5 Done Sent Done false
8687 Acked Read 10 Done Sent Done false
8688 Acked Flush 0 Done Sent Done false
8689 Acked Read 1 Done Sent Done false
8690 Acked Write 2 Done Sent Done false
8691 Acked Read 8 Done Sent Done false
8692 Acked Read 1 Done Sent Done false
8693 Acked Read 1 Done Sent Done false
8694 Acked Read 5 Done Sent Done false
8695 Acked Read 8 Done Sent Done false
8696 Acked Read 3 Done Sent Done false
8697 Acked Read 5 Done Sent Done false
8698 Acked Write 1 Done Sent Done false
8699 Acked Write 3 Done Sent Done false
8700 Acked Write 4 Done Sent Done false
8701 Acked Read 8 Done Sent Done false
8702 Acked Write 4 Done Sent Done false
8703 Acked Read 2 Done Sent Done false
8704 Acked Read 5 Done Sent Done false
8705 Acked Write 8 Done Sent Done false
8706 Acked Read 4 Done Sent Done false
8707 Acked Read 4 Done Sent Done false
8708 Acked Read 5 Done Sent Done false
8709 Acked Read 1 Done Sent Done false
8710 Acked Write 1 Done Sent Done false
8711 Acked Flush 0 Done Sent Done false
8712 Acked Read 2 Done Sent Done false
8713 Acked Read 4 Done Sent Done false
8714 Acked Write 5 Done Sent Done false
8715 Acked Read 1 Done Sent Done false
8716 Acked Read 3 Done Sent Done false
8717 Acked Write 6 Done Sent Done false
8718 Acked Read 7 Done Sent Done false
8719 Acked Read 9 Done Sent Done false
8720 Acked Read 1 Done Sent Done false
8721 Acked Read 7 Done Sent Done false
8722 Acked Read 5 Done Sent Done false
8723 Acked Read 5 Done Sent Done false
8724 Acked Read 9 Done Sent Done false
8725 Acked Write 7 Done Sent Done false
8726 Acked Flush 0 Done Sent Done false
8727 Acked Read 6 Done Sent Done false
8728 Acked Read 3 Done Sent Done false
8729 Acked Write 7 Done Sent Done false
8730 Acked Read 6 Done Sent Done false
8731 Acked Read 4 Done Sent Done false
8732 Acked Read 5 Done Sent Done false
8733 Acked Write 8 Done Sent Done false
8734 Acked Read 10 Done Sent Done false
8735 Acked Read 2 Done Sent Done false
8736 Acked Write 9 Done Sent Done false
8737 Acked Write 2 Done Sent Done false
8738 Acked Write 3 Done Sent Done false
8739 Acked Read 5 Done Sent Done false
8740 Acked Read 5 Done Sent Done false
8741 Acked Write 3 Done Sent Done false
8742 Acked Flush 0 Done Sent Done false
8743 Acked Read 2 Done Sent Done false
8744 Acked Write 4 Done Sent Done false
8745 Acked Flush 0 Done Sent Done false
8746 Acked Write 2 Done Sent Done false
8747 Acked Write 1 Done Sent Done false
8748 Acked Read 6 Done Sent Done false
8749 Acked Read 1 Done Sent Done false
8750 Acked Read 2 Done Sent Done false
8751 Acked Read 5 Done Sent Done false
8752 Acked Write 7 Done Sent Done false
8753 Acked Write 3 Done Sent Done false
8754 Acked Read 1 Done Sent Done false
8755 Acked Read 9 Done Sent Done false
8756 Acked Write 3 Done Sent Done false
8757 Acked Read 4 Done Sent Done false
8758 Acked Write 4 Done Sent Done false
8759 Acked Read 10 Done Sent Done false
8760 Acked Write 8 Done Sent Done false
8761 Acked Write 3 Done Sent Done false
8762 Acked Flush 0 Done Sent Done false
8763 Acked Read 2 Done Sent Done false
8764 Acked Read 7 Done Sent Done false
8765 Acked Write 3 Done Sent Done false
8766 Acked Write 2 Done Sent Done false
8767 Acked Write 8 Done Sent Done false
8768 Acked Read 2 Done Sent Done false
8769 Acked Read 6 Done Sent Done false
8770 Acked Read 9 Done Sent Done false
8771 Acked Flush 0 Done Sent Done false
8772 Acked Read 1 Done Sent Done false
8773 Acked Read 7 Done Sent Done false
8774 Acked Read 4 Done Sent Done false
8775 Acked Read 7 Done Sent Done false
8776 Acked Write 8 Done Sent Done false
8777 Acked Write 1 Done Sent Done false
8778 Acked Write 4 Done Sent Done false
8779 Acked Write 9 Done Sent Done false
8780 Acked Write 5 Done Sent Done false
8781 Acked Read 3 Done Sent Done false
8782 Acked Write 9 Done Sent Done false
8783 Acked Read 6 Done Sent Done false
8784 Acked Read 10 Done Sent Done false
8785 Acked Write 1 Done Sent Done false
8786 Acked Read 9 Done Sent Done false
8787 Acked Write 2 Done Sent Done false
8788 Acked Read 1 Done Sent Done false
8789 Acked Read 7 Done Sent Done false
8790 Acked Read 8 Done Sent Done false
8791 Acked Read 2 Done Sent Done false
8792 Acked Write 4 Done Sent Done false
8793 Acked Flush 0 Done Sent Done false
8794 Acked Read 7 Done Sent Done false
8795 Acked Write 7 Done Sent Done false
8796 Acked Write 9 Done Sent Done false
8797 Acked Read 10 Done Sent Done false
8798 Acked Read 3 Done Sent Done false
8799 Acked Write 2 Done Sent Done false
8800 Acked Write 5 Done Sent Done false
8801 Acked Flush 0 Done Sent Done false
8802 Acked Flush 0 Done Sent Done false
8803 Acked Write 4 Done Sent Done false
8804 Acked Read 10 Done Sent Done false
8805 Acked Read 9 Done Sent Done false
8806 Acked Flush 0 Done Sent Done false
8807 Acked Write 10 Done Sent Done false
8808 Acked Write 9 Done Sent Done false
8809 Acked Read 6 Done Sent Done false
8810 Acked Read 8 Done Sent Done false
8811 Acked Read 4 Done Sent Done false
8812 Acked Read 5 Done Sent Done false
8813 Acked Read 7 Done Sent Done false
8814 Acked Read 8 Done Sent Done false
8815 Acked Read 7 Done Sent Done false
8816 Acked Write 9 Done Sent Done false
8817 Acked Read 8 Done Sent Done false
8818 Acked Read 1 Done Sent Done false
8819 Acked Write 6 Done Sent Done false
8820 Acked Write 1 Done Sent Done false
8821 Acked Read 4 Done Sent Done false
8822 Acked Write 2 Done Sent Done false
8823 Acked Read 9 Done Sent Done false
8824 Acked Read 8 Done Sent Done false
8825 Acked Read 9 Done Sent Done false
8826 Acked Read 2 Done Sent Done false
8827 Acked Write 4 Done Sent Done false
8828 Acked Read 10 Done Sent Done false
8829 Acked Read 10 Done Sent Done false
8830 Acked Write 2 Done Sent Done false
8831 Acked Write 3 Done Sent Done false
8832 Acked Read 2 Done Sent Done false
8833 Acked Read 7 Done Sent Done false
8834 Acked Write 4 Done Sent Done false
8835 Acked Read 4 Done Sent Done false
8836 Acked Read 10 Done Sent Done false
8837 Acked Read 2 Done Sent Done false
8838 Acked Read 10 Done Sent Done false
8839 Acked Read 2 Done Sent Done false
8840 Acked Write 8 Done Sent Done false
8841 Acked Read 3 Done Sent Done false
8842 Acked Read 5 Done Sent Done false
8843 Acked Read 10 Done Sent Done false
8844 Acked Write 8 Done Sent Done false
8845 Acked Write 8 Done Sent Done false
8846 Acked Write 8 Done Sent Done false
8847 Acked Read 9 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8806)) Some(JobId(8546)) Some(JobId(8806))
Downstairs last five completed: 8546 8545 8544 8543 8542
Upstairs last five completed: 8847 8846 8845 8844 8843
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:59.042261582Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:00.044217457Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.159515445Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61551)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.159689764Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.161057546Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.16166906Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.161699382Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(8546))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.161717648Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.161734754Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] client re-new 301 jobs since flush Some(JobId(8546))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.16175143Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[1] Replayed 301 jobs since flush: Some(JobId(8546))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:00.161948872Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8848 8847 8846 8845 8844
Upstairs last five completed: 8848 8847 8846 8845 8844
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Feb 28 06:08:02.410 INFO request completed, latency_us: 318, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b9a61855-e02a-4b3b-af1e-45281f08aba9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:02.41134739Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:02.411419482Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:02.411443047Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:02.411486666Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:08:02.411 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9fa696d1-e837-4492-9007-2e9a1220e172, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:08:06.414 INFO request completed, latency_us: 299, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6d733b56-a0c2-4fe5-8831-9063cc81ef93, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 2068 len 24576
002/300 Write block 203 len 32768 data: 6 6 5 5 5 8 8 8
003/300 Read block 3436 len 16384
004/300 Write block 933 len 40960 data: 6 5 5 5 6 7 6 6 8 9
005/300 Read block 4939 len 28672
006/300 Write block 196 len 32768 data: 8 6 5 5 5 4 4 7
007/300 Read block 44 len 40960
008/300 Read block 3489 len 36864
009/300 Flush
010/300 Read block 4828 len 8192
011/300 Read block 972 len 20480
012/300 Read block 2868 len 16384
013/300 Write block 2591 len 20480 data: 4 3 3 4 4
014/300 Write block 2104 len 40960 data: 3 3 2 4 5 4 6 6 5 5
015/300 Write block 2742 len 28672 data: 5 5 4 4 4 4 6
016/300 Read block 4898 len 28672
017/300 Flush
018/300 Read block 1976 len 40960
019/300 Write block 881 len 24576 data: 2 2 2 2 3 3
020/300 Write block 1793 len 24576 data: 5 5 6 5 5 5
021/300 Read block 2158 len 12288
022/300 Read block 4011 len 24576
023/300 Write block 3622 len 32768 data: 5 6 6 6 5 5 5 5
024/300 Write block 1222 len 32768 data: 7 7 7 6 7 8 7 6
025/300 Read block 2283 len 36864
026/300 Flush
027/300 Write block 2635 len 4096 data: 4
028/300 Read block 4273 len 36864
029/300 Read block 4813 len 16384
030/300 Read block 4403 len 28672
031/300 Read block 4955 len 24576
032/300 Read block 3258 len 12288
033/300 Flush
034/300 Flush
035/300 Write block 4773 len 12288 data: 7 6 7
036/300 Read block 4485 len 24576
037/300 Read block 847 len 4096
038/300 Flush
039/300 Read block 936 len 4096
040/300 Read block 1276 len 24576
041/300 Read block 1014 len 12288
042/300 Write block 2780 len 32768 data: 5 5 4 7 6 5 6 6
043/300 Write block 4059 len 16384 data: 5 5 3 5
044/300 Read block 1413 len 8192
045/300 Read block 1513 len 8192
046/300 Read block 1041 len 12288
047/300 Read block 1062 len 4096
048/300 Read block 4104 len 20480
049/300 Write block 3935 len 40960 data: 7 8 7 9 10 10 7 6 6 7
050/300 Read block 57 len 4096
051/300 Flush
052/300 Read block 3412 len 12288
053/300 Write block 2321 len 36864 data: 5 5 6 5 5 6 5 3 3
054/300 Write block 4889 len 16384 data: 15 14 13 11
055/300 Read block 2667 len 36864
056/300 Write block 1319 len 32768 data: 7 7 7 8 8 6 6 7
057/300 Flush
058/300 Read block 1903 len 32768
059/300 Read block 2052 len 32768
060/300 Write block 141 len 16384 data: 5 4 3 6
061/300 Write block 81 len 28672 data: 9 7 7 5 4 4 3
062/300 Flush
063/300 Write block 1755 len 12288 data: 6 5 5
064/300 Write block 3969 len 8192 data: 3 4
065/300 Write block 4333 len 12288 data: 5 4 4
066/300 Read block 3534 len 4096
067/300 Flush
068/300 Read block 4461 len 28672
069/300 Read block 1389 len 36864
070/300 Flush
071/300 Read block 2112 len 20480
072/300 Read block 2939 len 12288
073/300 Flush
074/300 Read block 2871 len 16384
075/300 Read block 2773 len 24576
076/300 Write block 2927 len 20480 data: 5 3 4 4 7
077/300 Read block 622 len 20480
078/300 Read block 4734 len 40960
079/300 Read block 4053 len 12288
080/300 Write block 3539 len 36864 data: 5 5 5 5 2 2 5 6 5
081/300 Write block 1793 len 12288 data: 6 6 7
082/300 Write block 4280 len 20480 data: 6 5 4 3 3
083/300 Write block 2994 len 20480 data: 6 5 7 7 6
084/300 Read block 4924 len 36864
085/300 Write block 3316 len 36864 data: 4 3 2 3 5 6 7 7 7
086/300 Write block 4190 len 32768 data: 11 11 9 8 6 5 5 6
087/300 Write block 4513 len 24576 data: 10 11 8 4 5 6
088/300 Write block 4630 len 16384 data: 9 10 10 10
089/300 Flush
090/300 Write block 3115 len 8192 data: 7 7
091/300 Read block 4669 len 8192
092/300 Read block 4082 len 32768
093/300 Read block 4431 len 8192
094/300 Read block 1139 len 24576
095/300 Write block 2849 len 32768 data: 5 5 5 6 5 6 6 6
096/300 Write block 1286 len 28672 data: 5 6 6 7 7 8 7
097/300 Write block 966 len 4096 data: 4
098/300 Read block 4867 len 24576
099/300 Flush
100/300 Read block 4107 len 32768
101/300 Write block 3817 len 32768 data: 4 4 4 3 3 4 5 4
102/300 Write block 4372 len 24576 data: 3 3 3 7 7 8
103/300 Write block 4435 len 12288 data: 4 5 4
104/300 Read block 3930 len 12288
105/300 Write block 2529 len 28672 data: 8 6 4 4 4 4 3
106/300 Flush
107/300 Read block 2201 len 12288
108/300 Write block 1283 len 8192 data: 6 4
109/300 Write block 2872 len 12288 data: 7 6 6
110/300 Write block 15 len 20480 data: 3 3 4 6 6
111/300 Flush
112/300 Write block 4388 len 32768 data: 8 7 9 8 6 5 6 6
113/300 Write block 1497 len 32768 data: 3 3 4 4 5 4 5 4
114/300 Read block 1828 len 12288
115/300 Read block 3928 len 24576
116/300 Read block 1901 len 12288
117/300 Write block 3538 len 12288 data: 7 6 6
118/300 Read block 4215 len 12288
119/300 Read block 119 len 36864
120/300 Write block 1284 len 36864 data: 5 3 6 7 7 8 8 9 8
121/300 Write block 4010 len 40960 data: 6 6 5 4 3 4 3 3 5 5
122/300 Write block 1910 len 36864 data: 2 2 3 5 5 5 5 5 5
123/300 Read block 1751 len 36864
124/300 Read block 1889 len 28672
125/300 Read block 4537 len 28672
126/300 Read block 1562 len 32768
127/300 Write block 4373 len 12288 data: 4 4 8
128/300 Read block 4345 len 12288
129/300 Flush
130/300 Write block 2133 len 28672 data: 7 7 6 5 7 5 6
131/300 Read block 4477 len 28672
132/300 Read block 2485 len 32768
133/300 Write block 155 len 16384 data: 6 6 4 3
134/300 Read block 2628 len 20480
135/300 Read block 2737 len 32768
136/300 Write block 548 len 8192 data: 6 7
137/300 Write block 689 len 36864 data: 3 4 4 5 4 4 4 3 5
138/300 Write block 4267 len 12288 data: 8 8 7
139/300 Read block 1895 len 32768
140/300 Read block 4633 len 12288
141/300 Write block 662 len 40960 data: 9 8 8 8 5 7 8 7 8 8
142/300 Read block 2671 len 8192
143/300 Write block 3789 len 32768 data: 2 2 3 4 3 5 5 5
144/300 Read block 2162 len 32768
145/300 Read block 362 len 4096
146/300 Read block 2931 len 8192
147/300 Write block 4116 len 8192 data: 7 7
148/300 Write block 2139 len 16384 data: 7 7 6 6
149/300 Flush
150/300 Read block 1491 len 12288
151/300 Write block 4443 len 24576 data: 6 6 5 3 4 3
152/300 Write block 188 len 20480 data: 6 6 7 8 7
153/300 Read block 171 len 4096
154/300 Read block 1009 len 32768
155/300 Read block 971 len 36864
156/300 Write block 2264 len 12288 data: 8 6 6
157/300 Flush
158/300 Write block 2704 len 24576 data: 6 7 7 6 6 6
159/300 Read block 3564 len 16384
160/300 Read block 3241 len 40960
161/300 Read block 725 len 12288
162/300 Write block 2244 len 16384 data: 2 4 4 4
163/300 Write block 3007 len 32768 data: 4 4 5 5 5 4 4 4
164/300 Write block 1599 len 20480 data: 2 3 3 3 3
165/300 Read block 2650 len 40960
166/300 Write block 3897 len 24576 data: 3 4 4 4 4 4
167/300 Write block 2220 len 4096 data: 6
168/300 Read block 3825 len 8192
169/300 Read block 4055 len 40960
170/300 Write block 1485 len 24576 data: 8 8 7 8 9 10
171/300 Write block 2107 len 28672 data: 5 6 5 7 7 6 6
172/300 Read block 3191 len 32768
173/300 Read block 3188 len 20480
174/300 Read block 3643 len 8192
175/300 Flush
176/300 Write block 3401 len 12288 data: 4 4 4
177/300 Read block 3428 len 36864
178/300 Read block 1315 len 40960
179/300 Read block 4608 len 36864
180/300 Write block 905 len 4096 data: 6
181/300 Read block 1426 len 32768
182/300 Write block 1952 len 28672 data: 3 3 5 6 5 5 5
183/300 Read block 3776 len 16384
184/300 Read block 4249 len 20480
185/300 Write block 2080 len 16384 data: 3 4 4 6
186/300 Read block 4068 len 24576
187/300 Flush
188/300 Read block 1527 len 28672
189/300 Write block 131 len 36864 data: 5 4 6 5 6 6 6 5 5
190/300 Read block 3313 len 20480
191/300 Read block 4244 len 8192
192/300 Write block 1881 len 12288 data: 6 5 2
193/300 Read block 4989 len 36864
194/300 Read block 2605 len 16384
195/300 Read block 3379 len 36864
196/300 Write block 1801 len 8192 data: 5 7
197/300 Read block 1350 len 8192
198/300 Write block 895 len 40960 data: 4 4 5 5 5 5 7 8 6 5
199/300 Flush
200/300 Write block 3749 len 36864 data: 7 5 6 6 5 4 4 4 4
201/300 Write block 1764 len 24576 data: 5 5 5 4 2 3
202/300 Write block 4761 len 24576 data: 7 7 6 5 4 4
203/300 Read block 1031 len 28672
204/300 Read block 4420 len 32768
205/300 Read block 3112 len 24576
206/300 Read block 4042 len 12288
207/300 Read block 4156 len 40960
208/300 Write block 1982 len 28672 data: 7 9 9 7 7 6 7
209/300 Read block 1977 len 12288
210/300 Read block 856 len 4096
211/300 Read block 2597 len 36864
212/300 Flush
213/300 Read block 1515 len 4096
214/300 Write block 1144 len 12288 data: 6 8 8
215/300 Read block 778 len 12288
216/300 Flush
217/300 Read block 4123 len 16384
218/300 Read block 2261 len 4096
219/300 Write block 2765 len 4096 data: 4
220/300 Read block 820 len 40960
221/300 Read block 2449 len 28672
222/300 Write block 3651 len 28672 data: 4 4 4 3 3 3 4
223/300 Flush
224/300 Write block 2538 len 16384 data: 6 5 4 4
225/300 Write block 4373 len 40960 data: 5 5 9 8 9 8 9 9 8 5
226/300 Read block 3294 len 40960
227/300 Read block 4740 len 36864
228/300 Read block 15 len 12288
229/300 Write block 748 len 20480 data: 10 10 10 6 4
230/300 Write block 2717 len 28672 data: 10 8 6 5 5 5 4
231/300 Write block 4058 len 24576 data: 5 6 6 4 6 5
232/300 Write block 1733 len 16384 data: 5 4 4 4
233/300 Read block 407 len 24576
234/300 Write block 4360 len 40960 data: 5 3 3 4 5 4 3 3 3 4
235/300 Read block 1281 len 24576
236/300 Write block 4728 len 20480 data: 6 8 9 8 8
237/300 Read block 4190 len 8192
238/300 Write block 4320 len 40960 data: 8 9 8 9 8 7 6 7 7 5
239/300 Write block 3208 len 16384 data: 9 8 6 6
240/300 Write block 3280 len 24576 data: 4 3 4 3 3 3
241/300 Read block 423 len 20480
242/300 Write block 4595 len 24576 data: 10 12 11 11 8 8
243/300 Read block 1348 len 28672
244/300 Write block 66 len 32768 data: 4 3 3 3 3 4 4 6
245/300 Write block 3335 len 28672 data: 6 6 5 6 6 7 8
246/300 Write block 4140 len 8192 data: 5 6
247/300 Read block 3199 len 24576
248/300 Read block 3535 len 28672
249/300 Read block 1393 len 4096
250/300 Read block 1732 len 28672
251/300 Write block 978 len 24576 data: 6 5 6 5 5 6
252/300 Flush
253/300 Flush
254/300 Read block 2855 len 28672
255/300 Write block 1606 len 4096 data: 4
256/300 Write block 59 len 12288 data: 5 5 5
257/300 Write block 887 len 36864 data: 4 4 3 4 4 4 4 5 5
258/300 Read block 1448 len 8192
259/300 Write block 3701 len 8192 data: 4 5
260/300 Read block 536 len 40960
261/300 Read block 2050 len 12288
262/300 Read block 2334 len 36864
263/300 Write block 4000 len 8192 data: 5 4
264/300 Read block 1676 len 40960
265/300 Read block 256 len 16384
266/300 Write block 308 len 28672 data: 3 2 3 4 3 4 4
267/300 Write block 4491 len 32768 data: 6 5 5 6 7 8 6 6
268/300 Read block 877 len 12288
269/300 Flush
270/300 Write block 3837 len 40960 data: 3 5 5 6 5 5 6 6 7 6
271/300 Read block 4315 len 8192
272/300 Read block 4654 len 36864
273/300 Read block 1272 len 40960
274/300 Read block 3406 len 12288
275/300 Write block 231 len 16384 data: 6 4 4 6
276/300 Flush
277/300 Read block 3877 len 36864
278/300 Read block 2219 len 20480
279/300 Read block 3392 len 4096
280/300 Read block 2645 len 40960
281/300 Write block 2326 len 36864 data: 7 6 4 4 3 3 3 3 3
282/300 Write block 1926 len 4096 data: 4
283/300 Flush
284/300 Write block 1920 len 24576 data: 4 4 2 2 2 3
285/300 Read block 1748 len 40960
286/300 Read block 3423 len 20480
287/300 Read block 2621 len 8192
288/300 Read block 3289 len 16384
289/300 Write block 2206 len 20480 data: 4 3 4 6 5
290/300 Write block 741 len 36864 data: 5 7 8 10 9 9 8 11 11
291/300 Read block 3406 len 4096
292/300 Read block 242 len 28672
293/300 Read block 3727 len 40960
294/300 Write block 3826 len 32768 data: 4 4 4 5 3 4 4 4
295/300 Read block 602 len 16384
296/300 Write block 4241 len 24576 data: 3 3 3 2 2 3
297/300 Read block 1667 len 20480
298/300 Read block 3876 len 8192
299/300 Read block 1884 len 32768
300/300 Write block 3814 len 16384 data: 3 3 4 5
301/300 Flush
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:08:06.658 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1974e88b-c5a3-472e-ba2b-cf14c5563a82, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8849 Acked Read 6 Sent Done Done false
8850 Acked Write 8 Sent Done Done false
8851 Acked Read 4 Sent Done Done false
8852 Acked Write 10 Sent Done Done false
8853 Acked Read 7 Sent Done Done false
8854 Acked Write 8 Sent Done Done false
8855 Acked Read 10 Sent Done Done false
8856 Acked Read 9 Sent Done Done false
8857 Acked Flush 0 Sent Done Done false
8858 Acked Read 2 Sent Done Done false
8859 Acked Read 5 Sent Done Done false
8860 Acked Read 4 Sent Done Done false
8861 Acked Write 5 Sent Done Done false
8862 Acked Write 10 Sent Done Done false
8863 Acked Write 7 Sent Done Done false
8864 Acked Read 7 Sent Done Done false
8865 Acked Flush 0 Sent Done Done false
8866 Acked Read 10 Sent Done Done false
8867 Acked Write 6 Sent Done Done false
8868 Acked Write 6 Sent Done Done false
8869 Acked Read 3 Sent Done Done false
8870 Acked Read 6 Sent Done Done false
8871 Acked Write 8 Sent Done Done false
8872 Acked Write 8 Sent Done Done false
8873 Acked Read 9 Sent Done Done false
8874 Acked Flush 0 Sent Done Done false
8875 Acked Write 1 Sent Done Done false
8876 Acked Read 9 Sent Done Done false
8877 Acked Read 4 Sent Done Done false
8878 Acked Read 7 Sent Done Done false
8879 Acked Read 6 Sent Done Done false
8880 Acked Read 3 Sent Done Done false
8881 Acked Flush 0 Sent Done Done false
8882 Acked Flush 0 Sent Done Done false
8883 Acked Write 3 Sent Done Done false
8884 Acked Read 6 Sent Done Done false
8885 Acked Read 1 Sent Done Done false
8886 Acked Flush 0 Sent Done Done false
8887 Acked Read 1 Sent Done Done false
8888 Acked Read 6 Sent Done Done false
8889 Acked Read 3 Sent Done Done false
8890 Acked Write 8 Sent Done Done false
8891 Acked Write 4 Sent Done Done false
8892 Acked Read 2 Sent Done Done false
8893 Acked Read 2 Sent Done Done false
8894 Acked Read 3 Sent Done Done false
8895 Acked Read 1 Sent Done Done false
8896 Acked Read 5 Sent Done Done false
8897 Acked Write 10 Sent Done Done false
8898 Acked Read 1 Sent Done Done false
8899 Acked Flush 0 Sent Done Done false
8900 Acked Read 3 Sent Done Done false
8901 Acked Write 9 Sent Done Done false
8902 Acked Write 4 Sent Done Done false
8903 Acked Read 9 Sent Done Done false
8904 Acked Write 8 Sent Done Done false
8905 Acked Flush 0 Sent Done Done false
8906 Acked Read 8 Sent Done Done false
8907 Acked Read 8 Sent Done Done false
8908 Acked Write 4 Sent Done Done false
8909 Acked Write 7 Sent Done Done false
8910 Acked Flush 0 Sent Done Done false
8911 Acked Write 3 Sent Done Done false
8912 Acked Write 2 Sent Done Done false
8913 Acked Write 3 Sent Done Done false
8914 Acked Read 1 Sent Done Done false
8915 Acked Flush 0 Sent Done Done false
8916 Acked Read 7 Sent Done Done false
8917 Acked Read 9 Sent Done Done false
8918 Acked Flush 0 Sent Done Done false
8919 Acked Read 5 Sent Done Done false
8920 Acked Read 3 Sent Done Done false
8921 Acked Flush 0 Sent Done Done false
8922 Acked Read 4 Sent Done Done false
8923 Acked Read 6 Sent Done Done false
8924 Acked Write 5 Sent Done Done false
8925 Acked Read 5 Sent Done Done false
8926 Acked Read 10 Sent Done Done false
8927 Acked Read 3 Sent Done Done false
8928 Acked Write 9 Sent Done Done false
8929 Acked Write 3 Sent Done Done false
8930 Acked Write 5 Sent Done Done false
8931 Acked Write 5 Sent Done Done false
8932 Acked Read 9 Sent Done Done false
8933 Acked Write 9 Sent Done Done false
8934 Acked Write 8 Sent Done Done false
8935 Acked Write 6 Sent Done Done false
8936 Acked Write 4 Sent Done Done false
8937 Acked Flush 0 Sent Done Done false
8938 Acked Write 2 Sent Done Done false
8939 Acked Read 2 Sent Done Done false
8940 Acked Read 8 Sent Done Done false
8941 Acked Read 2 Sent Done Done false
8942 Acked Read 6 Sent Done Done false
8943 Acked Write 8 Sent Done Done false
8944 Acked Write 7 Sent Done Done false
8945 Acked Write 1 Sent Done Done false
8946 Acked Read 6 Sent Done Done false
8947 Acked Flush 0 Sent Done Done false
8948 Acked Read 8 Sent Done Done false
8949 Acked Write 8 Sent Done Done false
8950 Acked Write 6 Sent Done Done false
8951 Acked Write 3 Sent Done Done false
8952 Acked Read 3 Sent Done Done false
8953 Acked Write 7 Sent Done Done false
8954 Acked Flush 0 Sent Done Done false
8955 Acked Read 3 Sent Done Done false
8956 Acked Write 2 Sent Done Done false
8957 Acked Write 3 Sent Done Done false
8958 Acked Write 5 Sent Done Done false
8959 Acked Flush 0 Sent Done Done false
8960 Acked Write 8 Sent Done Done false
8961 Acked Write 8 Sent Done Done false
8962 Acked Read 3 Sent Done Done false
8963 Acked Read 6 Sent Done Done false
8964 Acked Read 3 Sent Done Done false
8965 Acked Write 3 Sent Done Done false
8966 Acked Read 3 Sent Done Done false
8967 Acked Read 9 Sent Done Done false
8968 Acked Write 9 Sent Done Done false
8969 Acked Write 10 Sent Done Done false
8970 Acked Write 9 Sent Done Done false
8971 Acked Read 9 Sent Done Done false
8972 Acked Read 7 Sent Done Done false
8973 Acked Read 7 Sent Done Done false
8974 Acked Read 8 Sent Done Done false
8975 Acked Write 3 Sent Done Done false
8976 Acked Read 3 Sent Done Done false
8977 Acked Flush 0 Sent Done Done false
8978 Acked Write 7 Sent Done Done false
8979 Acked Read 7 Sent Done Done false
8980 Acked Read 8 Sent Done Done false
8981 Acked Write 4 Sent Done Done false
8982 Acked Read 5 Sent Done Done false
8983 Acked Read 8 Sent Done Done false
8984 Acked Write 2 Sent Done Done false
8985 Acked Write 9 Sent Done Done false
8986 Acked Write 3 Sent Done Done false
8987 Acked Read 8 Sent Done Done false
8988 Acked Read 3 Sent Done Done false
8989 Acked Write 10 Sent Done Done false
8990 Acked Read 2 Sent Done Done false
8991 Acked Write 8 Sent Done Done false
8992 Acked Read 8 Sent Done Done false
8993 Acked Read 1 Sent Done Done false
8994 Acked Read 2 Sent Done Done false
8995 Acked Write 2 Sent Done Done false
8996 Acked Write 4 Sent Done Done false
8997 Acked Flush 0 Sent Done Done false
8998 Acked Read 3 Sent Done Done false
8999 Acked Write 6 Sent Done Done false
9000 Acked Write 5 Sent Done Done false
9001 Acked Read 1 Sent Done Done false
9002 Acked Read 8 Sent Done Done false
9003 Acked Read 9 Sent Done Done false
9004 Acked Write 3 Sent Done Done false
9005 Acked Flush 0 Sent Done Done false
9006 Acked Write 6 Sent Done Done false
9007 Acked Read 4 Sent Done Done false
9008 Acked Read 10 Sent Done Done false
9009 Acked Read 3 Sent Done Done false
9010 Acked Write 4 Sent Done Done false
9011 Acked Write 8 Sent Done Done false
9012 Acked Write 5 Sent Done Done false
9013 Acked Read 10 Sent Done Done false
9014 Acked Write 6 Sent Done Done false
9015 Acked Write 1 Sent Done Done false
9016 Acked Read 2 Sent Done Done false
9017 Acked Read 10 Sent Done Done false
9018 Acked Write 6 Sent Done Done false
9019 Acked Write 7 Sent Done Done false
9020 Acked Read 8 Sent Done Done false
9021 Acked Read 5 Sent Done Done false
9022 Acked Read 2 Sent Done Done false
9023 Acked Flush 0 Sent Done Done false
9024 Acked Write 3 Sent Done Done false
9025 Acked Read 9 Sent Done Done false
9026 Acked Read 10 Sent Done Done false
9027 Acked Read 9 Sent Done Done false
9028 Acked Write 1 Sent Done Done false
9029 Acked Read 8 Sent Done Done false
9030 Acked Write 7 Sent Done Done false
9031 Acked Read 4 Sent Done Done false
9032 Acked Read 5 Sent Done Done false
9033 Acked Write 4 Sent Done Done false
9034 Acked Read 6 Sent Done Done false
9035 Acked Flush 0 Sent Done Done false
9036 Acked Read 7 Sent Done Done false
9037 Acked Write 9 Sent Done Done false
9038 Acked Read 5 Sent Done Done false
9039 Acked Read 2 Sent Done Done false
9040 Acked Write 3 Sent Done Done false
9041 Acked Read 9 Sent Done Done false
9042 Acked Read 4 Sent Done Done false
9043 Acked Read 9 Sent Done Done false
9044 Acked Write 2 Sent Done Done false
9045 Acked Read 2 Sent Done Done false
9046 Acked Write 10 Sent Done Done false
9047 Acked Flush 0 Sent Done Done false
9048 Acked Write 9 Sent Done Done false
9049 Acked Write 6 Sent Done Done false
9050 Acked Write 6 Sent Done Done false
9051 Acked Read 7 Sent Done Done false
9052 Acked Read 8 Sent Done Done false
9053 Acked Read 6 Sent Done Done false
9054 Acked Read 3 Sent Done Done false
9055 Acked Read 10 Sent Done Done false
9056 Acked Write 7 Sent Done Done false
9057 Acked Read 3 Sent Done Done false
9058 Acked Read 1 Sent Done Done false
9059 Acked Read 9 Sent Done Done false
9060 Acked Flush 0 Sent Done Done false
9061 Acked Read 1 Sent Done Done false
9062 Acked Write 3 Sent Done Done false
9063 Acked Read 3 Sent Done Done false
9064 Acked Flush 0 Sent Done Done false
9065 Acked Read 4 Sent Done Done false
9066 Acked Read 1 Sent Done Done false
9067 Acked Write 1 Sent Done Done false
9068 Acked Read 10 Sent Done Done false
9069 Acked Read 7 Sent Done Done false
9070 Acked Write 7 Sent Done Done false
9071 Acked Flush 0 Sent Done Done false
9072 Acked Write 4 Sent Done Done false
9073 Acked Write 10 Sent Done Done false
9074 Acked Read 10 Sent Done Done false
9075 Acked Read 9 Sent Done Done false
9076 Acked Read 3 Sent Done Done false
9077 Acked Write 5 Sent Done Done false
9078 Acked Write 7 Sent Done Done false
9079 Acked Write 6 Sent Done Done false
9080 Acked Write 4 Sent Done Done false
9081 Acked Read 6 Sent Done Done false
9082 Acked Write 10 Sent Done Done false
9083 Acked Read 6 Sent Done Done false
9084 Acked Write 5 Sent Done Done false
9085 Acked Read 2 Sent Done Done false
9086 Acked Write 10 Sent Done Done false
9087 Acked Write 4 Sent Done Done false
9088 Acked Write 6 Sent Done Done false
9089 Acked Read 5 Sent Done Done false
9090 Acked Write 6 Sent Done Done false
9091 Acked Read 7 Sent Done Done false
9092 Acked Write 8 Sent Done Done false
9093 Acked Write 7 Sent Done Done false
9094 Acked Write 2 Sent Done Done false
9095 Acked Read 6 Sent Done Done false
9096 Acked Read 7 Sent Done Done false
9097 Acked Read 1 Sent Done Done false
9098 Acked Read 7 Sent Done Done false
9099 Acked Write 6 Sent Done Done false
9100 Acked Flush 0 Sent Done Done false
9101 Acked Flush 0 Sent Done Done false
9102 Acked Read 7 Sent Done Done false
9103 Acked Write 1 Sent Done Done false
9104 Acked Write 3 Sent Done Done false
9105 Acked Write 9 Sent Done Done false
9106 Acked Read 2 Sent Done Done false
9107 Acked Write 2 Sent Done Done false
9108 Acked Read 10 Sent Done Done false
9109 Acked Read 3 Sent Done Done false
9110 Acked Read 9 Sent Done Done false
9111 Acked Write 2 Sent Done Done false
9112 Acked Read 10 Sent Done Done false
9113 Acked Read 4 Sent Done Done false
9114 Acked Write 7 Sent Done Done false
9115 Acked Write 8 Sent Done Done false
9116 Acked Read 3 Sent Done Done false
9117 Acked Flush 0 Sent Done Done false
9118 Acked Write 10 Sent Done Done false
9119 Acked Read 2 Sent Done Done false
9120 Acked Read 9 Sent Done Done false
9121 Acked Read 10 Sent Done Done false
9122 Acked Read 3 Sent Done Done false
9123 Acked Write 4 Sent Done Done false
9124 Acked Flush 0 Sent Done Done false
9125 Acked Read 9 Sent Done Done false
9126 Acked Read 5 Sent Done Done false
9127 Acked Read 1 Sent Done Done false
9128 Acked Read 10 Sent Done Done false
9129 Acked Write 9 Sent Done Done false
9130 Acked Write 1 Sent Done Done false
9131 Acked Flush 0 Sent Done Done false
9132 Acked Write 6 Sent Done Done false
9133 Acked Read 10 Sent Done Done false
9134 Acked Read 5 Sent Done Done false
9135 Acked Read 2 Sent Done Done false
9136 Acked Read 4 Sent Done Done false
9137 Acked Write 5 Sent Done Done false
9138 Acked Write 9 Sent Done Done false
9139 Acked Read 1 Sent Done Done false
9140 Acked Read 7 Sent Done Done false
9141 Acked Read 10 Sent Done Done false
9142 Acked Write 8 Sent Done Done false
9143 Acked Read 4 Sent Done Done false
9144 Acked Write 6 Sent Done Done false
9145 Acked Read 5 Sent Done Done false
9146 Acked Read 2 Sent Done Done false
9147 Acked Read 8 Sent Done Done false
9148 Acked Write 4 Sent Done Done false
9149 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8848)) Some(JobId(9149)) Some(JobId(9149))
Downstairs last five completed: 8848 8847 8846 8845 8844
Upstairs last five completed: 9149 9148 9147 9146 9145
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:07.160044003Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:08.401804977Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1238)
[8810][0] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:09.057022619Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:10.059127308Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8849 Acked Read 6 Sent Done Done false
8850 Acked Write 8 Sent Done Done false
8851 Acked Read 4 Sent Done Done false
8852 Acked Write 10 Sent Done Done false
8853 Acked Read 7 Sent Done Done false
8854 Acked Write 8 Sent Done Done false
8855 Acked Read 10 Sent Done Done false
8856 Acked Read 9 Sent Done Done false
8857 Acked Flush 0 Sent Done Done false
8858 Acked Read 2 Sent Done Done false
8859 Acked Read 5 Sent Done Done false
8860 Acked Read 4 Sent Done Done false
8861 Acked Write 5 Sent Done Done false
8862 Acked Write 10 Sent Done Done false
8863 Acked Write 7 Sent Done Done false
8864 Acked Read 7 Sent Done Done false
8865 Acked Flush 0 Sent Done Done false
8866 Acked Read 10 Sent Done Done false
8867 Acked Write 6 Sent Done Done false
8868 Acked Write 6 Sent Done Done false
8869 Acked Read 3 Sent Done Done false
8870 Acked Read 6 Sent Done Done false
8871 Acked Write 8 Sent Done Done false
8872 Acked Write 8 Sent Done Done false
8873 Acked Read 9 Sent Done Done false
8874 Acked Flush 0 Sent Done Done false
8875 Acked Write 1 Sent Done Done false
8876 Acked Read 9 Sent Done Done false
8877 Acked Read 4 Sent Done Done false
8878 Acked Read 7 Sent Done Done false
8879 Acked Read 6 Sent Done Done false
8880 Acked Read 3 Sent Done Done false
8881 Acked Flush 0 Sent Done Done false
8882 Acked Flush 0 Sent Done Done false
8883 Acked Write 3 Sent Done Done false
8884 Acked Read 6 Sent Done Done false
8885 Acked Read 1 Sent Done Done false
8886 Acked Flush 0 Sent Done Done false
8887 Acked Read 1 Sent Done Done false
8888 Acked Read 6 Sent Done Done false
8889 Acked Read 3 Sent Done Done false
8890 Acked Write 8 Sent Done Done false
8891 Acked Write 4 Sent Done Done false
8892 Acked Read 2 Sent Done Done false
8893 Acked Read 2 Sent Done Done false
8894 Acked Read 3 Sent Done Done false
8895 Acked Read 1 Sent Done Done false
8896 Acked Read 5 Sent Done Done false
8897 Acked Write 10 Sent Done Done false
8898 Acked Read 1 Sent Done Done false
8899 Acked Flush 0 Sent Done Done false
8900 Acked Read 3 Sent Done Done false
8901 Acked Write 9 Sent Done Done false
8902 Acked Write 4 Sent Done Done false
8903 Acked Read 9 Sent Done Done false
8904 Acked Write 8 Sent Done Done false
8905 Acked Flush 0 Sent Done Done false
8906 Acked Read 8 Sent Done Done false
8907 Acked Read 8 Sent Done Done false
8908 Acked Write 4 Sent Done Done false
8909 Acked Write 7 Sent Done Done false
8910 Acked Flush 0 Sent Done Done false
8911 Acked Write 3 Sent Done Done false
8912 Acked Write 2 Sent Done Done false
8913 Acked Write 3 Sent Done Done false
8914 Acked Read 1 Sent Done Done false
8915 Acked Flush 0 Sent Done Done false
8916 Acked Read 7 Sent Done Done false
8917 Acked Read 9 Sent Done Done false
8918 Acked Flush 0 Sent Done Done false
8919 Acked Read 5 Sent Done Done false
8920 Acked Read 3 Sent Done Done false
8921 Acked Flush 0 Sent Done Done false
8922 Acked Read 4 Sent Done Done false
8923 Acked Read 6 Sent Done Done false
8924 Acked Write 5 Sent Done Done false
8925 Acked Read 5 Sent Done Done false
8926 Acked Read 10 Sent Done Done false
8927 Acked Read 3 Sent Done Done false
8928 Acked Write 9 Sent Done Done false
8929 Acked Write 3 Sent Done Done false
8930 Acked Write 5 Sent Done Done false
8931 Acked Write 5 Sent Done Done false
8932 Acked Read 9 Sent Done Done false
8933 Acked Write 9 Sent Done Done false
8934 Acked Write 8 Sent Done Done false
8935 Acked Write 6 Sent Done Done false
8936 Acked Write 4 Sent Done Done false
8937 Acked Flush 0 Sent Done Done false
8938 Acked Write 2 Sent Done Done false
8939 Acked Read 2 Sent Done Done false
8940 Acked Read 8 Sent Done Done false
8941 Acked Read 2 Sent Done Done false
8942 Acked Read 6 Sent Done Done false
8943 Acked Write 8 Sent Done Done false
8944 Acked Write 7 Sent Done Done false
8945 Acked Write 1 Sent Done Done false
8946 Acked Read 6 Sent Done Done false
8947 Acked Flush 0 Sent Done Done false
8948 Acked Read 8 Sent Done Done false
8949 Acked Write 8 Sent Done Done false
8950 Acked Write 6 Sent Done Done false
8951 Acked Write 3 Sent Done Done false
8952 Acked Read 3 Sent Done Done false
8953 Acked Write 7 Sent Done Done false
8954 Acked Flush 0 Sent Done Done false
8955 Acked Read 3 Sent Done Done false
8956 Acked Write 2 Sent Done Done false
8957 Acked Write 3 Sent Done Done false
8958 Acked Write 5 Sent Done Done false
8959 Acked Flush 0 Sent Done Done false
8960 Acked Write 8 Sent Done Done false
8961 Acked Write 8 Sent Done Done false
8962 Acked Read 3 Sent Done Done false
8963 Acked Read 6 Sent Done Done false
8964 Acked Read 3 Sent Done Done false
8965 Acked Write 3 Sent Done Done false
8966 Acked Read 3 Sent Done Done false
8967 Acked Read 9 Sent Done Done false
8968 Acked Write 9 Sent Done Done false
8969 Acked Write 10 Sent Done Done false
8970 Acked Write 9 Sent Done Done false
8971 Acked Read 9 Sent Done Done false
8972 Acked Read 7 Sent Done Done false
8973 Acked Read 7 Sent Done Done false
8974 Acked Read 8 Sent Done Done false
8975 Acked Write 3 Sent Done Done false
8976 Acked Read 3 Sent Done Done false
8977 Acked Flush 0 Sent Done Done false
8978 Acked Write 7 Sent Done Done false
8979 Acked Read 7 Sent Done Done false
8980 Acked Read 8 Sent Done Done false
8981 Acked Write 4 Sent Done Done false
8982 Acked Read 5 Sent Done Done false
8983 Acked Read 8 Sent Done Done false
8984 Acked Write 2 Sent Done Done false
8985 Acked Write 9 Sent Done Done false
8986 Acked Write 3 Sent Done Done false
8987 Acked Read 8 Sent Done Done false
8988 Acked Read 3 Sent Done Done false
8989 Acked Write 10 Sent Done Done false
8990 Acked Read 2 Sent Done Done false
8991 Acked Write 8 Sent Done Done false
8992 Acked Read 8 Sent Done Done false
8993 Acked Read 1 Sent Done Done false
8994 Acked Read 2 Sent Done Done false
8995 Acked Write 2 Sent Done Done false
8996 Acked Write 4 Sent Done Done false
8997 Acked Flush 0 Sent Done Done false
8998 Acked Read 3 Sent Done Done false
8999 Acked Write 6 Sent Done Done false
9000 Acked Write 5 Sent Done Done false
9001 Acked Read 1 Sent Done Done false
9002 Acked Read 8 Sent Done Done false
9003 Acked Read 9 Sent Done Done false
9004 Acked Write 3 Sent Done Done false
9005 Acked Flush 0 Sent Done Done false
9006 Acked Write 6 Sent Done Done false
9007 Acked Read 4 Sent Done Done false
9008 Acked Read 10 Sent Done Done false
9009 Acked Read 3 Sent Done Done false
9010 Acked Write 4 Sent Done Done false
9011 Acked Write 8 Sent Done Done false
9012 Acked Write 5 Sent Done Done false
9013 Acked Read 10 Sent Done Done false
9014 Acked Write 6 Sent Done Done false
9015 Acked Write 1 Sent Done Done false
9016 Acked Read 2 Sent Done Done false
9017 Acked Read 10 Sent Done Done false
9018 Acked Write 6 Sent Done Done false
9019 Acked Write 7 Sent Done Done false
9020 Acked Read 8 Sent Done Done false
9021 Acked Read 5 Sent Done Done false
9022 Acked Read 2 Sent Done Done false
9023 Acked Flush 0 Sent Done Done false
9024 Acked Write 3 Sent Done Done false
9025 Acked Read 9 Sent Done Done false
9026 Acked Read 10 Sent Done Done false
9027 Acked Read 9 Sent Done Done false
9028 Acked Write 1 Sent Done Done false
9029 Acked Read 8 Sent Done Done false
9030 Acked Write 7 Sent Done Done false
9031 Acked Read 4 Sent Done Done false
9032 Acked Read 5 Sent Done Done false
9033 Acked Write 4 Sent Done Done false
9034 Acked Read 6 Sent Done Done false
9035 Acked Flush 0 Sent Done Done false
9036 Acked Read 7 Sent Done Done false
9037 Acked Write 9 Sent Done Done false
9038 Acked Read 5 Sent Done Done false
9039 Acked Read 2 Sent Done Done false
9040 Acked Write 3 Sent Done Done false
9041 Acked Read 9 Sent Done Done false
9042 Acked Read 4 Sent Done Done false
9043 Acked Read 9 Sent Done Done false
9044 Acked Write 2 Sent Done Done false
9045 Acked Read 2 Sent Done Done false
9046 Acked Write 10 Sent Done Done false
9047 Acked Flush 0 Sent Done Done false
9048 Acked Write 9 Sent Done Done false
9049 Acked Write 6 Sent Done Done false
9050 Acked Write 6 Sent Done Done false
9051 Acked Read 7 Sent Done Done false
9052 Acked Read 8 Sent Done Done false
9053 Acked Read 6 Sent Done Done false
9054 Acked Read 3 Sent Done Done false
9055 Acked Read 10 Sent Done Done false
9056 Acked Write 7 Sent Done Done false
9057 Acked Read 3 Sent Done Done false
9058 Acked Read 1 Sent Done Done false
9059 Acked Read 9 Sent Done Done false
9060 Acked Flush 0 Sent Done Done false
9061 Acked Read 1 Sent Done Done false
9062 Acked Write 3 Sent Done Done false
9063 Acked Read 3 Sent Done Done false
9064 Acked Flush 0 Sent Done Done false
9065 Acked Read 4 Sent Done Done false
9066 Acked Read 1 Sent Done Done false
9067 Acked Write 1 Sent Done Done false
9068 Acked Read 10 Sent Done Done false
9069 Acked Read 7 Sent Done Done false
9070 Acked Write 7 Sent Done Done false
9071 Acked Flush 0 Sent Done Done false
9072 Acked Write 4 Sent Done Done false
9073 Acked Write 10 Sent Done Done false
9074 Acked Read 10 Sent Done Done false
9075 Acked Read 9 Sent Done Done false
9076 Acked Read 3 Sent Done Done false
9077 Acked Write 5 Sent Done Done false
9078 Acked Write 7 Sent Done Done false
9079 Acked Write 6 Sent Done Done false
9080 Acked Write 4 Sent Done Done false
9081 Acked Read 6 Sent Done Done false
9082 Acked Write 10 Sent Done Done false
9083 Acked Read 6 Sent Done Done false
9084 Acked Write 5 Sent Done Done false
9085 Acked Read 2 Sent Done Done false
9086 Acked Write 10 Sent Done Done false
9087 Acked Write 4 Sent Done Done false
9088 Acked Write 6 Sent Done Done false
9089 Acked Read 5 Sent Done Done false
9090 Acked Write 6 Sent Done Done false
9091 Acked Read 7 Sent Done Done false
9092 Acked Write 8 Sent Done Done false
9093 Acked Write 7 Sent Done Done false
9094 Acked Write 2 Sent Done Done false
9095 Acked Read 6 Sent Done Done false
9096 Acked Read 7 Sent Done Done false
9097 Acked Read 1 Sent Done Done false
9098 Acked Read 7 Sent Done Done false
9099 Acked Write 6 Sent Done Done false
9100 Acked Flush 0 Sent Done Done false
9101 Acked Flush 0 Sent Done Done false
9102 Acked Read 7 Sent Done Done false
9103 Acked Write 1 Sent Done Done false
9104 Acked Write 3 Sent Done Done false
9105 Acked Write 9 Sent Done Done false
9106 Acked Read 2 Sent Done Done false
9107 Acked Write 2 Sent Done Done false
9108 Acked Read 10 Sent Done Done false
9109 Acked Read 3 Sent Done Done false
9110 Acked Read 9 Sent Done Done false
9111 Acked Write 2 Sent Done Done false
9112 Acked Read 10 Sent Done Done false
9113 Acked Read 4 Sent Done Done false
9114 Acked Write 7 Sent Done Done false
9115 Acked Write 8 Sent Done Done false
9116 Acked Read 3 Sent Done Done false
9117 Acked Flush 0 Sent Done Done false
9118 Acked Write 10 Sent Done Done false
9119 Acked Read 2 Sent Done Done false
9120 Acked Read 9 Sent Done Done false
9121 Acked Read 10 Sent Done Done false
9122 Acked Read 3 Sent Done Done false
9123 Acked Write 4 Sent Done Done false
9124 Acked Flush 0 Sent Done Done false
9125 Acked Read 9 Sent Done Done false
9126 Acked Read 5 Sent Done Done false
9127 Acked Read 1 Sent Done Done false
9128 Acked Read 10 Sent Done Done false
9129 Acked Write 9 Sent Done Done false
9130 Acked Write 1 Sent Done Done false
9131 Acked Flush 0 Sent Done Done false
9132 Acked Write 6 Sent Done Done false
9133 Acked Read 10 Sent Done Done false
9134 Acked Read 5 Sent Done Done false
9135 Acked Read 2 Sent Done Done false
9136 Acked Read 4 Sent Done Done false
9137 Acked Write 5 Sent Done Done false
9138 Acked Write 9 Sent Done Done false
9139 Acked Read 1 Sent Done Done false
9140 Acked Read 7 Sent Done Done false
9141 Acked Read 10 Sent Done Done false
9142 Acked Write 8 Sent Done Done false
9143 Acked Read 4 Sent Done Done false
9144 Acked Write 6 Sent Done Done false
9145 Acked Read 5 Sent Done Done false
9146 Acked Read 2 Sent Done Done false
9147 Acked Read 8 Sent Done Done false
9148 Acked Write 4 Sent Done Done false
9149 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(8848)) Some(JobId(9149)) Some(JobId(9149))
Downstairs last five completed: 8848 8847 8846 8845 8844
Upstairs last five completed: 9149 9148 9147 9146 9145
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:11.162009405Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:12.062360669Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.412562111Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56235)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.412681903Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.413670576Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414232411Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414266243Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(8848))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414285048Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414307433Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(8848))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414325989Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(8848))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:12.414524851Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9149 9148 9147 9146 9145
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 28 06:08:14.664 INFO request completed, latency_us: 394, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4f4134da-3066-4fb3-9f33-25f2f200ed69, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:14.665601616Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:08:14.665 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:14.665650154Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
, latency_us: 154, response_code: 200, uri: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:14.66566652Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
/state/cid/2{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:14.665746761Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
, method: GET, req_id: be308082-160c-4f8a-b4f2-94a9c2808a1d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:08:18.668 INFO request completed, latency_us: 376, response_code: 200, uri: /state/cid/2, method: GET, req_id: b08a0b17-0a31-42c3-8f5b-13cc495fb6ec, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Read block 1110 len 24576
002/300 Read block 1886 len 8192
003/300 Read block 1641 len 40960
004/300 Flush
005/300 Write block 1477 len 28672 data: 5 5 5 4 5 5 7
006/300 Read block 361 len 16384
007/300 Read block 1669 len 36864
008/300 Write block 1292 len 36864 data: 9 8 6 5 5 6 7 6 6
009/300 Read block 239 len 40960
010/300 Read block 4077 len 32768
011/300 Read block 3263 len 4096
012/300 Read block 3719 len 40960
013/300 Write block 1634 len 12288 data: 5 4 5
014/300 Read block 3426 len 20480
015/300 Write block 1220 len 12288 data: 5 5 8
016/300 Write block 164 len 12288 data: 5 4 4
017/300 Read block 900 len 32768
018/300 Write block 1829 len 32768 data: 6 4 4 5 4 4 5 7
019/300 Write block 2370 len 12288 data: 4 5 4
020/300 Write block 2064 len 8192 data: 6 7
021/300 Write block 2582 len 8192 data: 7 7
022/300 Write block 3389 len 28672 data: 4 4 4 3 3 3 4
023/300 Write block 4579 len 12288 data: 3 3 3
024/300 Write block 3238 len 8192 data: 3 3
025/300 Write block 3067 len 36864 data: 6 5 5 5 5 6 7 7 6
026/300 Flush
027/300 Write block 2734 len 8192 data: 6 7
028/300 Read block 939 len 4096
029/300 Write block 4659 len 4096 data: 4
030/300 Write block 459 len 32768 data: 5 5 5 4 2 2 4 6
031/300 Write block 4209 len 24576 data: 5 5 5 7 8 6
032/300 Write block 4736 len 28672 data: 8 6 7 7 7 6 5
033/300 Write block 3558 len 4096 data: 5
034/300 Write block 2408 len 20480 data: 2 2 2 3 3
035/300 Read block 1542 len 16384
036/300 Read block 4180 len 36864
037/300 Read block 1944 len 36864
038/300 Flush
039/300 Read block 4214 len 16384
040/300 Write block 344 len 12288 data: 5 8 7
041/300 Write block 4203 len 4096 data: 4
042/300 Read block 1841 len 8192
043/300 Flush
044/300 Read block 4147 len 8192
045/300 Write block 4885 len 28672 data: 8 11 16 15 16 15 14
046/300 Write block 73 len 36864 data: 7 6 9 8 8 7 9 8 10
047/300 Write block 1658 len 32768 data: 6 6 8 7 6 7 7 10
048/300 Flush
049/300 Write block 3879 len 4096 data: 4
050/300 Read block 3291 len 28672
051/300 Read block 2752 len 36864
052/300 Read block 2920 len 36864
053/300 Write block 206 len 8192 data: 6 6
054/300 Write block 190 len 16384 data: 8 9 8 7
055/300 Read block 4395 len 24576
056/300 Read block 937 len 28672
057/300 Read block 4797 len 40960
058/300 Read block 1295 len 12288
059/300 Read block 4402 len 8192
060/300 Read block 2848 len 4096
061/300 Flush
062/300 Read block 3578 len 32768
063/300 Write block 2082 len 28672 data: 5 7 5 7 9 9 9
064/300 Flush
065/300 Write block 2457 len 12288 data: 9 9 7
066/300 Write block 719 len 20480 data: 5 6 6 4 5
067/300 Write block 3152 len 40960 data: 5 5 7 7 10 10 10 11 9 5
068/300 Flush
069/300 Write block 3027 len 28672 data: 6 6 6 6 6 7 6
070/300 Read block 2210 len 20480
071/300 Read block 490 len 12288
072/300 Read block 3735 len 12288
073/300 Write block 2219 len 32768 data: 8 7 4 4 4 4 5 6
074/300 Read block 535 len 36864
075/300 Read block 3739 len 40960
076/300 Write block 4295 len 20480 data: 5 5 5 6 4
077/300 Read block 1143 len 32768
078/300 Write block 3640 len 36864 data: 3 4 3 3 3 5 5 5 5
079/300 Read block 427 len 16384
080/300 Write block 1681 len 32768 data: 5 8 9 9 7 8 7 6
081/300 Read block 2847 len 32768
082/300 Write block 1444 len 16384 data: 6 7 5 4
083/300 Flush
084/300 Read block 1902 len 4096
085/300 Write block 2197 len 16384 data: 5 4 5 7
086/300 Write block 4984 len 28672 data: 7 6 5 5 2 3 3
087/300 Read block 4263 len 4096
088/300 Read block 1029 len 16384
089/300 Write block 2628 len 12288 data: 6 6 6
090/300 Write block 1704 len 4096 data: 6
091/300 Write block 4093 len 24576 data: 5 5 7 7 7 7
092/300 Write block 4173 len 12288 data: 4 5 5
093/300 Read block 2076 len 20480
094/300 Read block 583 len 36864
095/300 Write block 873 len 16384 data: 6 5 4 5
096/300 Write block 1199 len 40960 data: 5 4 4 4 5 4 5 4 4 4
097/300 Read block 288 len 32768
098/300 Flush
099/300 Write block 2611 len 36864 data: 6 4 4 3 4 4 4 4 4
100/300 Flush
101/300 Write block 1344 len 16384 data: 4 4 3 3
102/300 Read block 2393 len 12288
103/300 Read block 665 len 28672
104/300 Flush
105/300 Read block 2061 len 4096
106/300 Read block 4532 len 36864
107/300 Flush
108/300 Read block 2914 len 28672
109/300 Read block 790 len 20480
110/300 Read block 1112 len 12288
111/300 Write block 2742 len 4096 data: 6
112/300 Write block 1091 len 8192 data: 3 3
113/300 Read block 43 len 28672
114/300 Read block 3080 len 36864
115/300 Flush
116/300 Read block 1304 len 4096
117/300 Read block 2874 len 16384
118/300 Write block 3055 len 4096 data: 4
119/300 Read block 678 len 28672
120/300 Read block 3980 len 16384
121/300 Read block 1919 len 40960
122/300 Flush
123/300 Read block 3841 len 28672
124/300 Write block 4310 len 36864 data: 5 5 6 5 5 4 4 4 5
125/300 Read block 273 len 32768
126/300 Read block 2664 len 20480
127/300 Write block 1456 len 40960 data: 6 6 6 8 7 5 6 6 6 5
128/300 Read block 4383 len 4096
129/300 Flush
130/300 Write block 4019 len 32768 data: 6 5 5 5 5 4 4 3
131/300 Read block 3189 len 16384
132/300 Read block 4497 len 32768
133/300 Write block 1976 len 8192 data: 6 4
134/300 Read block 498 len 32768
135/300 Write block 2261 len 28672 data: 7 7 7 9 7 7 8
136/300 Write block 1655 len 16384 data: 4 6 5 7
137/300 Write block 4640 len 28672 data: 6 4 5 4 5 5 6
138/300 Read block 4557 len 32768
139/300 Write block 4073 len 28672 data: 4 4 5 6 7 6 7
140/300 Read block 2159 len 32768
141/300 Read block 3446 len 28672
142/300 Flush
143/300 Read block 1545 len 8192
144/300 Read block 3572 len 32768
145/300 Write block 2460 len 20480 data: 7 6 4 4 3
146/300 Read block 390 len 8192
147/300 Read block 944 len 24576
148/300 Read block 4411 len 32768
149/300 Flush
150/300 Read block 197 len 8192
151/300 Write block 2033 len 24576 data: 6 4 3 3 3 3
152/300 Read block 4825 len 20480
153/300 Read block 817 len 32768
154/300 Read block 2736 len 16384
155/300 Read block 1186 len 36864
156/300 Write block 1627 len 16384 data: 7 8 7 7
157/300 Read block 849 len 32768
158/300 Write block 2111 len 24576 data: 8 7 7 6 5 6
159/300 Read block 4818 len 28672
160/300 Read block 4977 len 12288
161/300 Write block 3495 len 8192 data: 5 4
162/300 Read block 4207 len 12288
163/300 Read block 2164 len 32768
164/300 Read block 1024 len 28672
165/300 Write block 3437 len 8192 data: 7 7
166/300 Read block 299 len 4096
167/300 Flush
168/300 Write block 1852 len 36864 data: 6 4 4 3 3 3 2 2 2
169/300 Read block 306 len 24576
170/300 Write block 3675 len 40960 data: 4 5 4 5 5 6 6 7 6 6
171/300 Read block 1455 len 32768
172/300 Write block 4841 len 16384 data: 7 8 7 6
173/300 Write block 4002 len 28672 data: 4 4 5 5 7 7 7
174/300 Read block 2875 len 20480
175/300 Write block 2610 len 24576 data: 4 7 5 5 4 5
176/300 Write block 1060 len 24576 data: 6 5 5 4 4 6
177/300 Read block 3229 len 24576
178/300 Read block 4622 len 40960
179/300 Write block 262 len 24576 data: 3 3 4 5 5 4
180/300 Read block 801 len 24576
181/300 Read block 3141 len 4096
182/300 Read block 4943 len 36864
183/300 Write block 3119 len 28672 data: 6 4 4 4 4 4 4
184/300 Read block 3631 len 40960
185/300 Read block 4438 len 20480
186/300 Write block 3707 len 20480 data: 8 6 5 4 4
187/300 Read block 4902 len 28672
188/300 Write block 1540 len 4096 data: 7
189/300 Read block 1763 len 40960
190/300 Write block 2833 len 36864 data: 4 3 4 4 5 5 5 5 6
191/300 Write block 3494 len 36864 data: 5 6 5 6 6 6 4 3 4
192/300 Read block 2447 len 16384
193/300 Read block 1970 len 36864
194/300 Read block 508 len 8192
195/300 Write block 1445 len 16384 data: 8 6 5 5
196/300 Read block 2579 len 32768
197/300 Write block 2865 len 32768 data: 4 5 5 5 5 6 7 8
198/300 Write block 291 len 40960 data: 6 7 6 5 7 4 5 5 5 3
199/300 Write block 1752 len 20480 data: 5 5 6 7 6
200/300 Read block 1772 len 4096
201/300 Read block 1700 len 8192
202/300 Write block 721 len 32768 data: 7 5 6 6 4 4 3 3
203/300 Read block 3563 len 12288
204/300 Write block 3306 len 8192 data: 8 9
205/300 Read block 1049 len 16384
206/300 Write block 4687 len 20480 data: 7 5 5 5 5
207/300 Read block 2430 len 4096
208/300 Write block 539 len 4096 data: 4
209/300 Read block 4169 len 40960
210/300 Write block 4275 len 16384 data: 6 5 6 6
211/300 Read block 2902 len 4096
212/300 Read block 1545 len 20480
213/300 Read block 3218 len 32768
214/300 Flush
215/300 Read block 1925 len 24576
216/300 Read block 2555 len 12288
217/300 Read block 3375 len 12288
218/300 Read block 1353 len 40960
219/300 Read block 1384 len 12288
220/300 Read block 4048 len 8192
221/300 Write block 2288 len 40960 data: 4 3 4 6 5 5 3 3 3 4
222/300 Flush
223/300 Read block 4109 len 16384
224/300 Read block 825 len 28672
225/300 Read block 2078 len 28672
226/300 Write block 3861 len 28672 data: 6 6 6 8 8 9 10
227/300 Read block 4896 len 12288
228/300 Flush
229/300 Read block 4299 len 8192
230/300 Write block 1309 len 12288 data: 7 7 8
231/300 Read block 4104 len 28672
232/300 Write block 4352 len 28672 data: 7 8 7 9 7 8 9
233/300 Read block 4000 len 28672
234/300 Read block 1356 len 28672
235/300 Read block 931 len 28672
236/300 Write block 2416 len 12288 data: 4 4 5
237/300 Read block 4047 len 40960
238/300 Flush
239/300 Read block 3814 len 40960
240/300 Write block 605 len 8192 data: 3 3
241/300 Read block 2917 len 20480
242/300 Write block 1222 len 40960 data: 9 8 8 7 8 9 8 7 7 4
243/300 Read block 2145 len 24576
244/300 Write block 3479 len 24576 data: 6 9 8 7 7 6
245/300 Write block 3142 len 4096 data: 4
246/300 Read block 504 len 36864
247/300 Read block 2964 len 4096
248/300 Write block 962 len 32768 data: 4 4 4 5 5 4 4 3
249/300 Read block 3897 len 36864
250/300 Read block 1246 len 8192
251/300 Read block 805 len 28672
252/300 Read block 480 len 32768
253/300 Read block 2163 len 8192
254/300 Read block 3596 len 12288
255/300 Write block 2315 len 4096 data: 3
256/300 Read block 276 len 36864
257/300 Read block 1156 len 4096
258/300 Read block 3710 len 24576
259/300 Read block 2844 len 40960
260/300 Write block 1496 len 12288 data: 5 4 4
261/300 Read block 2244 len 28672
262/300 Write block 12 len 8192 data: 5 3
263/300 Write block 3819 len 16384 data: 5 4 4 5
264/300 Read block 3587 len 20480
265/300 Read block 2295 len 4096
266/300 Write block 3617 len 16384 data: 6 5 6 4
267/300 Read block 3423 len 32768
268/300 Read block 892 len 20480
269/300 Read block 2900 len 36864
270/300 Read block 862 len 12288
271/300 Write block 4460 len 8192 data: 4 6
272/300 Read block 2181 len 36864
273/300 Write block 386 len 32768 data: 6 4 4 4 5 5 5 4
274/300 Write block 4642 len 40960 data: 6 5 6 6 7 9 8 8 7 7
275/300 Read block 2451 len 32768
276/300 Flush
277/300 Read block 4845 len 32768
278/300 Write block 3253 len 12288 data: 5 5 4
279/300 Read block 3270 len 32768
280/300 Read block 2269 len 28672
281/300 Write block 4213 len 32768 data: 9 7 7 7 6 3 2 2
282/300 Flush
283/300 Read block 970 len 24576
284/300 Write block 977 len 20480 data: 6 7 6 7 6
285/300 Write block 1341 len 28672 data: 7 7 6 5 5 4 4
286/300 Read block 3412 len 24576
287/300 Flush
288/300 Write block 920 len 32768 data: 7 7 7 6 6 6 7 7
289/300 Read block 4769 len 12288
290/300 Read block 3196 len 8192
291/300 Read block 2102 len 4096
292/300 Flush
293/300 Read block 1485 len 8192
294/300 Read block 286 len 28672
295/300 Write block 4653 len 28672 data: 9 9 7 5 4 4 5
296/300 Read block 2052 len 36864
297/300 Read block 853 len 12288
298/300 Write block 1180 len 20480 data: 8 7 5 4 6
299/300 Write block 4199 len 20480 data: 5 2 2 2 5
300/300 Write block 4495 len 12288 data: 8 9 7
301/300 Write block 702 len 32768 data: 7 7 8 6 7 7 6 8
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:08:18.917 INFO request completed, latency_us: 192, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8ba06243-e68f-4c20-ad36-5a06a50dbed5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9150 Acked Read 6 Done Done Sent false
9151 Acked Read 2 Done Done Sent false
9152 Acked Read 10 Done Done Sent false
9153 Acked Flush 0 Done Done Sent false
9154 Acked Write 7 Done Done Sent false
9155 Acked Read 4 Done Done Sent false
9156 Acked Read 9 Done Done Sent false
9157 Acked Write 9 Done Done Sent false
9158 Acked Read 10 Done Done Sent false
9159 Acked Read 8 Done Done Sent false
9160 Acked Read 1 Done Done Sent false
9161 Acked Read 10 Done Done Sent false
9162 Acked Write 3 Done Done Sent false
9163 Acked Read 5 Done Done Sent false
9164 Acked Write 3 Done Done Sent false
9165 Acked Write 3 Done Done Sent false
9166 Acked Read 8 Done Done Sent false
9167 Acked Write 8 Done Done Sent false
9168 Acked Write 3 Done Done Sent false
9169 Acked Write 2 Done Done Sent false
9170 Acked Write 2 Done Done Sent false
9171 Acked Write 7 Done Done Sent false
9172 Acked Write 3 Done Done Sent false
9173 Acked Write 2 Done Done Sent false
9174 Acked Write 9 Done Done Sent false
9175 Acked Flush 0 Done Done Sent false
9176 Acked Write 2 Done Done Sent false
9177 Acked Read 1 Done Done Sent false
9178 Acked Write 1 Done Done Sent false
9179 Acked Write 8 Done Done Sent false
9180 Acked Write 6 Done Done Sent false
9181 Acked Write 7 Done Done Sent false
9182 Acked Write 1 Done Done Sent false
9183 Acked Write 5 Done Done Sent false
9184 Acked Read 4 Done Done Sent false
9185 Acked Read 9 Done Done Sent false
9186 Acked Read 9 Done Done Sent false
9187 Acked Flush 0 Done Done Sent false
9188 Acked Read 4 Done Done Sent false
9189 Acked Write 3 Done Done Sent false
9190 Acked Write 1 Done Done Sent false
9191 Acked Read 2 Done Done Sent false
9192 Acked Flush 0 Done Done Sent false
9193 Acked Read 2 Done Done Sent false
9194 Acked Write 7 Done Done Sent false
9195 Acked Write 9 Done Done Sent false
9196 Acked Write 8 Done Done Sent false
9197 Acked Flush 0 Done Done Sent false
9198 Acked Write 1 Done Done Sent false
9199 Acked Read 7 Done Done Sent false
9200 Acked Read 9 Done Done Sent false
9201 Acked Read 9 Done Done Sent false
9202 Acked Write 2 Done Done Sent false
9203 Acked Write 4 Done Done Sent false
9204 Acked Read 6 Done Done Sent false
9205 Acked Read 7 Done Done Sent false
9206 Acked Read 10 Done Done Sent false
9207 Acked Read 3 Done Done Sent false
9208 Acked Read 2 Done Done Sent false
9209 Acked Read 1 Done Done Sent false
9210 Acked Flush 0 Done Done Sent false
9211 Acked Read 8 Done Done Sent false
9212 Acked Write 7 Done Done Sent false
9213 Acked Flush 0 Done Done Sent false
9214 Acked Write 3 Done Done Sent false
9215 Acked Write 5 Done Done Sent false
9216 Acked Write 10 Done Done Sent false
9217 Acked Flush 0 Done Done Sent false
9218 Acked Write 7 Done Done Sent false
9219 Acked Read 5 Done Done Sent false
9220 Acked Read 3 Done Done Sent false
9221 Acked Read 3 Done Done Sent false
9222 Acked Write 8 Done Done Sent false
9223 Acked Read 9 Done Done Sent false
9224 Acked Read 10 Done Done Sent false
9225 Acked Write 5 Done Done Sent false
9226 Acked Read 8 Done Done Sent false
9227 Acked Write 9 Done Done Sent false
9228 Acked Read 4 Done Done Sent false
9229 Acked Write 8 Done Done Sent false
9230 Acked Read 8 Done Done Sent false
9231 Acked Write 4 Done Done Sent false
9232 Acked Flush 0 Done Done Sent false
9233 Acked Read 1 Done Done Sent false
9234 Acked Write 4 Done Done Sent false
9235 Acked Write 7 Done Done Sent false
9236 Acked Read 1 Done Done Sent false
9237 Acked Read 4 Done Done Sent false
9238 Acked Write 3 Done Done Sent false
9239 Acked Write 1 Done Done Sent false
9240 Acked Write 6 Done Done Sent false
9241 Acked Write 3 Done Done Sent false
9242 Acked Read 5 Done Done Sent false
9243 Acked Read 9 Done Done Sent false
9244 Acked Write 4 Done Done Sent false
9245 Acked Write 10 Done Done Sent false
9246 Acked Read 8 Done Done Sent false
9247 Acked Flush 0 Done Done Sent false
9248 Acked Write 9 Done Done Sent false
9249 Acked Flush 0 Done Done Sent false
9250 Acked Write 4 Done Done Sent false
9251 Acked Read 3 Done Done Sent false
9252 Acked Read 7 Done Done Sent false
9253 Acked Flush 0 Done Done Sent false
9254 Acked Read 1 Done Done Sent false
9255 Acked Read 9 Done Done Sent false
9256 Acked Flush 0 Done Done Sent false
9257 Acked Read 7 Done Done Sent false
9258 Acked Read 5 Done Done Sent false
9259 Acked Read 3 Done Done Sent false
9260 Acked Write 1 Done Done Sent false
9261 Acked Write 2 Done Done Sent false
9262 Acked Read 7 Done Done Sent false
9263 Acked Read 9 Done Done Sent false
9264 Acked Flush 0 Done Done Sent false
9265 Acked Read 1 Done Done Sent false
9266 Acked Read 4 Done Done Sent false
9267 Acked Write 1 Done Done Sent false
9268 Acked Read 7 Done Done Sent false
9269 Acked Read 4 Done Done Sent false
9270 Acked Read 10 Done Done Sent false
9271 Acked Flush 0 Done Done Sent false
9272 Acked Read 7 Done Done Sent false
9273 Acked Write 9 Done Done Sent false
9274 Acked Read 8 Done Done Sent false
9275 Acked Read 5 Done Done Sent false
9276 Acked Write 10 Done Done Sent false
9277 Acked Read 1 Done Done Sent false
9278 Acked Flush 0 Done Done Sent false
9279 Acked Write 8 Done Done Sent false
9280 Acked Read 4 Done Done Sent false
9281 Acked Read 8 Done Done Sent false
9282 Acked Write 2 Done Done Sent false
9283 Acked Read 8 Done Done Sent false
9284 Acked Write 7 Done Done Sent false
9285 Acked Write 4 Done Done Sent false
9286 Acked Write 7 Done Done Sent false
9287 Acked Read 8 Done Done Sent false
9288 Acked Write 7 Done Done Sent false
9289 Acked Read 8 Done Done Sent false
9290 Acked Read 7 Done Done Sent false
9291 Acked Flush 0 Done Done Sent false
9292 Acked Read 2 Done Done Sent false
9293 Acked Read 8 Done Done Sent false
9294 Acked Write 5 Done Done Sent false
9295 Acked Read 2 Done Done Sent false
9296 Acked Read 6 Done Done Sent false
9297 Acked Read 8 Done Done Sent false
9298 Acked Flush 0 Done Done Sent false
9299 Acked Read 2 Done Done Sent false
9300 Acked Write 6 Done Done Sent false
9301 Acked Read 5 Done Done Sent false
9302 Acked Read 8 Done Done Sent false
9303 Acked Read 4 Done Done Sent false
9304 Acked Read 9 Done Done Sent false
9305 Acked Write 4 Done Done Sent false
9306 Acked Read 8 Done Done Sent false
9307 Acked Write 6 Done Done Sent false
9308 Acked Read 7 Done Done Sent false
9309 Acked Read 3 Done Done Sent false
9310 Acked Write 2 Done Done Sent false
9311 Acked Read 3 Done Done Sent false
9312 Acked Read 8 Done Done Sent false
9313 Acked Read 7 Done Done Sent false
9314 Acked Write 2 Done Done Sent false
9315 Acked Read 1 Done Done Sent false
9316 Acked Flush 0 Done Done Sent false
9317 Acked Write 9 Done Done Sent false
9318 Acked Read 6 Done Done Sent false
9319 Acked Write 10 Done Done Sent false
9320 Acked Read 8 Done Done Sent false
9321 Acked Write 4 Done Done Sent false
9322 Acked Write 7 Done Done Sent false
9323 Acked Read 5 Done Done Sent false
9324 Acked Write 6 Done Done Sent false
9325 Acked Write 6 Done Done Sent false
9326 Acked Read 6 Done Done Sent false
9327 Acked Read 10 Done Done Sent false
9328 Acked Write 6 Done Done Sent false
9329 Acked Read 6 Done Done Sent false
9330 Acked Read 1 Done Done Sent false
9331 Acked Read 9 Done Done Sent false
9332 Acked Write 7 Done Done Sent false
9333 Acked Read 10 Done Done Sent false
9334 Acked Read 5 Done Done Sent false
9335 Acked Write 5 Done Done Sent false
9336 Acked Read 7 Done Done Sent false
9337 Acked Write 1 Done Done Sent false
9338 Acked Read 10 Done Done Sent false
9339 Acked Write 9 Done Done Sent false
9340 Acked Write 9 Done Done Sent false
9341 Acked Read 4 Done Done Sent false
9342 Acked Read 9 Done Done Sent false
9343 Acked Read 2 Done Done Sent false
9344 Acked Write 4 Done Done Sent false
9345 Acked Read 8 Done Done Sent false
9346 Acked Write 8 Done Done Sent false
9347 Acked Write 10 Done Done Sent false
9348 Acked Write 5 Done Done Sent false
9349 Acked Read 1 Done Done Sent false
9350 Acked Read 2 Done Done Sent false
9351 Acked Write 8 Done Done Sent false
9352 Acked Read 3 Done Done Sent false
9353 Acked Write 2 Done Done Sent false
9354 Acked Read 4 Done Done Sent false
9355 Acked Write 5 Done Done Sent false
9356 Acked Read 1 Done Done Sent false
9357 Acked Write 1 Done Done Sent false
9358 Acked Read 10 Done Done Sent false
9359 Acked Write 4 Done Done Sent false
9360 Acked Read 1 Done Done Sent false
9361 Acked Read 5 Done Done Sent false
9362 Acked Read 8 Done Done Sent false
9363 Acked Flush 0 Done Done Sent false
9364 Acked Read 6 Done Done Sent false
9365 Acked Read 3 Done Done Sent false
9366 Acked Read 3 Done Done Sent false
9367 Acked Read 10 Done Done Sent false
9368 Acked Read 3 Done Done Sent false
9369 Acked Read 2 Done Done Sent false
9370 Acked Write 10 Done Done Sent false
9371 Acked Flush 0 Done Done Sent false
9372 Acked Read 4 Done Done Sent false
9373 Acked Read 7 Done Done Sent false
9374 Acked Read 7 Done Done Sent false
9375 Acked Write 7 Done Done Sent false
9376 Acked Read 3 Done Done Sent false
9377 Acked Flush 0 Done Done Sent false
9378 Acked Read 2 Done Done Sent false
9379 Acked Write 3 Done Done Sent false
9380 Acked Read 7 Done Done Sent false
9381 Acked Write 7 Done Done Sent false
9382 Acked Read 7 Done Done Sent false
9383 Acked Read 7 Done Done Sent false
9384 Acked Read 7 Done Done Sent false
9385 Acked Write 3 Done Done Sent false
9386 Acked Read 10 Done Done Sent false
9387 Acked Flush 0 Done Done Sent false
9388 Acked Read 10 Done Done Sent false
9389 Acked Write 2 Done Done Sent false
9390 Acked Read 5 Done Done Sent false
9391 Acked Write 10 Done Done Sent false
9392 Acked Read 6 Done Done Sent false
9393 Acked Write 6 Done Done Sent false
9394 Acked Write 1 Done Done Sent false
9395 Acked Read 9 Done Done Sent false
9396 Acked Read 1 Done Done Sent false
9397 Acked Write 8 Done Done Sent false
9398 Acked Read 9 Done Done Sent false
9399 Acked Read 2 Done Done Sent false
9400 Acked Read 7 Done Done Sent false
9401 Acked Read 8 Done Done Sent false
9402 Acked Read 2 Done Done Sent false
9403 Acked Read 3 Done Done Sent false
9404 Acked Write 1 Done Done Sent false
9405 Acked Read 9 Done Done Sent false
9406 Acked Read 1 Done Done Sent false
9407 Acked Read 6 Done Done Sent false
9408 Acked Read 10 Done Done Sent false
9409 Acked Write 3 Done Done Sent false
9410 Acked Read 7 Done Done Sent false
9411 Acked Write 2 Done Done Sent false
9412 Acked Write 4 Done Done Sent false
9413 Acked Read 5 Done Done Sent false
9414 Acked Read 1 Done Done Sent false
9415 Acked Write 4 Done Done Sent false
9416 Acked Read 8 Done Done Sent false
9417 Acked Read 5 Done Done Sent false
9418 Acked Read 9 Done Done Sent false
9419 Acked Read 3 Done Done Sent false
9420 Acked Write 2 Done Done Sent false
9421 Acked Read 9 Done Done Sent false
9422 Acked Write 8 Done Done Sent false
9423 Acked Write 10 Done Done Sent false
9424 Acked Read 8 Done Done Sent false
9425 Acked Flush 0 Done Done Sent false
9426 Acked Read 8 Done Done Sent false
9427 Acked Write 3 Done Done Sent false
9428 Acked Read 8 Done Done Sent false
9429 Acked Read 7 Done Done Sent false
9430 Acked Write 8 Done Done Sent false
9431 Acked Flush 0 Done Done Sent false
9432 Acked Read 6 Done Done Sent false
9433 Acked Write 5 Done Done Sent false
9434 Acked Write 7 Done Done Sent false
9435 Acked Read 6 Done Done Sent false
9436 Acked Flush 0 Done Done Sent false
9437 Acked Write 8 Done Done Sent false
9438 Acked Read 3 Done Done Sent false
9439 Acked Read 2 Done Done Sent false
9440 Acked Read 1 Done Done Sent false
9441 Acked Flush 0 Done Done Sent false
9442 Acked Read 2 Done Done Sent false
9443 Acked Read 7 Done Done Sent false
9444 Acked Write 7 Done Done Sent false
9445 Acked Read 9 Done Done Sent false
9446 Acked Read 3 Done Done Sent false
9447 Acked Write 5 Done Done Sent false
9448 Acked Write 5 Done Done Sent false
9449 Acked Write 3 Done Done Sent false
9450 Acked Write 8 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9450 9449 9448 9447 9446
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:19.419778975Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:20.076696791Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1239)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:21.077594888Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:22.078486298Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9150 Acked Read 6 Done Done Sent false
9151 Acked Read 2 Done Done Sent false
9152 Acked Read 10 Done Done Sent false
9153 Acked Flush 0 Done Done Sent false
9154 Acked Write 7 Done Done Sent false
9155 Acked Read 4 Done Done Sent false
9156 Acked Read 9 Done Done Sent false
9157 Acked Write 9 Done Done Sent false
9158 Acked Read 10 Done Done Sent false
9159 Acked Read 8 Done Done Sent false
9160 Acked Read 1 Done Done Sent false
9161 Acked Read 10 Done Done Sent false
9162 Acked Write 3 Done Done Sent false
9163 Acked Read 5 Done Done Sent false
9164 Acked Write 3 Done Done Sent false
9165 Acked Write 3 Done Done Sent false
9166 Acked Read 8 Done Done Sent false
9167 Acked Write 8 Done Done Sent false
9168 Acked Write 3 Done Done Sent false
9169 Acked Write 2 Done Done Sent false
9170 Acked Write 2 Done Done Sent false
9171 Acked Write 7 Done Done Sent false
9172 Acked Write 3 Done Done Sent false
9173 Acked Write 2 Done Done Sent false
9174 Acked Write 9 Done Done Sent false
9175 Acked Flush 0 Done Done Sent false
9176 Acked Write 2 Done Done Sent false
9177 Acked Read 1 Done Done Sent false
9178 Acked Write 1 Done Done Sent false
9179 Acked Write 8 Done Done Sent false
9180 Acked Write 6 Done Done Sent false
9181 Acked Write 7 Done Done Sent false
9182 Acked Write 1 Done Done Sent false
9183 Acked Write 5 Done Done Sent false
9184 Acked Read 4 Done Done Sent false
9185 Acked Read 9 Done Done Sent false
9186 Acked Read 9 Done Done Sent false
9187 Acked Flush 0 Done Done Sent false
9188 Acked Read 4 Done Done Sent false
9189 Acked Write 3 Done Done Sent false
9190 Acked Write 1 Done Done Sent false
9191 Acked Read 2 Done Done Sent false
9192 Acked Flush 0 Done Done Sent false
9193 Acked Read 2 Done Done Sent false
9194 Acked Write 7 Done Done Sent false
9195 Acked Write 9 Done Done Sent false
9196 Acked Write 8 Done Done Sent false
9197 Acked Flush 0 Done Done Sent false
9198 Acked Write 1 Done Done Sent false
9199 Acked Read 7 Done Done Sent false
9200 Acked Read 9 Done Done Sent false
9201 Acked Read 9 Done Done Sent false
9202 Acked Write 2 Done Done Sent false
9203 Acked Write 4 Done Done Sent false
9204 Acked Read 6 Done Done Sent false
9205 Acked Read 7 Done Done Sent false
9206 Acked Read 10 Done Done Sent false
9207 Acked Read 3 Done Done Sent false
9208 Acked Read 2 Done Done Sent false
9209 Acked Read 1 Done Done Sent false
9210 Acked Flush 0 Done Done Sent false
9211 Acked Read 8 Done Done Sent false
9212 Acked Write 7 Done Done Sent false
9213 Acked Flush 0 Done Done Sent false
9214 Acked Write 3 Done Done Sent false
9215 Acked Write 5 Done Done Sent false
9216 Acked Write 10 Done Done Sent false
9217 Acked Flush 0 Done Done Sent false
9218 Acked Write 7 Done Done Sent false
9219 Acked Read 5 Done Done Sent false
9220 Acked Read 3 Done Done Sent false
9221 Acked Read 3 Done Done Sent false
9222 Acked Write 8 Done Done Sent false
9223 Acked Read 9 Done Done Sent false
9224 Acked Read 10 Done Done Sent false
9225 Acked Write 5 Done Done Sent false
9226 Acked Read 8 Done Done Sent false
9227 Acked Write 9 Done Done Sent false
9228 Acked Read 4 Done Done Sent false
9229 Acked Write 8 Done Done Sent false
9230 Acked Read 8 Done Done Sent false
9231 Acked Write 4 Done Done Sent false
9232 Acked Flush 0 Done Done Sent false
9233 Acked Read 1 Done Done Sent false
9234 Acked Write 4 Done Done Sent false
9235 Acked Write 7 Done Done Sent false
9236 Acked Read 1 Done Done Sent false
9237 Acked Read 4 Done Done Sent false
9238 Acked Write 3 Done Done Sent false
9239 Acked Write 1 Done Done Sent false
9240 Acked Write 6 Done Done Sent false
9241 Acked Write 3 Done Done Sent false
9242 Acked Read 5 Done Done Sent false
9243 Acked Read 9 Done Done Sent false
9244 Acked Write 4 Done Done Sent false
9245 Acked Write 10 Done Done Sent false
9246 Acked Read 8 Done Done Sent false
9247 Acked Flush 0 Done Done Sent false
9248 Acked Write 9 Done Done Sent false
9249 Acked Flush 0 Done Done Sent false
9250 Acked Write 4 Done Done Sent false
9251 Acked Read 3 Done Done Sent false
9252 Acked Read 7 Done Done Sent false
9253 Acked Flush 0 Done Done Sent false
9254 Acked Read 1 Done Done Sent false
9255 Acked Read 9 Done Done Sent false
9256 Acked Flush 0 Done Done Sent false
9257 Acked Read 7 Done Done Sent false
9258 Acked Read 5 Done Done Sent false
9259 Acked Read 3 Done Done Sent false
9260 Acked Write 1 Done Done Sent false
9261 Acked Write 2 Done Done Sent false
9262 Acked Read 7 Done Done Sent false
9263 Acked Read 9 Done Done Sent false
9264 Acked Flush 0 Done Done Sent false
9265 Acked Read 1 Done Done Sent false
9266 Acked Read 4 Done Done Sent false
9267 Acked Write 1 Done Done Sent false
9268 Acked Read 7 Done Done Sent false
9269 Acked Read 4 Done Done Sent false
9270 Acked Read 10 Done Done Sent false
9271 Acked Flush 0 Done Done Sent false
9272 Acked Read 7 Done Done Sent false
9273 Acked Write 9 Done Done Sent false
9274 Acked Read 8 Done Done Sent false
9275 Acked Read 5 Done Done Sent false
9276 Acked Write 10 Done Done Sent false
9277 Acked Read 1 Done Done Sent false
9278 Acked Flush 0 Done Done Sent false
9279 Acked Write 8 Done Done Sent false
9280 Acked Read 4 Done Done Sent false
9281 Acked Read 8 Done Done Sent false
9282 Acked Write 2 Done Done Sent false
9283 Acked Read 8 Done Done Sent false
9284 Acked Write 7 Done Done Sent false
9285 Acked Write 4 Done Done Sent false
9286 Acked Write 7 Done Done Sent false
9287 Acked Read 8 Done Done Sent false
9288 Acked Write 7 Done Done Sent false
9289 Acked Read 8 Done Done Sent false
9290 Acked Read 7 Done Done Sent false
9291 Acked Flush 0 Done Done Sent false
9292 Acked Read 2 Done Done Sent false
9293 Acked Read 8 Done Done Sent false
9294 Acked Write 5 Done Done Sent false
9295 Acked Read 2 Done Done Sent false
9296 Acked Read 6 Done Done Sent false
9297 Acked Read 8 Done Done Sent false
9298 Acked Flush 0 Done Done Sent false
9299 Acked Read 2 Done Done Sent false
9300 Acked Write 6 Done Done Sent false
9301 Acked Read 5 Done Done Sent false
9302 Acked Read 8 Done Done Sent false
9303 Acked Read 4 Done Done Sent false
9304 Acked Read 9 Done Done Sent false
9305 Acked Write 4 Done Done Sent false
9306 Acked Read 8 Done Done Sent false
9307 Acked Write 6 Done Done Sent false
9308 Acked Read 7 Done Done Sent false
9309 Acked Read 3 Done Done Sent false
9310 Acked Write 2 Done Done Sent false
9311 Acked Read 3 Done Done Sent false
9312 Acked Read 8 Done Done Sent false
9313 Acked Read 7 Done Done Sent false
9314 Acked Write 2 Done Done Sent false
9315 Acked Read 1 Done Done Sent false
9316 Acked Flush 0 Done Done Sent false
9317 Acked Write 9 Done Done Sent false
9318 Acked Read 6 Done Done Sent false
9319 Acked Write 10 Done Done Sent false
9320 Acked Read 8 Done Done Sent false
9321 Acked Write 4 Done Done Sent false
9322 Acked Write 7 Done Done Sent false
9323 Acked Read 5 Done Done Sent false
9324 Acked Write 6 Done Done Sent false
9325 Acked Write 6 Done Done Sent false
9326 Acked Read 6 Done Done Sent false
9327 Acked Read 10 Done Done Sent false
9328 Acked Write 6 Done Done Sent false
9329 Acked Read 6 Done Done Sent false
9330 Acked Read 1 Done Done Sent false
9331 Acked Read 9 Done Done Sent false
9332 Acked Write 7 Done Done Sent false
9333 Acked Read 10 Done Done Sent false
9334 Acked Read 5 Done Done Sent false
9335 Acked Write 5 Done Done Sent false
9336 Acked Read 7 Done Done Sent false
9337 Acked Write 1 Done Done Sent false
9338 Acked Read 10 Done Done Sent false
9339 Acked Write 9 Done Done Sent false
9340 Acked Write 9 Done Done Sent false
9341 Acked Read 4 Done Done Sent false
9342 Acked Read 9 Done Done Sent false
9343 Acked Read 2 Done Done Sent false
9344 Acked Write 4 Done Done Sent false
9345 Acked Read 8 Done Done Sent false
9346 Acked Write 8 Done Done Sent false
9347 Acked Write 10 Done Done Sent false
9348 Acked Write 5 Done Done Sent false
9349 Acked Read 1 Done Done Sent false
9350 Acked Read 2 Done Done Sent false
9351 Acked Write 8 Done Done Sent false
9352 Acked Read 3 Done Done Sent false
9353 Acked Write 2 Done Done Sent false
9354 Acked Read 4 Done Done Sent false
9355 Acked Write 5 Done Done Sent false
9356 Acked Read 1 Done Done Sent false
9357 Acked Write 1 Done Done Sent false
9358 Acked Read 10 Done Done Sent false
9359 Acked Write 4 Done Done Sent false
9360 Acked Read 1 Done Done Sent false
9361 Acked Read 5 Done Done Sent false
9362 Acked Read 8 Done Done Sent false
9363 Acked Flush 0 Done Done Sent false
9364 Acked Read 6 Done Done Sent false
9365 Acked Read 3 Done Done Sent false
9366 Acked Read 3 Done Done Sent false
9367 Acked Read 10 Done Done Sent false
9368 Acked Read 3 Done Done Sent false
9369 Acked Read 2 Done Done Sent false
9370 Acked Write 10 Done Done Sent false
9371 Acked Flush 0 Done Done Sent false
9372 Acked Read 4 Done Done Sent false
9373 Acked Read 7 Done Done Sent false
9374 Acked Read 7 Done Done Sent false
9375 Acked Write 7 Done Done Sent false
9376 Acked Read 3 Done Done Sent false
9377 Acked Flush 0 Done Done Sent false
9378 Acked Read 2 Done Done Sent false
9379 Acked Write 3 Done Done Sent false
9380 Acked Read 7 Done Done Sent false
9381 Acked Write 7 Done Done Sent false
9382 Acked Read 7 Done Done Sent false
9383 Acked Read 7 Done Done Sent false
9384 Acked Read 7 Done Done Sent false
9385 Acked Write 3 Done Done Sent false
9386 Acked Read 10 Done Done Sent false
9387 Acked Flush 0 Done Done Sent false
9388 Acked Read 10 Done Done Sent false
9389 Acked Write 2 Done Done Sent false
9390 Acked Read 5 Done Done Sent false
9391 Acked Write 10 Done Done Sent false
9392 Acked Read 6 Done Done Sent false
9393 Acked Write 6 Done Done Sent false
9394 Acked Write 1 Done Done Sent false
9395 Acked Read 9 Done Done Sent false
9396 Acked Read 1 Done Done Sent false
9397 Acked Write 8 Done Done Sent false
9398 Acked Read 9 Done Done Sent false
9399 Acked Read 2 Done Done Sent false
9400 Acked Read 7 Done Done Sent false
9401 Acked Read 8 Done Done Sent false
9402 Acked Read 2 Done Done Sent false
9403 Acked Read 3 Done Done Sent false
9404 Acked Write 1 Done Done Sent false
9405 Acked Read 9 Done Done Sent false
9406 Acked Read 1 Done Done Sent false
9407 Acked Read 6 Done Done Sent false
9408 Acked Read 10 Done Done Sent false
9409 Acked Write 3 Done Done Sent false
9410 Acked Read 7 Done Done Sent false
9411 Acked Write 2 Done Done Sent false
9412 Acked Write 4 Done Done Sent false
9413 Acked Read 5 Done Done Sent false
9414 Acked Read 1 Done Done Sent false
9415 Acked Write 4 Done Done Sent false
9416 Acked Read 8 Done Done Sent false
9417 Acked Read 5 Done Done Sent false
9418 Acked Read 9 Done Done Sent false
9419 Acked Read 3 Done Done Sent false
9420 Acked Write 2 Done Done Sent false
9421 Acked Read 9 Done Done Sent false
9422 Acked Write 8 Done Done Sent false
9423 Acked Write 10 Done Done Sent false
9424 Acked Read 8 Done Done Sent false
9425 Acked Flush 0 Done Done Sent false
9426 Acked Read 8 Done Done Sent false
9427 Acked Write 3 Done Done Sent false
9428 Acked Read 8 Done Done Sent false
9429 Acked Read 7 Done Done Sent false
9430 Acked Write 8 Done Done Sent false
9431 Acked Flush 0 Done Done Sent false
9432 Acked Read 6 Done Done Sent false
9433 Acked Write 5 Done Done Sent false
9434 Acked Write 7 Done Done Sent false
9435 Acked Read 6 Done Done Sent false
9436 Acked Flush 0 Done Done Sent false
9437 Acked Write 8 Done Done Sent false
9438 Acked Read 3 Done Done Sent false
9439 Acked Read 2 Done Done Sent false
9440 Acked Read 1 Done Done Sent false
9441 Acked Flush 0 Done Done Sent false
9442 Acked Read 2 Done Done Sent false
9443 Acked Read 7 Done Done Sent false
9444 Acked Write 7 Done Done Sent false
9445 Acked Read 9 Done Done Sent false
9446 Acked Read 3 Done Done Sent false
9447 Acked Write 5 Done Done Sent false
9448 Acked Write 5 Done Done Sent false
9449 Acked Write 3 Done Done Sent false
9450 Acked Write 8 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9450 9449 9448 9447 9446
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:23.420823318Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:24.081699685Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.667022839Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36846)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.667206015Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.668535226Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669066729Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669089234Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(9149))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669106279Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669122606Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(9149))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669138731Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(9149))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:24.669319488Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9451 9450 9449 9448 9447
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 28 06:08:26.922[8830] Got stop action so:false kr:false
INFO request completed, latency_us: 422, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 29d28d84-867d-483e-b263-789c7811d6cc, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:26.923147833Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:26.923210388Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
Feb 28 06:08:26.923 INFO request completed, latency_us: 126{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:26.92324131Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6e35b112-8f94-4494-a048-d235735ff40d, remote_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:26.923309244Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
127.0.0.1:63908, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:08:30.925 INFO request completed, latency_us: 338, response_code: 200, uri: /state/cid/2, method: GET, req_id: 355d57f7-e0ba-4492-a819-da899d5d76cf, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Write block 336 len 40960 data: 6 7 8 5 5 7 6 6 6 9
002/300 Write block 2873 len 40960 data: 7 7 4 5 5 6 5 4 3 4
003/300 Read block 3945 len 16384
004/300 Read block 629 len 12288
005/300 Read block 1236 len 28672
006/300 Read block 880 len 16384
007/300 Read block 3563 len 8192
008/300 Read block 4438 len 8192
009/300 Write block 1806 len 40960 data: 4 3 3 5 6 6 5 5 5 4
010/300 Write block 2596 len 20480 data: 4 4 4 5 6
011/300 Read block 3356 len 40960
012/300 Read block 2383 len 40960
013/300 Read block 4339 len 16384
014/300 Flush
015/300 Write block 1717 len 32768 data: 6 5 4 6 7 9 9 9
016/300 Read block 851 len 12288
017/300 Write block 3171 len 16384 data: 4 4 3 3
018/300 Write block 4391 len 4096 data: 9
019/300 Read block 1329 len 28672
020/300 Write block 1030 len 4096 data: 3
021/300 Read block 3337 len 40960
022/300 Write block 1738 len 24576 data: 5 6 6 8 5 6
023/300 Write block 1926 len 16384 data: 5 6 5 6
024/300 Read block 3048 len 36864
025/300 Read block 666 len 28672
026/300 Read block 3802 len 24576
027/300 Flush
028/300 Read block 4645 len 24576
029/300 Read block 2952 len 28672
030/300 Write block 1255 len 40960 data: 9 8 7 5 3 4 8 8 7 9
031/300 Read block 4199 len 20480
032/300 Flush
033/300 Write block 508 len 28672 data: 6 7 4 4 3 3 3
034/300 Read block 4576 len 24576
035/300 Read block 254 len 36864
036/300 Read block 319 len 24576
037/300 Flush
038/300 Read block 2585 len 16384
039/300 Write block 4330 len 28672 data: 5 3 4 6 5 5 8
040/300 Flush
041/300 Read block 607 len 16384
042/300 Read block 3531 len 16384
043/300 Flush
044/300 Read block 3739 len 32768
045/300 Write block 4321 len 16384 data: 10 9 10 9
046/300 Write block 2752 len 4096 data: 6
047/300 Read block 692 len 16384
048/300 Write block 1310 len 32768 data: 8 9 8 8 4 4 5 4
049/300 Write block 2872 len 24576 data: 9 8 8 5 6 6
050/300 Write block 4814 len 24576 data: 3 4 4 4 5 5
051/300 Read block 298 len 20480
052/300 Flush
053/300 Read block 342 len 28672
054/300 Write block 1332 len 8192 data: 6 3
055/300 Read block 3663 len 36864
056/300 Read block 4929 len 12288
057/300 Read block 3970 len 40960
058/300 Read block 1096 len 32768
059/300 Read block 3852 len 24576
060/300 Write block 4436 len 4096 data: 6
061/300 Read block 3980 len 8192
062/300 Read block 476 len 40960
063/300 Read block 3587 len 28672
064/300 Write block 2881 len 12288 data: 4 5 5
065/300 Flush
066/300 Read block 211 len 28672
067/300 Read block 481 len 20480
068/300 Write block 3580 len 40960 data: 3 3 3 3 2 4 4 4 5 6
069/300 Read block 1489 len 36864
070/300 Read block 207 len 20480
071/300 Read block 988 len 28672
072/300 Write block 438 len 36864 data: 9 9 10 10 11 11 12 11 8
073/300 Flush
074/300 Flush
075/300 Flush
076/300 Read block 213 len 16384
077/300 Write block 3182 len 20480 data: 5 5 6 6 5
078/300 Read block 4271 len 36864
079/300 Read block 773 len 16384
080/300 Read block 2885 len 28672
081/300 Write block 3165 len 16384 data: 5 5 5 5
082/300 Read block 4928 len 32768
083/300 Read block 2132 len 20480
084/300 Write block 2550 len 24576 data: 5 5 5 5 5 5
085/300 Read block 3838 len 20480
086/300 Write block 110 len 28672 data: 5 6 6 5 4 5 3
087/300 Write block 2977 len 8192 data: 5 5
088/300 Read block 45 len 8192
089/300 Write block 206 len 12288 data: 7 7 9
090/300 Read block 2744 len 28672
091/300 Read block 2095 len 12288
092/300 Read block 1180 len 16384
093/300 Write block 1098 len 32768 data: 4 4 3 3 3 3 3 3
094/300 Write block 979 len 20480 data: 7 8 7 6 7
095/300 Write block 1716 len 8192 data: 6 7
096/300 Write block 4333 len 16384 data: 7 6 6 9
097/300 Read block 415 len 8192
098/300 Write block 2682 len 20480 data: 3 4 7 9 8
099/300 Read block 2824 len 40960
100/300 Write block 1474 len 36864 data: 7 5 4 6 6 6 5 6 6
101/300 Read block 1312 len 40960
102/300 Read block 1032 len 32768
103/300 Read block 1575 len 24576
104/300 Read block 624 len 4096
105/300 Write block 487 len 32768 data: 7 6 5 4 4 4 5 5
106/300 Flush
107/300 Read block 1345 len 12288
108/300 Read block 749 len 8192
109/300 Write block 1828 len 8192 data: 6 7
110/300 Write block 3631 len 40960 data: 4 4 6 6 5 5 4 3 3 4
111/300 Write block 3318 len 28672 data: 3 4 6 7 8 8 8
112/300 Read block 2793 len 40960
113/300 Write block 3945 len 36864 data: 5 4 4 4 3 3 3 3 3
114/300 Write block 2658 len 36864 data: 7 6 5 3 5 7 8 8 8
115/300 Write block 2314 len 36864 data: 5 4 4 5 7 7 6 6 6
116/300 Write block 1124 len 16384 data: 6 8 8 8
117/300 Read block 3536 len 36864
118/300 Write block 1130 len 32768 data: 7 8 7 6 8 8 9 9
119/300 Read block 2944 len 24576
120/300 Read block 3482 len 24576
121/300 Flush
122/300 Read block 961 len 20480
123/300 Read block 732 len 16384
124/300 Read block 3054 len 28672
125/300 Read block 3355 len 16384
126/300 Read block 498 len 12288
127/300 Write block 3682 len 28672 data: 8 7 7 6 6 5 6
128/300 Write block 2413 len 12288 data: 2 2 2
129/300 Read block 3286 len 16384
130/300 Read block 365 len 4096
131/300 Read block 3399 len 32768
132/300 Read block 1596 len 28672
133/300 Read block 269 len 28672
134/300 Write block 4801 len 12288 data: 4 4 3
135/300 Read block 3183 len 4096
136/300 Write block 4531 len 32768 data: 6 5 5 5 6 6 4 4
137/300 Write block 2181 len 20480 data: 6 6 6 7 8
138/300 Read block 4950 len 40960
139/300 Write block 4395 len 28672 data: 7 6 5 4 4 4 4
140/300 Write block 4929 len 12288 data: 3 4 4
141/300 Write block 4033 len 40960 data: 4 4 4 6 7 9 8 7 7 5
142/300 Flush
143/300 Write block 458 len 24576 data: 4 6 6 6 5 3
144/300 Read block 1294 len 24576
145/300 Read block 3377 len 36864
146/300 Read block 4603 len 8192
147/300 Write block 1835 len 40960 data: 6 8 6 7 7 7 5 4 6 6
148/300 Read block 1985 len 24576
149/300 Read block 391 len 32768
150/300 Flush
151/300 Read block 1462 len 12288
152/300 Read block 135 len 8192
153/300 Read block 1697 len 4096
154/300 Read block 685 len 24576
155/300 Write block 4417 len 40960 data: 3 3 5 4 4 3 3 2 2 4
156/300 Read block 2316 len 40960
157/300 Flush
158/300 Read block 4270 len 8192
159/300 Read block 220 len 12288
160/300 Read block 4698 len 8192
161/300 Write block 4692 len 40960 data: 6 5 5 6 7 7 6 8 6 6
162/300 Flush
163/300 Write block 4226 len 24576 data: 7 9 12 12 10 10
164/300 Write block 4971 len 32768 data: 9 8 7 6 5 5 5 7
165/300 Write block 4028 len 4096 data: 3
166/300 Write block 70 len 24576 data: 4 5 5 8 7 10
167/300 Read block 272 len 4096
168/300 Read block 4903 len 4096
169/300 Read block 1073 len 28672
170/300 Read block 4760 len 40960
171/300 Write block 1672 len 20480 data: 5 3 4 4 7
172/300 Read block 4385 len 16384
173/300 Write block 4143 len 32768 data: 7 6 5 5 5 5 5 5
174/300 Write block 3903 len 24576 data: 8 8 7 6 6 5
175/300 Read block 3812 len 12288
176/300 Read block 3149 len 32768
177/300 Write block 2081 len 20480 data: 5 6 8 6 8
178/300 Write block 4721 len 28672 data: 2 2 2 4 6 6 6
179/300 Write block 1959 len 12288 data: 4 3 3
180/300 Read block 1920 len 8192
181/300 Write block 1931 len 32768 data: 6 7 6 7 5 5 4 3
182/300 Read block 254 len 24576
183/300 Write block 165 len 40960 data: 5 5 3 2 2 3 4 6 7 8
184/300 Read block 4569 len 4096
185/300 Read block 3721 len 36864
186/300 Read block 3801 len 4096
187/300 Read block 3012 len 32768
188/300 Write block 4221 len 32768 data: 2 2 3 4 6 8 10 13
189/300 Write block 4810 len 8192 data: 6 5
190/300 Write block 3975 len 16384 data: 6 5 5 5
191/300 Read block 2017 len 36864
192/300 Write block 2481 len 24576 data: 5 4 6 7 7 6
193/300 Read block 823 len 20480
194/300 Write block 2578 len 16384 data: 4 6 8 7
195/300 Write block 4689 len 36864 data: 6 6 6 7 6 6 7 8 8
196/300 Flush
197/300 Read block 3246 len 12288
198/300 Read block 2078 len 4096
199/300 Read block 2032 len 40960
200/300 Write block 3863 len 20480 data: 7 9 9 10 11
201/300 Read block 1068 len 40960
202/300 Read block 1204 len 20480
203/300 Read block 3580 len 8192
204/300 Read block 1030 len 8192
205/300 Read block 1914 len 32768
206/300 Read block 2860 len 40960
207/300 Flush
208/300 Read block 3627 len 36864
209/300 Read block 683 len 36864
210/300 Read block 2519 len 40960
211/300 Write block 1589 len 4096 data: 5
212/300 Write block 53 len 40960 data: 6 6 6 6 6 7 6 6 6 5
213/300 Read block 3507 len 12288
214/300 Read block 2643 len 4096
215/300 Read block 3492 len 20480
216/300 Read block 1893 len 4096
217/300 Flush
218/300 Write block 4968 len 28672 data: 10 9 8 10 9 8 7
219/300 Read block 2392 len 36864
220/300 Flush
221/300 Write block 2586 len 32768 data: 6 5 4 3 3 5 4 4
222/300 Read block 3843 len 16384
223/300 Flush
224/300 Write block 1948 len 8192 data: 6 4
225/300 Write block 4318 len 8192 data: 6 5
226/300 Flush
227/300 Write block 736 len 16384 data: 5 4 4 5
228/300 Write block 3902 len 12288 data: 5 9 9
229/300 Write block 2964 len 8192 data: 6 6
230/300 Write block 976 len 36864 data: 6 7 8 8 9 8 7 8 5
231/300 Write block 3963 len 32768 data: 3 3 3 3 3 3 4 5
232/300 Write block 340 len 32768 data: 6 8 7 7 7 10 8 5
233/300 Write block 4276 len 16384 data: 6 7 7 6
234/300 Write block 2126 len 20480 data: 7 4 6 5 6
235/300 Write block 3246 len 24576 data: 5 5 4 8 8 7
236/300 Write block 2828 len 4096 data: 5
237/300 Read block 200 len 32768
238/300 Write block 1853 len 36864 data: 5 5 4 4 4 3 3 3 2
239/300 Flush
240/300 Read block 1070 len 12288
241/300 Read block 1605 len 24576
242/300 Write block 1488 len 24576 data: 9 10 11 9 9 8
243/300 Read block 2852 len 20480
244/300 Read block 513 len 4096
245/300 Read block 3067 len 8192
246/300 Read block 3893 len 36864
247/300 Read block 1405 len 8192
248/300 Write block 2721 len 40960 data: 6 6 5 4 4 4 4 4 6 9
249/300 Write block 953 len 8192 data: 4 4
250/300 Read block 546 len 16384
251/300 Write block 2427 len 40960 data: 6 5 4 5 6 6 5 5 6 6
252/300 Write block 3912 len 24576 data: 5 5 6 6 6 5
253/300 Read block 1764 len 20480
254/300 Read block 2979 len 40960
255/300 Read block 1611 len 16384
256/300 Read block 458 len 20480
257/300 Write block 1688 len 16384 data: 7 5 5 6
258/300 Read block 3620 len 32768
259/300 Write block 2617 len 28672 data: 5 5 5 4 3 4 4
260/300 Write block 3318 len 36864 data: 4 5 7 8 9 9 9 8 6
261/300 Write block 3617 len 20480 data: 7 6 7 5 5
262/300 Read block 1179 len 12288
263/300 Write block 1227 len 40960 data: 10 9 8 8 5 3 3 2 2 2
264/300 Write block 374 len 36864 data: 6 5 5 6 8 7 8 7 8
265/300 Read block 3045 len 20480
266/300 Flush
267/300 Write block 1151 len 36864 data: 11 11 10 9 10 9 9 7 7
268/300 Read block 1950 len 28672
269/300 Flush
270/300 Write block 3457 len 40960 data: 6 6 6 3 5 5 5 5 7 6
271/300 Write block 4998 len 8192 data: 5 3
272/300 Write block 674 len 16384 data: 7 7 8 5
273/300 Write block 1703 len 12288 data: 8 7 5
274/300 Read block 3112 len 16384
275/300 Write block 1744 len 32768 data: 5 5 5 8 7 7 7 4
276/300 Write block 3666 len 36864 data: 4 5 5 5 5 4 4 4 3
277/300 Write block 3269 len 12288 data: 6 5 6
278/300 Read block 4298 len 8192
279/300 Write block 4273 len 32768 data: 7 8 7 7 8 8 7 7
280/300 Read block 2541 len 24576
281/300 Write block 103 len 8192 data: 5 5
282/300 Write block 2252 len 16384 data: 3 4 5 4
283/300 Write block 4252 len 8192 data: 4 4
284/300 Write block 3030 len 16384 data: 7 7 8 7
285/300 Read block 1022 len 28672
286/300 Read block 2338 len 40960
287/300 Read block 3145 len 36864
288/300 Flush
289/300 Flush
290/300 Read block 3630 len 8192
291/300 Read block 1668 len 24576
292/300 Read block 4406 len 12288
293/300 Read block 67 len 24576
294/300 Read block 2086 len 16384
295/300 Write block 1804 len 24576 data: 4 4 5 4 4 6
296/300 Read block 4042 len 12288
297/300 Read block 2531 len 20480
298/300 Read block 4250 len 24576
299/300 Read block 1348 len 20480
300/300 Read block 257 len 8192
301/300 Read block 626 len 16384
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:08:31.173 INFO request completed, latency_us: 186, response_code: 204, uri: /start/cid/2, method: POST, req_id: bb8f102d-9991-4c6d-a094-f006a7096622, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Write 10 Done Done Sent false
9453 Acked Write 10 Done Done Sent false
9454 Acked Read 4 Done Done Sent false
9455 Acked Read 3 Done Done Sent false
9456 Acked Read 7 Done Done Sent false
9457 Acked Read 4 Done Done Sent false
9458 Acked Read 2 Done Done Sent false
9459 Acked Read 2 Done Done Sent false
9460 Acked Write 10 Done Done Sent false
9461 Acked Write 5 Done Done Sent false
9462 Acked Read 10 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 4 Done Done Sent false
9465 Acked Flush 0 Done Done Sent false
9466 Acked Write 8 Done Done Sent false
9467 Acked Read 3 Done Done Sent false
9468 Acked Write 4 Done Done Sent false
9469 Acked Write 1 Done Done Sent false
9470 Acked Read 7 Done Done Sent false
9471 Acked Write 1 Done Done Sent false
9472 Acked Read 10 Done Done Sent false
9473 Acked Write 6 Done Done Sent false
9474 Acked Write 4 Done Done Sent false
9475 Acked Read 9 Done Done Sent false
9476 Acked Read 7 Done Done Sent false
9477 Acked Read 6 Done Done Sent false
9478 Acked Flush 0 Done Done Sent false
9479 Acked Read 6 Done Done Sent false
9480 Acked Read 7 Done Done Sent false
9481 Acked Write 10 Done Done Sent false
9482 Acked Read 5 Done Done Sent false
9483 Acked Flush 0 Done Done Sent false
9484 Acked Write 7 Done Done Sent false
9485 Acked Read 6 Done Done Sent false
9486 Acked Read 9 Done Done Sent false
9487 Acked Read 6 Done Done Sent false
9488 Acked Flush 0 Done Done Sent false
9489 Acked Read 4 Done Done Sent false
9490 Acked Write 7 Done Done Sent false
9491 Acked Flush 0 Done Done Sent false
9492 Acked Read 4 Done Done Sent false
9493 Acked Read 4 Done Done Sent false
9494 Acked Flush 0 Done Done Sent false
9495 Acked Read 8 Done Done Sent false
9496 Acked Write 4 Done Done Sent false
9497 Acked Write 1 Done Done Sent false
9498 Acked Read 4 Done Done Sent false
9499 Acked Write 8 Done Done Sent false
9500 Acked Write 6 Done Done Sent false
9501 Acked Write 6 Done Done Sent false
9502 Acked Read 5 Done Done Sent false
9503 Acked Flush 0 Done Done Sent false
9504 Acked Read 7 Done Done Sent false
9505 Acked Write 2 Done Done Sent false
9506 Acked Read 9 Done Done Sent false
9507 Acked Read 3 Done Done Sent false
9508 Acked Read 10 Done Done Sent false
9509 Acked Read 8 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Write 1 Done Done Sent false
9512 Acked Read 2 Done Done Sent false
9513 Acked Read 10 Done Done Sent false
9514 Acked Read 7 Done Done Sent false
9515 Acked Write 3 Done Done Sent false
9516 Acked Flush 0 Done Done Sent false
9517 Acked Read 7 Done Done Sent false
9518 Acked Read 5 Done Done Sent false
9519 Acked Write 10 Done Done Sent false
9520 Acked Read 9 Done Done Sent false
9521 Acked Read 5 Done Done Sent false
9522 Acked Read 7 Done Done Sent false
9523 Acked Write 9 Done Done Sent false
9524 Acked Flush 0 Done Done Sent false
9525 Acked Flush 0 Done Done Sent false
9526 Acked Flush 0 Done Done Sent false
9527 Acked Read 4 Done Done Sent false
9528 Acked Write 5 Done Done Sent false
9529 Acked Read 9 Done Done Sent false
9530 Acked Read 4 Done Done Sent false
9531 Acked Read 7 Done Done Sent false
9532 Acked Write 4 Done Done Sent false
9533 Acked Read 8 Done Done Sent false
9534 Acked Read 5 Done Done Sent false
9535 Acked Write 6 Done Done Sent false
9536 Acked Read 5 Done Done Sent false
9537 Acked Write 7 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Read 2 Done Done Sent false
9540 Acked Write 3 Done Done Sent false
9541 Acked Read 7 Done Done Sent false
9542 Acked Read 3 Done Done Sent false
9543 Acked Read 4 Done Done Sent false
9544 Acked Write 8 Done Done Sent false
9545 Acked Write 5 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 4 Done Done Sent false
9548 Acked Read 2 Done Done Sent false
9549 Acked Write 5 Done Done Sent false
9550 Acked Read 10 Done Done Sent false
9551 Acked Write 9 Done Done Sent false
9552 Acked Read 10 Done Done Sent false
9553 Acked Read 8 Done Done Sent false
9554 Acked Read 6 Done Done Sent false
9555 Acked Read 1 Done Done Sent false
9556 Acked Write 8 Done Done Sent false
9557 Acked Flush 0 Done Done Sent false
9558 Acked Read 3 Done Done Sent false
9559 Acked Read 2 Done Done Sent false
9560 Acked Write 2 Done Done Sent false
9561 Acked Write 10 Done Done Sent false
9562 Acked Write 7 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Write 9 Done Done Sent false
9565 Acked Write 9 Done Done Sent false
9566 Acked Write 9 Done Done Sent false
9567 Acked Write 4 Done Done Sent false
9568 Acked Read 9 Done Done Sent false
9569 Acked Write 8 Done Done Sent false
9570 Acked Read 6 Done Done Sent false
9571 Acked Read 6 Done Done Sent false
9572 Acked Flush 0 Done Done Sent false
9573 Acked Read 5 Done Done Sent false
9574 Acked Read 4 Done Done Sent false
9575 Acked Read 7 Done Done Sent false
9576 Acked Read 4 Done Done Sent false
9577 Acked Read 3 Done Done Sent false
9578 Acked Write 7 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 4 Done Done Sent false
9581 Acked Read 1 Done Done Sent false
9582 Acked Read 8 Done Done Sent false
9583 Acked Read 7 Done Done Sent false
9584 Acked Read 7 Done Done Sent false
9585 Acked Write 3 Done Done Sent false
9586 Acked Read 1 Done Done Sent false
9587 Acked Write 8 Done Done Sent false
9588 Acked Write 5 Done Done Sent false
9589 Acked Read 10 Done Done Sent false
9590 Acked Write 7 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 10 Done Done Sent false
9593 Acked Flush 0 Done Done Sent false
9594 Acked Write 6 Done Done Sent false
9595 Acked Read 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Read 2 Done Done Sent false
9598 Acked Write 10 Done Done Sent false
9599 Acked Read 6 Done Done Sent false
9600 Acked Read 8 Done Done Sent false
9601 Acked Flush 0 Done Done Sent false
9602 Acked Read 3 Done Done Sent false
9603 Acked Read 2 Done Done Sent false
9604 Acked Read 1 Done Done Sent false
9605 Acked Read 6 Done Done Sent false
9606 Acked Write 10 Done Done Sent false
9607 Acked Read 10 Done Done Sent false
9608 Acked Flush 0 Done Done Sent false
9609 Acked Read 2 Done Done Sent false
9610 Acked Read 3 Done Done Sent false
9611 Acked Read 2 Done Done Sent false
9612 Acked Write 10 Done Done Sent false
9613 Acked Flush 0 Done Done Sent false
9614 Acked Write 6 Done Done Sent false
9615 Acked Write 8 Done Done Sent false
9616 Acked Write 1 Done Done Sent false
9617 Acked Write 6 Done Done Sent false
9618 Acked Read 1 Done Done Sent false
9619 Acked Read 1 Done Done Sent false
9620 Acked Read 7 Done Done Sent false
9621 Acked Read 10 Done Done Sent false
9622 Acked Write 5 Done Done Sent false
9623 Acked Read 4 Done Done Sent false
9624 Acked Write 8 Done Done Sent false
9625 Acked Write 6 Done Done Sent false
9626 Acked Read 3 Done Done Sent false
9627 Acked Read 8 Done Done Sent false
9628 Acked Write 5 Done Done Sent false
9629 Acked Write 7 Done Done Sent false
9630 Acked Write 3 Done Done Sent false
9631 Acked Read 2 Done Done Sent false
9632 Acked Write 8 Done Done Sent false
9633 Acked Read 6 Done Done Sent false
9634 Acked Write 10 Done Done Sent false
9635 Acked Read 1 Done Done Sent false
9636 Acked Read 9 Done Done Sent false
9637 Acked Read 1 Done Done Sent false
9638 Acked Read 8 Done Done Sent false
9639 Acked Write 8 Done Done Sent false
9640 Acked Write 2 Done Done Sent false
9641 Acked Write 4 Done Done Sent false
9642 Acked Read 9 Done Done Sent false
9643 Acked Write 6 Done Done Sent false
9644 Acked Read 5 Done Done Sent false
9645 Acked Write 4 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 3 Done Done Sent false
9649 Acked Read 1 Done Done Sent false
9650 Acked Read 10 Done Done Sent false
9651 Acked Write 5 Done Done Sent false
9652 Acked Read 10 Done Done Sent false
9653 Acked Read 5 Done Done Sent false
9654 Acked Read 2 Done Done Sent false
9655 Acked Read 2 Done Done Sent false
9656 Acked Read 8 Done Done Sent false
9657 Acked Read 10 Done Done Sent false
9658 Acked Flush 0 Done Done Sent false
9659 Acked Read 9 Done Done Sent false
9660 Acked Read 9 Done Done Sent false
9661 Acked Read 10 Done Done Sent false
9662 Acked Write 1 Done Done Sent false
9663 Acked Write 10 Done Done Sent false
9664 Acked Read 3 Done Done Sent false
9665 Acked Read 1 Done Done Sent false
9666 Acked Read 5 Done Done Sent false
9667 Acked Read 1 Done Done Sent false
9668 Acked Flush 0 Done Done Sent false
9669 Acked Write 7 Done Done Sent false
9670 Acked Read 9 Done Done Sent false
9671 Acked Flush 0 Done Done Sent false
9672 Acked Write 8 Done Done Sent false
9673 Acked Read 4 Done Done Sent false
9674 Acked Flush 0 Done Done Sent false
9675 Acked Write 2 Done Done Sent false
9676 Acked Write 2 Done Done Sent false
9677 Acked Flush 0 Done Done Sent false
9678 Acked Write 4 Done Done Sent false
9679 Acked Write 3 Done Done Sent false
9680 Acked Write 2 Done Done Sent false
9681 Acked Write 9 Done Done Sent false
9682 Acked Write 8 Done Done Sent false
9683 Acked Write 8 Done Done Sent false
9684 Acked Write 4 Done Done Sent false
9685 Acked Write 5 Done Done Sent false
9686 Acked Write 6 Done Done Sent false
9687 Acked Write 1 Done Done Sent false
9688 Acked Read 8 Done Done Sent false
9689 Acked Write 9 Done Done Sent false
9690 Acked Flush 0 Done Done Sent false
9691 Acked Read 3 Done Done Sent false
9692 Acked Read 6 Done Done Sent false
9693 Acked Write 6 Done Done Sent false
9694 Acked Read 5 Done Done Sent false
9695 Acked Read 1 Done Done Sent false
9696 Acked Read 2 Done Done Sent false
9697 Acked Read 9 Done Done Sent false
9698 Acked Read 2 Done Done Sent false
9699 Acked Write 10 Done Done Sent false
9700 Acked Write 2 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 10 Done Done Sent false
9703 Acked Write 6 Done Done Sent false
9704 Acked Read 5 Done Done Sent false
9705 Acked Read 10 Done Done Sent false
9706 Acked Read 4 Done Done Sent false
9707 Acked Read 5 Done Done Sent false
9708 Acked Write 4 Done Done Sent false
9709 Acked Read 8 Done Done Sent false
9710 Acked Write 7 Done Done Sent false
9711 Acked Write 9 Done Done Sent false
9712 Acked Write 5 Done Done Sent false
9713 Acked Read 3 Done Done Sent false
9714 Acked Write 10 Done Done Sent false
9715 Acked Write 9 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Flush 0 Done Done Sent false
9718 Acked Write 9 Done Done Sent false
9719 Acked Read 7 Done Done Sent false
9720 Acked Flush 0 Done Done Sent false
9721 Acked Write 10 Done Done Sent false
9722 Acked Write 2 Done Done Sent false
9723 Acked Write 4 Done Done Sent false
9724 Acked Write 3 Done Done Sent false
9725 Acked Read 4 Done Done Sent false
9726 Acked Write 8 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Write 3 Done Done Sent false
9729 Acked Read 2 Done Done Sent false
9730 Acked Write 8 Done Done Sent false
9731 Acked Read 6 Done Done Sent false
9732 Acked Write 2 Done Done Sent false
9733 Acked Write 4 Done Done Sent false
9734 Acked Write 2 Done Done Sent false
9735 Acked Write 4 Done Done Sent false
9736 Acked Read 7 Done Done Sent false
9737 Acked Read 10 Done Done Sent false
9738 Acked Read 9 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Flush 0 Done Done Sent false
9741 Acked Read 2 Done Done Sent false
9742 Acked Read 6 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 4 Done Done Sent false
9746 Acked Write 6 Done Done Sent false
9747 Acked Read 3 Done Done Sent false
9748 Acked Read 5 Done Done Sent false
9749 Acked Read 6 Done Done Sent false
9750 Acked Read 5 Done Done Sent false
9751 Acked Read 2 Done Done Sent false
9752 Acked Read 4 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9740)) Some(JobId(9740)) Some(JobId(9451))
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9752 9751 9750 9749 9748
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:32.093664903Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1240)
[8830][2] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:33.095176744Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:34.097570473Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:35.097830184Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Write 10 Done Done Sent false
9453 Acked Write 10 Done Done Sent false
9454 Acked Read 4 Done Done Sent false
9455 Acked Read 3 Done Done Sent false
9456 Acked Read 7 Done Done Sent false
9457 Acked Read 4 Done Done Sent false
9458 Acked Read 2 Done Done Sent false
9459 Acked Read 2 Done Done Sent false
9460 Acked Write 10 Done Done Sent false
9461 Acked Write 5 Done Done Sent false
9462 Acked Read 10 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 4 Done Done Sent false
9465 Acked Flush 0 Done Done Sent false
9466 Acked Write 8 Done Done Sent false
9467 Acked Read 3 Done Done Sent false
9468 Acked Write 4 Done Done Sent false
9469 Acked Write 1 Done Done Sent false
9470 Acked Read 7 Done Done Sent false
9471 Acked Write 1 Done Done Sent false
9472 Acked Read 10 Done Done Sent false
9473 Acked Write 6 Done Done Sent false
9474 Acked Write 4 Done Done Sent false
9475 Acked Read 9 Done Done Sent false
9476 Acked Read 7 Done Done Sent false
9477 Acked Read 6 Done Done Sent false
9478 Acked Flush 0 Done Done Sent false
9479 Acked Read 6 Done Done Sent false
9480 Acked Read 7 Done Done Sent false
9481 Acked Write 10 Done Done Sent false
9482 Acked Read 5 Done Done Sent false
9483 Acked Flush 0 Done Done Sent false
9484 Acked Write 7 Done Done Sent false
9485 Acked Read 6 Done Done Sent false
9486 Acked Read 9 Done Done Sent false
9487 Acked Read 6 Done Done Sent false
9488 Acked Flush 0 Done Done Sent false
9489 Acked Read 4 Done Done Sent false
9490 Acked Write 7 Done Done Sent false
9491 Acked Flush 0 Done Done Sent false
9492 Acked Read 4 Done Done Sent false
9493 Acked Read 4 Done Done Sent false
9494 Acked Flush 0 Done Done Sent false
9495 Acked Read 8 Done Done Sent false
9496 Acked Write 4 Done Done Sent false
9497 Acked Write 1 Done Done Sent false
9498 Acked Read 4 Done Done Sent false
9499 Acked Write 8 Done Done Sent false
9500 Acked Write 6 Done Done Sent false
9501 Acked Write 6 Done Done Sent false
9502 Acked Read 5 Done Done Sent false
9503 Acked Flush 0 Done Done Sent false
9504 Acked Read 7 Done Done Sent false
9505 Acked Write 2 Done Done Sent false
9506 Acked Read 9 Done Done Sent false
9507 Acked Read 3 Done Done Sent false
9508 Acked Read 10 Done Done Sent false
9509 Acked Read 8 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Write 1 Done Done Sent false
9512 Acked Read 2 Done Done Sent false
9513 Acked Read 10 Done Done Sent false
9514 Acked Read 7 Done Done Sent false
9515 Acked Write 3 Done Done Sent false
9516 Acked Flush 0 Done Done Sent false
9517 Acked Read 7 Done Done Sent false
9518 Acked Read 5 Done Done Sent false
9519 Acked Write 10 Done Done Sent false
9520 Acked Read 9 Done Done Sent false
9521 Acked Read 5 Done Done Sent false
9522 Acked Read 7 Done Done Sent false
9523 Acked Write 9 Done Done Sent false
9524 Acked Flush 0 Done Done Sent false
9525 Acked Flush 0 Done Done Sent false
9526 Acked Flush 0 Done Done Sent false
9527 Acked Read 4 Done Done Sent false
9528 Acked Write 5 Done Done Sent false
9529 Acked Read 9 Done Done Sent false
9530 Acked Read 4 Done Done Sent false
9531 Acked Read 7 Done Done Sent false
9532 Acked Write 4 Done Done Sent false
9533 Acked Read 8 Done Done Sent false
9534 Acked Read 5 Done Done Sent false
9535 Acked Write 6 Done Done Sent false
9536 Acked Read 5 Done Done Sent false
9537 Acked Write 7 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Read 2 Done Done Sent false
9540 Acked Write 3 Done Done Sent false
9541 Acked Read 7 Done Done Sent false
9542 Acked Read 3 Done Done Sent false
9543 Acked Read 4 Done Done Sent false
9544 Acked Write 8 Done Done Sent false
9545 Acked Write 5 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 4 Done Done Sent false
9548 Acked Read 2 Done Done Sent false
9549 Acked Write 5 Done Done Sent false
9550 Acked Read 10 Done Done Sent false
9551 Acked Write 9 Done Done Sent false
9552 Acked Read 10 Done Done Sent false
9553 Acked Read 8 Done Done Sent false
9554 Acked Read 6 Done Done Sent false
9555 Acked Read 1 Done Done Sent false
9556 Acked Write 8 Done Done Sent false
9557 Acked Flush 0 Done Done Sent false
9558 Acked Read 3 Done Done Sent false
9559 Acked Read 2 Done Done Sent false
9560 Acked Write 2 Done Done Sent false
9561 Acked Write 10 Done Done Sent false
9562 Acked Write 7 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Write 9 Done Done Sent false
9565 Acked Write 9 Done Done Sent false
9566 Acked Write 9 Done Done Sent false
9567 Acked Write 4 Done Done Sent false
9568 Acked Read 9 Done Done Sent false
9569 Acked Write 8 Done Done Sent false
9570 Acked Read 6 Done Done Sent false
9571 Acked Read 6 Done Done Sent false
9572 Acked Flush 0 Done Done Sent false
9573 Acked Read 5 Done Done Sent false
9574 Acked Read 4 Done Done Sent false
9575 Acked Read 7 Done Done Sent false
9576 Acked Read 4 Done Done Sent false
9577 Acked Read 3 Done Done Sent false
9578 Acked Write 7 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 4 Done Done Sent false
9581 Acked Read 1 Done Done Sent false
9582 Acked Read 8 Done Done Sent false
9583 Acked Read 7 Done Done Sent false
9584 Acked Read 7 Done Done Sent false
9585 Acked Write 3 Done Done Sent false
9586 Acked Read 1 Done Done Sent false
9587 Acked Write 8 Done Done Sent false
9588 Acked Write 5 Done Done Sent false
9589 Acked Read 10 Done Done Sent false
9590 Acked Write 7 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 10 Done Done Sent false
9593 Acked Flush 0 Done Done Sent false
9594 Acked Write 6 Done Done Sent false
9595 Acked Read 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Read 2 Done Done Sent false
9598 Acked Write 10 Done Done Sent false
9599 Acked Read 6 Done Done Sent false
9600 Acked Read 8 Done Done Sent false
9601 Acked Flush 0 Done Done Sent false
9602 Acked Read 3 Done Done Sent false
9603 Acked Read 2 Done Done Sent false
9604 Acked Read 1 Done Done Sent false
9605 Acked Read 6 Done Done Sent false
9606 Acked Write 10 Done Done Sent false
9607 Acked Read 10 Done Done Sent false
9608 Acked Flush 0 Done Done Sent false
9609 Acked Read 2 Done Done Sent false
9610 Acked Read 3 Done Done Sent false
9611 Acked Read 2 Done Done Sent false
9612 Acked Write 10 Done Done Sent false
9613 Acked Flush 0 Done Done Sent false
9614 Acked Write 6 Done Done Sent false
9615 Acked Write 8 Done Done Sent false
9616 Acked Write 1 Done Done Sent false
9617 Acked Write 6 Done Done Sent false
9618 Acked Read 1 Done Done Sent false
9619 Acked Read 1 Done Done Sent false
9620 Acked Read 7 Done Done Sent false
9621 Acked Read 10 Done Done Sent false
9622 Acked Write 5 Done Done Sent false
9623 Acked Read 4 Done Done Sent false
9624 Acked Write 8 Done Done Sent false
9625 Acked Write 6 Done Done Sent false
9626 Acked Read 3 Done Done Sent false
9627 Acked Read 8 Done Done Sent false
9628 Acked Write 5 Done Done Sent false
9629 Acked Write 7 Done Done Sent false
9630 Acked Write 3 Done Done Sent false
9631 Acked Read 2 Done Done Sent false
9632 Acked Write 8 Done Done Sent false
9633 Acked Read 6 Done Done Sent false
9634 Acked Write 10 Done Done Sent false
9635 Acked Read 1 Done Done Sent false
9636 Acked Read 9 Done Done Sent false
9637 Acked Read 1 Done Done Sent false
9638 Acked Read 8 Done Done Sent false
9639 Acked Write 8 Done Done Sent false
9640 Acked Write 2 Done Done Sent false
9641 Acked Write 4 Done Done Sent false
9642 Acked Read 9 Done Done Sent false
9643 Acked Write 6 Done Done Sent false
9644 Acked Read 5 Done Done Sent false
9645 Acked Write 4 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 3 Done Done Sent false
9649 Acked Read 1 Done Done Sent false
9650 Acked Read 10 Done Done Sent false
9651 Acked Write 5 Done Done Sent false
9652 Acked Read 10 Done Done Sent false
9653 Acked Read 5 Done Done Sent false
9654 Acked Read 2 Done Done Sent false
9655 Acked Read 2 Done Done Sent false
9656 Acked Read 8 Done Done Sent false
9657 Acked Read 10 Done Done Sent false
9658 Acked Flush 0 Done Done Sent false
9659 Acked Read 9 Done Done Sent false
9660 Acked Read 9 Done Done Sent false
9661 Acked Read 10 Done Done Sent false
9662 Acked Write 1 Done Done Sent false
9663 Acked Write 10 Done Done Sent false
9664 Acked Read 3 Done Done Sent false
9665 Acked Read 1 Done Done Sent false
9666 Acked Read 5 Done Done Sent false
9667 Acked Read 1 Done Done Sent false
9668 Acked Flush 0 Done Done Sent false
9669 Acked Write 7 Done Done Sent false
9670 Acked Read 9 Done Done Sent false
9671 Acked Flush 0 Done Done Sent false
9672 Acked Write 8 Done Done Sent false
9673 Acked Read 4 Done Done Sent false
9674 Acked Flush 0 Done Done Sent false
9675 Acked Write 2 Done Done Sent false
9676 Acked Write 2 Done Done Sent false
9677 Acked Flush 0 Done Done Sent false
9678 Acked Write 4 Done Done Sent false
9679 Acked Write 3 Done Done Sent false
9680 Acked Write 2 Done Done Sent false
9681 Acked Write 9 Done Done Sent false
9682 Acked Write 8 Done Done Sent false
9683 Acked Write 8 Done Done Sent false
9684 Acked Write 4 Done Done Sent false
9685 Acked Write 5 Done Done Sent false
9686 Acked Write 6 Done Done Sent false
9687 Acked Write 1 Done Done Sent false
9688 Acked Read 8 Done Done Sent false
9689 Acked Write 9 Done Done Sent false
9690 Acked Flush 0 Done Done Sent false
9691 Acked Read 3 Done Done Sent false
9692 Acked Read 6 Done Done Sent false
9693 Acked Write 6 Done Done Sent false
9694 Acked Read 5 Done Done Sent false
9695 Acked Read 1 Done Done Sent false
9696 Acked Read 2 Done Done Sent false
9697 Acked Read 9 Done Done Sent false
9698 Acked Read 2 Done Done Sent false
9699 Acked Write 10 Done Done Sent false
9700 Acked Write 2 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 10 Done Done Sent false
9703 Acked Write 6 Done Done Sent false
9704 Acked Read 5 Done Done Sent false
9705 Acked Read 10 Done Done Sent false
9706 Acked Read 4 Done Done Sent false
9707 Acked Read 5 Done Done Sent false
9708 Acked Write 4 Done Done Sent false
9709 Acked Read 8 Done Done Sent false
9710 Acked Write 7 Done Done Sent false
9711 Acked Write 9 Done Done Sent false
9712 Acked Write 5 Done Done Sent false
9713 Acked Read 3 Done Done Sent false
9714 Acked Write 10 Done Done Sent false
9715 Acked Write 9 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Flush 0 Done Done Sent false
9718 Acked Write 9 Done Done Sent false
9719 Acked Read 7 Done Done Sent false
9720 Acked Flush 0 Done Done Sent false
9721 Acked Write 10 Done Done Sent false
9722 Acked Write 2 Done Done Sent false
9723 Acked Write 4 Done Done Sent false
9724 Acked Write 3 Done Done Sent false
9725 Acked Read 4 Done Done Sent false
9726 Acked Write 8 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Write 3 Done Done Sent false
9729 Acked Read 2 Done Done Sent false
9730 Acked Write 8 Done Done Sent false
9731 Acked Read 6 Done Done Sent false
9732 Acked Write 2 Done Done Sent false
9733 Acked Write 4 Done Done Sent false
9734 Acked Write 2 Done Done Sent false
9735 Acked Write 4 Done Done Sent false
9736 Acked Read 7 Done Done Sent false
9737 Acked Read 10 Done Done Sent false
9738 Acked Read 9 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Flush 0 Done Done Sent false
9741 Acked Read 2 Done Done Sent false
9742 Acked Read 6 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 4 Done Done Sent false
9746 Acked Write 6 Done Done Sent false
9747 Acked Read 3 Done Done Sent false
9748 Acked Read 5 Done Done Sent false
9749 Acked Read 6 Done Done Sent false
9750 Acked Read 5 Done Done Sent false
9751 Acked Read 2 Done Done Sent false
9752 Acked Read 4 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9740)) Some(JobId(9740)) Some(JobId(9451))
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9752 9751 9750 9749 9748
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:36.100110841Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.923953941Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45461)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.924054727Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.925171359Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.925745462Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.925765707Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(9451))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.925779923Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.92579311Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(9451))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.925811926Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(9451))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:36.926068814Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 9753 9752 9751 9750 9749
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 28 06:08:39.179 INFO request completed, latency_us: 397, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4b5e5c4c-eed5-4ff3-8e4a-f942ac4f245c, remote_addr: 127.0.0.1:63908, local_addr[8810] Got stop action so:false kr:false
: 127.0.0.1:9998
Feb 28 06:08:39.180 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/0, method: GET, req_id: f080f39d-75e2-44b4-a93b-6bae9b47a348, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:39.180381114Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:39.18044225Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:39.180465654Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:39.180538167Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Feb 28 06:08:43.182 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/0, method: GET, req_id: 508914b3-5f4d-4e4e-98a5-85ab5d551d1a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Flush
003/300 Read block 3690 len 36864
004/300 Write block 2679 len 36864 data: 8 7 5 4 5 8 10 9 10
005/300 Read block 816 len 32768
006/300 Write block 2219 len 28672 data: 9 8 5 5 5 5 6
007/300 Write block 3232 len 40960 data: 2 3 2 2 3 3 4 4 2 2
008/300 Write block 1505 len 4096 data: 4
009/300 Write block 323 len 32768 data: 7 3 4 3 3 2 2 2
010/300 Flush
011/300 Write block 4740 len 28672 data: 8 7 6 4 5 4 3
012/300 Read block 2559 len 24576
013/300 Write block 3267 len 8192 data: 5 5
014/300 Read block 626 len 4096
015/300 Write block 2466 len 12288 data: 2 4 6
016/300 Read block 460 len 24576
017/300 Write block 3453 len 32768 data: 5 5 5 6 7 7 7 4
018/300 Read block 4801 len 28672
019/300 Write block 3453 len 16384 data: 6 6 6 7
020/300 Write block 2572 len 40960 data: 6 6 4 3 3 4 5 7 9 8
021/300 Read block 2523 len 16384
022/300 Write block 1266 len 28672 data: 6 5 5 4 4 5 5
023/300 Write block 1829 len 4096 data: 8
024/300 Write block 4256 len 40960 data: 3 3 3 3 4 3 5 5 5 5
025/300 Read block 4919 len 36864
026/300 Read block 4495 len 20480
027/300 Write block 3026 len 8192 data: 5 7
028/300 Read block 486 len 8192
029/300 Flush
030/300 Read block 586 len 16384
031/300 Read block 2161 len 28672
032/300 Read block 43 len 24576
033/300 Write block 4756 len 28672 data: 10 11 10 10 8 8 8
034/300 Read block 868 len 4096
035/300 Write block 918 len 24576 data: 7 7 8 8 8 7
036/300 Flush
037/300 Read block 1775 len 40960
038/300 Write block 1061 len 16384 data: 6 6 5 5
039/300 Read block 317 len 36864
040/300 Write block 2263 len 28672 data: 8 10 8 8 9 7 6
041/300 Read block 1114 len 28672
042/300 Write block 197 len 24576 data: 7 6 6 6 5 5
043/300 Read block 2628 len 8192
044/300 Read block 4808 len 20480
045/300 Write block 3214 len 8192 data: 8 8
046/300 Read block 568 len 36864
047/300 Read block 1912 len 28672
048/300 Read block 2929 len 12288
049/300 Write block 615 len 36864 data: 3 3 3 3 3 6 6 6 7
050/300 Write block 2467 len 32768 data: 5 7 4 4 5 7 7 6
051/300 Read block 2670 len 40960
052/300 Write block 3699 len 32768 data: 5 4 5 6 3 4 5 6
053/300 Write block 799 len 12288 data: 5 6 5
054/300 Write block 4215 len 28672 data: 8 8 7 4 3 3 3
055/300 Write block 4506 len 32768 data: 7 9 7 6 9 8 8 11
056/300 Write block 1586 len 4096 data: 7
057/300 Write block 1151 len 32768 data: 12 12 11 10 11 10 10 8
058/300 Write block 420 len 8192 data: 10 11
059/300 Read block 421 len 4096
060/300 Read block 4808 len 8192
061/300 Flush
062/300 Write block 3346 len 4096 data: 8
063/300 Read block 3033 len 24576
064/300 Read block 4915 len 28672
065/300 Read block 3485 len 8192
066/300 Write block 1347 len 20480 data: 5 3 4 5 6
067/300 Read block 202 len 16384
068/300 Write block 3241 len 36864 data: 3 4 3 4 4 6 6 5 9
069/300 Read block 4885 len 28672
070/300 Write block 1962 len 28672 data: 3 4 6 5 5 6 4
071/300 Read block 1214 len 32768
072/300 Read block 3084 len 16384
073/300 Read block 2137 len 28672
074/300 Read block 609 len 24576
075/300 Read block 2118 len 40960
076/300 Write block 2204 len 36864 data: 5 4 5 4 5 7 6 5 6
077/300 Write block 3012 len 40960 data: 5 5 5 6 4 5 5 5 5 5
078/300 Read block 2609 len 12288
079/300 Write block 1036 len 32768 data: 4 6 6 7 9 7 8 8
080/300 Write block 4331 len 24576 data: 4 5 8 7 7 10
081/300 Read block 1204 len 40960
082/300 Read block 4560 len 16384
083/300 Read block 3046 len 28672
084/300 Write block 4296 len 36864 data: 6 6 7 5 3 3 2 4 4
085/300 Write block 2686 len 28672 data: 10 11 7 7 6 6 6
086/300 Write block 681 len 40960 data: 7 6 6 5 4 3 3 3 4 5
087/300 Flush
088/300 Read block 4756 len 24576
089/300 Write block 3404 len 4096 data: 3
090/300 Write block 2294 len 40960 data: 4 4 4 5 5 4 4 4 5 5
091/300 Write block 3484 len 4096 data: 7
092/300 Read block 4868 len 24576
093/300 Read block 3886 len 8192
094/300 Read block 4171 len 12288
095/300 Read block 3048 len 40960
096/300 Read block 1776 len 28672
097/300 Flush
098/300 Flush
099/300 Write block 1530 len 24576 data: 6 5 5 5 7 6
100/300 Read block 15 len 20480
101/300 Write block 1050 len 20480 data: 7 7 8 8 8
102/300 Read block 1766 len 24576
103/300 Read block 4003 len 20480
104/300 Write block 1702 len 32768 data: 6 9 8 6 5 7 7 9
105/300 Read block 1170 len 24576
106/300 Read block 885 len 24576
107/300 Read block 439 len 28672
108/300 Flush
109/300 Read block 660 len 4096
110/300 Read block 1271 len 28672
111/300 Flush
112/300 Read block 4244 len 24576
113/300 Read block 1407 len 40960
114/300 Read block 3381 len 40960
115/300 Read block 550 len 28672
116/300 Read block 1463 len 12288
117/300 Write block 4457 len 24576 data: 4 4 4 5 7 6
118/300 Write block 2663 len 32768 data: 8 9 9 9 9 9 11 12
119/300 Read block 2518 len 36864
120/300 Write block 2730 len 20480 data: 10 7 7 8 7
121/300 Write block 1076 len 36864 data: 5 5 6 5 7 6 6 5 4
122/300 Read block 903 len 16384
123/300 Read block 4860 len 32768
124/300 Flush
125/300 Write block 211 len 32768 data: 8 6 6 8 10 8 9 9
126/300 Flush
127/300 Write block 3623 len 36864 data: 7 7 7 6 6 6 6 4 5
128/300 Read block 2086 len 28672
129/300 Flush
130/300 Write block 4927 len 20480 data: 7 5 4 5 5
131/300 Read block 4207 len 24576
132/300 Write block 558 len 28672 data: 6 6 5 7 7 7 7
133/300 Read block 3093 len 16384
134/300 Write block 1992 len 28672 data: 4 4 4 4 4 4 4
135/300 Read block 3913 len 36864
136/300 Write block 624 len 4096 data: 7
137/300 Read block 4257 len 4096
138/300 Flush
139/300 Read block 4497 len 28672
140/300 Write block 3053 len 28672 data: 4 4 5 6 3 4 4
141/300 Write block 2814 len 36864 data: 8 8 8 6 7 8 7 6 5
142/300 Read block 3235 len 36864
143/300 Read block 1948 len 36864
144/300 Write block 3 len 20480 data: 5 6 7 6 5
145/300 Read block 2277 len 28672
146/300 Flush
147/300 Read block 4964 len 12288
148/300 Write block 312 len 20480 data: 4 5 5 5 5
149/300 Read block 2846 len 32768
150/300 Read block 1964 len 4096
151/300 Write block 3277 len 4096 data: 4
152/300 Write block 827 len 20480 data: 4 3 6 6 6
153/300 Write block 1179 len 32768 data: 11 9 8 6 5 7 6 6
154/300 Flush
155/300 Read block 4061 len 36864
156/300 Read block 3747 len 16384
157/300 Flush
158/300 Flush
159/300 Read block 4733 len 36864
160/300 Write block 2250 len 8192 data: 3 3
161/300 Read block 2304 len 40960
162/300 Write block 3783 len 36864 data: 7 5 4 4 3 4 3 3 4
163/300 Read block 3878 len 12288
164/300 Write block 4764 len 28672 data: 6 5 5 5 6 7 10
165/300 Write block 644 len 36864 data: 7 7 6 7 8 6 6 5 5
166/300 Read block 4742 len 32768
167/300 Read block 2916 len 8192
168/300 Read block 1713 len 16384
169/300 Read block 3726 len 4096
170/300 Write block 746 len 12288 data: 10 9 12
171/300 Flush
172/300 Read block 1638 len 4096
173/300 Write block 1265 len 16384 data: 7 7 6 6
174/300 Read block 788 len 12288
175/300 Write block 850 len 4096 data: 3
176/300 Read block 28 len 4096
177/300 Write block 1227 len 12288 data: 11 10 9
178/300 Read block 2045 len 32768
179/300 Read block 3281 len 40960
180/300 Read block 3183 len 12288
181/300 Read block 1895 len 20480
182/300 Write block 4040 len 8192 data: 8 8
183/300 Write block 1747 len 32768 data: 9 8 8 8 5 6 6 7
184/300 Write block 3519 len 32768 data: 3 4 4 7 6 6 7 9
185/300 Write block 1583 len 40960 data: 8 6 6 8 7 6 6 5 5 4
186/300 Write block 4989 len 20480 data: 4 4 2 2 2
187/300 Read block 3250 len 32768
188/300 Read block 4832 len 28672
189/300 Read block 3115 len 24576
190/300 Read block 1214 len 20480
191/300 Read block 1751 len 32768
192/300 Write block 4 len 20480 data: 7 8 7 6 5
193/300 Write block 1982 len 36864 data: 8 10 10 8 8 7 8 7 4
194/300 Write block 3453 len 36864 data: 7 7 7 8 8 8 8 5 6
195/300 Write block 1154 len 28672 data: 11 12 11 11 9 8 6
196/300 Read block 1357 len 8192
197/300 Write block 2590 len 24576 data: 4 6 5 5 5 5
198/300 Read block 455 len 12288
199/300 Write block 1518 len 24576 data: 6 4 5 6 7 6
200/300 Write block 576 len 32768 data: 5 6 6 7 8 7 8 9
201/300 Read block 1163 len 36864
202/300 Write block 3906 len 4096 data: 7
203/300 Write block 3050 len 12288 data: 4 4 3
204/300 Read block 2226 len 24576
205/300 Write block 3736 len 12288 data: 6 6 5
206/300 Flush
207/300 Write block 2583 len 24576 data: 8 8 7 7 6 5
208/300 Flush
209/300 Flush
210/300 Write block 4253 len 16384 data: 5 4 5 4
211/300 Read block 3173 len 4096
212/300 Write block 3543 len 32768 data: 3 3 6 7 6 6 5 6
213/300 Read block 3532 len 16384
214/300 Read block 197 len 36864
215/300 Write block 1398 len 36864 data: 3 6 7 5 6 5 5 5 4
216/300 Read block 4290 len 20480
217/300 Write block 8 len 24576 data: 6 6 7 6 6 4
218/300 Read block 4610 len 16384
219/300 Write block 2505 len 28672 data: 3 3 3 4 5 5 7
220/300 Write block 2606 len 40960 data: 5 6 6 5 5 8 6 6 5 6
221/300 Flush
222/300 Read block 1253 len 12288
223/300 Read block 2416 len 12288
224/300 Read block 3000 len 28672
225/300 Read block 202 len 36864
226/300 Write block 2394 len 40960 data: 5 4 3 3 3 2 3 3 3 3
227/300 Read block 3407 len 36864
228/300 Read block 3581 len 32768
229/300 Write block 2447 len 36864 data: 7 6 5 5 6 7 6 7 9
230/300 Write block 997 len 36864 data: 5 5 6 6 7 7 6 5 5
231/300 Read block 4314 len 32768
232/300 Write block 2172 len 4096 data: 5
233/300 Flush
234/300 Read block 861 len 24576
235/300 Read block 268 len 24576
236/300 Write block 459 len 12288 data: 7 7 7
237/300 Read block 3841 len 28672
238/300 Read block 1142 len 8192
239/300 Read block 4504 len 40960
240/300 Read block 1688 len 16384
241/300 Write block 3047 len 12288 data: 4 4 4
242/300 Write block 961 len 36864 data: 4 5 5 5 6 6 5 5 4
243/300 Write block 3878 len 40960 data: 5 5 4 5 6 7 5 5 5 5
244/300 Read block 364 len 28672
245/300 Read block 983 len 16384
246/300 Read block 3779 len 36864
247/300 Read block 920 len 8192
248/300 Flush
249/300 Write block 4969 len 28672 data: 10 9 11 10 9 8 6
250/300 Read block 2228 len 16384
251/300 Read block 4840 len 8192
252/300 Write block 1276 len 16384 data: 5 3 4 4
253/300 Write block 3318 len 32768 data: 5 6 8 9 10 10 10 9
254/300 Write block 4346 len 28672 data: 9 10 8 8 7 7 8
255/300 Flush
256/300 Flush
257/300 Flush
258/300 Write block 3613 len 32768 data: 7 8 7 6 8 7 8 6
259/300 Read block 1900 len 12288
260/300 Read block 4014 len 40960
261/300 Write block 2096 len 8192 data: 7 7
262/300 Read block 1430 len 20480
263/300 Flush
264/300 Read block 3485 len 24576
265/300 Read block 886 len 28672
266/300 Read block 2766 len 16384
267/300 Write block 1424 len 20480 data: 6 7 7 7 4
268/300 Read block 1686 len 20480
269/300 Flush
270/300 Read block 3866 len 36864
271/300 Read block 1038 len 4096
272/300 Flush
273/300 Read block 857 len 16384
274/300 Read block 4747 len 36864
275/300 Write block 4403 len 12288 data: 5 6 10
276/300 Write block 652 len 28672 data: 6 5 5 5 5 6 7
277/300 Read block 3170 len 24576
278/300 Write block 3052 len 8192 data: 4 5
279/300 Write block 2131 len 4096 data: 7
280/300 Read block 2017 len 32768
281/300 Write block 2798 len 16384 data: 6 7 7 7
282/300 Write block 4926 len 28672 data: 8 8 6 5 6 6 4
283/300 Write block 1277 len 4096 data: 4
284/300 Write block 2440 len 32768 data: 6 6 8 7 7 7 7 8
285/300 Write block 1485 len 32768 data: 9 9 8 10 11 12 10 10
286/300 Read block 2267 len 28672
287/300 Write block 1201 len 36864 data: 5 5 6 5 6 5 5 5 4
288/300 Write block 1333 len 20480 data: 4 4 6 6 7
289/300 Read block 337 len 12288
290/300 Read block 597 len 16384
291/300 Read block 3258 len 28672
292/300 Read block 3795 len 4096
293/300 Write block 2514 len 12288 data: 5 4 5
294/300 Read block 3358 len 4096
295/300 Read block 4464 len 24576
296/300 Read block 4926 len 12288
297/300 Write block 1616 len 36864 data: 5 5 4 4 5 5 4 4 7
298/300 Read block 4546 len 32768
299/300 Write block 2896 len 4096 data: 6
300/300 Read block 103 len 40960
301/300 Read block 4363 len 36864
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 28 06:08:43.430 INFO request completed, latency_us: 183, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9906bd83-49c1-4854-85a3-3ca7f28651e4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9754 Acked Flush 0 Sent Done Done false
9755 Acked Flush 0 Sent Done Done false
9756 Acked Read 9 Sent Done Done false
9757 Acked Write 9 Sent Done Done false
9758 Acked Read 8 Sent Done Done false
9759 Acked Write 7 Sent Done Done false
9760 Acked Write 10 Sent Done Done false
9761 Acked Write 1 Sent Done Done false
9762 Acked Write 8 Sent Done Done false
9763 Acked Flush 0 Sent Done Done false
9764 Acked Write 7 Sent Done Done false
9765 Acked Read 6 Sent Done Done false
9766 Acked Write 2 Sent Done Done false
9767 Acked Read 1 Sent Done Done false
9768 Acked Write 3 Sent Done Done false
9769 Acked Read 6 Sent Done Done false
9770 Acked Write 8 Sent Done Done false
9771 Acked Read 7 Sent Done Done false
9772 Acked Write 4 Sent Done Done false
9773 Acked Write 10 Sent Done Done false
9774 Acked Read 4 Sent Done Done false
9775 Acked Write 7 Sent Done Done false
9776 Acked Write 1 Sent Done Done false
9777 Acked Write 10 Sent Done Done false
9778 Acked Read 9 Sent Done Done false
9779 Acked Read 5 Sent Done Done false
9780 Acked Write 2 Sent Done Done false
9781 Acked Read 2 Sent Done Done false
9782 Acked Flush 0 Sent Done Done false
9783 Acked Read 4 Sent Done Done false
9784 Acked Read 7 Sent Done Done false
9785 Acked Read 6 Sent Done Done false
9786 Acked Write 7 Sent Done Done false
9787 Acked Read 1 Sent Done Done false
9788 Acked Write 6 Sent Done Done false
9789 Acked Flush 0 Sent Done Done false
9790 Acked Read 10 Sent Done Done false
9791 Acked Write 4 Sent Done Done false
9792 Acked Read 9 Sent Done Done false
9793 Acked Write 7 Sent Done Done false
9794 Acked Read 7 Sent Done Done false
9795 Acked Write 6 Sent Done Done false
9796 Acked Read 2 Sent Done Done false
9797 Acked Read 5 Sent Done Done false
9798 Acked Write 2 Sent Done Done false
9799 Acked Read 9 Sent Done Done false
9800 Acked Read 7 Sent Done Done false
9801 Acked Read 3 Sent Done Done false
9802 Acked Write 9 Sent Done Done false
9803 Acked Write 8 Sent Done Done false
9804 Acked Read 10 Sent Done Done false
9805 Acked Write 8 Sent Done Done false
9806 Acked Write 3 Sent Done Done false
9807 Acked Write 7 Sent Done Done false
9808 Acked Write 8 Sent Done Done false
9809 Acked Write 1 Sent Done Done false
9810 Acked Write 8 Sent Done Done false
9811 Acked Write 2 Sent Done Done false
9812 Acked Read 1 Sent Done Done false
9813 Acked Read 2 Sent Done Done false
9814 Acked Flush 0 Sent Done Done false
9815 Acked Write 1 Sent Done Done false
9816 Acked Read 6 Sent Done Done false
9817 Acked Read 7 Sent Done Done false
9818 Acked Read 2 Sent Done Done false
9819 Acked Write 5 Sent Done Done false
9820 Acked Read 4 Sent Done Done false
9821 Acked Write 9 Sent Done Done false
9822 Acked Read 7 Sent Done Done false
9823 Acked Write 7 Sent Done Done false
9824 Acked Read 8 Sent Done Done false
9825 Acked Read 4 Sent Done Done false
9826 Acked Read 7 Sent Done Done false
9827 Acked Read 6 Sent Done Done false
9828 Acked Read 10 Sent Done Done false
9829 Acked Write 9 Sent Done Done false
9830 Acked Write 10 Sent Done Done false
9831 Acked Read 3 Sent Done Done false
9832 Acked Write 8 Sent Done Done false
9833 Acked Write 6 Sent Done Done false
9834 Acked Read 10 Sent Done Done false
9835 Acked Read 4 Sent Done Done false
9836 Acked Read 7 Sent Done Done false
9837 Acked Write 9 Sent Done Done false
9838 Acked Write 7 Sent Done Done false
9839 Acked Write 10 Sent Done Done false
9840 Acked Flush 0 Sent Done Done false
9841 Acked Read 6 Sent Done Done false
9842 Acked Write 1 Sent Done Done false
9843 Acked Write 10 Sent Done Done false
9844 Acked Write 1 Sent Done Done false
9845 Acked Read 6 Sent Done Done false
9846 Acked Read 2 Sent Done Done false
9847 Acked Read 3 Sent Done Done false
9848 Acked Read 10 Sent Done Done false
9849 Acked Read 7 Sent Done Done false
9850 Acked Flush 0 Sent Done Done false
9851 Acked Flush 0 Sent Done Done false
9852 Acked Write 6 Sent Done Done false
9853 Acked Read 5 Sent Done Done false
9854 Acked Write 5 Sent Done Done false
9855 Acked Read 6 Sent Done Done false
9856 Acked Read 5 Sent Done Done false
9857 Acked Write 8 Sent Done Done false
9858 Acked Read 6 Sent Done Done false
9859 Acked Read 6 Sent Done Done false
9860 Acked Read 7 Sent Done Done false
9861 Acked Flush 0 Sent Done Done false
9862 Acked Read 1 Sent Done Done false
9863 Acked Read 7 Sent Done Done false
9864 Acked Flush 0 Sent Done Done false
9865 Acked Read 6 Sent Done Done false
9866 Acked Read 10 Sent Done Done false
9867 Acked Read 10 Sent Done Done false
9868 Acked Read 7 Sent Done Done false
9869 Acked Read 3 Sent Done Done false
9870 Acked Write 6 Sent Done Done false
9871 Acked Write 8 Sent Done Done false
9872 Acked Read 9 Sent Done Done false
9873 Acked Write 5 Sent Done Done false
9874 Acked Write 9 Sent Done Done false
9875 Acked Read 4 Sent Done Done false
9876 Acked Read 8 Sent Done Done false
9877 Acked Flush 0 Sent Done Done false
9878 Acked Write 8 Sent Done Done false
9879 Acked Flush 0 Sent Done Done false
9880 Acked Write 9 Sent Done Done false
9881 Acked Read 7 Sent Done Done false
9882 Acked Flush 0 Sent Done Done false
9883 Acked Write 5 Sent Done Done false
9884 Acked Read 6 Sent Done Done false
9885 Acked Write 7 Sent Done Done false
9886 Acked Read 4 Sent Done Done false
9887 Acked Write 7 Sent Done Done false
9888 Acked Read 9 Sent Done Done false
9889 Acked Write 1 Sent Done Done false
9890 Acked Read 1 Sent Done Done false
9891 Acked Flush 0 Sent Done Done false
9892 Acked Read 7 Sent Done Done false
9893 Acked Write 7 Sent Done Done false
9894 Acked Write 9 Sent Done Done false
9895 Acked Read 9 Sent Done Done false
9896 Acked Read 9 Sent Done Done false
9897 Acked Write 5 Sent Done Done false
9898 Acked Read 7 Sent Done Done false
9899 Acked Flush 0 Sent Done Done false
9900 Acked Read 3 Sent Done Done false
9901 Acked Write 5 Sent Done Done false
9902 Acked Read 8 Sent Done Done false
9903 Acked Read 1 Sent Done Done false
9904 Acked Write 1 Sent Done Done false
9905 Acked Write 5 Sent Done Done false
9906 Acked Write 8 Sent Done Done false
9907 Acked Flush 0 Sent Done Done false
9908 Acked Read 9 Sent Done Done false
9909 Acked Read 4 Sent Done Done false
9910 Acked Flush 0 Sent Done Done false
9911 Acked Flush 0 Sent Done Done false
9912 Acked Read 9 Sent Done Done false
9913 Acked Write 2 Sent Done Done false
9914 Acked Read 10 Sent Done Done false
9915 Acked Write 9 Sent Done Done false
9916 Acked Read 3 Sent Done Done false
9917 Acked Write 7 Sent Done Done false
9918 Acked Write 9 Sent Done Done false
9919 Acked Read 8 Sent Done Done false
9920 Acked Read 2 Sent Done Done false
9921 Acked Read 4 Sent Done Done false
9922 Acked Read 1 Sent Done Done false
9923 Acked Write 3 Sent Done Done false
9924 Acked Flush 0 Sent Done Done false
9925 Acked Read 1 Sent Done Done false
9926 Acked Write 4 Sent Done Done false
9927 Acked Read 3 Sent Done Done false
9928 Acked Write 1 Sent Done Done false
9929 Acked Read 1 Sent Done Done false
9930 Acked Write 3 Sent Done Done false
9931 Acked Read 8 Sent Done Done false
9932 Acked Read 10 Sent Done Done false
9933 Acked Read 3 Sent Done Done false
9934 Acked Read 5 Sent Done Done false
9935 Acked Write 2 Sent Done Done false
9936 Acked Write 8 Sent Done Done false
9937 Acked Write 8 Sent Done Done false
9938 Acked Write 10 Sent Done Done false
9939 Acked Write 5 Sent Done Done false
9940 Acked Read 8 Sent Done Done false
9941 Acked Read 7 Sent Done Done false
9942 Acked Read 6 Sent Done Done false
9943 Acked Read 5 Sent Done Done false
9944 Acked Read 8 Sent Done Done false
9945 Acked Write 5 Sent Done Done false
9946 Acked Write 9 Sent Done Done false
9947 Acked Write 9 Sent Done Done false
9948 Acked Write 7 Sent Done Done false
9949 Acked Read 2 Sent Done Done false
9950 Acked Write 6 Sent Done Done false
9951 Acked Read 3 Sent Done Done false
9952 Acked Write 6 Sent Done Done false
9953 Acked Write 8 Sent Done Done false
9954 Acked Read 9 Sent Done Done false
9955 Acked Write 1 Sent Done Done false
9956 Acked Write 3 Sent Done Done false
9957 Acked Read 6 Sent Done Done false
9958 Acked Write 3 Sent Done Done false
9959 Acked Flush 0 Sent Done Done false
9960 Acked Write 6 Sent Done Done false
9961 Acked Flush 0 Sent Done Done false
9962 Acked Flush 0 Sent Done Done false
9963 Acked Write 4 Sent Done Done false
9964 Acked Read 1 Sent Done Done false
9965 Acked Write 8 Sent Done Done false
9966 Acked Read 4 Sent Done Done false
9967 Acked Read 9 Sent Done Done false
9968 Acked Write 9 Sent Done Done false
9969 Acked Read 5 Sent Done Done false
9970 Acked Write 6 Sent Done Done false
9971 Acked Read 4 Sent Done Done false
9972 Acked Write 7 Sent Done Done false
9973 Acked Write 10 Sent Done Done false
9974 Acked Flush 0 Sent Done Done false
9975 Acked Read 3 Sent Done Done false
9976 Acked Read 3 Sent Done Done false
9977 Acked Read 7 Sent Done Done false
9978 Acked Read 9 Sent Done Done false
9979 Acked Write 10 Sent Done Done false
9980 Acked Read 9 Sent Done Done false
9981 Acked Read 8 Sent Done Done false
9982 Acked Write 9 Sent Done Done false
9983 Acked Write 9 Sent Done Done false
9984 Acked Read 8 Sent Done Done false
9985 Acked Write 1 Sent Done Done false
9986 Acked Flush 0 Sent Done Done false
9987 Acked Read 6 Sent Done Done false
9988 Acked Read 6 Sent Done Done false
9989 Acked Write 3 Sent Done Done false
9990 Acked Read 7 Sent Done Done false
9991 Acked Read 2 Sent Done Done false
9992 Acked Read 10 Sent Done Done false
9993 Acked Read 4 Sent Done Done false
9994 Acked Write 3 Sent Done Done false
9995 Acked Write 9 Sent Done Done false
9996 Acked Write 10 Sent Done Done false
9997 Acked Read 7 Sent Done Done false
9998 Acked Read 4 Sent Done Done false
9999 Acked Read 9 Sent Done Done false
10000 Acked Read 2 Sent Done Done false
10001 Acked Flush 0 Sent Done Done false
10002 Acked Write 7 Sent Done Done false
10003 Acked Read 4 Sent Done Done false
10004 Acked Read 2 Sent Done Done false
10005 Acked Write 4 Sent Done Done false
10006 Acked Write 8 Sent Done Done false
10007 Acked Write 7 Sent Done Done false
10008 Acked Flush 0 Sent Done Done false
10009 Acked Flush 0 Sent Done Done false
10010 Acked Flush 0 Sent Done Done false
10011 Acked Write 8 Sent Done Done false
10012 Acked Read 3 Sent Done Done false
10013 Acked Read 10 Sent Done Done false
10014 Acked Write 2 Sent Done Done false
10015 Acked Read 5 Sent Done Done false
10016 Acked Flush 0 Sent Done Done false
10017 Acked Read 6 Sent Done Done false
10018 Acked Read 7 Sent Done Done false
10019 Acked Read 4 Sent Done Done false
10020 Acked Write 5 Sent Done Done false
10021 Acked Read 5 Sent Done Done false
10022 Acked Flush 0 Sent Done Done false
10023 Acked Read 9 Sent Done Done false
10024 Acked Read 1 Sent Done Done false
10025 Acked Flush 0 Sent Done Done false
10026 Acked Read 4 Sent Done Done false
10027 Acked Read 9 Sent Done Done false
10028 Acked Write 3 Sent Done Done false
10029 Acked Write 7 Sent Done Done false
10030 Acked Read 6 Sent Done Done false
10031 Acked Write 2 Sent Done Done false
10032 Acked Write 1 Sent Done Done false
10033 Acked Read 8 Sent Done Done false
10034 Acked Write 4 Sent Done Done false
10035 Acked Write 7 Sent Done Done false
10036 Acked Write 1 Sent Done Done false
10037 Acked Write 8 Sent Done Done false
10038 Acked Write 8 Sent Done Done false
10039 Acked Read 7 Sent Done Done false
10040 Acked Write 9 Sent Done Done false
10041 Acked Write 5 Sent Done Done false
10042 Acked Read 3 Sent Done Done false
10043 Acked Read 4 Sent Done Done false
10044 Acked Read 7 Sent Done Done false
10045 Acked Read 1 Sent Done Done false
10046 Acked Write 3 Sent Done Done false
10047 Acked Read 1 Sent Done Done false
10048 Acked Read 6 Sent Done Done false
10049 Acked Read 3 Sent Done Done false
10050 Acked Write 9 Sent Done Done false
10051 Acked Read 8 Sent Done Done false
10052 Acked Write 1 Sent Done Done false
10053 Acked Read 10 Sent Done Done false
10054 Acked Read 9 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9753)) Some(JobId(10025)) Some(JobId(10025))
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 10054 10053 10052 10051 10050
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:44.111944671Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:45.112880271Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:46.115176943Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1241)
[8810][0] reports Running
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:47.427795895Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9754 Acked Flush 0 Sent Done Done false
9755 Acked Flush 0 Sent Done Done false
9756 Acked Read 9 Sent Done Done false
9757 Acked Write 9 Sent Done Done false
9758 Acked Read 8 Sent Done Done false
9759 Acked Write 7 Sent Done Done false
9760 Acked Write 10 Sent Done Done false
9761 Acked Write 1 Sent Done Done false
9762 Acked Write 8 Sent Done Done false
9763 Acked Flush 0 Sent Done Done false
9764 Acked Write 7 Sent Done Done false
9765 Acked Read 6 Sent Done Done false
9766 Acked Write 2 Sent Done Done false
9767 Acked Read 1 Sent Done Done false
9768 Acked Write 3 Sent Done Done false
9769 Acked Read 6 Sent Done Done false
9770 Acked Write 8 Sent Done Done false
9771 Acked Read 7 Sent Done Done false
9772 Acked Write 4 Sent Done Done false
9773 Acked Write 10 Sent Done Done false
9774 Acked Read 4 Sent Done Done false
9775 Acked Write 7 Sent Done Done false
9776 Acked Write 1 Sent Done Done false
9777 Acked Write 10 Sent Done Done false
9778 Acked Read 9 Sent Done Done false
9779 Acked Read 5 Sent Done Done false
9780 Acked Write 2 Sent Done Done false
9781 Acked Read 2 Sent Done Done false
9782 Acked Flush 0 Sent Done Done false
9783 Acked Read 4 Sent Done Done false
9784 Acked Read 7 Sent Done Done false
9785 Acked Read 6 Sent Done Done false
9786 Acked Write 7 Sent Done Done false
9787 Acked Read 1 Sent Done Done false
9788 Acked Write 6 Sent Done Done false
9789 Acked Flush 0 Sent Done Done false
9790 Acked Read 10 Sent Done Done false
9791 Acked Write 4 Sent Done Done false
9792 Acked Read 9 Sent Done Done false
9793 Acked Write 7 Sent Done Done false
9794 Acked Read 7 Sent Done Done false
9795 Acked Write 6 Sent Done Done false
9796 Acked Read 2 Sent Done Done false
9797 Acked Read 5 Sent Done Done false
9798 Acked Write 2 Sent Done Done false
9799 Acked Read 9 Sent Done Done false
9800 Acked Read 7 Sent Done Done false
9801 Acked Read 3 Sent Done Done false
9802 Acked Write 9 Sent Done Done false
9803 Acked Write 8 Sent Done Done false
9804 Acked Read 10 Sent Done Done false
9805 Acked Write 8 Sent Done Done false
9806 Acked Write 3 Sent Done Done false
9807 Acked Write 7 Sent Done Done false
9808 Acked Write 8 Sent Done Done false
9809 Acked Write 1 Sent Done Done false
9810 Acked Write 8 Sent Done Done false
9811 Acked Write 2 Sent Done Done false
9812 Acked Read 1 Sent Done Done false
9813 Acked Read 2 Sent Done Done false
9814 Acked Flush 0 Sent Done Done false
9815 Acked Write 1 Sent Done Done false
9816 Acked Read 6 Sent Done Done false
9817 Acked Read 7 Sent Done Done false
9818 Acked Read 2 Sent Done Done false
9819 Acked Write 5 Sent Done Done false
9820 Acked Read 4 Sent Done Done false
9821 Acked Write 9 Sent Done Done false
9822 Acked Read 7 Sent Done Done false
9823 Acked Write 7 Sent Done Done false
9824 Acked Read 8 Sent Done Done false
9825 Acked Read 4 Sent Done Done false
9826 Acked Read 7 Sent Done Done false
9827 Acked Read 6 Sent Done Done false
9828 Acked Read 10 Sent Done Done false
9829 Acked Write 9 Sent Done Done false
9830 Acked Write 10 Sent Done Done false
9831 Acked Read 3 Sent Done Done false
9832 Acked Write 8 Sent Done Done false
9833 Acked Write 6 Sent Done Done false
9834 Acked Read 10 Sent Done Done false
9835 Acked Read 4 Sent Done Done false
9836 Acked Read 7 Sent Done Done false
9837 Acked Write 9 Sent Done Done false
9838 Acked Write 7 Sent Done Done false
9839 Acked Write 10 Sent Done Done false
9840 Acked Flush 0 Sent Done Done false
9841 Acked Read 6 Sent Done Done false
9842 Acked Write 1 Sent Done Done false
9843 Acked Write 10 Sent Done Done false
9844 Acked Write 1 Sent Done Done false
9845 Acked Read 6 Sent Done Done false
9846 Acked Read 2 Sent Done Done false
9847 Acked Read 3 Sent Done Done false
9848 Acked Read 10 Sent Done Done false
9849 Acked Read 7 Sent Done Done false
9850 Acked Flush 0 Sent Done Done false
9851 Acked Flush 0 Sent Done Done false
9852 Acked Write 6 Sent Done Done false
9853 Acked Read 5 Sent Done Done false
9854 Acked Write 5 Sent Done Done false
9855 Acked Read 6 Sent Done Done false
9856 Acked Read 5 Sent Done Done false
9857 Acked Write 8 Sent Done Done false
9858 Acked Read 6 Sent Done Done false
9859 Acked Read 6 Sent Done Done false
9860 Acked Read 7 Sent Done Done false
9861 Acked Flush 0 Sent Done Done false
9862 Acked Read 1 Sent Done Done false
9863 Acked Read 7 Sent Done Done false
9864 Acked Flush 0 Sent Done Done false
9865 Acked Read 6 Sent Done Done false
9866 Acked Read 10 Sent Done Done false
9867 Acked Read 10 Sent Done Done false
9868 Acked Read 7 Sent Done Done false
9869 Acked Read 3 Sent Done Done false
9870 Acked Write 6 Sent Done Done false
9871 Acked Write 8 Sent Done Done false
9872 Acked Read 9 Sent Done Done false
9873 Acked Write 5 Sent Done Done false
9874 Acked Write 9 Sent Done Done false
9875 Acked Read 4 Sent Done Done false
9876 Acked Read 8 Sent Done Done false
9877 Acked Flush 0 Sent Done Done false
9878 Acked Write 8 Sent Done Done false
9879 Acked Flush 0 Sent Done Done false
9880 Acked Write 9 Sent Done Done false
9881 Acked Read 7 Sent Done Done false
9882 Acked Flush 0 Sent Done Done false
9883 Acked Write 5 Sent Done Done false
9884 Acked Read 6 Sent Done Done false
9885 Acked Write 7 Sent Done Done false
9886 Acked Read 4 Sent Done Done false
9887 Acked Write 7 Sent Done Done false
9888 Acked Read 9 Sent Done Done false
9889 Acked Write 1 Sent Done Done false
9890 Acked Read 1 Sent Done Done false
9891 Acked Flush 0 Sent Done Done false
9892 Acked Read 7 Sent Done Done false
9893 Acked Write 7 Sent Done Done false
9894 Acked Write 9 Sent Done Done false
9895 Acked Read 9 Sent Done Done false
9896 Acked Read 9 Sent Done Done false
9897 Acked Write 5 Sent Done Done false
9898 Acked Read 7 Sent Done Done false
9899 Acked Flush 0 Sent Done Done false
9900 Acked Read 3 Sent Done Done false
9901 Acked Write 5 Sent Done Done false
9902 Acked Read 8 Sent Done Done false
9903 Acked Read 1 Sent Done Done false
9904 Acked Write 1 Sent Done Done false
9905 Acked Write 5 Sent Done Done false
9906 Acked Write 8 Sent Done Done false
9907 Acked Flush 0 Sent Done Done false
9908 Acked Read 9 Sent Done Done false
9909 Acked Read 4 Sent Done Done false
9910 Acked Flush 0 Sent Done Done false
9911 Acked Flush 0 Sent Done Done false
9912 Acked Read 9 Sent Done Done false
9913 Acked Write 2 Sent Done Done false
9914 Acked Read 10 Sent Done Done false
9915 Acked Write 9 Sent Done Done false
9916 Acked Read 3 Sent Done Done false
9917 Acked Write 7 Sent Done Done false
9918 Acked Write 9 Sent Done Done false
9919 Acked Read 8 Sent Done Done false
9920 Acked Read 2 Sent Done Done false
9921 Acked Read 4 Sent Done Done false
9922 Acked Read 1 Sent Done Done false
9923 Acked Write 3 Sent Done Done false
9924 Acked Flush 0 Sent Done Done false
9925 Acked Read 1 Sent Done Done false
9926 Acked Write 4 Sent Done Done false
9927 Acked Read 3 Sent Done Done false
9928 Acked Write 1 Sent Done Done false
9929 Acked Read 1 Sent Done Done false
9930 Acked Write 3 Sent Done Done false
9931 Acked Read 8 Sent Done Done false
9932 Acked Read 10 Sent Done Done false
9933 Acked Read 3 Sent Done Done false
9934 Acked Read 5 Sent Done Done false
9935 Acked Write 2 Sent Done Done false
9936 Acked Write 8 Sent Done Done false
9937 Acked Write 8 Sent Done Done false
9938 Acked Write 10 Sent Done Done false
9939 Acked Write 5 Sent Done Done false
9940 Acked Read 8 Sent Done Done false
9941 Acked Read 7 Sent Done Done false
9942 Acked Read 6 Sent Done Done false
9943 Acked Read 5 Sent Done Done false
9944 Acked Read 8 Sent Done Done false
9945 Acked Write 5 Sent Done Done false
9946 Acked Write 9 Sent Done Done false
9947 Acked Write 9 Sent Done Done false
9948 Acked Write 7 Sent Done Done false
9949 Acked Read 2 Sent Done Done false
9950 Acked Write 6 Sent Done Done false
9951 Acked Read 3 Sent Done Done false
9952 Acked Write 6 Sent Done Done false
9953 Acked Write 8 Sent Done Done false
9954 Acked Read 9 Sent Done Done false
9955 Acked Write 1 Sent Done Done false
9956 Acked Write 3 Sent Done Done false
9957 Acked Read 6 Sent Done Done false
9958 Acked Write 3 Sent Done Done false
9959 Acked Flush 0 Sent Done Done false
9960 Acked Write 6 Sent Done Done false
9961 Acked Flush 0 Sent Done Done false
9962 Acked Flush 0 Sent Done Done false
9963 Acked Write 4 Sent Done Done false
9964 Acked Read 1 Sent Done Done false
9965 Acked Write 8 Sent Done Done false
9966 Acked Read 4 Sent Done Done false
9967 Acked Read 9 Sent Done Done false
9968 Acked Write 9 Sent Done Done false
9969 Acked Read 5 Sent Done Done false
9970 Acked Write 6 Sent Done Done false
9971 Acked Read 4 Sent Done Done false
9972 Acked Write 7 Sent Done Done false
9973 Acked Write 10 Sent Done Done false
9974 Acked Flush 0 Sent Done Done false
9975 Acked Read 3 Sent Done Done false
9976 Acked Read 3 Sent Done Done false
9977 Acked Read 7 Sent Done Done false
9978 Acked Read 9 Sent Done Done false
9979 Acked Write 10 Sent Done Done false
9980 Acked Read 9 Sent Done Done false
9981 Acked Read 8 Sent Done Done false
9982 Acked Write 9 Sent Done Done false
9983 Acked Write 9 Sent Done Done false
9984 Acked Read 8 Sent Done Done false
9985 Acked Write 1 Sent Done Done false
9986 Acked Flush 0 Sent Done Done false
9987 Acked Read 6 Sent Done Done false
9988 Acked Read 6 Sent Done Done false
9989 Acked Write 3 Sent Done Done false
9990 Acked Read 7 Sent Done Done false
9991 Acked Read 2 Sent Done Done false
9992 Acked Read 10 Sent Done Done false
9993 Acked Read 4 Sent Done Done false
9994 Acked Write 3 Sent Done Done false
9995 Acked Write 9 Sent Done Done false
9996 Acked Write 10 Sent Done Done false
9997 Acked Read 7 Sent Done Done false
9998 Acked Read 4 Sent Done Done false
9999 Acked Read 9 Sent Done Done false
10000 Acked Read 2 Sent Done Done false
10001 Acked Flush 0 Sent Done Done false
10002 Acked Write 7 Sent Done Done false
10003 Acked Read 4 Sent Done Done false
10004 Acked Read 2 Sent Done Done false
10005 Acked Write 4 Sent Done Done false
10006 Acked Write 8 Sent Done Done false
10007 Acked Write 7 Sent Done Done false
10008 Acked Flush 0 Sent Done Done false
10009 Acked Flush 0 Sent Done Done false
10010 Acked Flush 0 Sent Done Done false
10011 Acked Write 8 Sent Done Done false
10012 Acked Read 3 Sent Done Done false
10013 Acked Read 10 Sent Done Done false
10014 Acked Write 2 Sent Done Done false
10015 Acked Read 5 Sent Done Done false
10016 Acked Flush 0 Sent Done Done false
10017 Acked Read 6 Sent Done Done false
10018 Acked Read 7 Sent Done Done false
10019 Acked Read 4 Sent Done Done false
10020 Acked Write 5 Sent Done Done false
10021 Acked Read 5 Sent Done Done false
10022 Acked Flush 0 Sent Done Done false
10023 Acked Read 9 Sent Done Done false
10024 Acked Read 1 Sent Done Done false
10025 Acked Flush 0 Sent Done Done false
10026 Acked Read 4 Sent Done Done false
10027 Acked Read 9 Sent Done Done false
10028 Acked Write 3 Sent Done Done false
10029 Acked Write 7 Sent Done Done false
10030 Acked Read 6 Sent Done Done false
10031 Acked Write 2 Sent Done Done false
10032 Acked Write 1 Sent Done Done false
10033 Acked Read 8 Sent Done Done false
10034 Acked Write 4 Sent Done Done false
10035 Acked Write 7 Sent Done Done false
10036 Acked Write 1 Sent Done Done false
10037 Acked Write 8 Sent Done Done false
10038 Acked Write 8 Sent Done Done false
10039 Acked Read 7 Sent Done Done false
10040 Acked Write 9 Sent Done Done false
10041 Acked Write 5 Sent Done Done false
10042 Acked Read 3 Sent Done Done false
10043 Acked Read 4 Sent Done Done false
10044 Acked Read 7 Sent Done Done false
10045 Acked Read 1 Sent Done Done false
10046 Acked Write 3 Sent Done Done false
10047 Acked Read 1 Sent Done Done false
10048 Acked Read 6 Sent Done Done false
10049 Acked Read 3 Sent Done Done false
10050 Acked Write 9 Sent Done Done false
10051 Acked Read 8 Sent Done Done false
10052 Acked Write 1 Sent Done Done false
10053 Acked Read 10 Sent Done Done false
10054 Acked Read 9 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(9753)) Some(JobId(10025)) Some(JobId(10025))
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 10054 10053 10052 10051 10050
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:48.119220621Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:49.12140657Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.181710808Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39060)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.181886356Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.183008667Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.183544368Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.183564734Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(9753))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.18357883Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.183592587Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] client re-new 301 jobs since flush Some(JobId(9753))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.183605804Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[0] Replayed 301 jobs since flush: Some(JobId(9753))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:49.18382835Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10055 10054 10053 10052 10051
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 28 06:08:51.437 INFO request completed, latency_us: 394, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2b316f51-2f4c-4d7d-9c53-15dbcb3a6db4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Feb 28 06:08:51.438 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9bb5706f-7d0f-4dab-bee5-520bdd8d94c5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:51.438155246Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:51.438212182Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:51.438229308Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:51.438310249Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Feb 28 06:08:55.440 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/2, method: GET, req_id: 33993828-a554-4b03-91b5-7b177879233d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Flush
003/300 Read block 3032 len 8192
004/300 Read block 4172 len 12288
005/300 Read block 2116 len 28672
006/300 Write block 1301 len 8192 data: 6 5
007/300 Write block 2186 len 12288 data: 9 7 8
008/300 Write block 4399 len 8192 data: 5 5
009/300 Read block 4664 len 12288
010/300 Write block 2229 len 8192 data: 7 9
011/300 Read block 1855 len 16384
012/300 Write block 1468 len 28672 data: 7 8 7 8 7 7 8
013/300 Read block 214 len 24576
014/300 Read block 1157 len 40960
015/300 Write block 2314 len 16384 data: 6 5 5 6
016/300 Read block 4182 len 36864
017/300 Read block 1734 len 24576
018/300 Read block 941 len 12288
019/300 Read block 3642 len 32768
020/300 Read block 3557 len 16384
021/300 Write block 3163 len 28672 data: 3 4 6 6 6 6 6
022/300 Write block 2854 len 20480 data: 7 7 7 7 7
023/300 Read block 1240 len 8192
024/300 Write block 4131 len 20480 data: 5 4 6 5 5
025/300 Flush
026/300 Read block 117 len 16384
027/300 Write block 2481 len 24576 data: 6 5 7 8 8 7
028/300 Read block 676 len 28672
029/300 Read block 680 len 28672
030/300 Write block 3480 len 40960 data: 10 9 8 8 8 6 7 5 5 6
031/300 Flush
032/300 Read block 3994 len 24576
033/300 Flush
034/300 Read block 3025 len 8192
035/300 Read block 583 len 8192
036/300 Read block 1304 len 16384
037/300 Read block 1346 len 20480
038/300 Read block 1237 len 36864
039/300 Read block 3629 len 12288
040/300 Write block 448 len 20480 data: 9 10 9 10 8
041/300 Write block 4556 len 24576 data: 3 4 4 4 4 5
042/300 Read block 1109 len 36864
043/300 Write block 2707 len 24576 data: 7 7 7 5 5 6
044/300 Read block 3536 len 40960
045/300 Write block 4978 len 28672 data: 8 7 6 6 8 7 8
046/300 Write block 4396 len 20480 data: 7 6 5 6 6
047/300 Write block 1117 len 16384 data: 3 4 4 6
048/300 Write block 532 len 36864 data: 5 5 6 5 4 5 4 5 4
049/300 Read block 2656 len 16384
050/300 Read block 4456 len 32768
051/300 Write block 3097 len 40960 data: 6 6 6 6 8 7 9 8 9 8
052/300 Write block 1015 len 4096 data: 6
053/300 Write block 2831 len 36864 data: 3 2 5 4 5 5 6 6 6
054/300 Write block 4340 len 40960 data: 8 7 10 9 9 8 10 11 9 9
055/300 Read block 834 len 4096
056/300 Read block 3441 len 4096
057/300 Flush
058/300 Read block 4932 len 20480
059/300 Read block 2742 len 28672
060/300 Read block 3063 len 4096
061/300 Read block 252 len 8192
062/300 Write block 3121 len 36864 data: 5 5 5 5 5 4 4 3 4
063/300 Write block 1081 len 8192 data: 7 7
064/300 Flush
065/300 Read block 1123 len 12288
066/300 Read block 1145 len 20480
067/300 Read block 2430 len 16384
068/300 Read block 412 len 36864
069/300 Flush
070/300 Read block 497 len 4096
071/300 Read block 1018 len 36864
072/300 Write block 2664 len 12288 data: 10 10 10
073/300 Read block 4788 len 8192
074/300 Flush
075/300 Write block 75 len 32768 data: 11 9 9 8 10 9 11 8
076/300 Read block 4462 len 36864
077/300 Read block 2263 len 36864
078/300 Read block 2260 len 28672
079/300 Read block 2776 len 36864
080/300 Read block 3299 len 28672
081/300 Write block 2141 len 16384 data: 7 7 6 5
082/300 Write block 102 len 16384 data: 4 6 6 5
083/300 Read block 332 len 4096
084/300 Read block 2770 len 16384
085/300 Write block 4479 len 36864 data: 6 5 7 7 7 6 7 8 8
086/300 Write block 2786 len 4096 data: 7
087/300 Read block 2465 len 28672
088/300 Read block 4943 len 8192
089/300 Read block 3267 len 20480
090/300 Read block 4219 len 36864
091/300 Read block 216 len 12288
092/300 Read block 3382 len 36864
093/300 Read block 4498 len 8192
094/300 Read block 3934 len 32768
095/300 Read block 80 len 40960
096/300 Write block 3005 len 16384 data: 3 5 5 5
097/300 Read block 4082 len 12288
098/300 Read block 1994 len 8192
099/300 Write block 2756 len 36864 data: 6 4 6 8 7 6 5 5 5
100/300 Write block 3705 len 40960 data: 6 7 9 7 6 5 5 3 2 3
101/300 Write block 3911 len 8192 data: 5 6
102/300 Flush
103/300 Write block 953 len 28672 data: 5 5 4 4 4 4 4
104/300 Read block 2451 len 28672
105/300 Write block 844 len 16384 data: 7 6 5 2
106/300 Read block 1191 len 8192
107/300 Read block 109 len 28672
108/300 Read block 1961 len 20480
109/300 Read block 2933 len 8192
110/300 Read block 489 len 28672
111/300 Write block 4054 len 12288 data: 4 5 5
112/300 Read block 1046 len 8192
113/300 Write block 4900 len 36864 data: 7 6 7 7 6 5 5 4 4
114/300 Read block 2166 len 8192
115/300 Read block 396 len 36864
116/300 Write block 2116 len 32768 data: 7 7 7 6 6 7 8 8
117/300 Read block 4954 len 20480
118/300 Read block 1114 len 20480
119/300 Read block 1356 len 12288
120/300 Read block 941 len 4096
121/300 Read block 3932 len 8192
122/300 Write block 4033 len 20480 data: 5 5 5 7 8
123/300 Write block 1595 len 36864 data: 4 4 4 2 3 4 4 4 4
124/300 Flush
125/300 Write block 2803 len 32768 data: 6 6 7 7 7 6 6 5
126/300 Write block 3989 len 28672 data: 6 6 6 5 6 7 6
127/300 Write block 1921 len 20480 data: 5 3 3 3 4
128/300 Read block 2032 len 16384
129/300 Read block 1498 len 12288
130/300 Write block 1130 len 36864 data: 8 9 8 7 9 9 10 10 8
131/300 Read block 4785 len 24576
132/300 Read block 3366 len 32768
133/300 Read block 2864 len 32768
134/300 Read block 216 len 28672
135/300 Read block 4868 len 8192
136/300 Read block 442 len 40960
137/300 Write block 79 len 8192 data: 11 10
138/300 Read block 4174 len 36864
139/300 Write block 3664 len 12288 data: 5 5 5
140/300 Read block 1584 len 12288
141/300 Write block 3749 len 16384 data: 8 6 7 7
142/300 Read block 4622 len 20480
143/300 Read block 4359 len 20480
144/300 Read block 2025 len 36864
145/300 Read block 4984 len 40960
146/300 Read block 916 len 36864
147/300 Read block 2842 len 36864
148/300 Write block 4220 len 20480 data: 4 4 3 4 5
149/300 Write block 4057 len 24576 data: 4 6 7 7 5 7
150/300 Read block 66 len 16384
151/300 Read block 3720 len 40960
152/300 Read block 3891 len 8192
153/300 Write block 4504 len 24576 data: 6 6 8 10 8 7
154/300 Read block 3903 len 24576
155/300 Read block 4665 len 4096
156/300 Read block 1944 len 40960
157/300 Write block 3046 len 12288 data: 3 5 5
158/300 Write block 1283 len 4096 data: 7
159/300 Write block 4116 len 12288 data: 8 8 5
160/300 Write block 4951 len 20480 data: 8 6 7 8 7
161/300 Read block 1981 len 36864
162/300 Write block 238 len 36864 data: 8 6 5 5 6 5 6 6 5
163/300 Read block 3394 len 20480
164/300 Read block 1505 len 28672
165/300 Write block 244 len 36864 data: 7 7 6 4 4 5 5 6 5
166/300 Read block 3773 len 20480
167/300 Read block 228 len 24576
168/300 Write block 414 len 4096 data: 10
169/300 Read block 4464 len 20480
170/300 Read block 1154 len 12288
171/300 Read block 4263 len 8192
172/300 Read block 2889 len 40960
173/300 Write block 3605 len 16384 data: 6 6 7 6
174/300 Read block 389 len 20480
175/300 Write block 729 len 12288 data: 2 3 3
176/300 Read block 2779 len 8192
177/300 Read block 966 len 32768
178/300 Write block 1695 len 16384 data: 5 6 6 5
179/300 Write block 4432 len 20480 data: 6 5 4 5 7
180/300 Read block 3144 len 12288
181/300 Read block 4731 len 4096
182/300 Write block 1420 len 40960 data: 6 7 6 6 7 8 8 8 5 3
183/300 Read block 2566 len 20480
184/300 Flush
185/300 Write block 3490 len 24576 data: 4 4 5 6 6 7
186/300 Flush
187/300 Read block 4233 len 36864
188/300 Read block 3781 len 28672
189/300 Write block 2597 len 20480 data: 5 5 6 7 8
190/300 Read block 3348 len 28672
191/300 Read block 2345 len 20480
192/300 Write block 2087 len 24576 data: 10 10 10 10 10 6
193/300 Read block 118 len 36864
194/300 Write block 1633 len 36864 data: 7 6 5 6 5 3 2 3 5
195/300 Read block 1662 len 24576
196/300 Read block 3739 len 8192
197/300 Read block 2352 len 8192
198/300 Write block 3607 len 36864 data: 8 7 5 7 8 8 8 9 8
199/300 Read block 493 len 16384
200/300 Read block 2756 len 28672
201/300 Read block 2321 len 16384
202/300 Read block 2301 len 8192
203/300 Write block 222 len 12288 data: 7 6 5
204/300 Read block 3373 len 40960
205/300 Read block 1640 len 24576
206/300 Write block 4302 len 8192 data: 3 5
207/300 Write block 2417 len 20480 data: 5 6 7 6 6
208/300 Write block 1869 len 4096 data: 10
209/300 Read block 4109 len 8192
210/300 Read block 480 len 24576
211/300 Write block 1810 len 12288 data: 7 7 6
212/300 Write block 3531 len 40960 data: 8 9 9 8 5 7 5 8 7 7
213/300 Read block 894 len 4096
214/300 Read block 760 len 32768
215/300 Read block 3442 len 4096
216/300 Read block 4777 len 28672
217/300 Write block 442 len 4096 data: 12
218/300 Read block 4011 len 32768
219/300 Read block 1488 len 12288
220/300 Write block 238 len 24576 data: 9 7 6 6 7 6
221/300 Write block 3024 len 8192 data: 5 4
222/300 Write block 4962 len 40960 data: 5 6 7 8 6 7 11 11 10 12
223/300 Flush
224/300 Read block 3582 len 8192
225/300 Read block 4633 len 24576
226/300 Write block 3603 len 20480 data: 4 5 7 7 9
227/300 Read block 3142 len 20480
228/300 Read block 304 len 28672
229/300 Write block 2264 len 32768 data: 11 9 9 10 8 7 6 5
230/300 Flush
231/300 Read block 4182 len 4096
232/300 Write block 2046 len 20480 data: 5 4 3 3 3
233/300 Read block 3993 len 12288
234/300 Read block 694 len 4096
235/300 Write block 1161 len 40960 data: 6 6 6 9 8 8 8 7 9 8
236/300 Read block 3069 len 36864
237/300 Flush
238/300 Write block 1163 len 20480 data: 7 10 9 9 9
239/300 Read block 1444 len 36864
240/300 Write block 475 len 32768 data: 8 9 8 8 7 8 8 8
241/300 Read block 4842 len 12288
242/300 Read block 1549 len 16384
243/300 Read block 3752 len 8192
244/300 Read block 2785 len 20480
245/300 Write block 2004 len 36864 data: 7 6 6 5 4 3 3 4 6
246/300 Read block 1944 len 20480
247/300 Read block 4888 len 16384
248/300 Read block 535 len 16384
249/300 Write block 3014 len 12288 data: 6 7 5
250/300 Flush
251/300 Flush
252/300 Read block 902 len 24576
253/300 Read block 1544 len 20480
254/300 Write block 2954 len 36864 data: 6 5 4 5 6 5 5 7 8
255/300 Read block 2015 len 16384
256/300 Write block 692 len 12288 data: 6 5 5
257/300 Flush
258/300 Write block 2506 len 12288 data: 4 4 5
259/300 Write block 4636 len 28672 data: 4 6 6 7 7 5 7
260/300 Write block 4837 len 32768 data: 8 9 8 8 8 9 8 7
261/300 Flush
262/300 Write block 2991 len 24576 data: 9 10 7 7 6 8
263/300 Write block 3726 len 32768 data: 4 3 4 4 4 7 5 6
264/300 Write block 2108 len 28672 data: 7 6 8 9 8 8 7
265/300 Read block 1322 len 36864
266/300 Write block 2329 len 12288 data: 5 4 4
267/300 Write block 2528 len 8192 data: 7 9
268/300 Read block 151 len 16384
269/300 Read block 4141 len 12288
270/300 Write block 840 len 32768 data: 4 5 5 5 8 7 6 3
271/300 Flush
272/300 Read block 619 len 12288
273/300 Read block 285 len 40960
274/300 Read block 2249 len 24576
275/300 Read block 2420 len 16384
276/300 Read block 2389 len 24576
277/300 Read block 4315 len 24576
278/300 Write block 4867 len 28672 data: 5 6 4 4 5 5 5
279/300 Write block 657 len 36864 data: 7 8 8 8 8 10 9 9 9
280/300 Read block 4477 len 12288
281/300 Read block 2639 len 12288
282/300 Write block 2086 len 16384 data: 10 11 11 11
283/300 Read block 4903 len 16384
284/300 Read block 1205 len 28672
285/300 Write block 144 len 20480 data: 7 5 6 6 6
286/300 Read block 683 len 20480
287/300 Read block 157 len 28672
288/300 Flush
289/300 Write block 3731 len 20480 data: 8 6 7 8 8
290/300 Read block 2805 len 8192
291/300 Read block 4052 len 28672
292/300 Write block 2985 len 4096 data: 4
293/300 Read block 1835 len 32768
294/300 Read block 456 len 28672
295/300 Write block 684 len 40960 data: 6 5 4 4 4 5 6 5 7 6
296/300 Read block 2915 len 4096
297/300 Read block 1691 len 4096
298/300 Read block 114 len 4096
299/300 Read block 913 len 28672
300/300 Read block 4052 len 36864
301/300 Read block 1176 len 28672
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 28 06:08:55.692 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 63d592a5-5598-435b-a1c9-31b7df4f190d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10056 Acked Flush 0 Done Done Sent false
10057 Acked Flush 0 Done Done Sent false
10058 Acked Read 2 Done Done Sent false
10059 Acked Read 3 Done Done Sent false
10060 Acked Read 7 Done Done Sent false
10061 Acked Write 2 Done Done Sent false
10062 Acked Write 3 Done Done Sent false
10063 Acked Write 2 Done Done Sent false
10064 Acked Read 3 Done Done Sent false
10065 Acked Write 2 Done Done Sent false
10066 Acked Read 4 Done Done Sent false
10067 Acked Write 7 Done Done Sent false
10068 Acked Read 6 Done Done Sent false
10069 Acked Read 10 Done Done Sent false
10070 Acked Write 4 Done Done Sent false
10071 Acked Read 9 Done Done Sent false
10072 Acked Read 6 Done Done Sent false
10073 Acked Read 3 Done Done Sent false
10074 Acked Read 8 Done Done Sent false
10075 Acked Read 4 Done Done Sent false
10076 Acked Write 7 Done Done Sent false
10077 Acked Write 5 Done Done Sent false
10078 Acked Read 2 Done Done Sent false
10079 Acked Write 5 Done Done Sent false
10080 Acked Flush 0 Done Done Sent false
10081 Acked Read 4 Done Done Sent false
10082 Acked Write 6 Done Done Sent false
10083 Acked Read 7 Done Done Sent false
10084 Acked Read 7 Done Done Sent false
10085 Acked Write 10 Done Done Sent false
10086 Acked Flush 0 Done Done Sent false
10087 Acked Read 6 Done Done Sent false
10088 Acked Flush 0 Done Done Sent false
10089 Acked Read 2 Done Done Sent false
10090 Acked Read 2 Done Done Sent false
10091 Acked Read 4 Done Done Sent false
10092 Acked Read 5 Done Done Sent false
10093 Acked Read 9 Done Done Sent false
10094 Acked Read 3 Done Done Sent false
10095 Acked Write 5 Done Done Sent false
10096 Acked Write 6 Done Done Sent false
10097 Acked Read 9 Done Done Sent false
10098 Acked Write 6 Done Done Sent false
10099 Acked Read 10 Done Done Sent false
10100 Acked Write 7 Done Done Sent false
10101 Acked Write 5 Done Done Sent false
10102 Acked Write 4 Done Done Sent false
10103 Acked Write 9 Done Done Sent false
10104 Acked Read 4 Done Done Sent false
10105 Acked Read 8 Done Done Sent false
10106 Acked Write 10 Done Done Sent false
10107 Acked Write 1 Done Done Sent false
10108 Acked Write 9 Done Done Sent false
10109 Acked Write 10 Done Done Sent false
10110 Acked Read 1 Done Done Sent false
10111 Acked Read 1 Done Done Sent false
10112 Acked Flush 0 Done Done Sent false
10113 Acked Read 5 Done Done Sent false
10114 Acked Read 7 Done Done Sent false
10115 Acked Read 1 Done Done Sent false
10116 Acked Read 2 Done Done Sent false
10117 Acked Write 9 Done Done Sent false
10118 Acked Write 2 Done Done Sent false
10119 Acked Flush 0 Done Done Sent false
10120 Acked Read 3 Done Done Sent false
10121 Acked Read 5 Done Done Sent false
10122 Acked Read 4 Done Done Sent false
10123 Acked Read 9 Done Done Sent false
10124 Acked Flush 0 Done Done Sent false
10125 Acked Read 1 Done Done Sent false
10126 Acked Read 9 Done Done Sent false
10127 Acked Write 3 Done Done Sent false
10128 Acked Read 2 Done Done Sent false
10129 Acked Flush 0 Done Done Sent false
10130 Acked Write 8 Done Done Sent false
10131 Acked Read 9 Done Done Sent false
10132 Acked Read 9 Done Done Sent false
10133 Acked Read 7 Done Done Sent false
10134 Acked Read 9 Done Done Sent false
10135 Acked Read 7 Done Done Sent false
10136 Acked Write 4 Done Done Sent false
10137 Acked Write 4 Done Done Sent false
10138 Acked Read 1 Done Done Sent false
10139 Acked Read 4 Done Done Sent false
10140 Acked Write 9 Done Done Sent false
10141 Acked Write 1 Done Done Sent false
10142 Acked Read 7 Done Done Sent false
10143 Acked Read 2 Done Done Sent false
10144 Acked Read 5 Done Done Sent false
10145 Acked Read 9 Done Done Sent false
10146 Acked Read 3 Done Done Sent false
10147 Acked Read 9 Done Done Sent false
10148 Acked Read 2 Done Done Sent false
10149 Acked Read 8 Done Done Sent false
10150 Acked Read 10 Done Done Sent false
10151 Acked Write 4 Done Done Sent false
10152 Acked Read 3 Done Done Sent false
10153 Acked Read 2 Done Done Sent false
10154 Acked Write 9 Done Done Sent false
10155 Acked Write 10 Done Done Sent false
10156 Acked Write 2 Done Done Sent false
10157 Acked Flush 0 Done Done Sent false
10158 Acked Write 7 Done Done Sent false
10159 Acked Read 7 Done Done Sent false
10160 Acked Write 4 Done Done Sent false
10161 Acked Read 2 Done Done Sent false
10162 Acked Read 7 Done Done Sent false
10163 Acked Read 5 Done Done Sent false
10164 Acked Read 2 Done Done Sent false
10165 Acked Read 7 Done Done Sent false
10166 Acked Write 3 Done Done Sent false
10167 Acked Read 2 Done Done Sent false
10168 Acked Write 9 Done Done Sent false
10169 Acked Read 2 Done Done Sent false
10170 Acked Read 9 Done Done Sent false
10171 Acked Write 8 Done Done Sent false
10172 Acked Read 5 Done Done Sent false
10173 Acked Read 5 Done Done Sent false
10174 Acked Read 3 Done Done Sent false
10175 Acked Read 1 Done Done Sent false
10176 Acked Read 2 Done Done Sent false
10177 Acked Write 5 Done Done Sent false
10178 Acked Write 9 Done Done Sent false
10179 Acked Flush 0 Done Done Sent false
10180 Acked Write 8 Done Done Sent false
10181 Acked Write 7 Done Done Sent false
10182 Acked Write 5 Done Done Sent false
10183 Acked Read 4 Done Done Sent false
10184 Acked Read 3 Done Done Sent false
10185 Acked Write 9 Done Done Sent false
10186 Acked Read 6 Done Done Sent false
10187 Acked Read 8 Done Done Sent false
10188 Acked Read 8 Done Done Sent false
10189 Acked Read 7 Done Done Sent false
10190 Acked Read 2 Done Done Sent false
10191 Acked Read 10 Done Done Sent false
10192 Acked Write 2 Done Done Sent false
10193 Acked Read 9 Done Done Sent false
10194 Acked Write 3 Done Done Sent false
10195 Acked Read 3 Done Done Sent false
10196 Acked Write 4 Done Done Sent false
10197 Acked Read 5 Done Done Sent false
10198 Acked Read 5 Done Done Sent false
10199 Acked Read 9 Done Done Sent false
10200 Acked Read 10 Done Done Sent false
10201 Acked Read 9 Done Done Sent false
10202 Acked Read 9 Done Done Sent false
10203 Acked Write 5 Done Done Sent false
10204 Acked Write 6 Done Done Sent false
10205 Acked Read 4 Done Done Sent false
10206 Acked Read 10 Done Done Sent false
10207 Acked Read 2 Done Done Sent false
10208 Acked Write 6 Done Done Sent false
10209 Acked Read 6 Done Done Sent false
10210 Acked Read 1 Done Done Sent false
10211 Acked Read 10 Done Done Sent false
10212 Acked Write 3 Done Done Sent false
10213 Acked Write 1 Done Done Sent false
10214 Acked Write 3 Done Done Sent false
10215 Acked Write 5 Done Done Sent false
10216 Acked Read 9 Done Done Sent false
10217 Acked Write 9 Done Done Sent false
10218 Acked Read 5 Done Done Sent false
10219 Acked Read 7 Done Done Sent false
10220 Acked Write 9 Done Done Sent false
10221 Acked Read 5 Done Done Sent false
10222 Acked Read 6 Done Done Sent false
10223 Acked Write 1 Done Done Sent false
10224 Acked Read 5 Done Done Sent false
10225 Acked Read 3 Done Done Sent false
10226 Acked Read 2 Done Done Sent false
10227 Acked Read 10 Done Done Sent false
10228 Acked Write 4 Done Done Sent false
10229 Acked Read 5 Done Done Sent false
10230 Acked Write 3 Done Done Sent false
10231 Acked Read 2 Done Done Sent false
10232 Acked Read 8 Done Done Sent false
10233 Acked Write 4 Done Done Sent false
10234 Acked Write 5 Done Done Sent false
10235 Acked Read 3 Done Done Sent false
10236 Acked Read 1 Done Done Sent false
10237 Acked Write 10 Done Done Sent false
10238 Acked Read 5 Done Done Sent false
10239 Acked Flush 0 Done Done Sent false
10240 Acked Write 6 Done Done Sent false
10241 Acked Flush 0 Done Done Sent false
10242 Acked Read 9 Done Done Sent false
10243 Acked Read 7 Done Done Sent false
10244 Acked Write 5 Done Done Sent false
10245 Acked Read 7 Done Done Sent false
10246 Acked Read 5 Done Done Sent false
10247 Acked Write 6 Done Done Sent false
10248 Acked Read 9 Done Done Sent false
10249 Acked Write 9 Done Done Sent false
10250 Acked Read 6 Done Done Sent false
10251 Acked Read 2 Done Done Sent false
10252 Acked Read 2 Done Done Sent false
10253 Acked Write 9 Done Done Sent false
10254 Acked Read 4 Done Done Sent false
10255 Acked Read 7 Done Done Sent false
10256 Acked Read 4 Done Done Sent false
10257 Acked Read 2 Done Done Sent false
10258 Acked Write 3 Done Done Sent false
10259 Acked Read 10 Done Done Sent false
10260 Acked Read 6 Done Done Sent false
10261 Acked Write 2 Done Done Sent false
10262 Acked Write 5 Done Done Sent false
10263 Acked Write 1 Done Done Sent false
10264 Acked Read 2 Done Done Sent false
10265 Acked Read 6 Done Done Sent false
10266 Acked Write 3 Done Done Sent false
10267 Acked Write 10 Done Done Sent false
10268 Acked Read 1 Done Done Sent false
10269 Acked Read 8 Done Done Sent false
10270 Acked Read 1 Done Done Sent false
10271 Acked Read 7 Done Done Sent false
10272 Acked Write 1 Done Done Sent false
10273 Acked Read 8 Done Done Sent false
10274 Acked Read 3 Done Done Sent false
10275 Acked Write 6 Done Done Sent false
10276 Acked Write 2 Done Done Sent false
10277 Acked Write 10 Done Done Sent false
10278 Acked Flush 0 Done Done Sent false
10279 Acked Read 2 Done Done Sent false
10280 Acked Read 6 Done Done Sent false
10281 Acked Write 5 Done Done Sent false
10282 Acked Read 5 Done Done Sent false
10283 Acked Read 7 Done Done Sent false
10284 Acked Write 8 Done Done Sent false
10285 Acked Flush 0 Done Done Sent false
10286 Acked Read 1 Done Done Sent false
10287 Acked Write 5 Done Done Sent false
10288 Acked Read 3 Done Done Sent false
10289 Acked Read 1 Done Done Sent false
10290 Acked Write 10 Done Done Sent false
10291 Acked Read 9 Done Done Sent false
10292 Acked Flush 0 Done Done Sent false
10293 Acked Write 5 Done Done Sent false
10294 Acked Read 9 Done Done Sent false
10295 Acked Write 8 Done Done Sent false
10296 Acked Read 3 Done Done Sent false
10297 Acked Read 4 Done Done Sent false
10298 Acked Read 2 Done Done Sent false
10299 Acked Read 5 Done Done Sent false
10300 Acked Write 9 Done Done Sent false
10301 Acked Read 5 Done Done Sent false
10302 Acked Read 4 Done Done Sent false
10303 Acked Read 4 Done Done Sent false
10304 Acked Write 3 Done Done Sent false
10305 Acked Flush 0 Done Done Sent false
10306 Acked Flush 0 Done Done Sent false
10307 Acked Read 6 Done Done Sent false
10308 Acked Read 5 Done Done Sent false
10309 Acked Write 9 Done Done Sent false
10310 Acked Read 4 Done Done Sent false
10311 Acked Write 3 Done Done Sent false
10312 Acked Flush 0 Done Done Sent false
10313 Acked Write 3 Done Done Sent false
10314 Acked Write 7 Done Done Sent false
10315 Acked Write 8 Done Done Sent false
10316 Acked Flush 0 Done Done Sent false
10317 Acked Write 6 Done Done Sent false
10318 Acked Write 8 Done Done Sent false
10319 Acked Write 7 Done Done Sent false
10320 Acked Read 9 Done Done Sent false
10321 Acked Write 3 Done Done Sent false
10322 Acked Write 2 Done Done Sent false
10323 Acked Read 4 Done Done Sent false
10324 Acked Read 3 Done Done Sent false
10325 Acked Write 8 Done Done Sent false
10326 Acked Flush 0 Done Done Sent false
10327 Acked Read 3 Done Done Sent false
10328 Acked Read 10 Done Done Sent false
10329 Acked Read 6 Done Done Sent false
10330 Acked Read 4 Done Done Sent false
10331 Acked Read 6 Done Done Sent false
10332 Acked Read 6 Done Done Sent false
10333 Acked Write 7 Done Done Sent false
10334 Acked Write 9 Done Done Sent false
10335 Acked Read 3 Done Done Sent false
10336 Acked Read 3 Done Done Sent false
10337 Acked Write 4 Done Done Sent false
10338 Acked Read 4 Done Done Sent false
10339 Acked Read 7 Done Done Sent false
10340 Acked Write 5 Done Done Sent false
10341 Acked Read 5 Done Done Sent false
10342 Acked Read 7 Done Done Sent false
10343 Acked Flush 0 Done Done Sent false
10344 Acked Write 5 Done Done Sent false
10345 Acked Read 2 Done Done Sent false
10346 Acked Read 7 Done Done Sent false
10347 Acked Write 1 Done Done Sent false
10348 Acked Read 8 Done Done Sent false
10349 Acked Read 7 Done Done Sent false
10350 Acked Write 10 Done Done Sent false
10351 Acked Read 1 Done Done Sent false
10352 Acked Read 1 Done Done Sent false
10353 Acked Read 1 Done Done Sent false
10354 Acked Read 7 Done Done Sent false
10355 Acked Read 9 Done Done Sent false
10356 Acked Read 7 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(10343)) Some(JobId(10343)) Some(JobId(10055))
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10356 10355 10354 10353 10352
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:56.194912197Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:57.136126618Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:58.138446826Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:59.140869189Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1242)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10056 Acked Flush 0 Done Done Sent false
10057 Acked Flush 0 Done Done Sent false
10058 Acked Read 2 Done Done Sent false
10059 Acked Read 3 Done Done Sent false
10060 Acked Read 7 Done Done Sent false
10061 Acked Write 2 Done Done Sent false
10062 Acked Write 3 Done Done Sent false
10063 Acked Write 2 Done Done Sent false
10064 Acked Read 3 Done Done Sent false
10065 Acked Write 2 Done Done Sent false
10066 Acked Read 4 Done Done Sent false
10067 Acked Write 7 Done Done Sent false
10068 Acked Read 6 Done Done Sent false
10069 Acked Read 10 Done Done Sent false
10070 Acked Write 4 Done Done Sent false
10071 Acked Read 9 Done Done Sent false
10072 Acked Read 6 Done Done Sent false
10073 Acked Read 3 Done Done Sent false
10074 Acked Read 8 Done Done Sent false
10075 Acked Read 4 Done Done Sent false
10076 Acked Write 7 Done Done Sent false
10077 Acked Write 5 Done Done Sent false
10078 Acked Read 2 Done Done Sent false
10079 Acked Write 5 Done Done Sent false
10080 Acked Flush 0 Done Done Sent false
10081 Acked Read 4 Done Done Sent false
10082 Acked Write 6 Done Done Sent false
10083 Acked Read 7 Done Done Sent false
10084 Acked Read 7 Done Done Sent false
10085 Acked Write 10 Done Done Sent false
10086 Acked Flush 0 Done Done Sent false
10087 Acked Read 6 Done Done Sent false
10088 Acked Flush 0 Done Done Sent false
10089 Acked Read 2 Done Done Sent false
10090 Acked Read 2 Done Done Sent false
10091 Acked Read 4 Done Done Sent false
10092 Acked Read 5 Done Done Sent false
10093 Acked Read 9 Done Done Sent false
10094 Acked Read 3 Done Done Sent false
10095 Acked Write 5 Done Done Sent false
10096 Acked Write 6 Done Done Sent false
10097 Acked Read 9 Done Done Sent false
10098 Acked Write 6 Done Done Sent false
10099 Acked Read 10 Done Done Sent false
10100 Acked Write 7 Done Done Sent false
10101 Acked Write 5 Done Done Sent false
10102 Acked Write 4 Done Done Sent false
10103 Acked Write 9 Done Done Sent false
10104 Acked Read 4 Done Done Sent false
10105 Acked Read 8 Done Done Sent false
10106 Acked Write 10 Done Done Sent false
10107 Acked Write 1 Done Done Sent false
10108 Acked Write 9 Done Done Sent false
10109 Acked Write 10 Done Done Sent false
10110 Acked Read 1 Done Done Sent false
10111 Acked Read 1 Done Done Sent false
10112 Acked Flush 0 Done Done Sent false
10113 Acked Read 5 Done Done Sent false
10114 Acked Read 7 Done Done Sent false
10115 Acked Read 1 Done Done Sent false
10116 Acked Read 2 Done Done Sent false
10117 Acked Write 9 Done Done Sent false
10118 Acked Write 2 Done Done Sent false
10119 Acked Flush 0 Done Done Sent false
10120 Acked Read 3 Done Done Sent false
10121 Acked Read 5 Done Done Sent false
10122 Acked Read 4 Done Done Sent false
10123 Acked Read 9 Done Done Sent false
10124 Acked Flush 0 Done Done Sent false
10125 Acked Read 1 Done Done Sent false
10126 Acked Read 9 Done Done Sent false
10127 Acked Write 3 Done Done Sent false
10128 Acked Read 2 Done Done Sent false
10129 Acked Flush 0 Done Done Sent false
10130 Acked Write 8 Done Done Sent false
10131 Acked Read 9 Done Done Sent false
10132 Acked Read 9 Done Done Sent false
10133 Acked Read 7 Done Done Sent false
10134 Acked Read 9 Done Done Sent false
10135 Acked Read 7 Done Done Sent false
10136 Acked Write 4 Done Done Sent false
10137 Acked Write 4 Done Done Sent false
10138 Acked Read 1 Done Done Sent false
10139 Acked Read 4 Done Done Sent false
10140 Acked Write 9 Done Done Sent false
10141 Acked Write 1 Done Done Sent false
10142 Acked Read 7 Done Done Sent false
10143 Acked Read 2 Done Done Sent false
10144 Acked Read 5 Done Done Sent false
10145 Acked Read 9 Done Done Sent false
10146 Acked Read 3 Done Done Sent false
10147 Acked Read 9 Done Done Sent false
10148 Acked Read 2 Done Done Sent false
10149 Acked Read 8 Done Done Sent false
10150 Acked Read 10 Done Done Sent false
10151 Acked Write 4 Done Done Sent false
10152 Acked Read 3 Done Done Sent false
10153 Acked Read 2 Done Done Sent false
10154 Acked Write 9 Done Done Sent false
10155 Acked Write 10 Done Done Sent false
10156 Acked Write 2 Done Done Sent false
10157 Acked Flush 0 Done Done Sent false
10158 Acked Write 7 Done Done Sent false
10159 Acked Read 7 Done Done Sent false
10160 Acked Write 4 Done Done Sent false
10161 Acked Read 2 Done Done Sent false
10162 Acked Read 7 Done Done Sent false
10163 Acked Read 5 Done Done Sent false
10164 Acked Read 2 Done Done Sent false
10165 Acked Read 7 Done Done Sent false
10166 Acked Write 3 Done Done Sent false
10167 Acked Read 2 Done Done Sent false
10168 Acked Write 9 Done Done Sent false
10169 Acked Read 2 Done Done Sent false
10170 Acked Read 9 Done Done Sent false
10171 Acked Write 8 Done Done Sent false
10172 Acked Read 5 Done Done Sent false
10173 Acked Read 5 Done Done Sent false
10174 Acked Read 3 Done Done Sent false
10175 Acked Read 1 Done Done Sent false
10176 Acked Read 2 Done Done Sent false
10177 Acked Write 5 Done Done Sent false
10178 Acked Write 9 Done Done Sent false
10179 Acked Flush 0 Done Done Sent false
10180 Acked Write 8 Done Done Sent false
10181 Acked Write 7 Done Done Sent false
10182 Acked Write 5 Done Done Sent false
10183 Acked Read 4 Done Done Sent false
10184 Acked Read 3 Done Done Sent false
10185 Acked Write 9 Done Done Sent false
10186 Acked Read 6 Done Done Sent false
10187 Acked Read 8 Done Done Sent false
10188 Acked Read 8 Done Done Sent false
10189 Acked Read 7 Done Done Sent false
10190 Acked Read 2 Done Done Sent false
10191 Acked Read 10 Done Done Sent false
10192 Acked Write 2 Done Done Sent false
10193 Acked Read 9 Done Done Sent false
10194 Acked Write 3 Done Done Sent false
10195 Acked Read 3 Done Done Sent false
10196 Acked Write 4 Done Done Sent false
10197 Acked Read 5 Done Done Sent false
10198 Acked Read 5 Done Done Sent false
10199 Acked Read 9 Done Done Sent false
10200 Acked Read 10 Done Done Sent false
10201 Acked Read 9 Done Done Sent false
10202 Acked Read 9 Done Done Sent false
10203 Acked Write 5 Done Done Sent false
10204 Acked Write 6 Done Done Sent false
10205 Acked Read 4 Done Done Sent false
10206 Acked Read 10 Done Done Sent false
10207 Acked Read 2 Done Done Sent false
10208 Acked Write 6 Done Done Sent false
10209 Acked Read 6 Done Done Sent false
10210 Acked Read 1 Done Done Sent false
10211 Acked Read 10 Done Done Sent false
10212 Acked Write 3 Done Done Sent false
10213 Acked Write 1 Done Done Sent false
10214 Acked Write 3 Done Done Sent false
10215 Acked Write 5 Done Done Sent false
10216 Acked Read 9 Done Done Sent false
10217 Acked Write 9 Done Done Sent false
10218 Acked Read 5 Done Done Sent false
10219 Acked Read 7 Done Done Sent false
10220 Acked Write 9 Done Done Sent false
10221 Acked Read 5 Done Done Sent false
10222 Acked Read 6 Done Done Sent false
10223 Acked Write 1 Done Done Sent false
10224 Acked Read 5 Done Done Sent false
10225 Acked Read 3 Done Done Sent false
10226 Acked Read 2 Done Done Sent false
10227 Acked Read 10 Done Done Sent false
10228 Acked Write 4 Done Done Sent false
10229 Acked Read 5 Done Done Sent false
10230 Acked Write 3 Done Done Sent false
10231 Acked Read 2 Done Done Sent false
10232 Acked Read 8 Done Done Sent false
10233 Acked Write 4 Done Done Sent false
10234 Acked Write 5 Done Done Sent false
10235 Acked Read 3 Done Done Sent false
10236 Acked Read 1 Done Done Sent false
10237 Acked Write 10 Done Done Sent false
10238 Acked Read 5 Done Done Sent false
10239 Acked Flush 0 Done Done Sent false
10240 Acked Write 6 Done Done Sent false
10241 Acked Flush 0 Done Done Sent false
10242 Acked Read 9 Done Done Sent false
10243 Acked Read 7 Done Done Sent false
10244 Acked Write 5 Done Done Sent false
10245 Acked Read 7 Done Done Sent false
10246 Acked Read 5 Done Done Sent false
10247 Acked Write 6 Done Done Sent false
10248 Acked Read 9 Done Done Sent false
10249 Acked Write 9 Done Done Sent false
10250 Acked Read 6 Done Done Sent false
10251 Acked Read 2 Done Done Sent false
10252 Acked Read 2 Done Done Sent false
10253 Acked Write 9 Done Done Sent false
10254 Acked Read 4 Done Done Sent false
10255 Acked Read 7 Done Done Sent false
10256 Acked Read 4 Done Done Sent false
10257 Acked Read 2 Done Done Sent false
10258 Acked Write 3 Done Done Sent false
10259 Acked Read 10 Done Done Sent false
10260 Acked Read 6 Done Done Sent false
10261 Acked Write 2 Done Done Sent false
10262 Acked Write 5 Done Done Sent false
10263 Acked Write 1 Done Done Sent false
10264 Acked Read 2 Done Done Sent false
10265 Acked Read 6 Done Done Sent false
10266 Acked Write 3 Done Done Sent false
10267 Acked Write 10 Done Done Sent false
10268 Acked Read 1 Done Done Sent false
10269 Acked Read 8 Done Done Sent false
10270 Acked Read 1 Done Done Sent false
10271 Acked Read 7 Done Done Sent false
10272 Acked Write 1 Done Done Sent false
10273 Acked Read 8 Done Done Sent false
10274 Acked Read 3 Done Done Sent false
10275 Acked Write 6 Done Done Sent false
10276 Acked Write 2 Done Done Sent false
10277 Acked Write 10 Done Done Sent false
10278 Acked Flush 0 Done Done Sent false
10279 Acked Read 2 Done Done Sent false
10280 Acked Read 6 Done Done Sent false
10281 Acked Write 5 Done Done Sent false
10282 Acked Read 5 Done Done Sent false
10283 Acked Read 7 Done Done Sent false
10284 Acked Write 8 Done Done Sent false
10285 Acked Flush 0 Done Done Sent false
10286 Acked Read 1 Done Done Sent false
10287 Acked Write 5 Done Done Sent false
10288 Acked Read 3 Done Done Sent false
10289 Acked Read 1 Done Done Sent false
10290 Acked Write 10 Done Done Sent false
10291 Acked Read 9 Done Done Sent false
10292 Acked Flush 0 Done Done Sent false
10293 Acked Write 5 Done Done Sent false
10294 Acked Read 9 Done Done Sent false
10295 Acked Write 8 Done Done Sent false
10296 Acked Read 3 Done Done Sent false
10297 Acked Read 4 Done Done Sent false
10298 Acked Read 2 Done Done Sent false
10299 Acked Read 5 Done Done Sent false
10300 Acked Write 9 Done Done Sent false
10301 Acked Read 5 Done Done Sent false
10302 Acked Read 4 Done Done Sent false
10303 Acked Read 4 Done Done Sent false
10304 Acked Write 3 Done Done Sent false
10305 Acked Flush 0 Done Done Sent false
10306 Acked Flush 0 Done Done Sent false
10307 Acked Read 6 Done Done Sent false
10308 Acked Read 5 Done Done Sent false
10309 Acked Write 9 Done Done Sent false
10310 Acked Read 4 Done Done Sent false
10311 Acked Write 3 Done Done Sent false
10312 Acked Flush 0 Done Done Sent false
10313 Acked Write 3 Done Done Sent false
10314 Acked Write 7 Done Done Sent false
10315 Acked Write 8 Done Done Sent false
10316 Acked Flush 0 Done Done Sent false
10317 Acked Write 6 Done Done Sent false
10318 Acked Write 8 Done Done Sent false
10319 Acked Write 7 Done Done Sent false
10320 Acked Read 9 Done Done Sent false
10321 Acked Write 3 Done Done Sent false
10322 Acked Write 2 Done Done Sent false
10323 Acked Read 4 Done Done Sent false
10324 Acked Read 3 Done Done Sent false
10325 Acked Write 8 Done Done Sent false
10326 Acked Flush 0 Done Done Sent false
10327 Acked Read 3 Done Done Sent false
10328 Acked Read 10 Done Done Sent false
10329 Acked Read 6 Done Done Sent false
10330 Acked Read 4 Done Done Sent false
10331 Acked Read 6 Done Done Sent false
10332 Acked Read 6 Done Done Sent false
10333 Acked Write 7 Done Done Sent false
10334 Acked Write 9 Done Done Sent false
10335 Acked Read 3 Done Done Sent false
10336 Acked Read 3 Done Done Sent false
10337 Acked Write 4 Done Done Sent false
10338 Acked Read 4 Done Done Sent false
10339 Acked Read 7 Done Done Sent false
10340 Acked Write 5 Done Done Sent false
10341 Acked Read 5 Done Done Sent false
10342 Acked Read 7 Done Done Sent false
10343 Acked Flush 0 Done Done Sent false
10344 Acked Write 5 Done Done Sent false
10345 Acked Read 2 Done Done Sent false
10346 Acked Read 7 Done Done Sent false
10347 Acked Write 1 Done Done Sent false
10348 Acked Read 8 Done Done Sent false
10349 Acked Read 7 Done Done Sent false
10350 Acked Write 10 Done Done Sent false
10351 Acked Read 1 Done Done Sent false
10352 Acked Read 1 Done Done Sent false
10353 Acked Read 1 Done Done Sent false
10354 Acked Read 7 Done Done Sent false
10355 Acked Read 9 Done Done Sent false
10356 Acked Read 7 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: Some(JobId(10343)) Some(JobId(10343)) Some(JobId(10055))
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10356 10355 10354 10353 10352
CLIENT: Up:0 ds:301 act:2
{"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:09:01.143795174Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.439094824Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39127)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.43940207Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.440306703Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.440799485Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.4408212Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"send last flush ID to this DS: Some(JobId(10055))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.440838216Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.440854672Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] client re-new 301 jobs since flush Some(JobId(10055))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.440871368Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
{"msg":"[2] Replayed 301 jobs since flush: Some(JobId(10055))","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:01.441068381Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10357 10356 10355 10354 10353
Upstairs last five completed: 10357 10356 10355 10354 10353
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Test replay has completed
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10357 10356 10355 10354 10353
Upstairs last five completed: 10357 10356 10355 10354 10353
CLIENT: Up:0 ds:0 act:3
CLIENT: All jobs finished, all DS active.
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:09:03.701048551Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211}
Replays:30 time: 8:00
Do final verify
Feb 28 06:09:03.772 INFO accepted connection, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
Feb 28 06:09:03.772 INFO request completed, latency_us: 214, response_code: 200, uri: /regioninfo, method: GET, req_id: 6a42cb1a-9855-46a5-a4e7-b99efe138b9d, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.773442741Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"task":"crutest"}
Feb 28 06:09:03.773 INFO request completed, latency_us: 89, response_code: 200, uri: /regioncount, method: GET, req_id: 690b5449-d7fd-4ab5-a79b-df6800be512f, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.773920557Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"task":"crutest"}
Feb 28 06:09:03.774 INFO request completed, latency_us: 107, response_code: 200, uri: /port/cid/0, method: GET, req_id: 99792fae-7a1c-4ed5-b688-96c27dabb11f, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
Feb 28 06:09:03.774 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: f311a2bb-f438-4815-925c-fd0e72454bc9, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
Feb 28 06:09:03.775 INFO request completed, latency_us: 97, response_code: 200, uri: /port/cid/2, method: GET, req_id: 091e50b8-92f1-489b-a4c5-9c9bcaac40f4, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.775525102Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.775706129Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0c103f9ae2997264530dbd443fadbdf83878dac4\",\n git_commit_timestamp: \"2025-02-28T05:45:10.000000000Z\",\n git_branch: \"renovate/uuid-1.x-lockfile\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.77574276Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.775760316Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.775775092Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
{"msg":"Crucible e43d6e3b-f875-4bf6-93ed-2a4796303496 has session id: c6ccf12c-5dc8-4d51-966e-c87d79da1b7f","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.776228803Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Upstairs opts: Upstairs UUID: e43d6e3b-f875-4bf6-93ed-2a4796303496, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.776276822Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Crucible stats registered with UUID: e43d6e3b-f875-4bf6-93ed-2a4796303496","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.776301576Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:09:03.776322421Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777128158Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"task":"crutest"}
{"msg":"The guest has requested activation with gen:3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.77716181Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
{"msg":"e43d6e3b-f875-4bf6-93ed-2a4796303496 active request set","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777199651Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777252728Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777292489Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777320082Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777347575Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777364901Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777379348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777396633Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777420558Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777446002Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58207)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777473875Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36863)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777499829Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52451)","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.777522743Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"io task","client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778190523Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778219056Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778234353Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778794198Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778888916Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.778926057Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779652053Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779677097Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[0]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779734743Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779757068Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779772114Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[1]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779786041Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779799208Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779812714Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[2]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779825921Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779839008Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779852585Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779864942Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-02-28T06:09:03.779877789Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Next flush: 962","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779893435Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779906052Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779919019Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779930596Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"0","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779945633Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"1","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.77995885Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"client":"2","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779971736Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.779983653Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"e43d6e3b-f875-4bf6-93ed-2a4796303496 is now active with session: c6ccf12c-5dc8-4d51-966e-c87d79da1b7f","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.77999625Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.780007778Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
{"msg":"The guest has finished waiting for activation with:3","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:09:03.780019475Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Loading write count information from file "/tmp/test_replay_verify.log"
Read and Verify all blocks (0..5000 range:false)
Wrote out file "/tmp/test_replay_verify.log"
Verify test completed
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Read 256 Done Done Done false
1001 Acked Read 256 Done Done Done false
1002 Acked Read 256 Done Done Done false
1003 Acked Read 256 Done Done Done false
1004 Acked Read 256 Done Done Done false
1005 Acked Read 256 Done Done Done false
1006 Acked Read 256 Done Done Done false
1007 Acked Read 256 Done Done Done false
1008 Acked Read 256 Done Done Done false
1009 Acked Read 256 Done Done Done false
1010 Acked Read 256 Done Done Done false
1011 Acked Read 256 Done Done Done false
1012 Acked Read 256 Done Done Done false
1013 Acked Read 256 Done Done Done false
1014 Acked Read 256 Done Done Done false
1015 Acked Read 256 Done Done Done false
1016 Acked Read 256 Done Done Done false
1017 Acked Read 136 Done Done Done false
1018 Acked Read 256 Done Done Done false
1019 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: None None None
Downstairs last five completed:
Upstairs last five completed: 1019 1018 1017 1016 1015
CLIENT: Up:0 ds:20 act:3
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1020 1019 1018 1017 1016
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:09:08.183034193Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247}
Feb 28 06:09:08.247 INFO accepted connection, remote_addr: 127.0.0.1:49565, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Shutdown
Shutdown
Feb 28 06:09:08.247 INFO request completed, latency_us: 165, response_code: 204, uriShut it down
:[8810] Disable keep_running
/shutdown, method[8810] Got stop action so:false kr:false
: POST, req_id: fdaa15a1-cb0e-42b4-b59d-a87000a72de1, remote_addr: 127.0.0.1:49565, local_addr: 127.0.0.1:9998
[8830] Disable keep_running
[8830] Got stop action so:false kr:false
[8820] Disable keep_running
[8820] Got stop action so:false kr:false
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
[8820][1] reports Exit
[8830][2] reports Exit
February 28, 2025 at 06:09:13 AM UTC Test ends with 0