starting January 27, 2025 at 07:36:14 PM UTC
Creating 3 downstairs regions
Creating region directory at: /var/tmp/dsc/region
Creating dsc directory at: /tmp/dsc
Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.33400872
Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.024296075
Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.021968076
Created 3 regions
Starting 3 downstairs
Using existing output directory "/tmp/dsc"
Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
start ds: 8810
start ds: 8820
start ds: 8830
start access at:127.0.0.1:9998
Starting downstairs at port 8820
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
Make output file at "/tmp/dsc/downstairs-8820.txt"
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8820][1] initial start wait reports Starting
[8810][0] initial start wait reports Starting
[8830][2] initial start wait reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1190)
[8810][0] initial start wait reports Running
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1191)
[8820][1] initial start wait reports Running
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
[8830][2] initial start wait reports Running
All downstairs are running
Jan 27 19:36:15.124 INFO listening, local_addr: 127.0.0.1:9998
Control access at:127.0.0.1:9998
Running initial fill
Jan 27 19:36:20.227 INFO accepted connection, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
Jan 27 19:36:20.229 INFO request completed, latency_us: 428, response_code: 200, uri: /regioninfo, method: GET, req_id: 905a059f-6b56-4611-bb8e-058218864013, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
Jan 27 19:36:20.231 INFO request completed, latency_us: 129, response_code: 200, uri: /regioncount, method: GET, req_id: cf8808cd-f9ed-41c6-ae34-8179e8a564d0, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.231492577Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"task":"crutest"}
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.232194855Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"task":"crutest"}
Jan 27 19:36:20.232 INFO request completed, latency_us: 158, response_code: 200, uri: /port/cid/0, method: GET, req_id: 2dea7c18-c4d0-4dd2-be59-2b23e63d206a, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
Jan 27 19:36:20.233 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/1, method: GET, req_id: 69a3e2a4-a67a-4085-b330-8336d3b3e3bf, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
Jan 27 19:36:20.233 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/2, method: GET, req_id: c16841da-f17a-4844-86dc-010a18e1e8ed, remote_addr: 127.0.0.1:54371, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.234223308Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.235503209Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.235676679Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.235702886Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.235721333Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
{"msg":"Crucible 8ef76689-d9b6-498b-aa3a-f8fde761b2ed has session id: 975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.237255874Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Upstairs opts: Upstairs UUID: 8ef76689-d9b6-498b-aa3a-f8fde761b2ed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.237331106Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Crucible stats registered with UUID: 8ef76689-d9b6-498b-aa3a-f8fde761b2ed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.237349063Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:20.237426685Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.239537388Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"task":"crutest"}
{"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.239573004Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
{"msg":"8ef76689-d9b6-498b-aa3a-f8fde761b2ed active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.239593322Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.23960635Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.23995304Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.239977657Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.239993175Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240006353Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240020302Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.24003755Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240051518Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240065407Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64905)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240088904Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37213)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240110931Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36100)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.240280152Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"io task","client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.242153063Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.24218201Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.242197108Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.243369061Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.243397838Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.243413466Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244542314Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244567871Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244861827Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244885224Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244901132Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244914691Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244927909Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244941368Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244954306Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244967355Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244981033Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.244993582Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.24500631Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245018799Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245031377Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"":"downstairs","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245044206Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245055804Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"0","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245070513Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"1","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245083341Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"client":"2","":"downstairs","session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245096059Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245107878Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"8ef76689-d9b6-498b-aa3a-f8fde761b2ed is now active with session: 975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245120307Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245131875Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196,"session_id":"975f6c7c-1b94-4fd0-82f5-bf7e8ea9bbe2"}
{"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:20.245143694Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
Activate successful
Wait for a query_work_queue command to finish before sending IO
Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Fill test
Read and Verify all blocks (0..5000 range:false)
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Read 256 Done Done Done false
1022 Acked Read 256 Done Done Done false
1023 Acked Read 256 Done Done Done false
1024 Acked Read 256 Done Done Done false
1025 Acked Read 256 Done Done Done false
1026 Acked Read 256 Done Done Done false
1027 Acked Read 256 Done Done Done false
1028 Acked Read 256 Done Done Done false
1029 Acked Read 256 Done Done Done false
1030 Acked Read 256 Done Done Done false
1031 Acked Read 256 Done Done Done false
1032 Acked Read 256 Done Done Done false
1033 Acked Read 256 Done Done Done false
1034 Acked Read 256 Done Done Done false
1035 Acked Read 256 Done Done Done false
1036 Acked Read 256 Done Done Done false
1037 Acked Read 256 Done Done Done false
1038 Acked Read 136 Done Done Done false
1039 Acked Read 256 Done Done Done false
1040 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1040 1039 1038 1037 1036
CLIENT: Up:0 ds:20 act:3
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1041 1040 1039 1038 1037
Upstairs last five completed: 1041 1040 1039 1038 1037
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:24.812184791Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1196}
Replay loop starts now January 27, 2025 at 07:36:24 PM UTC
Jan 27 19:36:24.882 INFO accepted connection, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
Jan 27 19:36:24.882 INFO request completed, latency_us: 167, response_code: 200, uri: /regioninfo, method: GET, req_id: d0fcb2d0-25b5-4862-a408-d994c080d4c2, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.88366416Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"task":"crutest"}
Jan 27 19:36:24.883 INFO request completed, latency_us: 105, response_code: 200, uri: /regioncount, method: GET, req_id: 4531ea96-5880-4a6b-a483-6ed5b74bc70b, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.884258791Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"task":"crutest"}
Jan 27 19:36:24.884 INFO request completed, latency_us: 127, response_code: 200, uri: /port/cid/0, method: GET, req_id: 08d8e7a9-0a5a-46d1-8e7c-549b34a7beac, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
Jan 27 19:36:24.885 INFO request completed, latency_us: 108, response_code: 200, uri: /port/cid/1, method: GET, req_id: 9f3c4a17-24db-4044-bcf4-10bad5bbc127, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
Jan 27 19:36:24.885 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/2, method: GET, req_id: 301193ca-dc64-4e5a-9ea9-a720c17dfc6f, remote_addr: 127.0.0.1:51525, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.885888691Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.88606612Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886100866Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886114965Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886129003Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
{"msg":"Crucible 0c5c0539-a805-490c-b9bd-b023a60aa5a2 has session id: edf4bf43-2393-40de-98dd-2a3c00d4e1dd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886636664Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Upstairs opts: Upstairs UUID: 0c5c0539-a805-490c-b9bd-b023a60aa5a2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886673309Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Crucible stats registered with UUID: 0c5c0539-a805-490c-b9bd-b023a60aa5a2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.886692937Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:24.886705646Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887573864Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"task":"crutest"}
{"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887607471Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
{"msg":"0c5c0539-a805-490c-b9bd-b023a60aa5a2 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887666913Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.88769357Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887723437Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887748174Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887776001Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887803868Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887832354Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887856891Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887880568Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887896157Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38964)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.887918944Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59571)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.888197962Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51845)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.888243546Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.888582447Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.888619242Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.888646369Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.889249559Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.889278876Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.889300493Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.889995382Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890030468Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890063934Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890079182Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.89009316Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890106429Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890119997Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890137045Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890150594Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890163582Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890177031Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890194798Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890217776Z"Activate successful
,"Wait for a query_work_queue command to finish before sending IO
hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890246912Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.89026371Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890276839Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890288368Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890301126Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890313605Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890326373Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890338232Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"0c5c0539-a805-490c-b9bd-b023a60aa5a2 is now active with session: edf4bf43-2393-40de-98dd-2a3c00d4e1dd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.89035085Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890365878Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.890378097Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Loading write count information from file "/tmp/test_replay_verify.log"
Jan 27 19:36:24.924 INFO accepted connection, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:36:24.925 INFO request completed, latency_us: 177, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7ccd7535-91bc-4a3e-aedf-0a62407e1b18, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8810] Got stop action so:false kr:false
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:24.926121286Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:24.926162881Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:24.926194938Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:24.926309915Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:36:24.926 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4bec99ae-3fe3-4f01-87b1-80046ce199e3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:36:28.929 INFO request completed, latency_us: 239, response_code: 200, uri: /state/cid/0, method: GET, req_id: df0cefa2-6edb-477f-ad19-fabd9daf85e7, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 2729 len 32768
002/300 Flush
003/300 Read block 4517 len 28672
004/300 Read block 486 len 24576
005/300 Write block 2026 len 36864 data: 2 2 2 2 2 2 2 2 2
006/300 Read block 3891 len 32768
007/300 Write block 1543 len 12288 data: 2 2 2
008/300 Write block 820 len 4096 data: 2
009/300 Read block 4454 len 24576
010/300 Write block 4680 len 8192 data: 2 2
011/300 Write block 4250 len 32768 data: 2 2 2 2 2 2 2 2
012/300 Read block 1184 len 20480
013/300 Read block 4074 len 16384
014/300 Write block 393 len 20480 data: 2 2 2 2 2
015/300 Write block 1537 len 36864 data: 2 2 2 2 2 2 3 3 3
016/300 Read block 2224 len 16384
017/300 Write block 373 len 8192 data: 2 2
018/300 Read block 1675 len 36864
019/300 Write block 1189 len 32768 data: 2 2 2 2 2 2 2 2
020/300 Read block 81 len 40960
021/300 Read block 4473 len 8192
022/300 Write block 4118 len 20480 data: 2 2 2 2 2
023/300 Write block 3569 len 28672 data: 2 2 2 2 2 2 2
024/300 Read block 783 len 8192
025/300 Write block 1353 len 4096 data: 2
026/300 Write block 4017 len 28672 data: 2 2 2 2 2 2 2
027/300 Write block 2288 len 20480 data: 2 2 2 2 2
028/300 Write block 3285 len 12288 data: 2 2 2
029/300 Flush
030/300 Write block 2181 len 16384 data: 2 2 2 2
031/300 Read block 4067 len 16384
032/300 Flush
033/300 Read block 1835 len 32768
034/300 Write block 4771 len 4096 data: 2
035/300 Write block 4995 len 20480 data: 2 2 2 2 2
036/300 Write block 1207 len 36864 data: 2 2 2 2 2 2 2 2 2
037/300 Write block 3364 len 36864 data: 2 2 2 2 2 2 2 2 2
038/300 Flush
039/300 Flush
040/300 Write block 91 len 8192 data: 2 2
041/300 Read block 357 len 24576
042/300 Read block 1712 len 4096
043/300 Write block 3783 len 32768 data: 2 2 2 2 2 2 2 2
044/300 Read block 35 len 16384
045/300 Write block 1370 len 32768 data: 2 2 2 2 2 2 2 2
046/300 Flush
047/300 Read block 2118 len 16384
048/300 Flush
049/300 Write block 4391 len 24576 data: 2 2 2 2 2 2
050/300 Read block 3503 len 20480
051/300 Read block 3711 len 4096
052/300 Read block 1508 len 4096
053/300 Read block 1868 len 36864
054/300 Read block 686 len 20480
055/300 Read block 1767 len 12288
056/300 Read block 3597 len 40960
057/300 Read block 3333 len 40960
058/300 Read block 1658 len 24576
059/300 Flush
060/300 Write block 4153 len 12288 data: 2 2 2
061/300 Write block 1456 len 20480 data: 2 2 2 2 2
062/300 Write block 3904 len 40960 data: 2 2 2 2 2 2 2 2 2 2
063/300 Write block 4331 len 32768 data: 2 2 2 2 2 2 2 2
064/300 Write block 4730 len 28672 data: 2 2 2 2 2 2 2
065/300 Flush
066/300 Read block 2957 len 20480
067/300 Read block 3315 len 8192
068/300 Flush
069/300 Read block 1262 len 4096
070/300 Write block 2068 len 4096 data: 2
071/300 Write block 3514 len 24576 data: 2 2 2 2 2 2
072/300 Write block 602 len 16384 data: 2 2 2 2
073/300 Write block 207 len 36864 data: 2 2 2 2 2 2 2 2 2
074/300 Read block 323 len 36864
075/300 Write block 3158 len 12288 data: 2 2 2
076/300 Write block 3426 len 32768 data: 2 2 2 2 2 2 2 2
077/300 Read block 4011 len 4096
078/300 Read block 63 len 12288
079/300 Read block 1225 len 16384
080/300 Flush
081/300 Read block 4267 len 28672
082/300 Read block 4834 len 36864
083/300 Write block 4260 len 28672 data: 2 2 2 2 2 2 2
084/300 Read block 2493 len 8192
085/300 Write block 1811 len 8192 data: 2 2
086/300 Read block 3864 len 36864
087/300 Read block 3224 len 24576
088/300 Write block 1794 len 16384 data: 2 2 2 2
089/300 Write block 4718 len 36864 data: 2 2 2 2 2 2 2 2 2
090/300 Write block 630 len 4096 data: 2
091/300 Read block 4134 len 32768
092/300 Write block 4238 len 24576 data: 2 2 2 2 2 2
093/300 Write block 2767 len 36864 data: 2 2 2 2 2 2 2 2 2
094/300 Read block 2352 len 36864
095/300 Read block 976 len 28672
096/300 Write block 3230 len 24576 data: 2 2 2 2 2 2
097/300 Flush
098/300 Read block 2959 len 16384
099/300 Read block 4219 len 8192
100/300 Write block 3773 len 24576 data: 2 2 2 2 2 2
101/300 Write block 1924 len 12288 data: 2 2 2
102/300 Read block 620 len 16384
103/300 Read block 3443 len 20480
104/300 Flush
105/300 Read block 2181 len 20480
106/300 Write block 4896 len 16384 data: 2 2 2 2
107/300 Write block 3757 len 24576 data: 2 2 2 2 2 2
108/300 Write block 1580 len 28672 data: 2 2 2 2 2 2 2
109/300 Read block 143 len 20480
110/300 Read block 1419 len 8192
111/300 Read block 2048 len 24576
112/300 Write block 1820 len 12288 data: 2 2 2
113/300 Read block 3256 len 16384
114/300 Read block 1382 len 16384
115/300 Flush
116/300 Read block 4360 len 32768
117/300 Write block 4799 len 28672 data: 2 2 2 2 2 2 2
118/300 Write block 1848 len 32768 data: 2 2 2 2 2 2 2 2
119/300 Read block 626 len 20480
120/300 Write block 2283 len 32768 data: 2 2 2 2 2 3 3 3
121/300 Read block 1893 len 16384
122/300 Write block 1102 len 24576 data: 2 2 2 2 2 2
123/300 Write block 3826 len 20480 data: 2 2 2 2 2
124/300 Write block 1575 len 8192 data: 2 2
125/300 Read block 2825 len 4096
126/300 Write block 4922 len 32768 data: 2 2 2 2 2 2 2 2
127/300 Write block 738 len 24576 data: 2 2 2 2 2 2
128/300 Write block 1233 len 36864 data: 2 2 2 2 2 2 2 2 2
129/300 Write block 4942 len 24576 data: 2 2 2 2 2 2
130/300 Write block 1551 len 4096 data: 2
131/300 Write block 1290 len 8192 data: 2 2
132/300 Read block 4085 len 24576
133/300 Write block 720 len 28672 data: 2 2 2 2 2 2 2
134/300 Write block 1943 len 24576 data: 2 2 2 2 2 2
135/300 Read block 3428 len 40960
136/300 Write block 3000 len 36864 data: 2 2 2 2 2 2 2 2 2
137/300 Write block 4560 len 40960 data: 2 2 2 2 2 2 2 2 2 2
138/300 Write block 2658 len 16384 data: 2 2 2 2
139/300 Read block 3276 len 32768
140/300 Write block 4107 len 28672 data: 2 2 2 2 2 2 2
141/300 Write block 274 len 36864 data: 2 2 2 2 2 2 2 2 2
142/300 Flush
143/300 Read block 2900 len 8192
144/300 Write block 4170 len 36864 data: 2 2 2 2 2 2 2 2 2
145/300 Write block 2638 len 28672 data: 2 2 2 2 2 2 2
146/300 Read block 3641 len 8192
147/300 Read block 2198 len 20480
148/300 Flush
149/300 Write block 4173 len 4096 data: 3
150/300 Read block 4626 len 20480
151/300 Read block 2587 len 40960
152/300 Flush
153/300 Write block 4109 len 20480 data: 3 3 3 3 3
154/300 Flush
155/300 Read block 566 len 8192
156/300 Write block 4196 len 8192 data: 2 2
157/300 Read block 785 len 16384
158/300 Read block 2783 len 40960
159/300 Write block 2042 len 28672 data: 2 2 2 2 2 2 2
160/300 Write block 3215 len 8192 data: 2 2
161/300 Write block 1018 len 12288 data: 2 2 2
162/300 Write block 43 len 32768 data: 2 2 2 2 2 2 2 2
163/300 Read block 3002 len 28672
164/300 Write block 656 len 16384 data: 2 2 2 2
165/300 Read block 2679 len 4096
166/300 Read block 256 len 36864
167/300 Write block 3904 len 4096 data: 3
168/300 Write block 3129 len 32768 data: 2 2 2 2 2 2 2 2
169/300 Write block 2339 len 40960 data: 2 2 2 2 2 2 2 2 2 2
170/300 Read block 2873 len 28672
171/300 Read block 1526 len 16384
172/300 Write block 1581 len 24576 data: 3 3 3 3 3 3
173/300 Read block 4431 len 24576
174/300 Read block 2525 len 8192
175/300 Read block 470 len 24576
176/300 Read block 4104 len 32768
177/300 Read block 385 len 36864
178/300 Read block 97 len 36864
179/300 Write block 505 len 8192 data: 2 2
180/300 Write block 4714 len 20480 data: 2 2 2 2 3
181/300 Read block 3608 len 32768
182/300 Read block 1694 len 28672
183/300 Read block 3868 len 28672
184/300 Write block 498 len 8192 data: 2 2
185/300 Write block 3176 len 16384 data: 2 2 2 2
186/300 Flush
187/300 Write block 3896 len 16384 data: 2 2 2 2
188/300 Write block 904 len 4096 data: 2
189/300 Write block 4375 len 36864 data: 2 2 2 2 2 2 2 2 2
190/300 Write block 3037 len 20480 data: 2 2 2 2 2
191/300 Write block 1571 len 32768 data: 2 2 2 2 3 3 2 2
192/300 Read block 1344 len 4096
193/300 Flush
194/300 Read block 4626 len 32768
195/300 Read block 814 len 8192
196/300 Write block 4988 len 8192 data: 2 2
197/300 Read block 969 len 20480
198/300 Flush
199/300 Flush
200/300 Read block 3899 len 16384
201/300 Read block 4361 len 8192
202/300 Write block 2248 len 16384 data: 2 2 2 2
203/300 Read block 2884 len 40960
204/300 Read block 1486 len 4096
205/300 Write block 1375 len 40960 data: 3 3 3 2 2 2 2 2 2 2
206/300 Write block 3123 len 20480 data: 2 2 2 2 2
207/300 Write block 3161 len 12288 data: 2 2 2
208/300 Read block 1772 len 40960
209/300 Write block 3218 len 32768 data: 2 2 2 2 2 2 2 2
210/300 Read block 4960 len 16384
211/300 Write block 2239 len 40960 data: 2 2 2 2 2 2 2 2 2 3
212/300 Read block 3621 len 12288
213/300 Write block 2470 len 4096 data: 2
214/300 Read block 1372 len 28672
215/300 Write block 800 len 4096 data: 2
216/300 Read block 2101 len 8192
217/300 Write block 1029 len 8192 data: 2 2
218/300 Read block 109 len 16384
219/300 Read block 3343 len 32768
220/300 Write block 4693 len 36864 data: 2 2 2 2 2 2 2 2 2
221/300 Read block 4623 len 36864
222/300 Write block 131 len 4096 data: 2
223/300 Read block 2011 len 8192
224/300 Read block 1051 len 32768
225/300 Flush
226/300 Write block 4887 len 8192 data: 2 2
227/300 Read block 2260 len 36864
228/300 Write block 713 len 20480 data: 2 2 2 2 2
229/300 Read block 1172 len 24576
230/300 Read block 3350 len 36864
231/300 Read block 1834 len 24576
232/300 Flush
233/300 Write block 1129 len 28672 data: 2 2 2 2 2 2 2
234/300 Write block 575 len 40960 data: 2 2 2 2 2 2 2 2 2 2
235/300 Write block 2848 len 24576 data: 2 2 2 2 2 2
236/300 Write block 4277 len 8192 data: 2 2
237/300 Write block 1537 len 32768 data: 3 3 3 3 3 3 4 4
238/300 Flush
239/300 Read block 4565 len 12288
240/300 Write block 1249 len 4096 data: 2
241/300 Read block 3901 len 28672
242/300 Write block 1810 len 8192 data: 2 3
243/300 Write block 4325 len 28672 data: 2 2 2 2 2 2 3
244/300 Flush
245/300 Read block 4194 len 24576
246/300 Write block 2794 len 36864 data: 2 2 2 2 2 2 2 2 2
247/300 Read block 2286 len 20480
248/300 Write block 1952 len 40960 data: 2 2 2 2 2 2 2 2 2 2
249/300 Flush
250/300 Read block 2422 len 32768
251/300 Write block 3122 len 16384 data: 2 3 3 3
252/300 Write block 1417 len 24576 data: 2 2 2 2 2 2
253/300 Write block 4498 len 12288 data: 2 2 2
254/300 Write block 2473 len 20480 data: 2 2 2 2 2
255/300 Read block 4071 len 4096
256/300 Read block 953 len 36864
257/300 Read block 3474 len 28672
258/300 Read block 3454 len 32768
259/300 Read block 4991 len 8192
260/300 Write block 4958 len 20480 data: 2 2 2 2 2
261/300 Read block 1947 len 4096
262/300 Read block 2321 len 8192
263/300 Flush
264/300 Write block 2576 len 24576 data: 2 2 2 2 2 2
265/300 Read block 2474 len 36864
266/300 Write block 3565 len 4096 data: 2
267/300 Write block 1534 len 8192 data: 2 2
268/300 Write block 4134 len 40960 data: 2 2 2 2 2 2 2 2 2 2
269/300 Read block 4547 len 24576
270/300 Write block 3403 len 28672 data: 2 2 2 2 2 2 2
271/300 Read block 1627 len 36864
272/300 Read block 2818 len 28672
273/300 Write block 1410 len 36864 data: 2 2 2 2 2 2 2 3 3
274/300 Read block 1301 len 32768
275/300 Write block 4728 len 24576 data: 2 2 3 3 3 3
276/300 Read block 4760 len 28672
277/300 Read block 1070 len 4096
278/300 Read block 2709 len 16384
279/300 Read block 1590 len 12288
280/300 Write block 1049 len 12288 data: 2 2 2
281/300 Read block 4667 len 8192
282/300 Read block 3053 len 8192
283/300 Read block 579 len 24576
284/300 Write block 4790 len 28672 data: 2 2 2 2 2 2 2
285/300 Write block 1398 len 36864 data: 2 2 2 2 2 2 2 2 2
286/300 Read block 802 len 20480
287/300 Read block 2104 len 8192
288/300 Write block 4954 len 8192 data: 2 2
289/300 Read block 2288 len 36864
290/300 Read block 2166 len 20480
291/300 Write block 186 len 40960 data: 2 2 2 2 2 2 2 2 2 2
292/300 Write block 1641 len 24576 data: 2 2 2 2 2 2
293/300 Write block 2673 len 36864 data: 2 2 2 2 2 2 2 2 2
294/300 Read block 4729 len 8192
295/300 Read block 1371 len 12288
296/300 Write block 4492 len 20480 data: 2 2 2 2 2
297/300 Write block 856 len 28672 data: 2 2 2 2 2 2 2
298/300 Write block 2633 len 40960 data: 2 2 2 2 2 3 3 3 3 3
299/300 Write block 1391 len 20480 data: 2 2 2 2 2
300/300 Write block 1342 len 40960 data: 2 2 2 2 2 2 2 2 2 2
301/300 Write block 1785 len 28672 data: 2 2 2 2 2 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:36:29.154 INFO request completed, latency_us: 234, response_code: 204, uri: /start/cid/0, method: POST, req_id: 13fd5f6b-12bc-49c9-b874-0587d3d1a304, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Read 8 Sent Done Done false
1001 Acked Flush 0 Sent Done Done false
1002 Acked Read 7 Sent Done Done false
1003 Acked Read 6 Sent Done Done false
1004 Acked Write 9 Sent Done Done false
1005 Acked Read 8 Sent Done Done false
1006 Acked Write 3 Sent Done Done false
1007 Acked Write 1 Sent Done Done false
1008 Acked Read 6 Sent Done Done false
1009 Acked Write 2 Sent Done Done false
1010 Acked Write 8 Sent Done Done false
1011 Acked Read 5 Sent Done Done false
1012 Acked Read 4 Sent Done Done false
1013 Acked Write 5 Sent Done Done false
1014 Acked Write 9 Sent Done Done false
1015 Acked Read 4 Sent Done Done false
1016 Acked Write 2 Sent Done Done false
1017 Acked Read 9 Sent Done Done false
1018 Acked Write 8 Sent Done Done false
1019 Acked Read 10 Sent Done Done false
1020 Acked Read 2 Sent Done Done false
1021 Acked Write 5 Sent Done Done false
1022 Acked Write 7 Sent Done Done false
1023 Acked Read 2 Sent Done Done false
1024 Acked Write 1 Sent Done Done false
1025 Acked Write 7 Sent Done Done false
1026 Acked Write 5 Sent Done Done false
1027 Acked Write 3 Sent Done Done false
1028 Acked Flush 0 Sent Done Done false
1029 Acked Write 4 Sent Done Done false
1030 Acked Read 4 Sent Done Done false
1031 Acked Flush 0 Sent Done Done false
1032 Acked Read 8 Sent Done Done false
1033 Acked Write 1 Sent Done Done false
1034 Acked Write 5 Sent Done Done false
1035 Acked Write 9 Sent Done Done false
1036 Acked Write 9 Sent Done Done false
1037 Acked Flush 0 Sent Done Done false
1038 Acked Flush 0 Sent Done Done false
1039 Acked Write 2 Sent Done Done false
1040 Acked Read 6 Sent Done Done false
1041 Acked Read 1 Sent Done Done false
1042 Acked Write 8 Sent Done Done false
1043 Acked Read 4 Sent Done Done false
1044 Acked Write 8 Sent Done Done false
1045 Acked Flush 0 Sent Done Done false
1046 Acked Read 4 Sent Done Done false
1047 Acked Flush 0 Sent Done Done false
1048 Acked Write 6 Sent Done Done false
1049 Acked Read 5 Sent Done Done false
1050 Acked Read 1 Sent Done Done false
1051 Acked Read 1 Sent Done Done false
1052 Acked Read 9 Sent Done Done false
1053 Acked Read 5 Sent Done Done false
1054 Acked Read 3 Sent Done Done false
1055 Acked Read 10 Sent Done Done false
1056 Acked Read 10 Sent Done Done false
1057 Acked Read 6 Sent Done Done false
1058 Acked Flush 0 Sent Done Done false
1059 Acked Write 3 Sent Done Done false
1060 Acked Write 5 Sent Done Done false
1061 Acked Write 10 Sent Done Done false
1062 Acked Write 8 Sent Done Done false
1063 Acked Write 7 Sent Done Done false
1064 Acked Flush 0 Sent Done Done false
1065 Acked Read 5 Sent Done Done false
1066 Acked Read 2 Sent Done Done false
1067 Acked Flush 0 Sent Done Done false
1068 Acked Read 1 Sent Done Done false
1069 Acked Write 1 Sent Done Done false
1070 Acked Write 6 Sent Done Done false
1071 Acked Write 4 Sent Done Done false
1072 Acked Write 9 Sent Done Done false
1073 Acked Read 9 Sent Done Done false
1074 Acked Write 3 Sent Done Done false
1075 Acked Write 8 Sent Done Done false
1076 Acked Read 1 Sent Done Done false
1077 Acked Read 3 Sent Done Done false
1078 Acked Read 4 Sent Done Done false
1079 Acked Flush 0 Sent Done Done false
1080 Acked Read 7 Sent Done Done false
1081 Acked Read 9 Sent Done Done false
1082 Acked Write 7 Sent Done Done false
1083 Acked Read 2 Sent Done Done false
1084 Acked Write 2 Sent Done Done false
1085 Acked Read 9 Sent Done Done false
1086 Acked Read 6 Sent Done Done false
1087 Acked Write 4 Sent Done Done false
1088 Acked Write 9 Sent Done Done false
1089 Acked Write 1 Sent Done Done false
1090 Acked Read 8 Sent Done Done false
1091 Acked Write 6 Sent Done Done false
1092 Acked Write 9 Sent Done Done false
1093 Acked Read 9 Sent Done Done false
1094 Acked Read 7 Sent Done Done false
1095 Acked Write 6 Sent Done Done false
1096 Acked Flush 0 Sent Done Done false
1097 Acked Read 4 Sent Done Done false
1098 Acked Read 2 Sent Done Done false
1099 Acked Write 6 Sent Done Done false
1100 Acked Write 3 Sent Done Done false
1101 Acked Read 4 Sent Done Done false
1102 Acked Read 5 Sent Done Done false
1103 Acked Flush 0 Sent Done Done false
1104 Acked Read 5 Sent Done Done false
1105 Acked Write 4 Sent Done Done false
1106 Acked Write 6 Sent Done Done false
1107 Acked Write 7 Sent Done Done false
1108 Acked Read 5 Sent Done Done false
1109 Acked Read 2 Sent Done Done false
1110 Acked Read 6 Sent Done Done false
1111 Acked Write 3 Sent Done Done false
1112 Acked Read 4 Sent Done Done false
1113 Acked Read 4 Sent Done Done false
1114 Acked Flush 0 Sent Done Done false
1115 Acked Read 8 Sent Done Done false
1116 Acked Write 7 Sent Done Done false
1117 Acked Write 8 Sent Done Done false
1118 Acked Read 5 Sent Done Done false
1119 Acked Write 8 Sent Done Done false
1120 Acked Read 4 Sent Done Done false
1121 Acked Write 6 Sent Done Done false
1122 Acked Write 5 Sent Done Done false
1123 Acked Write 2 Sent Done Done false
1124 Acked Read 1 Sent Done Done false
1125 Acked Write 8 Sent Done Done false
1126 Acked Write 6 Sent Done Done false
1127 Acked Write 9 Sent Done Done false
1128 Acked Write 6 Sent Done Done false
1129 Acked Write 1 Sent Done Done false
1130 Acked Write 2 Sent Done Done false
1131 Acked Read 6 Sent Done Done false
1132 Acked Write 7 Sent Done Done false
1133 Acked Write 6 Sent Done Done false
1134 Acked Read 10 Sent Done Done false
1135 Acked Write 9 Sent Done Done false
1136 Acked Write 10 Sent Done Done false
1137 Acked Write 4 Sent Done Done false
1138 Acked Read 8 Sent Done Done false
1139 Acked Write 7 Sent Done Done false
1140 Acked Write 9 Sent Done Done false
1141 Acked Flush 0 Sent Done Done false
1142 Acked Read 2 Sent Done Done false
1143 Acked Write 9 Sent Done Done false
1144 Acked Write 7 Sent Done Done false
1145 Acked Read 2 Sent Done Done false
1146 Acked Read 5 Sent Done Done false
1147 Acked Flush 0 Sent Done Done false
1148 Acked Write 1 Sent Done Done false
1149 Acked Read 5 Sent Done Done false
1150 Acked Read 10 Sent Done Done false
1151 Acked Flush 0 Sent Done Done false
1152 Acked Write 5 Sent Done Done false
1153 Acked Flush 0 Sent Done Done false
1154 Acked Read 2 Sent Done Done false
1155 Acked Write 2 Sent Done Done false
1156 Acked Read 4 Sent Done Done false
1157 Acked Read 10 Sent Done Done false
1158 Acked Write 7 Sent Done Done false
1159 Acked Write 2 Sent Done Done false
1160 Acked Write 3 Sent Done Done false
1161 Acked Write 8 Sent Done Done false
1162 Acked Read 7 Sent Done Done false
1163 Acked Write 4 Sent Done Done false
1164 Acked Read 1 Sent Done Done false
1165 Acked Read 9 Sent Done Done false
1166 Acked Write 1 Sent Done Done false
1167 Acked Write 8 Sent Done Done false
1168 Acked Write 10 Sent Done Done false
1169 Acked Read 7 Sent Done Done false
1170 Acked Read 4 Sent Done Done false
1171 Acked Write 6 Sent Done Done false
1172 Acked Read 6 Sent Done Done false
1173 Acked Read 2 Sent Done Done false
1174 Acked Read 6 Sent Done Done false
1175 Acked Read 8 Sent Done Done false
1176 Acked Read 9 Sent Done Done false
1177 Acked Read 9 Sent Done Done false
1178 Acked Write 2 Sent Done Done false
1179 Acked Write 5 Sent Done Done false
1180 Acked Read 8 Sent Done Done false
1181 Acked Read 7 Sent Done Done false
1182 Acked Read 7 Sent Done Done false
1183 Acked Write 2 Sent Done Done false
1184 Acked Write 4 Sent Done Done false
1185 Acked Flush 0 Sent Done Done false
1186 Acked Write 4 Sent Done Done false
1187 Acked Write 1 Sent Done Done false
1188 Acked Write 9 Sent Done Done false
1189 Acked Write 5 Sent Done Done false
1190 Acked Write 8 Sent Done Done false
1191 Acked Read 1 Sent Done Done false
1192 Acked Flush 0 Sent Done Done false
1193 Acked Read 8 Sent Done Done false
1194 Acked Read 2 Sent Done Done false
1195 Acked Write 2 Sent Done Done false
1196 Acked Read 5 Sent Done Done false
1197 Acked Flush 0 Sent Done Done false
1198 Acked Flush 0 Sent Done Done false
1199 Acked Read 4 Sent Done Done false
1200 Acked Read 2 Sent Done Done false
1201 Acked Write 4 Sent Done Done false
1202 Acked Read 10 Sent Done Done false
1203 Acked Read 1 Sent Done Done false
1204 Acked Write 10 Sent Done Done false
1205 Acked Write 5 Sent Done Done false
1206 Acked Write 3 Sent Done Done false
1207 Acked Read 10 Sent Done Done false
1208 Acked Write 8 Sent Done Done false
1209 Acked Read 4 Sent Done Done false
1210 Acked Write 10 Sent Done Done false
1211 Acked Read 3 Sent Done Done false
1212 Acked Write 1 Sent Done Done false
1213 Acked Read 7 Sent Done Done false
1214 Acked Write 1 Sent Done Done false
1215 Acked Read 2 Sent Done Done false
1216 Acked Write 2 Sent Done Done false
1217 Acked Read 4 Sent Done Done false
1218 Acked Read 8 Sent Done Done false
1219 Acked Write 9 Sent Done Done false
1220 Acked Read 9 Sent Done Done false
1221 Acked Write 1 Sent Done Done false
1222 Acked Read 2 Sent Done Done false
1223 Acked Read 8 Sent Done Done false
1224 Acked Flush 0 Sent Done Done false
1225 Acked Write 2 Sent Done Done false
1226 Acked Read 9 Sent Done Done false
1227 Acked Write 5 Sent Done Done false
1228 Acked Read 6 Sent Done Done false
1229 Acked Read 9 Sent Done Done false
1230 Acked Read 6 Sent Done Done false
1231 Acked Flush 0 Sent Done Done false
1232 Acked Write 7 Sent Done Done false
1233 Acked Write 10 Sent Done Done false
1234 Acked Write 6 Sent Done Done false
1235 Acked Write 2 Sent Done Done false
1236 Acked Write 8 Sent Done Done false
1237 Acked Flush 0 Sent Done Done false
1238 Acked Read 3 Sent Done Done false
1239 Acked Write 1 Sent Done Done false
1240 Acked Read 7 Sent Done Done false
1241 Acked Write 2 Sent Done Done false
1242 Acked Write 7 Sent Done Done false
1243 Acked Flush 0 Sent Done Done false
1244 Acked Read 6 Sent Done Done false
1245 Acked Write 9 Sent Done Done false
1246 Acked Read 5 Sent Done Done false
1247 Acked Write 10 Sent Done Done false
1248 Acked Flush 0 Sent Done Done false
1249 Acked Read 8 Sent Done Done false
1250 Acked Write 4 Sent Done Done false
1251 Acked Write 6 Sent Done Done false
1252 Acked Write 3 Sent Done Done false
1253 Acked Write 5 Sent Done Done false
1254 Acked Read 1 Sent Done Done false
1255 Acked Read 9 Sent Done Done false
1256 Acked Read 7 Sent Done Done false
1257 Acked Read 8 Sent Done Done false
1258 Acked Read 2 Sent Done Done false
1259 Acked Write 5 Sent Done Done false
1260 Acked Read 1 Sent Done Done false
1261 Acked Read 2 Sent Done Done false
1262 Acked Flush 0 Sent Done Done false
1263 Acked Write 6 Sent Done Done false
1264 Acked Read 9 Sent Done Done false
1265 Acked Write 1 Sent Done Done false
1266 Acked Write 2 Sent Done Done false
1267 Acked Write 10 Sent Done Done false
1268 Acked Read 6 Sent Done Done false
1269 Acked Write 7 Sent Done Done false
1270 Acked Read 9 Sent Done Done false
1271 Acked Read 7 Sent Done Done false
1272 Acked Write 9 Sent Done Done false
1273 Acked Read 8 Sent Done Done false
1274 Acked Write 6 Sent Done Done false
1275 Acked Read 7 Sent Done Done false
1276 Acked Read 1 Sent Done Done false
1277 Acked Read 4 Sent Done Done false
1278 Acked Read 3 Sent Done Done false
1279 Acked Write 3 Sent Done Done false
1280 Acked Read 2 Sent Done Done false
1281 Acked Read 2 Sent Done Done false
1282 Acked Read 6 Sent Done Done false
1283 Acked Write 7 Sent Done Done false
1284 Acked Write 9 Sent Done Done false
1285 Acked Read 5 Sent Done Done false
1286 Acked Read 2 Sent Done Done false
1287 Acked Write 2 Sent Done Done false
1288 Acked Read 9 Sent Done Done false
1289 Acked Read 5 Sent Done Done false
1290 Acked Write 10 Sent Done Done false
1291 Acked Write 6 Sent Done Done false
1292 Acked Write 9 Sent Done Done false
1293 Acked Read 2 Sent Done Done false
1294 Acked Read 3 Sent Done Done false
1295 Acked Write 5 Sent Done Done false
1296 Acked Write 7 Sent Done Done false
1297 Acked Write 10 Sent Done Done false
1298 Acked Write 5 Sent Done Done false
1299 Acked Write 10 Sent Done Done false
1300 Acked Write 7 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 1262 1262
Downstairs last five completed:
Upstairs last five completed: 1300 1299 1298 1297 1296
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Read 8 Sent Done Done false
1001 Acked Flush 0 Sent Done Done false
1002 Acked Read 7 Sent Done Done false
1003 Acked Read 6 Sent Done Done false
1004 Acked Write 9 Sent Done Done false
1005 Acked Read 8 Sent Done Done false
1006 Acked Write 3 Sent Done Done false
1007 Acked Write 1 Sent Done Done false
1008 Acked Read 6 Sent Done Done false
1009 Acked Write 2 Sent Done Done false
1010 Acked Write 8 Sent Done Done false
1011 Acked Read 5 Sent Done Done false
1012 Acked Read 4 Sent Done Done false
1013 Acked Write 5 Sent Done Done false
1014 Acked Write 9 Sent Done Done false
1015 Acked Read 4 Sent Done Done false
1016 Acked Write 2 Sent Done Done false
1017 Acked Read 9 Sent Done Done false
1018 Acked Write 8 Sent Done Done false
1019 Acked Read 10 Sent Done Done false
1020 Acked Read 2 Sent Done Done false
1021 Acked Write 5 Sent Done Done false
1022 Acked Write 7 Sent Done Done false
1023 Acked Read 2 Sent Done Done false
1024 Acked Write 1 Sent Done Done false
1025 Acked Write 7 Sent Done Done false
1026 Acked Write 5 Sent Done Done false
1027 Acked Write 3 Sent Done Done false
1028 Acked Flush 0 Sent Done Done false
1029 Acked Write 4 Sent Done Done false
1030 Acked Read 4 Sent Done Done false
1031 Acked Flush 0 Sent Done Done false
1032 Acked Read 8 Sent Done Done false
1033 Acked Write 1 Sent Done Done false
1034 Acked Write 5 Sent Done Done false
1035 Acked Write 9 Sent Done Done false
1036 Acked Write 9 Sent Done Done false
1037 Acked Flush 0 Sent Done Done false
1038 Acked Flush 0 Sent Done Done false
1039 Acked Write 2 Sent Done Done false
1040 Acked Read 6 Sent Done Done false
1041 Acked Read 1 Sent Done Done false
1042 Acked Write 8 Sent Done Done false
1043 Acked Read 4 Sent Done Done false
1044 Acked Write 8 Sent Done Done false
1045 Acked Flush 0 Sent Done Done false
1046 Acked Read 4 Sent Done Done false
1047 Acked Flush 0 Sent Done Done false
1048 Acked Write 6 Sent Done Done false
1049 Acked Read 5 Sent Done Done false
1050 Acked Read 1 Sent Done Done false
1051 Acked Read 1 Sent Done Done false
1052 Acked Read 9 Sent Done Done false
1053 Acked Read 5 Sent Done Done false
1054 Acked Read 3 Sent Done Done false
1055 Acked Read 10 Sent Done Done false
1056 Acked Read 10 Sent Done Done false
1057 Acked Read 6 Sent Done Done false
1058 Acked Flush 0 Sent Done Done false
1059 Acked Write 3 Sent Done Done false
1060 Acked Write 5 Sent Done Done false
1061 Acked Write 10 Sent Done Done false
1062 Acked Write 8 Sent Done Done false
1063 Acked Write 7 Sent Done Done false
1064 Acked Flush 0 Sent Done Done false
1065 Acked Read 5 Sent Done Done false
1066 Acked Read 2 Sent Done Done false
1067 Acked Flush 0 Sent Done Done false
1068 Acked Read 1 Sent Done Done false
1069 Acked Write 1 Sent Done Done false
1070 Acked Write 6 Sent Done Done false
1071 Acked Write 4 Sent Done Done false
1072 Acked Write 9 Sent Done Done false
1073 Acked Read 9 Sent Done Done false
1074 Acked Write 3 Sent Done Done false
1075 Acked Write 8 Sent Done Done false
1076 Acked Read 1 Sent Done Done false
1077 Acked Read 3 Sent Done Done false
1078 Acked Read 4 Sent Done Done false
1079 Acked Flush 0 Sent Done Done false
1080 Acked Read 7 Sent Done Done false
1081 Acked Read 9 Sent Done Done false
1082 Acked Write 7 Sent Done Done false
1083 Acked Read 2 Sent Done Done false
1084 Acked Write 2 Sent Done Done false
1085 Acked Read 9 Sent Done Done false
1086 Acked Read 6 Sent Done Done false
1087 Acked Write 4 Sent Done Done false
1088 Acked Write 9 Sent Done Done false
1089 Acked Write 1 Sent Done Done false
1090 Acked Read 8 Sent Done Done false
1091 Acked Write 6 Sent Done Done false
1092 Acked Write 9 Sent Done Done false
1093 Acked Read 9 Sent Done Done false
1094 Acked Read 7 Sent Done Done false
1095 Acked Write 6 Sent Done Done false
1096 Acked Flush 0 Sent Done Done false
1097 Acked Read 4 Sent Done Done false
1098 Acked Read 2 Sent Done Done false
1099 Acked Write 6 Sent Done Done false
1100 Acked Write 3 Sent Done Done false
1101 Acked Read 4 Sent Done Done false
1102 Acked Read 5 Sent Done Done false
1103 Acked Flush 0 Sent Done Done false
1104 Acked Read 5 Sent Done Done false
1105 Acked Write 4 Sent Done Done false
1106 Acked Write 6 Sent Done Done false
1107 Acked Write 7 Sent Done Done false
1108 Acked Read 5 Sent Done Done false
1109 Acked Read 2 Sent Done Done false
1110 Acked Read 6 Sent Done Done false
1111 Acked Write 3 Sent Done Done false
1112 Acked Read 4 Sent Done Done false
1113 Acked Read 4 Sent Done Done false
1114 Acked Flush 0 Sent Done Done false
1115 Acked Read 8 Sent Done Done false
1116 Acked Write 7 Sent Done Done false
1117 Acked Write 8 Sent Done Done false
1118 Acked Read 5 Sent Done Done false
1119 Acked Write 8 Sent Done Done false
1120 Acked Read 4 Sent Done Done false
1121 Acked Write 6 Sent Done Done false
1122 Acked Write 5 Sent Done Done false
1123 Acked Write 2 Sent Done Done false
1124 Acked Read 1 Sent Done Done false
1125 Acked Write 8 Sent Done Done false
1126 Acked Write 6 Sent Done Done false
1127 Acked Write 9 Sent Done Done false
1128 Acked Write 6 Sent Done Done false
1129 Acked Write 1 Sent Done Done false
1130 Acked Write 2 Sent Done Done false
1131 Acked Read 6 Sent Done Done false
1132 Acked Write 7 Sent Done Done false
1133 Acked Write 6 Sent Done Done false
1134 Acked Read 10 Sent Done Done false
1135 Acked Write 9 Sent Done Done false
1136 Acked Write 10 Sent Done Done false
1137 Acked Write 4 Sent Done Done false
1138 Acked Read 8 Sent Done Done false
1139 Acked Write 7 Sent Done Done false
1140 Acked Write 9 Sent Done Done false
1141 Acked Flush 0 Sent Done Done false
1142 Acked Read 2 Sent Done Done false
1143 Acked Write 9 Sent Done Done false
1144 Acked Write 7 Sent Done Done false
1145 Acked Read 2 Sent Done Done false
1146 Acked Read 5 Sent Done Done false
1147 Acked Flush 0 Sent Done Done false
1148 Acked Write 1 Sent Done Done false
1149 Acked Read 5 Sent Done Done false
1150 Acked Read 10 Sent Done Done false
1151 Acked Flush 0 Sent Done Done false
1152 Acked Write 5 Sent Done Done false
1153 Acked Flush 0 Sent Done Done false
1154 Acked Read 2 Sent Done Done false
1155 Acked Write 2 Sent Done Done false
1156 Acked Read 4 Sent Done Done false
1157 Acked Read 10 Sent Done Done false
1158 Acked Write 7 Sent Done Done false
1159 Acked Write 2 Sent Done Done false
1160 Acked Write 3 Sent Done Done false
1161 Acked Write 8 Sent Done Done false
1162 Acked Read 7 Sent Done Done false
1163 Acked Write 4 Sent Done Done false
1164 Acked Read 1 Sent Done Done false
1165 Acked Read 9 Sent Done Done false
1166 Acked Write 1 Sent Done Done false
1167 Acked Write 8 Sent Done Done false
1168 Acked Write 10 Sent Done Done false
1169 Acked Read 7 Sent Done Done false
1170 Acked Read 4 Sent Done Done false
1171 Acked Write 6 Sent Done Done false
1172 Acked Read 6 Sent Done Done false
1173 Acked Read 2 Sent Done Done false
1174 Acked Read 6 Sent Done Done false
1175 Acked Read 8 Sent Done Done false
1176 Acked Read 9 Sent Done Done false
1177 Acked Read 9 Sent Done Done false
1178 Acked Write 2 Sent Done Done false
1179 Acked Write 5 Sent Done Done false
1180 Acked Read 8 Sent Done Done false
1181 Acked Read 7 Sent Done Done false
1182 Acked Read 7 Sent Done Done false
1183 Acked Write 2 Sent Done Done false
1184 Acked Write 4 Sent Done Done false
1185 Acked Flush 0 Sent Done Done false
1186 Acked Write 4 Sent Done Done false
1187 Acked Write 1 Sent Done Done false
1188 Acked Write 9 Sent Done Done false
1189 Acked Write 5 Sent Done Done false
1190 Acked Write 8 Sent Done Done false
1191 Acked Read 1 Sent Done Done false
1192 Acked Flush 0 Sent Done Done false
1193 Acked Read 8 Sent Done Done false
1194 Acked Read 2 Sent Done Done false
1195 Acked Write 2 Sent Done Done false
1196 Acked Read 5 Sent Done Done false
1197 Acked Flush 0 Sent Done Done false
1198 Acked Flush 0 Sent Done Done false
1199 Acked Read 4 Sent Done Done false
1200 Acked Read 2 Sent Done Done false
1201 Acked Write 4 Sent Done Done false
1202 Acked Read 10 Sent Done Done false
1203 Acked Read 1 Sent Done Done false
1204 Acked Write 10 Sent Done Done false
1205 Acked Write 5 Sent Done Done false
1206 Acked Write 3 Sent Done Done false
1207 Acked Read 10 Sent Done Done false
1208 Acked Write 8 Sent Done Done false
1209 Acked Read 4 Sent Done Done false
1210 Acked Write 10 Sent Done Done false
1211 Acked Read 3 Sent Done Done false
1212 Acked Write 1 Sent Done Done false
1213 Acked Read 7 Sent Done Done false
1214 Acked Write 1 Sent Done Done false
1215 Acked Read 2 Sent Done Done false
1216 Acked Write 2 Sent Done Done false
1217 Acked Read 4 Sent Done Done false
1218 Acked Read 8 Sent Done Done false
1219 Acked Write 9 Sent Done Done false
1220 Acked Read 9 Sent Done Done false
1221 Acked Write 1 Sent Done Done false
1222 Acked Read 2 Sent Done Done false
1223 Acked Read 8 Sent Done Done false
1224 Acked Flush 0 Sent Done Done false
1225 Acked Write 2 Sent Done Done false
1226 Acked Read 9 Sent Done Done false
1227 Acked Write 5 Sent Done Done false
1228 Acked Read 6 Sent Done Done false
1229 Acked Read 9 Sent Done Done false
1230 Acked Read 6 Sent Done Done false
1231 Acked Flush 0 Sent Done Done false
1232 Acked Write 7 Sent Done Done false
1233 Acked Write 10 Sent Done Done false
1234 Acked Write 6 Sent Done Done false
1235 Acked Write 2 Sent Done Done false
1236 Acked Write 8 Sent Done Done false
1237 Acked Flush 0 Sent Done Done false
1238 Acked Read 3 Sent Done Done false
1239 Acked Write 1 Sent Done Done false
1240 Acked Read 7 Sent Done Done false
1241 Acked Write 2 Sent Done Done false
1242 Acked Write 7 Sent Done Done false
1243 Acked Flush 0 Sent Done Done false
1244 Acked Read 6 Sent Done Done false
1245 Acked Write 9 Sent Done Done false
1246 Acked Read 5 Sent Done Done false
1247 Acked Write 10 Sent Done Done false
1248 Acked Flush 0 Sent Done Done false
1249 Acked Read 8 Sent Done Done false
1250 Acked Write 4 Sent Done Done false
1251 Acked Write 6 Sent Done Done false
1252 Acked Write 3 Sent Done Done false
1253 Acked Write 5 Sent Done Done false
1254 Acked Read 1 Sent Done Done false
1255 Acked Read 9 Sent Done Done false
1256 Acked Read 7 Sent Done Done false
1257 Acked Read 8 Sent Done Done false
1258 Acked Read 2 Sent Done Done false
1259 Acked Write 5 Sent Done Done false
1260 Acked Read 1 Sent Done Done false
1261 Acked Read 2 Sent Done Done false
1262 Acked Flush 0 Sent Done Done false
1263 Acked Write 6 Sent Done Done false
1264 Acked Read 9 Sent Done Done false
1265 Acked Write 1 Sent Done Done false
1266 Acked Write 2 Sent Done Done false
1267 Acked Write 10 Sent Done Done false
1268 Acked Read 6 Sent Done Done false
1269 Acked Write 7 Sent Done Done false
1270 Acked Read 9 Sent Done Done false
1271 Acked Read 7 Sent Done Done false
1272 Acked Write 9 Sent Done Done false
1273 Acked Read 8 Sent Done Done false
1274 Acked Write 6 Sent Done Done false
1275 Acked Read 7 Sent Done Done false
1276 Acked Read 1 Sent Done Done false
1277 Acked Read 4 Sent Done Done false
1278 Acked Read 3 Sent Done Done false
1279 Acked Write 3 Sent Done Done false
1280 Acked Read 2 Sent Done Done false
1281 Acked Read 2 Sent Done Done false
1282 Acked Read 6 Sent Done Done false
1283 Acked Write 7 Sent Done Done false
1284 Acked Write 9 Sent Done Done false
1285 Acked Read 5 Sent Done Done false
1286 Acked Read 2 Sent Done Done false
1287 Acked Write 2 Sent Done Done false
1288 Acked Read 9 Sent Done Done false
1289 Acked Read 5 Sent Done Done false
1290 Acked Write 10 Sent Done Done false
1291 Acked Write 6 Sent Done Done false
1292 Acked Write 9 Sent Done Done false
1293 Acked Read 2 Sent Done Done false
1294 Acked Read 3 Sent Done Done false
1295 Acked Write 5 Sent Done Done false
1296 Acked Write 7 Sent Done Done false
1297 Acked Write 10 Sent Done Done false
1298 Acked Write 5 Sent Done Done false
1299 Acked Write 10 Sent Done Done false
1300 Acked Write 7 Sent Done Done false
1301 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 1301 1301
Downstairs last five completed:
Upstairs last five completed: 1301 1300 1299 1298 1297
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.927592802Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58544)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.927724296Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.928862374Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.929433827Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.929472622Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.92949174Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.929691257Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.929718844Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:34.930057984Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1301 1300 1299 1298 1297
Upstairs last five completed: 1301 1300 1299 1298 1297
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:36:37.160 INFO request completed, latency_us: 357[8810] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 6a7b3b1d-86a2-489e-9056-7fa604c3b0f7, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:37.161668924Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:37.161727968Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:37.161750945Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:36:37.161 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9a0b2a35-811c-4468-be91-47f1d91d1f40, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:37.161802439Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:36:41.164 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/0, method: GET, req_id: 02bda8f9-355b-4ac9-999c-79b4a3aaeb66, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2423 len 32768 data: 2 2 2 2 2 2 2 2
002/300 Read block 1088 len 36864
003/300 Write block 2161 len 24576 data: 2 2 2 2 2 2
004/300 Flush
005/300 Write block 1535 len 16384 data: 3 2 4 4
006/300 Read block 2819 len 12288
007/300 Read block 955 len 32768
008/300 Write block 4271 len 24576 data: 2 2 2 2 2 2
009/300 Write block 2853 len 36864 data: 3 2 2 2 2 2 2 2 2
010/300 Read block 3341 len 32768
011/300 Write block 1553 len 8192 data: 2 2
012/300 Write block 4520 len 12288 data: 2 2 2
013/300 Read block 2824 len 20480
014/300 Read block 2327 len 36864
015/300 Read block 678 len 28672
016/300 Write block 122 len 28672 data: 2 2 2 2 2 2 2
017/300 Read block 4878 len 32768
018/300 Flush
019/300 Write block 1713 len 20480 data: 2 2 2 2 2
020/300 Write block 2203 len 24576 data: 2 2 2 2 2 2
021/300 Read block 1158 len 16384
022/300 Write block 2522 len 12288 data: 2 2 2
023/300 Flush
024/300 Write block 4246 len 20480 data: 2 2 2 2 3
025/300 Flush
026/300 Write block 2378 len 20480 data: 2 2 2 2 2
027/300 Write block 2836 len 40960 data: 2 2 2 2 2 2 2 2 2 2
028/300 Write block 919 len 32768 data: 2 2 2 2 2 2 2 2
029/300 Write block 2136 len 4096 data: 2
030/300 Read block 936 len 8192
031/300 Read block 209 len 20480
032/300 Write block 1029 len 36864 data: 3 3 2 2 2 2 2 2 2
033/300 Flush
034/300 Write block 3408 len 12288 data: 3 3 2
035/300 Write block 4709 len 16384 data: 2 2 2 2
036/300 Read block 602 len 16384
037/300 Flush
038/300 Read block 781 len 32768
039/300 Write block 2201 len 12288 data: 2 2 3
040/300 Write block 1373 len 32768 data: 3 3 4 4 4 3 3 3
041/300 Write block 120 len 32768 data: 2 2 3 3 3 3 3 3
042/300 Read block 4963 len 40960
043/300 Write block 4567 len 8192 data: 3 3
044/300 Flush
045/300 Read block 2302 len 36864
046/300 Write block 2102 len 16384 data: 2 2 2 2
047/300 Read block 4686 len 12288
048/300 Write block 1437 len 28672 data: 2 2 2 2 2 2 2
049/300 Write block 300 len 12288 data: 2 2 2
050/300 Read block 4500 len 16384
051/300 Write block 2560 len 28672 data: 2 2 2 2 2 2 2
052/300 Read block 4818 len 16384
053/300 Read block 1558 len 4096
054/300 Write block 3387 len 28672 data: 2 2 2 2 2 2 2
055/300 Read block 3223 len 4096
056/300 Read block 2545 len 24576
057/300 Write block 485 len 28672 data: 2 2 2 2 2 2 2
058/300 Read block 3664 len 20480
059/300 Read block 3400 len 4096
060/300 Read block 3973 len 40960
061/300 Read block 1595 len 4096
062/300 Write block 2323 len 8192 data: 2 2
063/300 Flush
064/300 Write block 3556 len 36864 data: 2 2 2 2 2 2 2 2 2
065/300 Read block 3480 len 4096
066/300 Read block 1402 len 36864
067/300 Write block 2076 len 40960 data: 2 2 2 2 2 2 2 2 2 2
068/300 Write block 598 len 32768 data: 2 2 2 2 3 3 3 3
069/300 Write block 648 len 8192 data: 2 2
070/300 Flush
071/300 Read block 2002 len 8192
072/300 Read block 2023 len 4096
073/300 Flush
074/300 Flush
075/300 Read block 3601 len 16384
076/300 Write block 2846 len 4096 data: 2
077/300 Flush
078/300 Read block 4224 len 32768
079/300 Read block 1850 len 20480
080/300 Write block 866 len 8192 data: 2 2
081/300 Read block 2131 len 32768
082/300 Read block 3060 len 24576
083/300 Write block 3231 len 20480 data: 3 3 3 3 3
084/300 Read block 716 len 4096
085/300 Write block 718 len 28672 data: 2 2 3 3 3 3 3
086/300 Write block 4160 len 36864 data: 2 2 2 2 2 2 2 2 2
087/300 Read block 18 len 4096
088/300 Flush
089/300 Read block 952 len 40960
090/300 Read block 3454 len 28672
091/300 Write block 3582 len 8192 data: 2 2
092/300 Flush
093/300 Read block 3047 len 24576
094/300 Read block 3487 len 20480
095/300 Write block 16 len 8192 data: 2 2
096/300 Write block 3320 len 28672 data: 2 2 2 2 2 2 2
097/300 Write block 2410 len 28672 data: 2 2 2 2 2 2 2
098/300 Write block 557 len 36864 data: 2 2 2 2 2 2 2 2 2
099/300 Write block 3908 len 24576 data: 3 3 3 3 3 3
100/300 Read block 2088 len 16384
101/300 Write block 3293 len 24576 data: 2 2 2 2 2 2
102/300 Flush
103/300 Read block 4506 len 40960
104/300 Flush
105/300 Flush
106/300 Read block 1288 len 20480
107/300 Write block 545 len 24576 data: 2 2 2 2 2 2
108/300 Write block 1126 len 12288 data: 2 2 2
109/300 Read block 3547 len 16384
110/300 Write block 1042 len 24576 data: 2 2 2 2 2 2
111/300 Read block 2757 len 12288
112/300 Write block 1507 len 8192 data: 2 2
113/300 Read block 1716 len 32768
114/300 Read block 1327 len 32768
115/300 Read block 3998 len 16384
116/300 Read block 4414 len 24576
117/300 Write block 3176 len 28672 data: 3 3 3 3 2 2 2
118/300 Read block 2356 len 16384
119/300 Write block 4502 len 24576 data: 2 2 2 2 2 2
120/300 Read block 1696 len 20480
121/300 Write block 1815 len 12288 data: 2 2 2
122/300 Write block 3766 len 32768 data: 2 2 2 2 2 2 2 3
123/300 Write block 1480 len 20480 data: 2 2 2 2 2
124/300 Read block 671 len 24576
125/300 Flush
126/300 Read block 4500 len 28672
127/300 Write block 372 len 36864 data: 2 3 3 2 2 2 2 2 2
128/300 Read block 338 len 28672
129/300 Write block 1983 len 24576 data: 2 2 2 2 2 2
130/300 Read block 890 len 12288
131/300 Read block 2900 len 12288
132/300 Read block 1777 len 12288
133/300 Flush
134/300 Write block 3304 len 12288 data: 2 2 2
135/300 Write block 2682 len 28672 data: 2 2 2 2 2 2 2
136/300 Read block 3347 len 4096
137/300 Read block 3162 len 24576
138/300 Read block 4869 len 12288
139/300 Write block 3455 len 12288 data: 2 2 2
140/300 Read block 3594 len 24576
141/300 Write block 4242 len 24576 data: 3 3 2 2 3 3
142/300 Read block 494 len 16384
143/300 Read block 4053 len 8192
144/300 Write block 709 len 28672 data: 2 2 2 2 3 3 3
145/300 Read block 4167 len 28672
146/300 Write block 4439 len 32768 data: 2 2 2 2 2 2 2 2
147/300 Read block 2284 len 20480
148/300 Write block 2018 len 24576 data: 2 2 2 2 2 2
149/300 Write block 4245 len 28672 data: 3 4 4 3 3 4 3
150/300 Read block 2658 len 32768
151/300 Write block 4013 len 36864 data: 2 2 2 2 3 3 3 3 3
152/300 Read block 3972 len 4096
153/300 Read block 1705 len 8192
154/300 Flush
155/300 Read block 4718 len 32768
156/300 Write block 3904 len 36864 data: 4 3 3 3 4 4 4 4 4
157/300 Write block 13 len 36864 data: 2 2 2 3 3 2 2 2 2
158/300 Read block 3425 len 40960
159/300 Read block 3159 len 24576
160/300 Read block 3343 len 24576
161/300 Read block 2318 len 36864
162/300 Write block 1789 len 28672 data: 3 3 3 2 2 3 3
163/300 Read block 1170 len 20480
164/300 Write block 4788 len 40960 data: 2 2 3 3 3 3 3 3 3 2
165/300 Read block 2406 len 32768
166/300 Read block 3702 len 8192
167/300 Read block 898 len 32768
168/300 Write block 2790 len 24576 data: 2 2 2 2 3 3
169/300 Read block 346 len 8192
170/300 Write block 3872 len 28672 data: 2 2 2 2 2 2 2
171/300 Read block 4494 len 8192
172/300 Read block 1097 len 24576
173/300 Read block 4877 len 36864
174/300 Read block 977 len 36864
175/300 Read block 285 len 16384
176/300 Write block 4526 len 12288 data: 2 2 2
177/300 Read block 2313 len 24576
178/300 Read block 4239 len 4096
179/300 Read block 3371 len 32768
180/300 Read block 393 len 16384
181/300 Read block 242 len 32768
182/300 Read block 104 len 4096
183/300 Write block 2437 len 40960 data: 2 2 2 2 2 2 2 2 2 2
184/300 Write block 2437 len 40960 data: 3 3 3 3 3 3 3 3 3 3
185/300 Read block 2037 len 12288
186/300 Flush
187/300 Read block 2489 len 40960
188/300 Flush
189/300 Read block 2604 len 4096
190/300 Write block 320 len 28672 data: 2 2 2 2 2 2 2
191/300 Write block 1221 len 20480 data: 2 2 2 2 2
192/300 Read block 1151 len 24576
193/300 Write block 713 len 8192 data: 4 4
194/300 Read block 2952 len 20480
195/300 Write block 4985 len 12288 data: 2 2 2
196/300 Read block 3337 len 16384
197/300 Write block 238 len 24576 data: 2 2 2 2 2 2
198/300 Write block 4824 len 8192 data: 2 2
199/300 Write block 4938 len 28672 data: 2 2 2 2 3 3 3
200/300 Write block 1616 len 16384 data: 2 2 2 2
201/300 Read block 2951 len 40960
202/300 Read block 4430 len 8192
203/300 Write block 1303 len 40960 data: 2 2 2 2 2 2 2 2 2 2
204/300 Write block 3222 len 16384 data: 3 3 3 3
205/300 Read block 3206 len 8192
206/300 Write block 2493 len 8192 data: 2 2
207/300 Read block 4966 len 36864
208/300 Write block 3559 len 28672 data: 3 3 3 3 3 3 3
209/300 Write block 2928 len 12288 data: 2 2 2
210/300 Read block 2953 len 24576
211/300 Read block 4608 len 24576
212/300 Write block 4034 len 16384 data: 2 2 2 2
213/300 Flush
214/300 Flush
215/300 Read block 1128 len 24576
216/300 Write block 184 len 12288 data: 2 2 3
217/300 Read block 296 len 40960
218/300 Write block 3921 len 16384 data: 2 2 2 2
219/300 Write block 1536 len 40960 data: 3 5 5 4 4 4 4 5 5 4
220/300 Write block 587 len 32768 data: 2 2 2 2 2 2 2 2
221/300 Read block 3984 len 4096
222/300 Read block 3318 len 24576
223/300 Write block 4304 len 40960 data: 2 2 2 2 2 2 2 2 2 2
224/300 Read block 193 len 8192
225/300 Read block 3018 len 8192
226/300 Write block 1812 len 28672 data: 3 2 2 3 3 3 2
227/300 Write block 4330 len 40960 data: 3 4 3 3 3 3 3 3 3 2
228/300 Read block 464 len 20480
229/300 Read block 548 len 4096
230/300 Read block 279 len 12288
231/300 Write block 4189 len 40960 data: 2 2 2 2 2 2 2 3 3 2
232/300 Flush
233/300 Read block 1433 len 12288
234/300 Write block 2255 len 16384 data: 2 2 2 2
235/300 Read block 3159 len 28672
236/300 Read block 1849 len 20480
237/300 Read block 1727 len 16384
238/300 Read block 1410 len 40960
239/300 Write block 4810 len 8192 data: 2 2
240/300 Read block 2362 len 32768
241/300 Write block 4008 len 4096 data: 2
242/300 Write block 3680 len 4096 data: 2
243/300 Flush
244/300 Read block 22 len 24576
245/300 Read block 3306 len 24576
246/300 Write block 2736 len 12288 data: 2 2 2
247/300 Flush
248/300 Read block 647 len 32768
249/300 Read block 3616 len 32768
250/300 Read block 1534 len 20480
251/300 Read block 3702 len 20480
252/300 Write block 1456 len 12288 data: 3 3 3
253/300 Read block 3951 len 32768
254/300 Read block 1628 len 40960
255/300 Write block 4053 len 24576 data: 2 2 2 2 2 2
256/300 Write block 1850 len 8192 data: 3 3
257/300 Write block 2677 len 32768 data: 3 3 3 3 3 3 3 3
258/300 Read block 3762 len 8192
259/300 Read block 3522 len 20480
260/300 Write block 4271 len 32768 data: 3 3 3 3 3 3 3 3
261/300 Write block 3725 len 20480 data: 2 2 2 2 2
262/300 Write block 2406 len 20480 data: 2 2 2 2 3
263/300 Write block 2385 len 28672 data: 2 2 2 2 2 2 2
264/300 Flush
265/300 Write block 2193 len 36864 data: 2 2 2 2 2 2 2 2 3
266/300 Write block 1306 len 20480 data: 3 3 3 3 3
267/300 Read block 3958 len 20480
268/300 Read block 662 len 32768
269/300 Read block 2328 len 32768
270/300 Write block 4853 len 12288 data: 2 2 2
271/300 Write block 607 len 28672 data: 2 2 2 2 2 2 2
272/300 Read block 3670 len 4096
273/300 Write block 3314 len 28672 data: 2 2 2 2 2 2 3
274/300 Flush
275/300 Read block 2234 len 36864
276/300 Flush
277/300 Read block 4915 len 24576
278/300 Read block 2166 len 32768
279/300 Read block 1682 len 32768
280/300 Read block 3235 len 36864
281/300 Read block 1870 len 4096
282/300 Write block 531 len 32768 data: 2 2 2 2 2 2 2 2
283/300 Write block 1768 len 12288 data: 2 2 2
284/300 Flush
285/300 Write block 2947 len 40960 data: 2 2 2 2 2 2 2 2 2 2
286/300 Read block 3418 len 28672
287/300 Write block 1719 len 36864 data: 2 2 2 2 2 2 2 2 2
288/300 Write block 1234 len 24576 data: 3 3 3 3 3 3
289/300 Read block 4149 len 20480
290/300 Write block 930 len 16384 data: 2 2 2 2
291/300 Read block 4770 len 20480
292/300 Read block 950 len 4096
293/300 Read block 2236 len 16384
294/300 Write block 4509 len 40960 data: 2 2 2 2 2 2 2 2 2 2
295/300 Read block 3168 len 20480
296/300 Flush
297/300 Read block 149 len 12288
298/300 Read block 2831 len 24576
299/300 Write block 4563 len 12288 data: 3 3 3
300/300 Write block 2486 len 20480 data: 2 2 2 2 2
301/300 Read block 4987 len 16384
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:36:41.398 INFO request completed, latency_us: 195, response_code: 204, uri: /start/cid/0, method: POST, req_id: a0b80ea4-980c-47be-9e54-496b960c6e61, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1302 Acked Write 8 Sent Done Done false
1303 Acked Read 9 Sent Done Done false
1304 Acked Write 6 Sent Done Done false
1305 Acked Flush 0 Sent Done Done false
1306 Acked Write 4 Sent Done Done false
1307 Acked Read 3 Sent Done Done false
1308 Acked Read 8 Sent Done Done false
1309 Acked Write 6 Sent Done Done false
1310 Acked Write 9 Sent Done Done false
1311 Acked Read 8 Sent Done Done false
1312 Acked Write 2 Sent Done Done false
1313 Acked Write 3 Sent Done Done false
1314 Acked Read 5 Sent Done Done false
1315 Acked Read 9 Sent Done Done false
1316 Acked Read 7 Sent Done Done false
1317 Acked Write 7 Sent Done Done false
1318 Acked Read 8 Sent Done Done false
1319 Acked Flush 0 Sent Done Done false
1320 Acked Write 5 Sent Done Done false
1321 Acked Write 6 Sent Done Done false
1322 Acked Read 4 Sent Done Done false
1323 Acked Write 3 Sent Done Done false
1324 Acked Flush 0 Sent Done Done false
1325 Acked Write 5 Sent Done Done false
1326 Acked Flush 0 Sent Done Done false
1327 Acked Write 5 Sent Done Done false
1328 Acked Write 10 Sent Done Done false
1329 Acked Write 8 Sent Done Done false
1330 Acked Write 1 Sent Done Done false
1331 Acked Read 2 Sent Done Done false
1332 Acked Read 5 Sent Done Done false
1333 Acked Write 9 Sent Done Done false
1334 Acked Flush 0 Sent Done Done false
1335 Acked Write 3 Sent Done Done false
1336 Acked Write 4 Sent Done Done false
1337 Acked Read 4 Sent Done Done false
1338 Acked Flush 0 Sent Done Done false
1339 Acked Read 8 Sent Done Done false
1340 Acked Write 3 Sent Done Done false
1341 Acked Write 8 Sent Done Done false
1342 Acked Write 8 Sent Done Done false
1343 Acked Read 10 Sent Done Done false
1344 Acked Write 2 Sent Done Done false
1345 Acked Flush 0 Sent Done Done false
1346 Acked Read 9 Sent Done Done false
1347 Acked Write 4 Sent Done Done false
1348 Acked Read 3 Sent Done Done false
1349 Acked Write 7 Sent Done Done false
1350 Acked Write 3 Sent Done Done false
1351 Acked Read 4 Sent Done Done false
1352 Acked Write 7 Sent Done Done false
1353 Acked Read 4 Sent Done Done false
1354 Acked Read 1 Sent Done Done false
1355 Acked Write 7 Sent Done Done false
1356 Acked Read 1 Sent Done Done false
1357 Acked Read 6 Sent Done Done false
1358 Acked Write 7 Sent Done Done false
1359 Acked Read 5 Sent Done Done false
1360 Acked Read 1 Sent Done Done false
1361 Acked Read 10 Sent Done Done false
1362 Acked Read 1 Sent Done Done false
1363 Acked Write 2 Sent Done Done false
1364 Acked Flush 0 Sent Done Done false
1365 Acked Write 9 Sent Done Done false
1366 Acked Read 1 Sent Done Done false
1367 Acked Read 9 Sent Done Done false
1368 Acked Write 10 Sent Done Done false
1369 Acked Write 8 Sent Done Done false
1370 Acked Write 2 Sent Done Done false
1371 Acked Flush 0 Sent Done Done false
1372 Acked Read 2 Sent Done Done false
1373 Acked Read 1 Sent Done Done false
1374 Acked Flush 0 Sent Done Done false
1375 Acked Flush 0 Sent Done Done false
1376 Acked Read 4 Sent Done Done false
1377 Acked Write 1 Sent Done Done false
1378 Acked Flush 0 Sent Done Done false
1379 Acked Read 8 Sent Done Done false
1380 Acked Read 5 Sent Done Done false
1381 Acked Write 2 Sent Done Done false
1382 Acked Read 8 Sent Done Done false
1383 Acked Read 6 Sent Done Done false
1384 Acked Write 5 Sent Done Done false
1385 Acked Read 1 Sent Done Done false
1386 Acked Write 7 Sent Done Done false
1387 Acked Write 9 Sent Done Done false
1388 Acked Read 1 Sent Done Done false
1389 Acked Flush 0 Sent Done Done false
1390 Acked Read 10 Sent Done Done false
1391 Acked Read 7 Sent Done Done false
1392 Acked Write 2 Sent Done Done false
1393 Acked Flush 0 Sent Done Done false
1394 Acked Read 6 Sent Done Done false
1395 Acked Read 5 Sent Done Done false
1396 Acked Write 2 Sent Done Done false
1397 Acked Write 7 Sent Done Done false
1398 Acked Write 7 Sent Done Done false
1399 Acked Write 9 Sent Done Done false
1400 Acked Write 6 Sent Done Done false
1401 Acked Read 4 Sent Done Done false
1402 Acked Write 6 Sent Done Done false
1403 Acked Flush 0 Sent Done Done false
1404 Acked Read 10 Sent Done Done false
1405 Acked Flush 0 Sent Done Done false
1406 Acked Flush 0 Sent Done Done false
1407 Acked Read 5 Sent Done Done false
1408 Acked Write 6 Sent Done Done false
1409 Acked Write 3 Sent Done Done false
1410 Acked Read 4 Sent Done Done false
1411 Acked Write 6 Sent Done Done false
1412 Acked Read 3 Sent Done Done false
1413 Acked Write 2 Sent Done Done false
1414 Acked Read 8 Sent Done Done false
1415 Acked Read 8 Sent Done Done false
1416 Acked Read 4 Sent Done Done false
1417 Acked Read 6 Sent Done Done false
1418 Acked Write 7 Sent Done Done false
1419 Acked Read 4 Sent Done Done false
1420 Acked Write 6 Sent Done Done false
1421 Acked Read 5 Sent Done Done false
1422 Acked Write 3 Sent Done Done false
1423 Acked Write 8 Sent Done Done false
1424 Acked Write 5 Sent Done Done false
1425 Acked Read 6 Sent Done Done false
1426 Acked Flush 0 Sent Done Done false
1427 Acked Read 7 Sent Done Done false
1428 Acked Write 9 Sent Done Done false
1429 Acked Read 7 Sent Done Done false
1430 Acked Write 6 Sent Done Done false
1431 Acked Read 3 Sent Done Done false
1432 Acked Read 3 Sent Done Done false
1433 Acked Read 3 Sent Done Done false
1434 Acked Flush 0 Sent Done Done false
1435 Acked Write 3 Sent Done Done false
1436 Acked Write 7 Sent Done Done false
1437 Acked Read 1 Sent Done Done false
1438 Acked Read 6 Sent Done Done false
1439 Acked Read 3 Sent Done Done false
1440 Acked Write 3 Sent Done Done false
1441 Acked Read 6 Sent Done Done false
1442 Acked Write 6 Sent Done Done false
1443 Acked Read 4 Sent Done Done false
1444 Acked Read 2 Sent Done Done false
1445 Acked Write 7 Sent Done Done false
1446 Acked Read 7 Sent Done Done false
1447 Acked Write 8 Sent Done Done false
1448 Acked Read 5 Sent Done Done false
1449 Acked Write 6 Sent Done Done false
1450 Acked Write 7 Sent Done Done false
1451 Acked Read 8 Sent Done Done false
1452 Acked Write 9 Sent Done Done false
1453 Acked Read 1 Sent Done Done false
1454 Acked Read 2 Sent Done Done false
1455 Acked Flush 0 Sent Done Done false
1456 Acked Read 8 Sent Done Done false
1457 Acked Write 9 Sent Done Done false
1458 Acked Write 9 Sent Done Done false
1459 Acked Read 10 Sent Done Done false
1460 Acked Read 6 Sent Done Done false
1461 Acked Read 6 Sent Done Done false
1462 Acked Read 9 Sent Done Done false
1463 Acked Write 7 Sent Done Done false
1464 Acked Read 5 Sent Done Done false
1465 Acked Write 10 Sent Done Done false
1466 Acked Read 8 Sent Done Done false
1467 Acked Read 2 Sent Done Done false
1468 Acked Read 8 Sent Done Done false
1469 Acked Write 6 Sent Done Done false
1470 Acked Read 2 Sent Done Done false
1471 Acked Write 7 Sent Done Done false
1472 Acked Read 2 Sent Done Done false
1473 Acked Read 6 Sent Done Done false
1474 Acked Read 9 Sent Done Done false
1475 Acked Read 9 Sent Done Done false
1476 Acked Read 4 Sent Done Done false
1477 Acked Write 3 Sent Done Done false
1478 Acked Read 6 Sent Done Done false
1479 Acked Read 1 Sent Done Done false
1480 Acked Read 8 Sent Done Done false
1481 Acked Read 4 Sent Done Done false
1482 Acked Read 8 Sent Done Done false
1483 Acked Read 1 Sent Done Done false
1484 Acked Write 10 Sent Done Done false
1485 Acked Write 10 Sent Done Done false
1486 Acked Read 3 Sent Done Done false
1487 Acked Flush 0 Sent Done Done false
1488 Acked Read 10 Sent Done Done false
1489 Acked Flush 0 Sent Done Done false
1490 Acked Read 1 Sent Done Done false
1491 Acked Write 7 Sent Done Done false
1492 Acked Write 5 Sent Done Done false
1493 Acked Read 6 Sent Done Done false
1494 Acked Write 2 Sent Done Done false
1495 Acked Read 5 Sent Done Done false
1496 Acked Write 3 Sent Done Done false
1497 Acked Read 4 Sent Done Done false
1498 Acked Write 6 Sent Done Done false
1499 Acked Write 2 Sent Done Done false
1500 Acked Write 7 Sent Done Done false
1501 Acked Write 4 Sent Done Done false
1502 Acked Read 10 Sent Done Done false
1503 Acked Read 2 Sent Done Done false
1504 Acked Write 10 Sent Done Done false
1505 Acked Write 4 Sent Done Done false
1506 Acked Read 2 Sent Done Done false
1507 Acked Write 2 Sent Done Done false
1508 Acked Read 9 Sent Done Done false
1509 Acked Write 7 Sent Done Done false
1510 Acked Write 3 Sent Done Done false
1511 Acked Read 6 Sent Done Done false
1512 Acked Read 6 Sent Done Done false
1513 Acked Write 4 Sent Done Done false
1514 Acked Flush 0 Sent Done Done false
1515 Acked Flush 0 Sent Done Done false
1516 Acked Read 6 Sent Done Done false
1517 Acked Write 3 Sent Done Done false
1518 Acked Read 10 Sent Done Done false
1519 Acked Write 4 Sent Done Done false
1520 Acked Write 10 Sent Done Done false
1521 Acked Write 8 Sent Done Done false
1522 Acked Read 1 Sent Done Done false
1523 Acked Read 6 Sent Done Done false
1524 Acked Write 10 Sent Done Done false
1525 Acked Read 2 Sent Done Done false
1526 Acked Read 2 Sent Done Done false
1527 Acked Write 7 Sent Done Done false
1528 Acked Write 10 Sent Done Done false
1529 Acked Read 5 Sent Done Done false
1530 Acked Read 1 Sent Done Done false
1531 Acked Read 3 Sent Done Done false
1532 Acked Write 10 Sent Done Done false
1533 Acked Flush 0 Sent Done Done false
1534 Acked Read 3 Sent Done Done false
1535 Acked Write 4 Sent Done Done false
1536 Acked Read 7 Sent Done Done false
1537 Acked Read 5 Sent Done Done false
1538 Acked Read 4 Sent Done Done false
1539 Acked Read 10 Sent Done Done false
1540 Acked Write 2 Sent Done Done false
1541 Acked Read 8 Sent Done Done false
1542 Acked Write 1 Sent Done Done false
1543 Acked Write 1 Sent Done Done false
1544 Acked Flush 0 Sent Done Done false
1545 Acked Read 6 Sent Done Done false
1546 Acked Read 6 Sent Done Done false
1547 Acked Write 3 Sent Done Done false
1548 Acked Flush 0 Sent Done Done false
1549 Acked Read 8 Sent Done Done false
1550 Acked Read 8 Sent Done Done false
1551 Acked Read 5 Sent Done Done false
1552 Acked Read 5 Sent Done Done false
1553 Acked Write 3 Sent Done Done false
1554 Acked Read 8 Sent Done Done false
1555 Acked Read 10 Sent Done Done false
1556 Acked Write 6 Sent Done Done false
1557 Acked Write 2 Sent Done Done false
1558 Acked Write 8 Sent Done Done false
1559 Acked Read 2 Sent Done Done false
1560 Acked Read 5 Sent Done Done false
1561 Acked Write 8 Sent Done Done false
1562 Acked Write 5 Sent Done Done false
1563 Acked Write 5 Sent Done Done false
1564 Acked Write 7 Sent Done Done false
1565 Acked Flush 0 Sent Done Done false
1566 Acked Write 9 Sent Done Done false
1567 Acked Write 5 Sent Done Done false
1568 Acked Read 5 Sent Done Done false
1569 Acked Read 8 Sent Done Done false
1570 Acked Read 8 Sent Done Done false
1571 Acked Write 3 Sent Done Done false
1572 Acked Write 7 Sent Done Done false
1573 Acked Read 1 Sent Done Done false
1574 Acked Write 7 Sent Done Done false
1575 Acked Flush 0 Sent Done Done false
1576 Acked Read 9 Sent Done Done false
1577 Acked Flush 0 Sent Done Done false
1578 Acked Read 6 Sent Done Done false
1579 Acked Read 8 Sent Done Done false
1580 Acked Read 8 Sent Done Done false
1581 Acked Read 9 Sent Done Done false
1582 Acked Read 1 Sent Done Done false
1583 Acked Write 8 Sent Done Done false
1584 Acked Write 3 Sent Done Done false
1585 Acked Flush 0 Sent Done Done false
1586 Acked Write 10 Sent Done Done false
1587 Acked Read 7 Sent Done Done false
1588 Acked Write 9 Sent Done Done false
1589 Acked Write 6 Sent Done Done false
1590 Acked Read 5 Sent Done Done false
1591 Acked Write 4 Sent Done Done false
1592 Acked Read 5 Sent Done Done false
1593 Acked Read 1 Sent Done Done false
1594 Acked Read 4 Sent Done Done false
1595 Acked Write 10 Sent Done Done false
1596 Acked Read 5 Sent Done Done false
1597 Acked Flush 0 Sent Done Done false
1598 Acked Read 3 Sent Done Done false
1599 Acked Read 6 Sent Done Done false
1600 Acked Write 3 Sent Done Done false
1601 Acked Write 5 Sent Done Done false
1602 Acked Read 4 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1301 1597 1597
Downstairs last five completed: 1301 1300 1299 1298 1297
Upstairs last five completed: 1602 1601 1600 1599 1598
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1202)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1302 Acked Write 8 Sent Done Done false
1303 Acked Read 9 Sent Done Done false
1304 Acked Write 6 Sent Done Done false
1305 Acked Flush 0 Sent Done Done false
1306 Acked Write 4 Sent Done Done false
1307 Acked Read 3 Sent Done Done false
1308 Acked Read 8 Sent Done Done false
1309 Acked Write 6 Sent Done Done false
1310 Acked Write 9 Sent Done Done false
1311 Acked Read 8 Sent Done Done false
1312 Acked Write 2 Sent Done Done false
1313 Acked Write 3 Sent Done Done false
1314 Acked Read 5 Sent Done Done false
1315 Acked Read 9 Sent Done Done false
1316 Acked Read 7 Sent Done Done false
1317 Acked Write 7 Sent Done Done false
1318 Acked Read 8 Sent Done Done false
1319 Acked Flush 0 Sent Done Done false
1320 Acked Write 5 Sent Done Done false
1321 Acked Write 6 Sent Done Done false
1322 Acked Read 4 Sent Done Done false
1323 Acked Write 3 Sent Done Done false
1324 Acked Flush 0 Sent Done Done false
1325 Acked Write 5 Sent Done Done false
1326 Acked Flush 0 Sent Done Done false
1327 Acked Write 5 Sent Done Done false
1328 Acked Write 10 Sent Done Done false
1329 Acked Write 8 Sent Done Done false
1330 Acked Write 1 Sent Done Done false
1331 Acked Read 2 Sent Done Done false
1332 Acked Read 5 Sent Done Done false
1333 Acked Write 9 Sent Done Done false
1334 Acked Flush 0 Sent Done Done false
1335 Acked Write 3 Sent Done Done false
1336 Acked Write 4 Sent Done Done false
1337 Acked Read 4 Sent Done Done false
1338 Acked Flush 0 Sent Done Done false
1339 Acked Read 8 Sent Done Done false
1340 Acked Write 3 Sent Done Done false
1341 Acked Write 8 Sent Done Done false
1342 Acked Write 8 Sent Done Done false
1343 Acked Read 10 Sent Done Done false
1344 Acked Write 2 Sent Done Done false
1345 Acked Flush 0 Sent Done Done false
1346 Acked Read 9 Sent Done Done false
1347 Acked Write 4 Sent Done Done false
1348 Acked Read 3 Sent Done Done false
1349 Acked Write 7 Sent Done Done false
1350 Acked Write 3 Sent Done Done false
1351 Acked Read 4 Sent Done Done false
1352 Acked Write 7 Sent Done Done false
1353 Acked Read 4 Sent Done Done false
1354 Acked Read 1 Sent Done Done false
1355 Acked Write 7 Sent Done Done false
1356 Acked Read 1 Sent Done Done false
1357 Acked Read 6 Sent Done Done false
1358 Acked Write 7 Sent Done Done false
1359 Acked Read 5 Sent Done Done false
1360 Acked Read 1 Sent Done Done false
1361 Acked Read 10 Sent Done Done false
1362 Acked Read 1 Sent Done Done false
1363 Acked Write 2 Sent Done Done false
1364 Acked Flush 0 Sent Done Done false
1365 Acked Write 9 Sent Done Done false
1366 Acked Read 1 Sent Done Done false
1367 Acked Read 9 Sent Done Done false
1368 Acked Write 10 Sent Done Done false
1369 Acked Write 8 Sent Done Done false
1370 Acked Write 2 Sent Done Done false
1371 Acked Flush 0 Sent Done Done false
1372 Acked Read 2 Sent Done Done false
1373 Acked Read 1 Sent Done Done false
1374 Acked Flush 0 Sent Done Done false
1375 Acked Flush 0 Sent Done Done false
1376 Acked Read 4 Sent Done Done false
1377 Acked Write 1 Sent Done Done false
1378 Acked Flush 0 Sent Done Done false
1379 Acked Read 8 Sent Done Done false
1380 Acked Read 5 Sent Done Done false
1381 Acked Write 2 Sent Done Done false
1382 Acked Read 8 Sent Done Done false
1383 Acked Read 6 Sent Done Done false
1384 Acked Write 5 Sent Done Done false
1385 Acked Read 1 Sent Done Done false
1386 Acked Write 7 Sent Done Done false
1387 Acked Write 9 Sent Done Done false
1388 Acked Read 1 Sent Done Done false
1389 Acked Flush 0 Sent Done Done false
1390 Acked Read 10 Sent Done Done false
1391 Acked Read 7 Sent Done Done false
1392 Acked Write 2 Sent Done Done false
1393 Acked Flush 0 Sent Done Done false
1394 Acked Read 6 Sent Done Done false
1395 Acked Read 5 Sent Done Done false
1396 Acked Write 2 Sent Done Done false
1397 Acked Write 7 Sent Done Done false
1398 Acked Write 7 Sent Done Done false
1399 Acked Write 9 Sent Done Done false
1400 Acked Write 6 Sent Done Done false
1401 Acked Read 4 Sent Done Done false
1402 Acked Write 6 Sent Done Done false
1403 Acked Flush 0 Sent Done Done false
1404 Acked Read 10 Sent Done Done false
1405 Acked Flush 0 Sent Done Done false
1406 Acked Flush 0 Sent Done Done false
1407 Acked Read 5 Sent Done Done false
1408 Acked Write 6 Sent Done Done false
1409 Acked Write 3 Sent Done Done false
1410 Acked Read 4 Sent Done Done false
1411 Acked Write 6 Sent Done Done false
1412 Acked Read 3 Sent Done Done false
1413 Acked Write 2 Sent Done Done false
1414 Acked Read 8 Sent Done Done false
1415 Acked Read 8 Sent Done Done false
1416 Acked Read 4 Sent Done Done false
1417 Acked Read 6 Sent Done Done false
1418 Acked Write 7 Sent Done Done false
1419 Acked Read 4 Sent Done Done false
1420 Acked Write 6 Sent Done Done false
1421 Acked Read 5 Sent Done Done false
1422 Acked Write 3 Sent Done Done false
1423 Acked Write 8 Sent Done Done false
1424 Acked Write 5 Sent Done Done false
1425 Acked Read 6 Sent Done Done false
1426 Acked Flush 0 Sent Done Done false
1427 Acked Read 7 Sent Done Done false
1428 Acked Write 9 Sent Done Done false
1429 Acked Read 7 Sent Done Done false
1430 Acked Write 6 Sent Done Done false
1431 Acked Read 3 Sent Done Done false
1432 Acked Read 3 Sent Done Done false
1433 Acked Read 3 Sent Done Done false
1434 Acked Flush 0 Sent Done Done false
1435 Acked Write 3 Sent Done Done false
1436 Acked Write 7 Sent Done Done false
1437 Acked Read 1 Sent Done Done false
1438 Acked Read 6 Sent Done Done false
1439 Acked Read 3 Sent Done Done false
1440 Acked Write 3 Sent Done Done false
1441 Acked Read 6 Sent Done Done false
1442 Acked Write 6 Sent Done Done false
1443 Acked Read 4 Sent Done Done false
1444 Acked Read 2 Sent Done Done false
1445 Acked Write 7 Sent Done Done false
1446 Acked Read 7 Sent Done Done false
1447 Acked Write 8 Sent Done Done false
1448 Acked Read 5 Sent Done Done false
1449 Acked Write 6 Sent Done Done false
1450 Acked Write 7 Sent Done Done false
1451 Acked Read 8 Sent Done Done false
1452 Acked Write 9 Sent Done Done false
1453 Acked Read 1 Sent Done Done false
1454 Acked Read 2 Sent Done Done false
1455 Acked Flush 0 Sent Done Done false
1456 Acked Read 8 Sent Done Done false
1457 Acked Write 9 Sent Done Done false
1458 Acked Write 9 Sent Done Done false
1459 Acked Read 10 Sent Done Done false
1460 Acked Read 6 Sent Done Done false
1461 Acked Read 6 Sent Done Done false
1462 Acked Read 9 Sent Done Done false
1463 Acked Write 7 Sent Done Done false
1464 Acked Read 5 Sent Done Done false
1465 Acked Write 10 Sent Done Done false
1466 Acked Read 8 Sent Done Done false
1467 Acked Read 2 Sent Done Done false
1468 Acked Read 8 Sent Done Done false
1469 Acked Write 6 Sent Done Done false
1470 Acked Read 2 Sent Done Done false
1471 Acked Write 7 Sent Done Done false
1472 Acked Read 2 Sent Done Done false
1473 Acked Read 6 Sent Done Done false
1474 Acked Read 9 Sent Done Done false
1475 Acked Read 9 Sent Done Done false
1476 Acked Read 4 Sent Done Done false
1477 Acked Write 3 Sent Done Done false
1478 Acked Read 6 Sent Done Done false
1479 Acked Read 1 Sent Done Done false
1480 Acked Read 8 Sent Done Done false
1481 Acked Read 4 Sent Done Done false
1482 Acked Read 8 Sent Done Done false
1483 Acked Read 1 Sent Done Done false
1484 Acked Write 10 Sent Done Done false
1485 Acked Write 10 Sent Done Done false
1486 Acked Read 3 Sent Done Done false
1487 Acked Flush 0 Sent Done Done false
1488 Acked Read 10 Sent Done Done false
1489 Acked Flush 0 Sent Done Done false
1490 Acked Read 1 Sent Done Done false
1491 Acked Write 7 Sent Done Done false
1492 Acked Write 5 Sent Done Done false
1493 Acked Read 6 Sent Done Done false
1494 Acked Write 2 Sent Done Done false
1495 Acked Read 5 Sent Done Done false
1496 Acked Write 3 Sent Done Done false
1497 Acked Read 4 Sent Done Done false
1498 Acked Write 6 Sent Done Done false
1499 Acked Write 2 Sent Done Done false
1500 Acked Write 7 Sent Done Done false
1501 Acked Write 4 Sent Done Done false
1502 Acked Read 10 Sent Done Done false
1503 Acked Read 2 Sent Done Done false
1504 Acked Write 10 Sent Done Done false
1505 Acked Write 4 Sent Done Done false
1506 Acked Read 2 Sent Done Done false
1507 Acked Write 2 Sent Done Done false
1508 Acked Read 9 Sent Done Done false
1509 Acked Write 7 Sent Done Done false
1510 Acked Write 3 Sent Done Done false
1511 Acked Read 6 Sent Done Done false
1512 Acked Read 6 Sent Done Done false
1513 Acked Write 4 Sent Done Done false
1514 Acked Flush 0 Sent Done Done false
1515 Acked Flush 0 Sent Done Done false
1516 Acked Read 6 Sent Done Done false
1517 Acked Write 3 Sent Done Done false
1518 Acked Read 10 Sent Done Done false
1519 Acked Write 4 Sent Done Done false
1520 Acked Write 10 Sent Done Done false
1521 Acked Write 8 Sent Done Done false
1522 Acked Read 1 Sent Done Done false
1523 Acked Read 6 Sent Done Done false
1524 Acked Write 10 Sent Done Done false
1525 Acked Read 2 Sent Done Done false
1526 Acked Read 2 Sent Done Done false
1527 Acked Write 7 Sent Done Done false
1528 Acked Write 10 Sent Done Done false
1529 Acked Read 5 Sent Done Done false
1530 Acked Read 1 Sent Done Done false
1531 Acked Read 3 Sent Done Done false
1532 Acked Write 10 Sent Done Done false
1533 Acked Flush 0 Sent Done Done false
1534 Acked Read 3 Sent Done Done false
1535 Acked Write 4 Sent Done Done false
1536 Acked Read 7 Sent Done Done false
1537 Acked Read 5 Sent Done Done false
1538 Acked Read 4 Sent Done Done false
1539 Acked Read 10 Sent Done Done false
1540 Acked Write 2 Sent Done Done false
1541 Acked Read 8 Sent Done Done false
1542 Acked Write 1 Sent Done Done false
1543 Acked Write 1 Sent Done Done false
1544 Acked Flush 0 Sent Done Done false
1545 Acked Read 6 Sent Done Done false
1546 Acked Read 6 Sent Done Done false
1547 Acked Write 3 Sent Done Done false
1548 Acked Flush 0 Sent Done Done false
1549 Acked Read 8 Sent Done Done false
1550 Acked Read 8 Sent Done Done false
1551 Acked Read 5 Sent Done Done false
1552 Acked Read 5 Sent Done Done false
1553 Acked Write 3 Sent Done Done false
1554 Acked Read 8 Sent Done Done false
1555 Acked Read 10 Sent Done Done false
1556 Acked Write 6 Sent Done Done false
1557 Acked Write 2 Sent Done Done false
1558 Acked Write 8 Sent Done Done false
1559 Acked Read 2 Sent Done Done false
1560 Acked Read 5 Sent Done Done false
1561 Acked Write 8 Sent Done Done false
1562 Acked Write 5 Sent Done Done false
1563 Acked Write 5 Sent Done Done false
1564 Acked Write 7 Sent Done Done false
1565 Acked Flush 0 Sent Done Done false
1566 Acked Write 9 Sent Done Done false
1567 Acked Write 5 Sent Done Done false
1568 Acked Read 5 Sent Done Done false
1569 Acked Read 8 Sent Done Done false
1570 Acked Read 8 Sent Done Done false
1571 Acked Write 3 Sent Done Done false
1572 Acked Write 7 Sent Done Done false
1573 Acked Read 1 Sent Done Done false
1574 Acked Write 7 Sent Done Done false
1575 Acked Flush 0 Sent Done Done false
1576 Acked Read 9 Sent Done Done false
1577 Acked Flush 0 Sent Done Done false
1578 Acked Read 6 Sent Done Done false
1579 Acked Read 8 Sent Done Done false
1580 Acked Read 8 Sent Done Done false
1581 Acked Read 9 Sent Done Done false
1582 Acked Read 1 Sent Done Done false
1583 Acked Write 8 Sent Done Done false
1584 Acked Write 3 Sent Done Done false
1585 Acked Flush 0 Sent Done Done false
1586 Acked Write 10 Sent Done Done false
1587 Acked Read 7 Sent Done Done false
1588 Acked Write 9 Sent Done Done false
1589 Acked Write 6 Sent Done Done false
1590 Acked Read 5 Sent Done Done false
1591 Acked Write 4 Sent Done Done false
1592 Acked Read 5 Sent Done Done false
1593 Acked Read 1 Sent Done Done false
1594 Acked Read 4 Sent Done Done false
1595 Acked Write 10 Sent Done Done false
1596 Acked Read 5 Sent Done Done false
1597 Acked Flush 0 Sent Done Done false
1598 Acked Read 3 Sent Done Done false
1599 Acked Read 6 Sent Done Done false
1600 Acked Write 3 Sent Done Done false
1601 Acked Write 5 Sent Done Done false
1602 Acked Read 4 Sent Done Done false
1603 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1301 1603 1603
Downstairs last five completed: 1301 1300 1299 1298 1297
Upstairs last five completed: 1603 1602 1601 1600 1599
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.162466189Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33893)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.162813858Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.163800543Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.164437518Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.164458206Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 1301","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.164472505Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 1301","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.164757411Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 1301","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.164777369Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 1301","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:47.165088922Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1603 1602 1601 1600 1599
Upstairs last five completed: 1603 1602 1601 1600 1599
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:36:49.403 INFO request completed, latency_us: 315[8830] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d7d16f20-7f19-4cc3-8d0b-02182f0f4a84, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:36:49.404 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/2, method: GET, req_id: 319d1c4b-4c75-4ef3-8a8f-8c654d7db738, remote_addr: 127.0.0.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:49.404146486Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:49.404216517Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:49.404252453Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:49.404337363Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:36:53.406 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/2, method: GET, req_id: 128e64e5-6436-4f1b-baab-06fe59abe7a4, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 265 len 36864 data: 2 2 2 2 2 2 2 2 2
002/300 Read block 4297 len 4096
003/300 Flush
004/300 Flush
005/300 Read block 1619 len 36864
006/300 Read block 2484 len 24576
007/300 Read block 196 len 8192
008/300 Read block 1366 len 16384
009/300 Write block 4704 len 40960 data: 2 2 2 2 2 3 3 3 3 2
010/300 Read block 631 len 4096
011/300 Read block 776 len 20480
012/300 Read block 1002 len 40960
013/300 Write block 4621 len 24576 data: 2 2 2 2 2 2
014/300 Write block 192 len 40960 data: 3 3 3 3 2 2 2 2 2 2
015/300 Write block 4803 len 36864 data: 3 3 3 2 2 2 2 3 3
016/300 Read block 3601 len 40960
017/300 Read block 2044 len 12288
018/300 Read block 2967 len 20480
019/300 Flush
020/300 Write block 3665 len 40960 data: 2 2 2 2 2 2 2 2 2 2
021/300 Write block 4970 len 8192 data: 2 2
022/300 Read block 2206 len 20480
023/300 Write block 2663 len 12288 data: 2 2 2
024/300 Read block 3191 len 12288
025/300 Write block 4189 len 32768 data: 3 3 3 3 3 3 3 4
026/300 Read block 2748 len 12288
027/300 Write block 2096 len 4096 data: 2
028/300 Write block 3397 len 28672 data: 2 2 2 2 2 2 3
029/300 Read block 4431 len 36864
030/300 Write block 2186 len 8192 data: 2 2
031/300 Write block 474 len 40960 data: 2 2 2 2 2 2 2 2 2 2
032/300 Read block 1147 len 16384
033/300 Read block 2543 len 12288
034/300 Flush
035/300 Flush
036/300 Write block 317 len 20480 data: 2 2 2 3 3
037/300 Flush
038/300 Read block 2405 len 24576
039/300 Read block 1971 len 40960
040/300 Read block 4881 len 8192
041/300 Write block 1772 len 40960 data: 2 2 2 2 2 2 2 2 2 2
042/300 Read block 4413 len 28672
043/300 Write block 3019 len 28672 data: 2 2 2 2 2 2 2
044/300 Read block 656 len 12288
045/300 Read block 1667 len 40960
046/300 Read block 3338 len 4096
047/300 Read block 3342 len 24576
048/300 Read block 1202 len 36864
049/300 Write block 48 len 28672 data: 3 3 3 2 2 2 2
050/300 Write block 4229 len 40960 data: 2 2 2 2 2 2 2 2 2 3
051/300 Write block 1597 len 28672 data: 2 2 2 2 2 2 2
052/300 Read block 152 len 8192
053/300 Read block 891 len 32768
054/300 Read block 1851 len 36864
055/300 Read block 3138 len 32768
056/300 Read block 2809 len 8192
057/300 Read block 2433 len 4096
058/300 Read block 1164 len 8192
059/300 Read block 1554 len 32768
060/300 Read block 629 len 20480
061/300 Flush
062/300 Read block 1653 len 12288
063/300 Read block 1894 len 20480
064/300 Read block 2383 len 28672
065/300 Flush
066/300 Write block 3918 len 8192 data: 2 2
067/300 Flush
068/300 Read block 3264 len 20480
069/300 Read block 117 len 40960
070/300 Read block 4022 len 20480
071/300 Read block 1719 len 24576
072/300 Read block 4054 len 28672
073/300 Read block 1008 len 24576
074/300 Write block 974 len 28672 data: 2 2 2 2 2 2 2
075/300 Flush
076/300 Read block 4013 len 24576
077/300 Write block 2616 len 36864 data: 2 2 2 2 2 2 2 2 2
078/300 Read block 3531 len 28672
079/300 Write block 1329 len 20480 data: 2 2 2 2 2
080/300 Read block 1007 len 8192
081/300 Read block 4423 len 32768
082/300 Read block 833 len 32768
083/300 Read block 1590 len 28672
084/300 Write block 1018 len 16384 data: 3 3 3 2
085/300 Write block 1923 len 28672 data: 2 3 3 3 2 2 2
086/300 Write block 3325 len 8192 data: 3 3
087/300 Read block 981 len 32768
088/300 Read block 1618 len 4096
089/300 Read block 3875 len 20480
090/300 Read block 3961 len 16384
091/300 Write block 1251 len 12288 data: 2 2 2
092/300 Read block 3254 len 16384
093/300 Read block 2677 len 40960
094/300 Read block 2059 len 36864
095/300 Write block 4100 len 32768 data: 2 2 2 2 2 2 2 3
096/300 Write block 1070 len 24576 data: 2 2 2 2 2 2
097/300 Flush
098/300 Read block 3334 len 12288
099/300 Read block 3019 len 8192
100/300 Write block 3599 len 8192 data: 2 2
101/300 Read block 1493 len 32768
102/300 Write block 4521 len 40960 data: 3 3 2 2 2 3 3 3 2 2
103/300 Read block 4951 len 12288
104/300 Write block 3936 len 24576 data: 2 2 2 2 2 2
105/300 Read block 2130 len 40960
106/300 Write block 694 len 32768 data: 2 2 2 2 2 2 2 2
107/300 Read block 409 len 40960
108/300 Flush
109/300 Read block 2484 len 4096
110/300 Read block 518 len 32768
111/300 Read block 2308 len 12288
112/300 Read block 3904 len 4096
113/300 Write block 1527 len 4096 data: 2
114/300 Write block 3038 len 36864 data: 3 3 3 3 2 2 2 2 2
115/300 Read block 2568 len 12288
116/300 Write block 1095 len 28672 data: 2 2 2 2 2 2 2
117/300 Write block 4797 len 36864 data: 3 2 3 3 3 3 4 4 4
118/300 Write block 2057 len 16384 data: 2 2 2 2
119/300 Write block 3773 len 28672 data: 4 3 3 3 3 3 2
120/300 Write block 401 len 40960 data: 2 2 2 2 2 2 2 2 2 2
121/300 Read block 4091 len 28672
122/300 Write block 2184 len 16384 data: 3 2 3 3
123/300 Read block 2585 len 36864
124/300 Flush
125/300 Read block 3055 len 16384
126/300 Flush
127/300 Flush
128/300 Write block 445 len 4096 data: 2
129/300 Read block 4843 len 4096
130/300 Read block 4414 len 28672
131/300 Write block 1974 len 20480 data: 2 2 2 2 2
132/300 Flush
133/300 Write block 3499 len 8192 data: 2 2
134/300 Write block 62 len 36864 data: 2 2 2 2 2 2 2 2 2
135/300 Read block 962 len 16384
136/300 Read block 840 len 32768
137/300 Read block 1145 len 12288
138/300 Flush
139/300 Write block 4198 len 8192 data: 3 2
140/300 Read block 1477 len 36864
141/300 Read block 1417 len 40960
142/300 Read block 469 len 20480
143/300 Write block 603 len 28672 data: 4 4 4 2 3 3 3
144/300 Read block 1750 len 20480
145/300 Read block 2229 len 32768
146/300 Read block 955 len 20480
147/300 Write block 1855 len 20480 data: 3 2 2 2 2
148/300 Write block 4601 len 40960 data: 2 2 2 2 2 2 2 2 2 2
149/300 Flush
150/300 Read block 4298 len 8192
151/300 Read block 4740 len 24576
152/300 Write block 507 len 20480 data: 2 2 2 2 2
153/300 Write block 4001 len 40960 data: 2 2 2 2 2 2 2 3 2 2
154/300 Read block 3204 len 36864
155/300 Write block 4169 len 40960 data: 2 3 3 3 4 3 3 3 3 3
156/300 Write block 2718 len 20480 data: 2 2 2 2 2
157/300 Write block 4713 len 20480 data: 3 3 3 3 3
158/300 Write block 1852 len 40960 data: 3 3 3 4 3 3 3 3 2 2
159/300 Read block 4228 len 40960
160/300 Read block 2002 len 8192
161/300 Write block 3449 len 36864 data: 2 2 2 2 2 2 3 3 3
162/300 Read block 1862 len 16384
163/300 Flush
164/300 Read block 831 len 16384
165/300 Flush
166/300 Write block 2765 len 32768 data: 2 2 3 3 3 3 3 3
167/300 Write block 2635 len 28672 data: 3 3 3 4 4 4 4
168/300 Read block 3494 len 4096
169/300 Write block 2059 len 16384 data: 3 3 2 2
170/300 Read block 173 len 20480
171/300 Write block 3504 len 8192 data: 2 2
172/300 Read block 1285 len 12288
173/300 Read block 3676 len 20480
174/300 Flush
175/300 Read block 4922 len 4096
176/300 Flush
177/300 Read block 181 len 24576
178/300 Write block 3665 len 36864 data: 3 3 3 3 3 3 3 3 3
179/300 Read block 2374 len 16384
180/300 Read block 3486 len 4096
181/300 Flush
182/300 Read block 3516 len 20480
183/300 Read block 591 len 4096
184/300 Flush
185/300 Write block 4361 len 36864 data: 2 2 2 2 2 2 2 2 2
186/300 Read block 1187 len 8192
187/300 Read block 2771 len 40960
188/300 Flush
189/300 Write block 1376 len 32768 data: 5 5 4 4 4 3 3 3
190/300 Read block 1647 len 28672
191/300 Write block 280 len 28672 data: 3 3 3 2 2 2 2
192/300 Read block 2198 len 24576
193/300 Write block 3952 len 32768 data: 2 2 2 2 2 2 2 2
194/300 Read block 3637 len 32768
195/300 Read block 775 len 24576
196/300 Read block 2082 len 32768
197/300 Read block 3409 len 12288
198/300 Read block 758 len 24576
199/300 Read block 2942 len 8192
200/300 Read block 2838 len 32768
201/300 Flush
202/300 Write block 3332 len 4096 data: 2
203/300 Read block 4271 len 12288
204/300 Read block 2557 len 8192
205/300 Write block 4353 len 4096 data: 2
206/300 Write block 982 len 32768 data: 2 2 2 2 2 2 2 2
207/300 Flush
208/300 Write block 4908 len 16384 data: 2 2 2 2
209/300 Read block 3683 len 8192
210/300 Read block 3565 len 40960
211/300 Read block 2068 len 40960
212/300 Write block 3239 len 16384 data: 2 2 2 2
213/300 Flush
214/300 Read block 4018 len 28672
215/300 Flush
216/300 Read block 3992 len 4096
217/300 Write block 351 len 32768 data: 2 2 2 2 2 2 2 2
218/300 Read block 2250 len 12288
219/300 Read block 2455 len 36864
220/300 Write block 4118 len 24576 data: 3 3 3 3 3 2
221/300 Read block 2421 len 32768
222/300 Write block 653 len 4096 data: 2
223/300 Read block 3080 len 4096
224/300 Write block 4875 len 40960 data: 2 2 2 2 2 2 2 2 2 2
225/300 Read block 1862 len 32768
226/300 Read block 3333 len 24576
227/300 Write block 1332 len 36864 data: 3 3 2 2 2 2 2 2 2
228/300 Write block 3136 len 32768 data: 3 2 2 2 2 2 2 2
229/300 Write block 3243 len 4096 data: 2
230/300 Read block 2112 len 16384
231/300 Write block 4681 len 32768 data: 3 2 2 2 2 2 2 2
232/300 Read block 2371 len 20480
233/300 Write block 871 len 36864 data: 2 2 2 2 2 2 2 2 2
234/300 Read block 1139 len 32768
235/300 Flush
236/300 Write block 2064 len 28672 data: 2 2 2 2 3 2 2
237/300 Read block 3988 len 8192
238/300 Read block 3291 len 12288
239/300 Read block 3561 len 28672
240/300 Write block 2285 len 32768 data: 3 3 3 4 4 4 3 3
241/300 Write block 1985 len 20480 data: 3 3 3 3 2
242/300 Read block 2441 len 8192
243/300 Write block 4165 len 4096 data: 3
244/300 Write block 1184 len 8192 data: 2 2
245/300 Write block 713 len 16384 data: 5 5 4 3
246/300 Read block 243 len 28672
247/300 Write block 1555 len 24576 data: 2 2 2 2 2 2
248/300 Read block 2710 len 20480
249/300 Write block 4413 len 28672 data: 2 2 2 2 2 2 2
250/300 Flush
251/300 Write block 2862 len 24576 data: 2 2 2 2 2 2
252/300 Flush
253/300 Read block 2054 len 8192
254/300 Read block 4030 len 40960
255/300 Read block 4293 len 12288
256/300 Read block 237 len 8192
257/300 Write block 1470 len 16384 data: 2 2 2 2
258/300 Flush
259/300 Flush
260/300 Write block 1672 len 20480 data: 2 2 2 2 2
261/300 Write block 3902 len 32768 data: 2 2 5 4 4 4 5 5
262/300 Read block 4970 len 40960
263/300 Write block 1057 len 20480 data: 2 2 2 2 2
264/300 Write block 4141 len 40960 data: 3 3 3 2 2 2 2 2 2 2
265/300 Write block 1473 len 16384 data: 3 2 2 2
266/300 Flush
267/300 Read block 4310 len 24576
268/300 Read block 4852 len 28672
269/300 Write block 3861 len 32768 data: 2 2 2 2 2 2 2 2
270/300 Write block 2402 len 8192 data: 2 2
271/300 Read block 2970 len 24576
272/300 Write block 439 len 20480 data: 2 2 2 2 2
273/300 Write block 4764 len 8192 data: 2 2
274/300 Read block 3384 len 36864
275/300 Read block 2705 len 4096
276/300 Read block 3447 len 40960
277/300 Read block 2854 len 24576
278/300 Read block 56 len 40960
279/300 Read block 3364 len 24576
280/300 Flush
281/300 Read block 3302 len 40960
282/300 Read block 4386 len 8192
283/300 Write block 3326 len 28672 data: 4 2 2 2 2 2 3
284/300 Write block 3057 len 20480 data: 2 2 2 2 2
285/300 Read block 4919 len 28672
286/300 Write block 3453 len 40960 data: 3 3 4 4 4 2 2 2 2 2
287/300 Write block 2048 len 16384 data: 3 2 2 2
288/300 Write block 1223 len 16384 data: 3 3 3 2
289/300 Read block 4307 len 28672
290/300 Read block 3608 len 32768
291/300 Read block 3010 len 8192
292/300 Read block 3321 len 40960
293/300 Read block 4930 len 28672
294/300 Read block 801 len 16384
295/300 Read block 3255 len 20480
296/300 Write block 1201 len 28672 data: 2 2 2 2 2 2 3
297/300 Read block 1736 len 12288
298/300 Write block 2144 len 32768 data: 2 2 2 2 2 2 2 2
299/300 Read block 1453 len 32768
300/300 Read block 2099 len 16384
301/300 Write block 2296 len 4096 data: 2
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:36:53.650 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/2, method: POST, req_id: f6dc7369-aef6-4af2-a4c1-a449a20809e9, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1604 Acked Write 9 Done Done Sent false
1605 Acked Read 1 Done Done Sent false
1606 Acked Flush 0 Done Done Sent false
1607 Acked Flush 0 Done Done Sent false
1608 Acked Read 9 Done Done Sent false
1609 Acked Read 6 Done Done Sent false
1610 Acked Read 2 Done Done Sent false
1611 Acked Read 4 Done Done Sent false
1612 Acked Write 10 Done Done Sent false
1613 Acked Read 1 Done Done Sent false
1614 Acked Read 5 Done Done Sent false
1615 Acked Read 10 Done Done Sent false
1616 Acked Write 6 Done Done Sent false
1617 Acked Write 10 Done Done Sent false
1618 Acked Write 9 Done Done Sent false
1619 Acked Read 10 Done Done Sent false
1620 Acked Read 3 Done Done Sent false
1621 Acked Read 5 Done Done Sent false
1622 Acked Flush 0 Done Done Sent false
1623 Acked Write 10 Done Done Sent false
1624 Acked Write 2 Done Done Sent false
1625 Acked Read 5 Done Done Sent false
1626 Acked Write 3 Done Done Sent false
1627 Acked Read 3 Done Done Sent false
1628 Acked Write 8 Done Done Sent false
1629 Acked Read 3 Done Done Sent false
1630 Acked Write 1 Done Done Sent false
1631 Acked Write 7 Done Done Sent false
1632 Acked Read 9 Done Done Sent false
1633 Acked Write 2 Done Done Sent false
1634 Acked Write 10 Done Done Sent false
1635 Acked Read 4 Done Done Sent false
1636 Acked Read 3 Done Done Sent false
1637 Acked Flush 0 Done Done Sent false
1638 Acked Flush 0 Done Done Sent false
1639 Acked Write 5 Done Done Sent false
1640 Acked Flush 0 Done Done Sent false
1641 Acked Read 6 Done Done Sent false
1642 Acked Read 10 Done Done Sent false
1643 Acked Read 2 Done Done Sent false
1644 Acked Write 10 Done Done Sent false
1645 Acked Read 7 Done Done Sent false
1646 Acked Write 7 Done Done Sent false
1647 Acked Read 3 Done Done Sent false
1648 Acked Read 10 Done Done Sent false
1649 Acked Read 1 Done Done Sent false
1650 Acked Read 6 Done Done Sent false
1651 Acked Read 9 Done Done Sent false
1652 Acked Write 7 Done Done Sent false
1653 Acked Write 10 Done Done Sent false
1654 Acked Write 7 Done Done Sent false
1655 Acked Read 2 Done Done Sent false
1656 Acked Read 8 Done Done Sent false
1657 Acked Read 9 Done Done Sent false
1658 Acked Read 8 Done Done Sent false
1659 Acked Read 2 Done Done Sent false
1660 Acked Read 1 Done Done Sent false
1661 Acked Read 2 Done Done Sent false
1662 Acked Read 8 Done Done Sent false
1663 Acked Read 5 Done Done Sent false
1664 Acked Flush 0 Done Done Sent false
1665 Acked Read 3 Done Done Sent false
1666 Acked Read 5 Done Done Sent false
1667 Acked Read 7 Done Done Sent false
1668 Acked Flush 0 Done Done Sent false
1669 Acked Write 2 Done Done Sent false
1670 Acked Flush 0 Done Done Sent false
1671 Acked Read 5 Done Done Sent false
1672 Acked Read 10 Done Done Sent false
1673 Acked Read 5 Done Done Sent false
1674 Acked Read 6 Done Done Sent false
1675 Acked Read 7 Done Done Sent false
1676 Acked Read 6 Done Done Sent false
1677 Acked Write 7 Done Done Sent false
1678 Acked Flush 0 Done Done Sent false
1679 Acked Read 6 Done Done Sent false
1680 Acked Write 9 Done Done Sent false
1681 Acked Read 7 Done Done Sent false
1682 Acked Write 5 Done Done Sent false
1683 Acked Read 2 Done Done Sent false
1684 Acked Read 8 Done Done Sent false
1685 Acked Read 8 Done Done Sent false
1686 Acked Read 7 Done Done Sent false
1687 Acked Write 4 Done Done Sent false
1688 Acked Write 7 Done Done Sent false
1689 Acked Write 2 Done Done Sent false
1690 Acked Read 8 Done Done Sent false
1691 Acked Read 1 Done Done Sent false
1692 Acked Read 5 Done Done Sent false
1693 Acked Read 4 Done Done Sent false
1694 Acked Write 3 Done Done Sent false
1695 Acked Read 4 Done Done Sent false
1696 Acked Read 10 Done Done Sent false
1697 Acked Read 9 Done Done Sent false
1698 Acked Write 8 Done Done Sent false
1699 Acked Write 6 Done Done Sent false
1700 Acked Flush 0 Done Done Sent false
1701 Acked Read 3 Done Done Sent false
1702 Acked Read 2 Done Done Sent false
1703 Acked Write 2 Done Done Sent false
1704 Acked Read 8 Done Done Sent false
1705 Acked Write 10 Done Done Sent false
1706 Acked Read 3 Done Done Sent false
1707 Acked Write 6 Done Done Sent false
1708 Acked Read 10 Done Done Sent false
1709 Acked Write 8 Done Done Sent false
1710 Acked Read 10 Done Done Sent false
1711 Acked Flush 0 Done Done Sent false
1712 Acked Read 1 Done Done Sent false
1713 Acked Read 8 Done Done Sent false
1714 Acked Read 3 Done Done Sent false
1715 Acked Read 1 Done Done Sent false
1716 Acked Write 1 Done Done Sent false
1717 Acked Write 9 Done Done Sent false
1718 Acked Read 3 Done Done Sent false
1719 Acked Write 7 Done Done Sent false
1720 Acked Write 9 Done Done Sent false
1721 Acked Write 4 Done Done Sent false
1722 Acked Write 7 Done Done Sent false
1723 Acked Write 10 Done Done Sent false
1724 Acked Read 7 Done Done Sent false
1725 Acked Write 4 Done Done Sent false
1726 Acked Read 9 Done Done Sent false
1727 Acked Flush 0 Done Done Sent false
1728 Acked Read 4 Done Done Sent false
1729 Acked Flush 0 Done Done Sent false
1730 Acked Flush 0 Done Done Sent false
1731 Acked Write 1 Done Done Sent false
1732 Acked Read 1 Done Done Sent false
1733 Acked Read 7 Done Done Sent false
1734 Acked Write 5 Done Done Sent false
1735 Acked Flush 0 Done Done Sent false
1736 Acked Write 2 Done Done Sent false
1737 Acked Write 9 Done Done Sent false
1738 Acked Read 4 Done Done Sent false
1739 Acked Read 8 Done Done Sent false
1740 Acked Read 3 Done Done Sent false
1741 Acked Flush 0 Done Done Sent false
1742 Acked Write 2 Done Done Sent false
1743 Acked Read 9 Done Done Sent false
1744 Acked Read 10 Done Done Sent false
1745 Acked Read 5 Done Done Sent false
1746 Acked Write 7 Done Done Sent false
1747 Acked Read 5 Done Done Sent false
1748 Acked Read 8 Done Done Sent false
1749 Acked Read 5 Done Done Sent false
1750 Acked Write 5 Done Done Sent false
1751 Acked Write 10 Done Done Sent false
1752 Acked Flush 0 Done Done Sent false
1753 Acked Read 2 Done Done Sent false
1754 Acked Read 6 Done Done Sent false
1755 Acked Write 5 Done Done Sent false
1756 Acked Write 10 Done Done Sent false
1757 Acked Read 9 Done Done Sent false
1758 Acked Write 10 Done Done Sent false
1759 Acked Write 5 Done Done Sent false
1760 Acked Write 5 Done Done Sent false
1761 Acked Write 10 Done Done Sent false
1762 Acked Read 10 Done Done Sent false
1763 Acked Read 2 Done Done Sent false
1764 Acked Write 9 Done Done Sent false
1765 Acked Read 4 Done Done Sent false
1766 Acked Flush 0 Done Done Sent false
1767 Acked Read 4 Done Done Sent false
1768 Acked Flush 0 Done Done Sent false
1769 Acked Write 8 Done Done Sent false
1770 Acked Write 7 Done Done Sent false
1771 Acked Read 1 Done Done Sent false
1772 Acked Write 4 Done Done Sent false
1773 Acked Read 5 Done Done Sent false
1774 Acked Write 2 Done Done Sent false
1775 Acked Read 3 Done Done Sent false
1776 Acked Read 5 Done Done Sent false
1777 Acked Flush 0 Done Done Sent false
1778 Acked Read 1 Done Done Sent false
1779 Acked Flush 0 Done Done Sent false
1780 Acked Read 6 Done Done Sent false
1781 Acked Write 9 Done Done Sent false
1782 Acked Read 4 Done Done Sent false
1783 Acked Read 1 Done Done Sent false
1784 Acked Flush 0 Done Done Sent false
1785 Acked Read 5 Done Done Sent false
1786 Acked Read 1 Done Done Sent false
1787 Acked Flush 0 Done Done Sent false
1788 Acked Write 9 Done Done Sent false
1789 Acked Read 2 Done Done Sent false
1790 Acked Read 10 Done Done Sent false
1791 Acked Flush 0 Done Done Sent false
1792 Acked Write 8 Done Done Sent false
1793 Acked Read 7 Done Done Sent false
1794 Acked Write 7 Done Done Sent false
1795 Acked Read 6 Done Done Sent false
1796 Acked Write 8 Done Done Sent false
1797 Acked Read 8 Done Done Sent false
1798 Acked Read 6 Done Done Sent false
1799 Acked Read 8 Done Done Sent false
1800 Acked Read 3 Done Done Sent false
1801 Acked Read 6 Done Done Sent false
1802 Acked Read 2 Done Done Sent false
1803 Acked Read 8 Done Done Sent false
1804 Acked Flush 0 Done Done Sent false
1805 Acked Write 1 Done Done Sent false
1806 Acked Read 3 Done Done Sent false
1807 Acked Read 2 Done Done Sent false
1808 Acked Write 1 Done Done Sent false
1809 Acked Write 8 Done Done Sent false
1810 Acked Flush 0 Done Done Sent false
1811 Acked Write 4 Done Done Sent false
1812 Acked Read 2 Done Done Sent false
1813 Acked Read 10 Done Done Sent false
1814 Acked Read 10 Done Done Sent false
1815 Acked Write 4 Done Done Sent false
1816 Acked Flush 0 Done Done Sent false
1817 Acked Read 7 Done Done Sent false
1818 Acked Flush 0 Done Done Sent false
1819 Acked Read 1 Done Done Sent false
1820 Acked Write 8 Done Done Sent false
1821 Acked Read 3 Done Done Sent false
1822 Acked Read 9 Done Done Sent false
1823 Acked Write 6 Done Done Sent false
1824 Acked Read 8 Done Done Sent false
1825 Acked Write 1 Done Done Sent false
1826 Acked Read 1 Done Done Sent false
1827 Acked Write 10 Done Done Sent false
1828 Acked Read 8 Done Done Sent false
1829 Acked Read 6 Done Done Sent false
1830 Acked Write 9 Done Done Sent false
1831 Acked Write 8 Done Done Sent false
1832 Acked Write 1 Done Done Sent false
1833 Acked Read 4 Done Done Sent false
1834 Acked Write 8 Done Done Sent false
1835 Acked Read 5 Done Done Sent false
1836 Acked Write 9 Done Done Sent false
1837 Acked Read 8 Done Done Sent false
1838 Acked Flush 0 Done Done Sent false
1839 Acked Write 7 Done Done Sent false
1840 Acked Read 2 Done Done Sent false
1841 Acked Read 3 Done Done Sent false
1842 Acked Read 7 Done Done Sent false
1843 Acked Write 8 Done Done Sent false
1844 Acked Write 5 Done Done Sent false
1845 Acked Read 2 Done Done Sent false
1846 Acked Write 1 Done Done Sent false
1847 Acked Write 2 Done Done Sent false
1848 Acked Write 4 Done Done Sent false
1849 Acked Read 7 Done Done Sent false
1850 Acked Write 6 Done Done Sent false
1851 Acked Read 5 Done Done Sent false
1852 Acked Write 7 Done Done Sent false
1853 Acked Flush 0 Done Done Sent false
1854 Acked Write 6 Done Done Sent false
1855 Acked Flush 0 Done Done Sent false
1856 Acked Read 2 Done Done Sent false
1857 Acked Read 10 Done Done Sent false
1858 Acked Read 3 Done Done Sent false
1859 Acked Read 2 Done Done Sent false
1860 Acked Write 4 Done Done Sent false
1861 Acked Flush 0 Done Done Sent false
1862 Acked Flush 0 Done Done Sent false
1863 Acked Write 5 Done Done Sent false
1864 Acked Write 8 Done Done Sent false
1865 Acked Read 10 Done Done Sent false
1866 Acked Write 5 Done Done Sent false
1867 Acked Write 10 Done Done Sent false
1868 Acked Write 4 Done Done Sent false
1869 Acked Flush 0 Done Done Sent false
1870 Acked Read 6 Done Done Sent false
1871 Acked Read 7 Done Done Sent false
1872 Acked Write 8 Done Done Sent false
1873 Acked Write 2 Done Done Sent false
1874 Acked Read 6 Done Done Sent false
1875 Acked Write 5 Done Done Sent false
1876 Acked Write 2 Done Done Sent false
1877 Acked Read 9 Done Done Sent false
1878 Acked Read 1 Done Done Sent false
1879 Acked Read 10 Done Done Sent false
1880 Acked Read 6 Done Done Sent false
1881 Acked Read 10 Done Done Sent false
1882 Acked Read 6 Done Done Sent false
1883 Acked Flush 0 Done Done Sent false
1884 Acked Read 10 Done Done Sent false
1885 Acked Read 2 Done Done Sent false
1886 Acked Write 7 Done Done Sent false
1887 Acked Write 5 Done Done Sent false
1888 Acked Read 7 Done Done Sent false
1889 Acked Write 10 Done Done Sent false
1890 Acked Write 4 Done Done Sent false
1891 Acked Write 4 Done Done Sent false
1892 Acked Read 7 Done Done Sent false
1893 Acked Read 8 Done Done Sent false
1894 Acked Read 2 Done Done Sent false
1895 Acked Read 10 Done Done Sent false
1896 Acked Read 7 Done Done Sent false
1897 Acked Read 4 Done Done Sent false
1898 Acked Read 5 Done Done Sent false
1899 Acked Write 7 Done Done Sent false
1900 Acked Read 3 Done Done Sent false
1901 Acked Write 8 Done Done Sent false
1902 Acked Read 8 Done Done Sent false
1903 Acked Read 4 Done Done Sent false
1904 Acked Write 1 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1883 1883 1603
Downstairs last five completed: 1603 1602 1601 1600 1599
Upstairs last five completed: 1904 1903 1902 1901 1900
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1203)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1604 Acked Write 9 Done Done Sent false
1605 Acked Read 1 Done Done Sent false
1606 Acked Flush 0 Done Done Sent false
1607 Acked Flush 0 Done Done Sent false
1608 Acked Read 9 Done Done Sent false
1609 Acked Read 6 Done Done Sent false
1610 Acked Read 2 Done Done Sent false
1611 Acked Read 4 Done Done Sent false
1612 Acked Write 10 Done Done Sent false
1613 Acked Read 1 Done Done Sent false
1614 Acked Read 5 Done Done Sent false
1615 Acked Read 10 Done Done Sent false
1616 Acked Write 6 Done Done Sent false
1617 Acked Write 10 Done Done Sent false
1618 Acked Write 9 Done Done Sent false
1619 Acked Read 10 Done Done Sent false
1620 Acked Read 3 Done Done Sent false
1621 Acked Read 5 Done Done Sent false
1622 Acked Flush 0 Done Done Sent false
1623 Acked Write 10 Done Done Sent false
1624 Acked Write 2 Done Done Sent false
1625 Acked Read 5 Done Done Sent false
1626 Acked Write 3 Done Done Sent false
1627 Acked Read 3 Done Done Sent false
1628 Acked Write 8 Done Done Sent false
1629 Acked Read 3 Done Done Sent false
1630 Acked Write 1 Done Done Sent false
1631 Acked Write 7 Done Done Sent false
1632 Acked Read 9 Done Done Sent false
1633 Acked Write 2 Done Done Sent false
1634 Acked Write 10 Done Done Sent false
1635 Acked Read 4 Done Done Sent false
1636 Acked Read 3 Done Done Sent false
1637 Acked Flush 0 Done Done Sent false
1638 Acked Flush 0 Done Done Sent false
1639 Acked Write 5 Done Done Sent false
1640 Acked Flush 0 Done Done Sent false
1641 Acked Read 6 Done Done Sent false
1642 Acked Read 10 Done Done Sent false
1643 Acked Read 2 Done Done Sent false
1644 Acked Write 10 Done Done Sent false
1645 Acked Read 7 Done Done Sent false
1646 Acked Write 7 Done Done Sent false
1647 Acked Read 3 Done Done Sent false
1648 Acked Read 10 Done Done Sent false
1649 Acked Read 1 Done Done Sent false
1650 Acked Read 6 Done Done Sent false
1651 Acked Read 9 Done Done Sent false
1652 Acked Write 7 Done Done Sent false
1653 Acked Write 10 Done Done Sent false
1654 Acked Write 7 Done Done Sent false
1655 Acked Read 2 Done Done Sent false
1656 Acked Read 8 Done Done Sent false
1657 Acked Read 9 Done Done Sent false
1658 Acked Read 8 Done Done Sent false
1659 Acked Read 2 Done Done Sent false
1660 Acked Read 1 Done Done Sent false
1661 Acked Read 2 Done Done Sent false
1662 Acked Read 8 Done Done Sent false
1663 Acked Read 5 Done Done Sent false
1664 Acked Flush 0 Done Done Sent false
1665 Acked Read 3 Done Done Sent false
1666 Acked Read 5 Done Done Sent false
1667 Acked Read 7 Done Done Sent false
1668 Acked Flush 0 Done Done Sent false
1669 Acked Write 2 Done Done Sent false
1670 Acked Flush 0 Done Done Sent false
1671 Acked Read 5 Done Done Sent false
1672 Acked Read 10 Done Done Sent false
1673 Acked Read 5 Done Done Sent false
1674 Acked Read 6 Done Done Sent false
1675 Acked Read 7 Done Done Sent false
1676 Acked Read 6 Done Done Sent false
1677 Acked Write 7 Done Done Sent false
1678 Acked Flush 0 Done Done Sent false
1679 Acked Read 6 Done Done Sent false
1680 Acked Write 9 Done Done Sent false
1681 Acked Read 7 Done Done Sent false
1682 Acked Write 5 Done Done Sent false
1683 Acked Read 2 Done Done Sent false
1684 Acked Read 8 Done Done Sent false
1685 Acked Read 8 Done Done Sent false
1686 Acked Read 7 Done Done Sent false
1687 Acked Write 4 Done Done Sent false
1688 Acked Write 7 Done Done Sent false
1689 Acked Write 2 Done Done Sent false
1690 Acked Read 8 Done Done Sent false
1691 Acked Read 1 Done Done Sent false
1692 Acked Read 5 Done Done Sent false
1693 Acked Read 4 Done Done Sent false
1694 Acked Write 3 Done Done Sent false
1695 Acked Read 4 Done Done Sent false
1696 Acked Read 10 Done Done Sent false
1697 Acked Read 9 Done Done Sent false
1698 Acked Write 8 Done Done Sent false
1699 Acked Write 6 Done Done Sent false
1700 Acked Flush 0 Done Done Sent false
1701 Acked Read 3 Done Done Sent false
1702 Acked Read 2 Done Done Sent false
1703 Acked Write 2 Done Done Sent false
1704 Acked Read 8 Done Done Sent false
1705 Acked Write 10 Done Done Sent false
1706 Acked Read 3 Done Done Sent false
1707 Acked Write 6 Done Done Sent false
1708 Acked Read 10 Done Done Sent false
1709 Acked Write 8 Done Done Sent false
1710 Acked Read 10 Done Done Sent false
1711 Acked Flush 0 Done Done Sent false
1712 Acked Read 1 Done Done Sent false
1713 Acked Read 8 Done Done Sent false
1714 Acked Read 3 Done Done Sent false
1715 Acked Read 1 Done Done Sent false
1716 Acked Write 1 Done Done Sent false
1717 Acked Write 9 Done Done Sent false
1718 Acked Read 3 Done Done Sent false
1719 Acked Write 7 Done Done Sent false
1720 Acked Write 9 Done Done Sent false
1721 Acked Write 4 Done Done Sent false
1722 Acked Write 7 Done Done Sent false
1723 Acked Write 10 Done Done Sent false
1724 Acked Read 7 Done Done Sent false
1725 Acked Write 4 Done Done Sent false
1726 Acked Read 9 Done Done Sent false
1727 Acked Flush 0 Done Done Sent false
1728 Acked Read 4 Done Done Sent false
1729 Acked Flush 0 Done Done Sent false
1730 Acked Flush 0 Done Done Sent false
1731 Acked Write 1 Done Done Sent false
1732 Acked Read 1 Done Done Sent false
1733 Acked Read 7 Done Done Sent false
1734 Acked Write 5 Done Done Sent false
1735 Acked Flush 0 Done Done Sent false
1736 Acked Write 2 Done Done Sent false
1737 Acked Write 9 Done Done Sent false
1738 Acked Read 4 Done Done Sent false
1739 Acked Read 8 Done Done Sent false
1740 Acked Read 3 Done Done Sent false
1741 Acked Flush 0 Done Done Sent false
1742 Acked Write 2 Done Done Sent false
1743 Acked Read 9 Done Done Sent false
1744 Acked Read 10 Done Done Sent false
1745 Acked Read 5 Done Done Sent false
1746 Acked Write 7 Done Done Sent false
1747 Acked Read 5 Done Done Sent false
1748 Acked Read 8 Done Done Sent false
1749 Acked Read 5 Done Done Sent false
1750 Acked Write 5 Done Done Sent false
1751 Acked Write 10 Done Done Sent false
1752 Acked Flush 0 Done Done Sent false
1753 Acked Read 2 Done Done Sent false
1754 Acked Read 6 Done Done Sent false
1755 Acked Write 5 Done Done Sent false
1756 Acked Write 10 Done Done Sent false
1757 Acked Read 9 Done Done Sent false
1758 Acked Write 10 Done Done Sent false
1759 Acked Write 5 Done Done Sent false
1760 Acked Write 5 Done Done Sent false
1761 Acked Write 10 Done Done Sent false
1762 Acked Read 10 Done Done Sent false
1763 Acked Read 2 Done Done Sent false
1764 Acked Write 9 Done Done Sent false
1765 Acked Read 4 Done Done Sent false
1766 Acked Flush 0 Done Done Sent false
1767 Acked Read 4 Done Done Sent false
1768 Acked Flush 0 Done Done Sent false
1769 Acked Write 8 Done Done Sent false
1770 Acked Write 7 Done Done Sent false
1771 Acked Read 1 Done Done Sent false
1772 Acked Write 4 Done Done Sent false
1773 Acked Read 5 Done Done Sent false
1774 Acked Write 2 Done Done Sent false
1775 Acked Read 3 Done Done Sent false
1776 Acked Read 5 Done Done Sent false
1777 Acked Flush 0 Done Done Sent false
1778 Acked Read 1 Done Done Sent false
1779 Acked Flush 0 Done Done Sent false
1780 Acked Read 6 Done Done Sent false
1781 Acked Write 9 Done Done Sent false
1782 Acked Read 4 Done Done Sent false
1783 Acked Read 1 Done Done Sent false
1784 Acked Flush 0 Done Done Sent false
1785 Acked Read 5 Done Done Sent false
1786 Acked Read 1 Done Done Sent false
1787 Acked Flush 0 Done Done Sent false
1788 Acked Write 9 Done Done Sent false
1789 Acked Read 2 Done Done Sent false
1790 Acked Read 10 Done Done Sent false
1791 Acked Flush 0 Done Done Sent false
1792 Acked Write 8 Done Done Sent false
1793 Acked Read 7 Done Done Sent false
1794 Acked Write 7 Done Done Sent false
1795 Acked Read 6 Done Done Sent false
1796 Acked Write 8 Done Done Sent false
1797 Acked Read 8 Done Done Sent false
1798 Acked Read 6 Done Done Sent false
1799 Acked Read 8 Done Done Sent false
1800 Acked Read 3 Done Done Sent false
1801 Acked Read 6 Done Done Sent false
1802 Acked Read 2 Done Done Sent false
1803 Acked Read 8 Done Done Sent false
1804 Acked Flush 0 Done Done Sent false
1805 Acked Write 1 Done Done Sent false
1806 Acked Read 3 Done Done Sent false
1807 Acked Read 2 Done Done Sent false
1808 Acked Write 1 Done Done Sent false
1809 Acked Write 8 Done Done Sent false
1810 Acked Flush 0 Done Done Sent false
1811 Acked Write 4 Done Done Sent false
1812 Acked Read 2 Done Done Sent false
1813 Acked Read 10 Done Done Sent false
1814 Acked Read 10 Done Done Sent false
1815 Acked Write 4 Done Done Sent false
1816 Acked Flush 0 Done Done Sent false
1817 Acked Read 7 Done Done Sent false
1818 Acked Flush 0 Done Done Sent false
1819 Acked Read 1 Done Done Sent false
1820 Acked Write 8 Done Done Sent false
1821 Acked Read 3 Done Done Sent false
1822 Acked Read 9 Done Done Sent false
1823 Acked Write 6 Done Done Sent false
1824 Acked Read 8 Done Done Sent false
1825 Acked Write 1 Done Done Sent false
1826 Acked Read 1 Done Done Sent false
1827 Acked Write 10 Done Done Sent false
1828 Acked Read 8 Done Done Sent false
1829 Acked Read 6 Done Done Sent false
1830 Acked Write 9 Done Done Sent false
1831 Acked Write 8 Done Done Sent false
1832 Acked Write 1 Done Done Sent false
1833 Acked Read 4 Done Done Sent false
1834 Acked Write 8 Done Done Sent false
1835 Acked Read 5 Done Done Sent false
1836 Acked Write 9 Done Done Sent false
1837 Acked Read 8 Done Done Sent false
1838 Acked Flush 0 Done Done Sent false
1839 Acked Write 7 Done Done Sent false
1840 Acked Read 2 Done Done Sent false
1841 Acked Read 3 Done Done Sent false
1842 Acked Read 7 Done Done Sent false
1843 Acked Write 8 Done Done Sent false
1844 Acked Write 5 Done Done Sent false
1845 Acked Read 2 Done Done Sent false
1846 Acked Write 1 Done Done Sent false
1847 Acked Write 2 Done Done Sent false
1848 Acked Write 4 Done Done Sent false
1849 Acked Read 7 Done Done Sent false
1850 Acked Write 6 Done Done Sent false
1851 Acked Read 5 Done Done Sent false
1852 Acked Write 7 Done Done Sent false
1853 Acked Flush 0 Done Done Sent false
1854 Acked Write 6 Done Done Sent false
1855 Acked Flush 0 Done Done Sent false
1856 Acked Read 2 Done Done Sent false
1857 Acked Read 10 Done Done Sent false
1858 Acked Read 3 Done Done Sent false
1859 Acked Read 2 Done Done Sent false
1860 Acked Write 4 Done Done Sent false
1861 Acked Flush 0 Done Done Sent false
1862 Acked Flush 0 Done Done Sent false
1863 Acked Write 5 Done Done Sent false
1864 Acked Write 8 Done Done Sent false
1865 Acked Read 10 Done Done Sent false
1866 Acked Write 5 Done Done Sent false
1867 Acked Write 10 Done Done Sent false
1868 Acked Write 4 Done Done Sent false
1869 Acked Flush 0 Done Done Sent false
1870 Acked Read 6 Done Done Sent false
1871 Acked Read 7 Done Done Sent false
1872 Acked Write 8 Done Done Sent false
1873 Acked Write 2 Done Done Sent false
1874 Acked Read 6 Done Done Sent false
1875 Acked Write 5 Done Done Sent false
1876 Acked Write 2 Done Done Sent false
1877 Acked Read 9 Done Done Sent false
1878 Acked Read 1 Done Done Sent false
1879 Acked Read 10 Done Done Sent false
1880 Acked Read 6 Done Done Sent false
1881 Acked Read 10 Done Done Sent false
1882 Acked Read 6 Done Done Sent false
1883 Acked Flush 0 Done Done Sent false
1884 Acked Read 10 Done Done Sent false
1885 Acked Read 2 Done Done Sent false
1886 Acked Write 7 Done Done Sent false
1887 Acked Write 5 Done Done Sent false
1888 Acked Read 7 Done Done Sent false
1889 Acked Write 10 Done Done Sent false
1890 Acked Write 4 Done Done Sent false
1891 Acked Write 4 Done Done Sent false
1892 Acked Read 7 Done Done Sent false
1893 Acked Read 8 Done Done Sent false
1894 Acked Read 2 Done Done Sent false
1895 Acked Read 10 Done Done Sent false
1896 Acked Read 7 Done Done Sent false
1897 Acked Read 4 Done Done Sent false
1898 Acked Read 5 Done Done Sent false
1899 Acked Write 7 Done Done Sent false
1900 Acked Read 3 Done Done Sent false
1901 Acked Write 8 Done Done Sent false
1902 Acked Read 8 Done Done Sent false
1903 Acked Read 4 Done Done Sent false
1904 Acked Write 1 Done Done Sent false
1905 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1905 1905 1603
Downstairs last five completed: 1603 1602 1601 1600 1599
Upstairs last five completed: 1905 1904 1903 1902 1901
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.405580733Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37391)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.405751454Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.407173528Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.407740881Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.407766448Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 1603","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.407783636Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 1603","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.408034937Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 1603","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.408058584Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 1603","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:59.408308895Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1905 1904 1903 1902 1901
Upstairs last five completed: 1905 1904 1903 1902 1901
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:37:01.656 INFO request completed, latency_us: 359, [8830] Got stop action so:false kr:false
response_code: 204, uri: /stop/cid/2, method: POST, req_id: 91c7e762-c6fb-4ffd-bbb3-6d6786754410, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:37:01.656 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2f09869b-b533-44b5-9d6f-42c4cc1b6370{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.656966138Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.657049178Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.657086784Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.65712523Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:37:05.660 INFO request completed, latency_us: 330, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2b8880fd-dad6-45a8-ab94-b482b38256e7, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 337 len 28672 data: 2 2 2 2 2 2 2
002/300 Read block 3414 len 12288
003/300 Write block 1361 len 28672 data: 2 2 2 2 2 2 2
004/300 Write block 3250 len 20480 data: 2 2 2 2 2
005/300 Write block 2978 len 24576 data: 2 2 2 2 2 2
006/300 Read block 2115 len 24576
007/300 Read block 3047 len 4096
008/300 Read block 848 len 32768
009/300 Read block 2382 len 20480
010/300 Write block 4540 len 32768 data: 2 2 2 2 2 2 2 2
011/300 Flush
012/300 Read block 4060 len 28672
013/300 Read block 3697 len 36864
014/300 Read block 2949 len 4096
015/300 Flush
016/300 Flush
017/300 Write block 1810 len 20480 data: 3 4 4 3 3
018/300 Write block 2287 len 16384 data: 4 5 5 5
019/300 Write block 1101 len 8192 data: 3 3
020/300 Write block 2268 len 8192 data: 2 2
021/300 Read block 3145 len 32768
022/300 Read block 4615 len 12288
023/300 Write block 4466 len 24576 data: 2 2 2 2 2 2
024/300 Write block 222 len 32768 data: 2 2 2 2 2 2 2 2
025/300 Read block 124 len 36864
026/300 Read block 4528 len 20480
027/300 Write block 3635 len 24576 data: 2 2 2 2 2 2
028/300 Read block 2517 len 20480
029/300 Read block 3033 len 40960
030/300 Write block 2770 len 20480 data: 4 4 4 3 3
031/300 Read block 3315 len 24576
032/300 Read block 2594 len 8192
033/300 Write block 3863 len 4096 data: 3
034/300 Read block 4124 len 40960
035/300 Write block 2393 len 4096 data: 2
036/300 Write block 2997 len 28672 data: 2 2 2 3 3 3 3
037/300 Read block 4733 len 4096
038/300 Write block 3270 len 36864 data: 2 2 2 2 2 2 2 2 2
039/300 Write block 1263 len 16384 data: 2 2 2 2
040/300 Flush
041/300 Write block 4633 len 28672 data: 2 2 2 2 2 2 2
042/300 Read block 1479 len 16384
043/300 Read block 4370 len 4096
044/300 Read block 4600 len 28672
045/300 Write block 2227 len 24576 data: 2 2 2 2 2 2
046/300 Read block 1983 len 28672
047/300 Write block 3130 len 16384 data: 3 3 3 3
048/300 Read block 1762 len 24576
049/300 Read block 343 len 16384
050/300 Read block 1603 len 16384
051/300 Write block 4248 len 12288 data: 4 4 5
052/300 Write block 4383 len 40960 data: 3 2 2 2 2 2 2 2 3 3
053/300 Read block 3561 len 12288
054/300 Read block 3810 len 36864
055/300 Flush
056/300 Read block 777 len 24576
057/300 Write block 1363 len 32768 data: 3 3 3 3 3 2 2 3
058/300 Read block 3560 len 28672
059/300 Read block 350 len 8192
060/300 Read block 4442 len 12288
061/300 Read block 1238 len 24576
062/300 Read block 3203 len 32768
063/300 Read block 1088 len 8192
064/300 Read block 1477 len 32768
065/300 Write block 371 len 8192 data: 2 3
066/300 Read block 4755 len 20480
067/300 Read block 313 len 40960
068/300 Write block 28 len 12288 data: 2 2 2
069/300 Read block 2792 len 32768
070/300 Read block 4514 len 32768
071/300 Read block 856 len 8192
072/300 Write block 1297 len 4096 data: 2
073/300 Read block 2482 len 24576
074/300 Flush
075/300 Write block 1412 len 32768 data: 3 3 3 3 3 4 4 3
076/300 Write block 3235 len 16384 data: 4 2 2 2
077/300 Write block 102 len 36864 data: 2 2 2 2 2 2 2 2 2
078/300 Flush
079/300 Write block 2933 len 4096 data: 2
080/300 Write block 4116 len 8192 data: 2 2
081/300 Read block 1300 len 16384
082/300 Write block 4568 len 40960 data: 4 3 2 2 2 2 2 2 2 2
083/300 Read block 971 len 28672
084/300 Read block 2980 len 36864
085/300 Read block 1460 len 32768
086/300 Write block 2556 len 8192 data: 2 2
087/300 Flush
088/300 Write block 1543 len 40960 data: 6 6 5 2 2 2 2 2 3 2
089/300 Write block 4912 len 20480 data: 2 2 2 2 2
090/300 Read block 640 len 16384
091/300 Write block 4468 len 28672 data: 3 3 3 3 2 2 2
092/300 Read block 1686 len 8192
093/300 Flush
094/300 Read block 2016 len 32768
095/300 Write block 669 len 40960 data: 2 2 2 2 2 2 2 2 2 2
096/300 Write block 120 len 20480 data: 3 3 4 4 4
097/300 Write block 1358 len 16384 data: 2 2 2 3
098/300 Write block 4100 len 4096 data: 3
099/300 Read block 4705 len 16384
100/300 Read block 3604 len 32768
101/300 Write block 4963 len 36864 data: 2 2 2 2 2 2 2 3 3
102/300 Write block 3335 len 12288 data: 2 2 2
103/300 Read block 4768 len 8192
104/300 Flush
105/300 Write block 3046 len 36864 data: 3 2 2 2 2 2 2 2 2
106/300 Read block 4095 len 36864
107/300 Write block 866 len 36864 data: 3 3 2 2 2 3 3 3 3
108/300 Read block 743 len 28672
109/300 Write block 2563 len 40960 data: 3 3 3 3 2 2 2 2 2 2
110/300 Write block 2851 len 8192 data: 3 3
111/300 Read block 1966 len 4096
112/300 Read block 1470 len 20480
113/300 Read block 2570 len 32768
114/300 Read block 3451 len 28672
115/300 Read block 3954 len 32768
116/300 Read block 3528 len 40960
117/300 Read block 3968 len 24576
118/300 Write block 1196 len 4096 data: 3
119/300 Write block 3145 len 32768 data: 2 2 2 2 2 2 2 2
120/300 Read block 3689 len 16384
121/300 Flush
122/300 Read block 2764 len 36864
123/300 Write block 1674 len 24576 data: 3 3 3 2 2 2
124/300 Read block 3884 len 12288
125/300 Read block 525 len 4096
126/300 Read block 3202 len 4096
127/300 Read block 105 len 12288
128/300 Read block 555 len 8192
129/300 Read block 3544 len 40960
130/300 Flush
131/300 Write block 3752 len 20480 data: 2 2 2 2 2
132/300 Write block 3727 len 20480 data: 3 3 3 2 2
133/300 Flush
134/300 Write block 3151 len 4096 data: 3
135/300 Write block 397 len 40960 data: 3 2 2 2 3 3 3 3 3 3
136/300 Read block 2531 len 24576
137/300 Flush
138/300 Read block 3390 len 36864
139/300 Read block 12 len 36864
140/300 Write block 1219 len 24576 data: 2 2 3 3 4 4
141/300 Flush
142/300 Flush
143/300 Read block 322 len 36864
144/300 Write block 3621 len 16384 data: 2 2 2 2
145/300 Write block 4643 len 20480 data: 2 2 2 2 2
146/300 Write block 4464 len 8192 data: 2 2
147/300 Flush
148/300 Write block 1047 len 8192 data: 3 2
149/300 Read block 1271 len 16384
150/300 Write block 2808 len 36864 data: 2 2 2 2 2 2 2 2 2
151/300 Read block 4916 len 24576
152/300 Read block 3213 len 8192
153/300 Read block 1881 len 8192
154/300 Write block 3153 len 20480 data: 2 2 2 2 2
155/300 Write block 2302 len 12288 data: 2 2 2
156/300 Write block 952 len 40960 data: 2 2 2 2 2 2 2 2 2 2
157/300 Flush
158/300 Write block 3009 len 4096 data: 2
159/300 Write block 648 len 32768 data: 3 3 2 2 2 3 2 2
160/300 Read block 2504 len 32768
161/300 Read block 4653 len 4096
162/300 Write block 585 len 36864 data: 2 2 3 3 3 3 3 3 3
163/300 Read block 368 len 24576
164/300 Read block 468 len 28672
165/300 Write block 2245 len 12288 data: 3 3 3
166/300 Write block 148 len 12288 data: 2 2 2
167/300 Write block 4144 len 8192 data: 3 3
168/300 Read block 4969 len 36864
169/300 Write block 3372 len 36864 data: 3 2 2 2 2 2 2 2 2
170/300 Write block 1110 len 8192 data: 2 2
171/300 Write block 604 len 12288 data: 5 5 3
172/300 Read block 4002 len 32768
173/300 Write block 1324 len 24576 data: 2 2 2 2 2 3
174/300 Write block 2207 len 8192 data: 3 3
175/300 Read block 2160 len 24576
176/300 Write block 4735 len 32768 data: 3 3 2 2 2 2 2 2
177/300 Write block 830 len 12288 data: 2 2 2
178/300 Read block 107 len 24576
179/300 Read block 2682 len 32768
180/300 Read block 4892 len 40960
181/300 Flush
182/300 Read block 4112 len 4096
183/300 Read block 101 len 40960
184/300 Flush
185/300 Flush
186/300 Read block 3237 len 16384
187/300 Read block 4788 len 32768
188/300 Read block 1559 len 32768
189/300 Read block 1465 len 24576
190/300 Flush
191/300 Read block 3577 len 28672
192/300 Read block 3635 len 4096
193/300 Write block 3135 len 40960 data: 3 4 3 3 3 3 3 3 3 2
194/300 Read block 715 len 20480
195/300 Read block 643 len 40960
196/300 Read block 1252 len 40960
197/300 Write block 3622 len 16384 data: 3 3 3 2
198/300 Write block 1301 len 16384 data: 2 2 3 3
199/300 Read block 978 len 28672
200/300 Write block 275 len 28672 data: 3 3 3 3 3 4 4
201/300 Read block 4538 len 8192
202/300 Flush
203/300 Read block 4391 len 4096
204/300 Write block 4514 len 28672 data: 3 3 3 3 3 2 3
205/300 Read block 1871 len 16384
206/300 Read block 563 len 4096
207/300 Read block 4430 len 8192
208/300 Read block 4784 len 28672
209/300 Write block 807 len 40960 data: 2 2 2 2 2 2 2 2 2 2
210/300 Flush
211/300 Flush
212/300 Read block 141 len 24576
213/300 Read block 2503 len 32768
214/300 Read block 1590 len 4096
215/300 Write block 2371 len 32768 data: 2 2 2 2 2 2 2 3
216/300 Read block 2801 len 28672
217/300 Write block 1627 len 36864 data: 2 2 2 2 2 2 2 2 2
218/300 Read block 1891 len 4096
219/300 Read block 1508 len 12288
220/300 Write block 295 len 36864 data: 2 2 2 2 2 3 3 3 2
221/300 Read block 2557 len 36864
222/300 Write block 3056 len 32768 data: 2 3 3 3 3 3 2 2
223/300 Write block 3463 len 20480 data: 2 2 2 2 2
224/300 Read block 2946 len 12288
225/300 Read block 3853 len 8192
226/300 Read block 1520 len 36864
227/300 Write block 2011 len 4096 data: 2
228/300 Read block 164 len 40960
229/300 Flush
230/300 Write block 738 len 32768 data: 3 3 3 3 3 3 2 2
231/300 Read block 1716 len 36864
232/300 Read block 3283 len 16384
233/300 Write block 4118 len 24576 data: 4 4 4 4 4 3
234/300 Flush
235/300 Write block 2779 len 28672 data: 2 2 2 2 2 2 2
236/300 Write block 4807 len 16384 data: 3 3 3 4
237/300 Flush
238/300 Write block 2353 len 8192 data: 2 2
239/300 Write block 2837 len 40960 data: 3 3 3 3 3 3 3 3 3 3
240/300 Flush
241/300 Write block 4737 len 4096 data: 3
242/300 Read block 3758 len 12288
243/300 Read block 2976 len 16384
244/300 Write block 2350 len 28672 data: 2 2 2 3 3 2 2
245/300 Write block 1354 len 36864 data: 2 2 2 2 3 3 3 4 3
246/300 Read block 1875 len 40960
247/300 Read block 2501 len 32768
248/300 Read block 292 len 8192
249/300 Write block 37 len 32768 data: 2 2 2 2 2 2 3 3
250/300 Flush
251/300 Flush
252/300 Read block 3691 len 32768
253/300 Read block 3773 len 28672
254/300 Read block 569 len 40960
255/300 Write block 2400 len 24576 data: 2 2 3 3 2 2
256/300 Read block 1350 len 24576
257/300 Write block 868 len 4096 data: 3
258/300 Write block 420 len 28672 data: 2 2 2 2 2 2 2
259/300 Write block 564 len 4096 data: 3
260/300 Read block 4196 len 8192
261/300 Write block 4666 len 32768 data: 2 2 2 2 2 2 2 2
262/300 Read block 4420 len 20480
263/300 Read block 513 len 36864
264/300 Write block 829 len 28672 data: 2 3 3 3 2 2 2
265/300 Read block 377 len 8192
266/300 Read block 1598 len 40960
267/300 Write block 1 len 8192 data: 2 2
268/300 Write block 713 len 16384 data: 6 6 5 4
269/300 Flush
270/300 Read block 3492 len 12288
271/300 Read block 4225 len 4096
272/300 Read block 2728 len 32768
273/300 Flush
274/300 Write block 166 len 24576 data: 2 2 2 2 2 2
275/300 Write block 4190 len 12288 data: 4 4 4
276/300 Read block 3079 len 20480
277/300 Write block 3614 len 32768 data: 2 2 2 2 2 2 2 3
278/300 Read block 907 len 32768
279/300 Read block 4691 len 16384
280/300 Read block 3091 len 20480
281/300 Read block 3882 len 40960
282/300 Write block 2463 len 4096 data: 2
283/300 Write block 266 len 4096 data: 3
284/300 Write block 2335 len 28672 data: 2 2 2 2 3 3 3
285/300 Write block 364 len 24576 data: 2 2 2 2 2 2
286/300 Write block 4646 len 16384 data: 3 3 2 2
287/300 Read block 3172 len 40960
288/300 Write block 3331 len 40960 data: 3 4 2 2 3 3 3 2 2 2
289/300 Flush
290/300 Write block 2801 len 8192 data: 3 3
291/300 Flush
292/300 Write block 1398 len 16384 data: 3 3 3 3
293/300 Write block 1009 len 36864 data: 2 2 2 2 2 2 2 2 2
294/300 Read block 590 len 36864
295/300 Read block 245 len 12288
296/300 Write block 959 len 24576 data: 3 3 3 2 2 2
297/300 Read block 2560 len 12288
298/300 Read block 4849 len 16384
299/300 Write block 3688 len 40960 data: 2 2 2 2 2 2 2 2 2 2
300/300 Write block 2483 len 12288 data: 2 2 2
301/300 Read block 4236 len 24576
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:37:05.895 INFO request completed, latency_us: 176, response_code: 204, uri: /start/cid/2, method: POST, req_id: e21cf74d-f3d8-4596-8cc3-eecb39ef6728, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1906 Acked Write 7 Done Done Sent false
1907 Acked Read 3 Done Done Sent false
1908 Acked Write 7 Done Done Sent false
1909 Acked Write 5 Done Done Sent false
1910 Acked Write 6 Done Done Sent false
1911 Acked Read 6 Done Done Sent false
1912 Acked Read 1 Done Done Sent false
1913 Acked Read 8 Done Done Sent false
1914 Acked Read 5 Done Done Sent false
1915 Acked Write 8 Done Done Sent false
1916 Acked Flush 0 Done Done Sent false
1917 Acked Read 7 Done Done Sent false
1918 Acked Read 9 Done Done Sent false
1919 Acked Read 1 Done Done Sent false
1920 Acked Flush 0 Done Done Sent false
1921 Acked Flush 0 Done Done Sent false
1922 Acked Write 5 Done Done Sent false
1923 Acked Write 4 Done Done Sent false
1924 Acked Write 2 Done Done Sent false
1925 Acked Write 2 Done Done Sent false
1926 Acked Read 8 Done Done Sent false
1927 Acked Read 3 Done Done Sent false
1928 Acked Write 6 Done Done Sent false
1929 Acked Write 8 Done Done Sent false
1930 Acked Read 9 Done Done Sent false
1931 Acked Read 5 Done Done Sent false
1932 Acked Write 6 Done Done Sent false
1933 Acked Read 5 Done Done Sent false
1934 Acked Read 10 Done Done Sent false
1935 Acked Write 5 Done Done Sent false
1936 Acked Read 6 Done Done Sent false
1937 Acked Read 2 Done Done Sent false
1938 Acked Write 1 Done Done Sent false
1939 Acked Read 10 Done Done Sent false
1940 Acked Write 1 Done Done Sent false
1941 Acked Write 7 Done Done Sent false
1942 Acked Read 1 Done Done Sent false
1943 Acked Write 9 Done Done Sent false
1944 Acked Write 4 Done Done Sent false
1945 Acked Flush 0 Done Done Sent false
1946 Acked Write 7 Done Done Sent false
1947 Acked Read 4 Done Done Sent false
1948 Acked Read 1 Done Done Sent false
1949 Acked Read 7 Done Done Sent false
1950 Acked Write 6 Done Done Sent false
1951 Acked Read 7 Done Done Sent false
1952 Acked Write 4 Done Done Sent false
1953 Acked Read 6 Done Done Sent false
1954 Acked Read 4 Done Done Sent false
1955 Acked Read 4 Done Done Sent false
1956 Acked Write 3 Done Done Sent false
1957 Acked Write 10 Done Done Sent false
1958 Acked Read 3 Done Done Sent false
1959 Acked Read 9 Done Done Sent false
1960 Acked Flush 0 Done Done Sent false
1961 Acked Read 6 Done Done Sent false
1962 Acked Write 8 Done Done Sent false
1963 Acked Read 7 Done Done Sent false
1964 Acked Read 2 Done Done Sent false
1965 Acked Read 3 Done Done Sent false
1966 Acked Read 6 Done Done Sent false
1967 Acked Read 8 Done Done Sent false
1968 Acked Read 2 Done Done Sent false
1969 Acked Read 8 Done Done Sent false
1970 Acked Write 2 Done Done Sent false
1971 Acked Read 5 Done Done Sent false
1972 Acked Read 10 Done Done Sent false
1973 Acked Write 3 Done Done Sent false
1974 Acked Read 8 Done Done Sent false
1975 Acked Read 8 Done Done Sent false
1976 Acked Read 2 Done Done Sent false
1977 Acked Write 1 Done Done Sent false
1978 Acked Read 6 Done Done Sent false
1979 Acked Flush 0 Done Done Sent false
1980 Acked Write 8 Done Done Sent false
1981 Acked Write 4 Done Done Sent false
1982 Acked Write 9 Done Done Sent false
1983 Acked Flush 0 Done Done Sent false
1984 Acked Write 1 Done Done Sent false
1985 Acked Write 2 Done Done Sent false
1986 Acked Read 4 Done Done Sent false
1987 Acked Write 10 Done Done Sent false
1988 Acked Read 7 Done Done Sent false
1989 Acked Read 9 Done Done Sent false
1990 Acked Read 8 Done Done Sent false
1991 Acked Write 2 Done Done Sent false
1992 Acked Flush 0 Done Done Sent false
1993 Acked Write 10 Done Done Sent false
1994 Acked Write 5 Done Done Sent false
1995 Acked Read 4 Done Done Sent false
1996 Acked Write 7 Done Done Sent false
1997 Acked Read 2 Done Done Sent false
1998 Acked Flush 0 Done Done Sent false
1999 Acked Read 8 Done Done Sent false
2000 Acked Write 10 Done Done Sent false
2001 Acked Write 5 Done Done Sent false
2002 Acked Write 4 Done Done Sent false
2003 Acked Write 1 Done Done Sent false
2004 Acked Read 4 Done Done Sent false
2005 Acked Read 8 Done Done Sent false
2006 Acked Write 9 Done Done Sent false
2007 Acked Write 3 Done Done Sent false
2008 Acked Read 2 Done Done Sent false
2009 Acked Flush 0 Done Done Sent false
2010 Acked Write 9 Done Done Sent false
2011 Acked Read 9 Done Done Sent false
2012 Acked Write 9 Done Done Sent false
2013 Acked Read 7 Done Done Sent false
2014 Acked Write 10 Done Done Sent false
2015 Acked Write 2 Done Done Sent false
2016 Acked Read 1 Done Done Sent false
2017 Acked Read 5 Done Done Sent false
2018 Acked Read 8 Done Done Sent false
2019 Acked Read 7 Done Done Sent false
2020 Acked Read 8 Done Done Sent false
2021 Acked Read 10 Done Done Sent false
2022 Acked Read 6 Done Done Sent false
2023 Acked Write 1 Done Done Sent false
2024 Acked Write 8 Done Done Sent false
2025 Acked Read 4 Done Done Sent false
2026 Acked Flush 0 Done Done Sent false
2027 Acked Read 9 Done Done Sent false
2028 Acked Write 6 Done Done Sent false
2029 Acked Read 3 Done Done Sent false
2030 Acked Read 1 Done Done Sent false
2031 Acked Read 1 Done Done Sent false
2032 Acked Read 3 Done Done Sent false
2033 Acked Read 2 Done Done Sent false
2034 Acked Read 10 Done Done Sent false
2035 Acked Flush 0 Done Done Sent false
2036 Acked Write 5 Done Done Sent false
2037 Acked Write 5 Done Done Sent false
2038 Acked Flush 0 Done Done Sent false
2039 Acked Write 1 Done Done Sent false
2040 Acked Write 10 Done Done Sent false
2041 Acked Read 6 Done Done Sent false
2042 Acked Flush 0 Done Done Sent false
2043 Acked Read 9 Done Done Sent false
2044 Acked Read 9 Done Done Sent false
2045 Acked Write 6 Done Done Sent false
2046 Acked Flush 0 Done Done Sent false
2047 Acked Flush 0 Done Done Sent false
2048 Acked Read 9 Done Done Sent false
2049 Acked Write 4 Done Done Sent false
2050 Acked Write 5 Done Done Sent false
2051 Acked Write 2 Done Done Sent false
2052 Acked Flush 0 Done Done Sent false
2053 Acked Write 2 Done Done Sent false
2054 Acked Read 4 Done Done Sent false
2055 Acked Write 9 Done Done Sent false
2056 Acked Read 6 Done Done Sent false
2057 Acked Read 2 Done Done Sent false
2058 Acked Read 2 Done Done Sent false
2059 Acked Write 5 Done Done Sent false
2060 Acked Write 3 Done Done Sent false
2061 Acked Write 10 Done Done Sent false
2062 Acked Flush 0 Done Done Sent false
2063 Acked Write 1 Done Done Sent false
2064 Acked Write 8 Done Done Sent false
2065 Acked Read 8 Done Done Sent false
2066 Acked Read 1 Done Done Sent false
2067 Acked Write 9 Done Done Sent false
2068 Acked Read 6 Done Done Sent false
2069 Acked Read 7 Done Done Sent false
2070 Acked Write 3 Done Done Sent false
2071 Acked Write 3 Done Done Sent false
2072 Acked Write 2 Done Done Sent false
2073 Acked Read 9 Done Done Sent false
2074 Acked Write 9 Done Done Sent false
2075 Acked Write 2 Done Done Sent false
2076 Acked Write 3 Done Done Sent false
2077 Acked Read 8 Done Done Sent false
2078 Acked Write 6 Done Done Sent false
2079 Acked Write 2 Done Done Sent false
2080 Acked Read 6 Done Done Sent false
2081 Acked Write 8 Done Done Sent false
2082 Acked Write 3 Done Done Sent false
2083 Acked Read 6 Done Done Sent false
2084 Acked Read 8 Done Done Sent false
2085 Acked Read 10 Done Done Sent false
2086 Acked Flush 0 Done Done Sent false
2087 Acked Read 1 Done Done Sent false
2088 Acked Read 10 Done Done Sent false
2089 Acked Flush 0 Done Done Sent false
2090 Acked Flush 0 Done Done Sent false
2091 Acked Read 4 Done Done Sent false
2092 Acked Read 8 Done Done Sent false
2093 Acked Read 8 Done Done Sent false
2094 Acked Read 6 Done Done Sent false
2095 Acked Flush 0 Done Done Sent false
2096 Acked Read 7 Done Done Sent false
2097 Acked Read 1 Done Done Sent false
2098 Acked Write 10 Done Done Sent false
2099 Acked Read 5 Done Done Sent false
2100 Acked Read 10 Done Done Sent false
2101 Acked Read 10 Done Done Sent false
2102 Acked Write 4 Done Done Sent false
2103 Acked Write 4 Done Done Sent false
2104 Acked Read 7 Done Done Sent false
2105 Acked Write 7 Done Done Sent false
2106 Acked Read 2 Done Done Sent false
2107 Acked Flush 0 Done Done Sent false
2108 Acked Read 1 Done Done Sent false
2109 Acked Write 7 Done Done Sent false
2110 Acked Read 4 Done Done Sent false
2111 Acked Read 1 Done Done Sent false
2112 Acked Read 2 Done Done Sent false
2113 Acked Read 7 Done Done Sent false
2114 Acked Write 10 Done Done Sent false
2115 Acked Flush 0 Done Done Sent false
2116 Acked Flush 0 Done Done Sent false
2117 Acked Read 6 Done Done Sent false
2118 Acked Read 8 Done Done Sent false
2119 Acked Read 1 Done Done Sent false
2120 Acked Write 8 Done Done Sent false
2121 Acked Read 7 Done Done Sent false
2122 Acked Write 9 Done Done Sent false
2123 Acked Read 1 Done Done Sent false
2124 Acked Read 3 Done Done Sent false
2125 Acked Write 9 Done Done Sent false
2126 Acked Read 9 Done Done Sent false
2127 Acked Write 8 Done Done Sent false
2128 Acked Write 5 Done Done Sent false
2129 Acked Read 3 Done Done Sent false
2130 Acked Read 2 Done Done Sent false
2131 Acked Read 9 Done Done Sent false
2132 Acked Write 1 Done Done Sent false
2133 Acked Read 10 Done Done Sent false
2134 Acked Flush 0 Done Done Sent false
2135 Acked Write 8 Done Done Sent false
2136 Acked Read 9 Done Done Sent false
2137 Acked Read 4 Done Done Sent false
2138 Acked Write 6 Done Done Sent false
2139 Acked Flush 0 Done Done Sent false
2140 Acked Write 7 Done Done Sent false
2141 Acked Write 4 Done Done Sent false
2142 Acked Flush 0 Done Done Sent false
2143 Acked Write 2 Done Done Sent false
2144 Acked Write 10 Done Done Sent false
2145 Acked Flush 0 Done Done Sent false
2146 Acked Write 1 Done Done Sent false
2147 Acked Read 3 Done Done Sent false
2148 Acked Read 4 Done Done Sent false
2149 Acked Write 7 Done Done Sent false
2150 Acked Write 9 Done Done Sent false
2151 Acked Read 10 Done Done Sent false
2152 Acked Read 8 Done Done Sent false
2153 Acked Read 2 Done Done Sent false
2154 Acked Write 8 Done Done Sent false
2155 Acked Flush 0 Done Done Sent false
2156 Acked Flush 0 Done Done Sent false
2157 Acked Read 8 Done Done Sent false
2158 Acked Read 7 Done Done Sent false
2159 Acked Read 10 Done Done Sent false
2160 Acked Write 6 Done Done Sent false
2161 Acked Read 6 Done Done Sent false
2162 Acked Write 1 Done Done Sent false
2163 Acked Write 7 Done Done Sent false
2164 Acked Write 1 Done Done Sent false
2165 Acked Read 2 Done Done Sent false
2166 Acked Write 8 Done Done Sent false
2167 Acked Read 5 Done Done Sent false
2168 Acked Read 9 Done Done Sent false
2169 Acked Write 7 Done Done Sent false
2170 Acked Read 2 Done Done Sent false
2171 Acked Read 10 Done Done Sent false
2172 Acked Write 2 Done Done Sent false
2173 Acked Write 4 Done Done Sent false
2174 Acked Flush 0 Done Done Sent false
2175 Acked Read 3 Done Done Sent false
2176 Acked Read 1 Done Done Sent false
2177 Acked Read 8 Done Done Sent false
2178 Acked Flush 0 Done Done Sent false
2179 Acked Write 6 Done Done Sent false
2180 Acked Write 3 Done Done Sent false
2181 Acked Read 5 Done Done Sent false
2182 Acked Write 8 Done Done Sent false
2183 Acked Read 8 Done Done Sent false
2184 Acked Read 4 Done Done Sent false
2185 Acked Read 5 Done Done Sent false
2186 Acked Read 10 Done Done Sent false
2187 Acked Write 1 Done Done Sent false
2188 Acked Write 1 Done Done Sent false
2189 Acked Write 7 Done Done Sent false
2190 Acked Write 6 Done Done Sent false
2191 Acked Write 4 Done Done Sent false
2192 Acked Read 10 Done Done Sent false
2193 Acked Write 10 Done Done Sent false
2194 Acked Flush 0 Done Done Sent false
2195 Acked Write 2 Done Done Sent false
2196 Acked Flush 0 Done Done Sent false
2197 Acked Write 4 Done Done Sent false
2198 Acked Write 9 Done Done Sent false
2199 Acked Read 9 Done Done Sent false
2200 Acked Read 3 Done Done Sent false
2201 Acked Write 6 Done Done Sent false
2202 Acked Read 3 Done Done Sent false
2203 Acked Read 4 Done Done Sent false
2204 Acked Write 10 Done Done Sent false
2205 Acked Write 3 Done Done Sent false
2206 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2196 2196 1905
Downstairs last five completed: 1905 1904 1903 1902 1901
Upstairs last five completed: 2206 2205 2204 2203 2202
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1204)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1906 Acked Write 7 Done Done Sent false
1907 Acked Read 3 Done Done Sent false
1908 Acked Write 7 Done Done Sent false
1909 Acked Write 5 Done Done Sent false
1910 Acked Write 6 Done Done Sent false
1911 Acked Read 6 Done Done Sent false
1912 Acked Read 1 Done Done Sent false
1913 Acked Read 8 Done Done Sent false
1914 Acked Read 5 Done Done Sent false
1915 Acked Write 8 Done Done Sent false
1916 Acked Flush 0 Done Done Sent false
1917 Acked Read 7 Done Done Sent false
1918 Acked Read 9 Done Done Sent false
1919 Acked Read 1 Done Done Sent false
1920 Acked Flush 0 Done Done Sent false
1921 Acked Flush 0 Done Done Sent false
1922 Acked Write 5 Done Done Sent false
1923 Acked Write 4 Done Done Sent false
1924 Acked Write 2 Done Done Sent false
1925 Acked Write 2 Done Done Sent false
1926 Acked Read 8 Done Done Sent false
1927 Acked Read 3 Done Done Sent false
1928 Acked Write 6 Done Done Sent false
1929 Acked Write 8 Done Done Sent false
1930 Acked Read 9 Done Done Sent false
1931 Acked Read 5 Done Done Sent false
1932 Acked Write 6 Done Done Sent false
1933 Acked Read 5 Done Done Sent false
1934 Acked Read 10 Done Done Sent false
1935 Acked Write 5 Done Done Sent false
1936 Acked Read 6 Done Done Sent false
1937 Acked Read 2 Done Done Sent false
1938 Acked Write 1 Done Done Sent false
1939 Acked Read 10 Done Done Sent false
1940 Acked Write 1 Done Done Sent false
1941 Acked Write 7 Done Done Sent false
1942 Acked Read 1 Done Done Sent false
1943 Acked Write 9 Done Done Sent false
1944 Acked Write 4 Done Done Sent false
1945 Acked Flush 0 Done Done Sent false
1946 Acked Write 7 Done Done Sent false
1947 Acked Read 4 Done Done Sent false
1948 Acked Read 1 Done Done Sent false
1949 Acked Read 7 Done Done Sent false
1950 Acked Write 6 Done Done Sent false
1951 Acked Read 7 Done Done Sent false
1952 Acked Write 4 Done Done Sent false
1953 Acked Read 6 Done Done Sent false
1954 Acked Read 4 Done Done Sent false
1955 Acked Read 4 Done Done Sent false
1956 Acked Write 3 Done Done Sent false
1957 Acked Write 10 Done Done Sent false
1958 Acked Read 3 Done Done Sent false
1959 Acked Read 9 Done Done Sent false
1960 Acked Flush 0 Done Done Sent false
1961 Acked Read 6 Done Done Sent false
1962 Acked Write 8 Done Done Sent false
1963 Acked Read 7 Done Done Sent false
1964 Acked Read 2 Done Done Sent false
1965 Acked Read 3 Done Done Sent false
1966 Acked Read 6 Done Done Sent false
1967 Acked Read 8 Done Done Sent false
1968 Acked Read 2 Done Done Sent false
1969 Acked Read 8 Done Done Sent false
1970 Acked Write 2 Done Done Sent false
1971 Acked Read 5 Done Done Sent false
1972 Acked Read 10 Done Done Sent false
1973 Acked Write 3 Done Done Sent false
1974 Acked Read 8 Done Done Sent false
1975 Acked Read 8 Done Done Sent false
1976 Acked Read 2 Done Done Sent false
1977 Acked Write 1 Done Done Sent false
1978 Acked Read 6 Done Done Sent false
1979 Acked Flush 0 Done Done Sent false
1980 Acked Write 8 Done Done Sent false
1981 Acked Write 4 Done Done Sent false
1982 Acked Write 9 Done Done Sent false
1983 Acked Flush 0 Done Done Sent false
1984 Acked Write 1 Done Done Sent false
1985 Acked Write 2 Done Done Sent false
1986 Acked Read 4 Done Done Sent false
1987 Acked Write 10 Done Done Sent false
1988 Acked Read 7 Done Done Sent false
1989 Acked Read 9 Done Done Sent false
1990 Acked Read 8 Done Done Sent false
1991 Acked Write 2 Done Done Sent false
1992 Acked Flush 0 Done Done Sent false
1993 Acked Write 10 Done Done Sent false
1994 Acked Write 5 Done Done Sent false
1995 Acked Read 4 Done Done Sent false
1996 Acked Write 7 Done Done Sent false
1997 Acked Read 2 Done Done Sent false
1998 Acked Flush 0 Done Done Sent false
1999 Acked Read 8 Done Done Sent false
2000 Acked Write 10 Done Done Sent false
2001 Acked Write 5 Done Done Sent false
2002 Acked Write 4 Done Done Sent false
2003 Acked Write 1 Done Done Sent false
2004 Acked Read 4 Done Done Sent false
2005 Acked Read 8 Done Done Sent false
2006 Acked Write 9 Done Done Sent false
2007 Acked Write 3 Done Done Sent false
2008 Acked Read 2 Done Done Sent false
2009 Acked Flush 0 Done Done Sent false
2010 Acked Write 9 Done Done Sent false
2011 Acked Read 9 Done Done Sent false
2012 Acked Write 9 Done Done Sent false
2013 Acked Read 7 Done Done Sent false
2014 Acked Write 10 Done Done Sent false
2015 Acked Write 2 Done Done Sent false
2016 Acked Read 1 Done Done Sent false
2017 Acked Read 5 Done Done Sent false
2018 Acked Read 8 Done Done Sent false
2019 Acked Read 7 Done Done Sent false
2020 Acked Read 8 Done Done Sent false
2021 Acked Read 10 Done Done Sent false
2022 Acked Read 6 Done Done Sent false
2023 Acked Write 1 Done Done Sent false
2024 Acked Write 8 Done Done Sent false
2025 Acked Read 4 Done Done Sent false
2026 Acked Flush 0 Done Done Sent false
2027 Acked Read 9 Done Done Sent false
2028 Acked Write 6 Done Done Sent false
2029 Acked Read 3 Done Done Sent false
2030 Acked Read 1 Done Done Sent false
2031 Acked Read 1 Done Done Sent false
2032 Acked Read 3 Done Done Sent false
2033 Acked Read 2 Done Done Sent false
2034 Acked Read 10 Done Done Sent false
2035 Acked Flush 0 Done Done Sent false
2036 Acked Write 5 Done Done Sent false
2037 Acked Write 5 Done Done Sent false
2038 Acked Flush 0 Done Done Sent false
2039 Acked Write 1 Done Done Sent false
2040 Acked Write 10 Done Done Sent false
2041 Acked Read 6 Done Done Sent false
2042 Acked Flush 0 Done Done Sent false
2043 Acked Read 9 Done Done Sent false
2044 Acked Read 9 Done Done Sent false
2045 Acked Write 6 Done Done Sent false
2046 Acked Flush 0 Done Done Sent false
2047 Acked Flush 0 Done Done Sent false
2048 Acked Read 9 Done Done Sent false
2049 Acked Write 4 Done Done Sent false
2050 Acked Write 5 Done Done Sent false
2051 Acked Write 2 Done Done Sent false
2052 Acked Flush 0 Done Done Sent false
2053 Acked Write 2 Done Done Sent false
2054 Acked Read 4 Done Done Sent false
2055 Acked Write 9 Done Done Sent false
2056 Acked Read 6 Done Done Sent false
2057 Acked Read 2 Done Done Sent false
2058 Acked Read 2 Done Done Sent false
2059 Acked Write 5 Done Done Sent false
2060 Acked Write 3 Done Done Sent false
2061 Acked Write 10 Done Done Sent false
2062 Acked Flush 0 Done Done Sent false
2063 Acked Write 1 Done Done Sent false
2064 Acked Write 8 Done Done Sent false
2065 Acked Read 8 Done Done Sent false
2066 Acked Read 1 Done Done Sent false
2067 Acked Write 9 Done Done Sent false
2068 Acked Read 6 Done Done Sent false
2069 Acked Read 7 Done Done Sent false
2070 Acked Write 3 Done Done Sent false
2071 Acked Write 3 Done Done Sent false
2072 Acked Write 2 Done Done Sent false
2073 Acked Read 9 Done Done Sent false
2074 Acked Write 9 Done Done Sent false
2075 Acked Write 2 Done Done Sent false
2076 Acked Write 3 Done Done Sent false
2077 Acked Read 8 Done Done Sent false
2078 Acked Write 6 Done Done Sent false
2079 Acked Write 2 Done Done Sent false
2080 Acked Read 6 Done Done Sent false
2081 Acked Write 8 Done Done Sent false
2082 Acked Write 3 Done Done Sent false
2083 Acked Read 6 Done Done Sent false
2084 Acked Read 8 Done Done Sent false
2085 Acked Read 10 Done Done Sent false
2086 Acked Flush 0 Done Done Sent false
2087 Acked Read 1 Done Done Sent false
2088 Acked Read 10 Done Done Sent false
2089 Acked Flush 0 Done Done Sent false
2090 Acked Flush 0 Done Done Sent false
2091 Acked Read 4 Done Done Sent false
2092 Acked Read 8 Done Done Sent false
2093 Acked Read 8 Done Done Sent false
2094 Acked Read 6 Done Done Sent false
2095 Acked Flush 0 Done Done Sent false
2096 Acked Read 7 Done Done Sent false
2097 Acked Read 1 Done Done Sent false
2098 Acked Write 10 Done Done Sent false
2099 Acked Read 5 Done Done Sent false
2100 Acked Read 10 Done Done Sent false
2101 Acked Read 10 Done Done Sent false
2102 Acked Write 4 Done Done Sent false
2103 Acked Write 4 Done Done Sent false
2104 Acked Read 7 Done Done Sent false
2105 Acked Write 7 Done Done Sent false
2106 Acked Read 2 Done Done Sent false
2107 Acked Flush 0 Done Done Sent false
2108 Acked Read 1 Done Done Sent false
2109 Acked Write 7 Done Done Sent false
2110 Acked Read 4 Done Done Sent false
2111 Acked Read 1 Done Done Sent false
2112 Acked Read 2 Done Done Sent false
2113 Acked Read 7 Done Done Sent false
2114 Acked Write 10 Done Done Sent false
2115 Acked Flush 0 Done Done Sent false
2116 Acked Flush 0 Done Done Sent false
2117 Acked Read 6 Done Done Sent false
2118 Acked Read 8 Done Done Sent false
2119 Acked Read 1 Done Done Sent false
2120 Acked Write 8 Done Done Sent false
2121 Acked Read 7 Done Done Sent false
2122 Acked Write 9 Done Done Sent false
2123 Acked Read 1 Done Done Sent false
2124 Acked Read 3 Done Done Sent false
2125 Acked Write 9 Done Done Sent false
2126 Acked Read 9 Done Done Sent false
2127 Acked Write 8 Done Done Sent false
2128 Acked Write 5 Done Done Sent false
2129 Acked Read 3 Done Done Sent false
2130 Acked Read 2 Done Done Sent false
2131 Acked Read 9 Done Done Sent false
2132 Acked Write 1 Done Done Sent false
2133 Acked Read 10 Done Done Sent false
2134 Acked Flush 0 Done Done Sent false
2135 Acked Write 8 Done Done Sent false
2136 Acked Read 9 Done Done Sent false
2137 Acked Read 4 Done Done Sent false
2138 Acked Write 6 Done Done Sent false
2139 Acked Flush 0 Done Done Sent false
2140 Acked Write 7 Done Done Sent false
2141 Acked Write 4 Done Done Sent false
2142 Acked Flush 0 Done Done Sent false
2143 Acked Write 2 Done Done Sent false
2144 Acked Write 10 Done Done Sent false
2145 Acked Flush 0 Done Done Sent false
2146 Acked Write 1 Done Done Sent false
2147 Acked Read 3 Done Done Sent false
2148 Acked Read 4 Done Done Sent false
2149 Acked Write 7 Done Done Sent false
2150 Acked Write 9 Done Done Sent false
2151 Acked Read 10 Done Done Sent false
2152 Acked Read 8 Done Done Sent false
2153 Acked Read 2 Done Done Sent false
2154 Acked Write 8 Done Done Sent false
2155 Acked Flush 0 Done Done Sent false
2156 Acked Flush 0 Done Done Sent false
2157 Acked Read 8 Done Done Sent false
2158 Acked Read 7 Done Done Sent false
2159 Acked Read 10 Done Done Sent false
2160 Acked Write 6 Done Done Sent false
2161 Acked Read 6 Done Done Sent false
2162 Acked Write 1 Done Done Sent false
2163 Acked Write 7 Done Done Sent false
2164 Acked Write 1 Done Done Sent false
2165 Acked Read 2 Done Done Sent false
2166 Acked Write 8 Done Done Sent false
2167 Acked Read 5 Done Done Sent false
2168 Acked Read 9 Done Done Sent false
2169 Acked Write 7 Done Done Sent false
2170 Acked Read 2 Done Done Sent false
2171 Acked Read 10 Done Done Sent false
2172 Acked Write 2 Done Done Sent false
2173 Acked Write 4 Done Done Sent false
2174 Acked Flush 0 Done Done Sent false
2175 Acked Read 3 Done Done Sent false
2176 Acked Read 1 Done Done Sent false
2177 Acked Read 8 Done Done Sent false
2178 Acked Flush 0 Done Done Sent false
2179 Acked Write 6 Done Done Sent false
2180 Acked Write 3 Done Done Sent false
2181 Acked Read 5 Done Done Sent false
2182 Acked Write 8 Done Done Sent false
2183 Acked Read 8 Done Done Sent false
2184 Acked Read 4 Done Done Sent false
2185 Acked Read 5 Done Done Sent false
2186 Acked Read 10 Done Done Sent false
2187 Acked Write 1 Done Done Sent false
2188 Acked Write 1 Done Done Sent false
2189 Acked Write 7 Done Done Sent false
2190 Acked Write 6 Done Done Sent false
2191 Acked Write 4 Done Done Sent false
2192 Acked Read 10 Done Done Sent false
2193 Acked Write 10 Done Done Sent false
2194 Acked Flush 0 Done Done Sent false
2195 Acked Write 2 Done Done Sent false
2196 Acked Flush 0 Done Done Sent false
2197 Acked Write 4 Done Done Sent false
2198 Acked Write 9 Done Done Sent false
2199 Acked Read 9 Done Done Sent false
2200 Acked Read 3 Done Done Sent false
2201 Acked Write 6 Done Done Sent false
2202 Acked Read 3 Done Done Sent false
2203 Acked Read 4 Done Done Sent false
2204 Acked Write 10 Done Done Sent false
2205 Acked Write 3 Done Done Sent false
2206 Acked Read 6 Done Done Sent false
2207 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2207 2207 1905
Downstairs last five completed: 1905 1904 1903 1902 1901
Upstairs last five completed: 2207 2206 2205 2204 2203
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.658305816Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49560)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.658459538Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.659420636Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.660042173Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.66007066Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 1905","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.660086058Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 1905","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.660322851Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 1905","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.660344878Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 1905","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:11.660566152Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2207 2206 2205 2204 2203
Upstairs last five completed: 2207 2206 2205 2204 2203
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:37:13.899 INFO [8810] Got stop action so:false kr:false
request completed, latency_us: 367, response_code: 204, uri: /stop/cid/0, method: POST, req_id: e3202517-f3d4-4353-9906-0b6cd855b192, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:13.900697057Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:37:13.900 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/0, method: GET, req_id: d1fa1e2c-b3c6-48c4-9136-52912ddca85d, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:13.900792557Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:13.900836371Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:13.900867418Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:37:17.903 INFO request completed, latency_us: 364, response_code: 200, uri: /state/cid/0, method: GET, req_id: f35e5b79-5f1b-4228-a79d-dfb24713d66c, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 2972 len 16384
002/300 Flush
003/300 Write block 194 len 28672 data: 4 4 3 3 3 3 3
004/300 Write block 4190 len 32768 data: 5 5 5 4 4 4 5 4
005/300 Write block 568 len 24576 data: 2 2 2 2 2 2
006/300 Read block 673 len 16384
007/300 Write block 2977 len 20480 data: 2 3 3 3 3
008/300 Write block 3926 len 32768 data: 2 2 2 2 2 2 2 2
009/300 Read block 4221 len 28672
010/300 Write block 76 len 32768 data: 2 2 2 2 2 2 2 2
011/300 Read block 2573 len 20480
012/300 Read block 4906 len 36864
013/300 Write block 4922 len 8192 data: 3 3
014/300 Read block 3179 len 20480
015/300 Write block 2663 len 16384 data: 3 3 3 2
016/300 Read block 2032 len 36864
017/300 Read block 1777 len 36864
018/300 Write block 3720 len 4096 data: 2
019/300 Read block 1982 len 4096
020/300 Read block 4401 len 12288
021/300 Read block 3264 len 36864
022/300 Read block 3882 len 36864
023/300 Read block 296 len 8192
024/300 Write block 1028 len 36864 data: 2 4 4 3 3 3 3 3 3
025/300 Read block 2078 len 32768
026/300 Write block 3735 len 12288 data: 2 2 2
027/300 Write block 4405 len 16384 data: 2 2 2 2
028/300 Read block 3473 len 16384
029/300 Flush
030/300 Read block 1388 len 8192
031/300 Read block 2933 len 28672
032/300 Read block 1808 len 32768
033/300 Read block 4707 len 8192
034/300 Read block 1414 len 20480
035/300 Write block 2359 len 8192 data: 2 2
036/300 Write block 3931 len 28672 data: 3 3 3 2 2 3 3
037/300 Write block 910 len 20480 data: 2 2 2 2 2
038/300 Flush
039/300 Write block 88 len 4096 data: 2
040/300 Read block 1435 len 12288
041/300 Read block 2694 len 16384
042/300 Read block 2811 len 36864
043/300 Read block 3500 len 20480
044/300 Read block 1550 len 4096
045/300 Read block 4114 len 36864
046/300 Write block 3178 len 28672 data: 4 4 3 3 3 2 2
047/300 Read block 439 len 36864
048/300 Write block 4243 len 28672 data: 4 3 4 5 5 5 5
049/300 Write block 502 len 20480 data: 2 2 2 3 3
050/300 Read block 853 len 20480
051/300 Write block 941 len 28672 data: 2 2 2 2 2 2 2
052/300 Read block 4729 len 32768
053/300 Write block 4284 len 12288 data: 2 2 2
054/300 Read block 714 len 8192
055/300 Read block 4035 len 20480
056/300 Read block 16 len 28672
057/300 Read block 135 len 16384
058/300 Read block 4442 len 8192
059/300 Write block 2364 len 32768 data: 2 2 2 2 2 2 2 3
060/300 Write block 3658 len 28672 data: 2 2 2 2 2 2 2
061/300 Read block 2634 len 12288
062/300 Flush
063/300 Write block 346 len 12288 data: 2 2 2
064/300 Write block 2391 len 16384 data: 3 2 3 2
065/300 Read block 4994 len 12288
066/300 Write block 3292 len 20480 data: 2 3 3 3 3
067/300 Write block 4042 len 32768 data: 2 2 2 2 2 2 2 2
068/300 Write block 206 len 4096 data: 2
069/300 Read block 3541 len 20480
070/300 Read block 2185 len 40960
071/300 Write block 2222 len 24576 data: 2 2 2 2 2 3
072/300 Read block 3305 len 32768
073/300 Write block 96 len 36864 data: 2 2 2 2 2 2 3 3 3
074/300 Write block 4336 len 24576 data: 4 4 4 3 2 2
075/300 Write block 1509 len 4096 data: 2
076/300 Flush
077/300 Read block 2981 len 24576
078/300 Write block 1425 len 16384 data: 2 2 2 2
079/300 Write block 4531 len 24576 data: 2 2 2 2 2 2
080/300 Read block 2661 len 12288
081/300 Read block 889 len 16384
082/300 Read block 3016 len 12288
083/300 Flush
084/300 Read block 1016 len 12288
085/300 Read block 1913 len 24576
086/300 Write block 4193 len 40960 data: 5 5 5 6 5 4 3 2 2 2
087/300 Read block 4997 len 8192
088/300 Read block 2067 len 12288
089/300 Read block 1548 len 24576
090/300 Read block 4766 len 24576
091/300 Write block 1428 len 8192 data: 3 2
092/300 Write block 451 len 32768 data: 2 2 2 2 2 2 2 2
093/300 Write block 3880 len 36864 data: 2 2 2 2 2 2 2 2 2
094/300 Read block 1712 len 40960
095/300 Read block 1830 len 24576
096/300 Read block 4531 len 20480
097/300 Read block 2577 len 36864
098/300 Read block 1643 len 28672
099/300 Read block 928 len 36864
100/300 Flush
101/300 Write block 1076 len 20480 data: 2 2 2 2 2
102/300 Flush
103/300 Read block 3328 len 8192
104/300 Read block 1445 len 4096
105/300 Read block 773 len 12288
106/300 Write block 3094 len 8192 data: 2 2
107/300 Read block 251 len 12288
108/300 Read block 1381 len 8192
109/300 Read block 3008 len 24576
110/300 Flush
111/300 Write block 1995 len 20480 data: 2 2 2 2 2
112/300 Write block 1031 len 16384 data: 4 4 4 4
113/300 Write block 492 len 40960 data: 2 2 2 2 2 2 3 3 2 2
114/300 Write block 4103 len 4096 data: 3
115/300 Read block 4867 len 8192
116/300 Write block 2162 len 32768 data: 3 3 3 3 3 2 2 2
117/300 Write block 2979 len 36864 data: 4 4 4 3 3 2 2 2 2
118/300 Flush
119/300 Read block 3419 len 4096
120/300 Read block 3402 len 4096
121/300 Write block 1845 len 32768 data: 2 2 2 3 3 4 4 4
122/300 Read block 1170 len 8192
123/300 Write block 76 len 16384 data: 3 3 3 3
124/300 Read block 231 len 40960
125/300 Read block 1900 len 32768
126/300 Read block 1781 len 36864
127/300 Read block 1322 len 24576
128/300 Write block 2557 len 40960 data: 3 2 2 3 3 3 4 4 4 4
129/300 Write block 3774 len 36864 data: 4 4 4 4 4 3 2 2 2
130/300 Flush
131/300 Read block 1687 len 16384
132/300 Read block 3053 len 40960
133/300 Write block 4594 len 40960 data: 2 2 2 2 2 2 2 3 3 3
134/300 Write block 4708 len 12288 data: 3 4 4
135/300 Write block 2598 len 12288 data: 2 2 2
136/300 Write block 4077 len 8192 data: 2 2
137/300 Write block 1019 len 12288 data: 4 4 3
138/300 Flush
139/300 Read block 786 len 16384
140/300 Read block 764 len 28672
141/300 Read block 3603 len 12288
142/300 Flush
143/300 Write block 742 len 32768 data: 4 4 3 3 2 2 2 2
144/300 Read block 1802 len 20480
145/300 Read block 4158 len 12288
146/300 Write block 2274 len 8192 data: 2 2
147/300 Read block 4741 len 24576
148/300 Read block 3899 len 28672
149/300 Read block 2096 len 8192
150/300 Read block 3465 len 4096
151/300 Read block 4937 len 24576
152/300 Write block 3091 len 20480 data: 2 2 2 3 3
153/300 Write block 3919 len 16384 data: 3 2 3 3
154/300 Read block 4487 len 28672
155/300 Read block 4679 len 16384
156/300 Write block 4355 len 20480 data: 2 2 2 2 2
157/300 Write block 4733 len 36864 data: 4 3 4 4 4 3 3 3 3
158/300 Read block 4335 len 16384
159/300 Read block 3797 len 4096
160/300 Read block 3265 len 20480
161/300 Read block 3394 len 4096
162/300 Read block 1987 len 40960
163/300 Read block 813 len 16384
164/300 Read block 1475 len 20480
165/300 Read block 3492 len 12288
166/300 Flush
167/300 Read block 4321 len 32768
168/300 Read block 788 len 32768
169/300 Read block 4104 len 4096
170/300 Read block 459 len 24576
171/300 Read block 3992 len 28672
172/300 Write block 1478 len 20480 data: 2 2 3 3 3
173/300 Flush
174/300 Read block 4246 len 28672
175/300 Flush
176/300 Read block 1325 len 16384
177/300 Read block 3465 len 4096
178/300 Write block 1420 len 40960 data: 3 3 3 2 2 3 3 3 4 3
179/300 Write block 3397 len 8192 data: 3 3
180/300 Flush
181/300 Flush
182/300 Read block 2932 len 12288
183/300 Flush
184/300 Read block 3129 len 20480
185/300 Read block 1752 len 40960
186/300 Write block 552 len 16384 data: 2 2 2 2
187/300 Write block 4411 len 12288 data: 2 2 3
188/300 Read block 1224 len 4096
189/300 Read block 3635 len 24576
190/300 Read block 2885 len 36864
191/300 Write block 1098 len 20480 data: 3 3 3 4 4
192/300 Read block 3438 len 32768
193/300 Read block 1443 len 4096
194/300 Read block 1378 len 4096
195/300 Read block 264 len 28672
196/300 Read block 1840 len 24576
197/300 Read block 4744 len 16384
198/300 Read block 3218 len 20480
199/300 Write block 2286 len 12288 data: 4 5 6
200/300 Write block 312 len 28672 data: 2 2 2 2 2 3 3
201/300 Read block 4718 len 24576
202/300 Write block 2026 len 28672 data: 3 3 3 3 3 3 3
203/300 Read block 4351 len 40960
204/300 Read block 4798 len 12288
205/300 Read block 3528 len 20480
206/300 Write block 3388 len 16384 data: 3 3 3 3
207/300 Write block 2830 len 20480 data: 2 2 2 2 2
208/300 Write block 2448 len 40960 data: 2 2 2 2 2 2 2 2 2 2
209/300 Write block 3520 len 28672 data: 2 2 2 2 2 2 2
210/300 Read block 347 len 8192
211/300 Read block 2563 len 8192
212/300 Write block 4437 len 24576 data: 2 2 3 3 3 3
213/300 Read block 919 len 12288
214/300 Read block 4208 len 24576
215/300 Write block 1314 len 32768 data: 2 2 2 2 2 2 2 2
216/300 Read block 2358 len 40960
217/300 Write block 3747 len 16384 data: 2 2 2 2
218/300 Write block 623 len 12288 data: 2 2 2
219/300 Read block 3029 len 16384
220/300 Write block 418 len 8192 data: 2 2
221/300 Write block 3271 len 16384 data: 3 3 3 3
222/300 Write block 2253 len 12288 data: 2 2 3
223/300 Read block 528 len 40960
224/300 Read block 1733 len 36864
225/300 Write block 2969 len 32768 data: 2 2 2 2 2 2 2 2
226/300 Write block 2205 len 4096 data: 3
227/300 Read block 2403 len 24576
228/300 Write block 3826 len 36864 data: 3 3 3 3 3 2 2 2 2
229/300 Write block 3150 len 12288 data: 3 4 3
230/300 Write block 1164 len 36864 data: 2 2 2 2 2 2 2 2 2
231/300 Write block 2709 len 8192 data: 2 2
232/300 Write block 3023 len 36864 data: 3 3 3 2 2 2 2 2 2
233/300 Write block 1930 len 4096 data: 2
234/300 Write block 4967 len 24576 data: 3 3 3 4 4 2
235/300 Read block 4365 len 16384
236/300 Flush
237/300 Write block 2718 len 36864 data: 3 3 3 3 3 2 2 2 2
238/300 Read block 3912 len 40960
239/300 Write block 2124 len 8192 data: 2 2
240/300 Read block 3055 len 32768
241/300 Write block 3576 len 4096 data: 2
242/300 Read block 1950 len 4096
243/300 Write block 4034 len 16384 data: 3 3 3 3
244/300 Read block 316 len 36864
245/300 Read block 3476 len 40960
246/300 Write block 288 len 28672 data: 2 2 2 2 2 2 2
247/300 Read block 4746 len 16384
248/300 Flush
249/300 Read block 298 len 36864
250/300 Write block 255 len 16384 data: 2 2 2 2
251/300 Read block 3557 len 32768
252/300 Write block 4829 len 20480 data: 2 2 2 2 2
253/300 Flush
254/300 Read block 4851 len 36864
255/300 Flush
256/300 Write block 2255 len 8192 data: 4 3
257/300 Write block 2164 len 28672 data: 4 4 4 3 3 3 2
258/300 Write block 225 len 12288 data: 3 3 3
259/300 Read block 521 len 28672
260/300 Flush
261/300 Flush
262/300 Write block 3845 len 20480 data: 2 2 2 2 2
263/300 Write block 4902 len 4096 data: 2
264/300 Write block 4863 len 8192 data: 2 2
265/300 Read block 3102 len 28672
266/300 Read block 582 len 28672
267/300 Write block 280 len 40960 data: 5 5 4 3 3 3 3 2 3 3
268/300 Read block 1045 len 32768
269/300 Write block 4882 len 4096 data: 3
270/300 Flush
271/300 Write block 789 len 36864 data: 2 2 2 2 2 2 2 2 2
272/300 Read block 1615 len 28672
273/300 Read block 4497 len 32768
274/300 Write block 4907 len 8192 data: 2 3
275/300 Read block 2427 len 36864
276/300 Write block 1464 len 36864 data: 2 2 2 2 2 2 3 3 3
277/300 Read block 293 len 24576
278/300 Read block 414 len 24576
279/300 Write block 4957 len 32768 data: 2 3 3 3 3 3 3 3
280/300 Write block 2751 len 24576 data: 2 2 2 2 2 2
281/300 Read block 2452 len 8192
282/300 Flush
283/300 Read block 3547 len 16384
284/300 Write block 4830 len 4096 data: 3
285/300 Flush
286/300 Read block 1152 len 16384
287/300 Write block 1843 len 24576 data: 2 2 3 3 3 4
288/300 Read block 40 len 36864
289/300 Read block 1823 len 40960
290/300 Write block 794 len 16384 data: 3 3 3 3
291/300 Read block 156 len 28672
292/300 Write block 3708 len 32768 data: 2 2 2 2 2 2 2 2
293/300 Write block 2619 len 28672 data: 3 3 3 3 3 3 2
294/300 Read block 1724 len 24576
295/300 Write block 4943 len 36864 data: 4 4 3 3 3 2 2 2 2
296/300 Read block 4265 len 32768
297/300 Write block 1854 len 36864 data: 4 5 4 4 4 4 3 3 2
298/300 Write block 735 len 12288 data: 2 2 2
299/300 Read block 3418 len 20480
300/300 Flush
301/300 Write block 588 len 36864 data: 4 4 4 4 4 4 3 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:37:18.143 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/0, method: POST, req_id: e1249de8-2e63-44ad-884a-9b010a6a24be, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2208 Acked Read 4 Sent Done Done false
2209 Acked Flush 0 Sent Done Done false
2210 Acked Write 7 Sent Done Done false
2211 Acked Write 8 Sent Done Done false
2212 Acked Write 6 Sent Done Done false
2213 Acked Read 4 Sent Done Done false
2214 Acked Write 5 Sent Done Done false
2215 Acked Write 8 Sent Done Done false
2216 Acked Read 7 Sent Done Done false
2217 Acked Write 8 Sent Done Done false
2218 Acked Read 5 Sent Done Done false
2219 Acked Read 9 Sent Done Done false
2220 Acked Write 2 Sent Done Done false
2221 Acked Read 5 Sent Done Done false
2222 Acked Write 4 Sent Done Done false
2223 Acked Read 9 Sent Done Done false
2224 Acked Read 9 Sent Done Done false
2225 Acked Write 1 Sent Done Done false
2226 Acked Read 1 Sent Done Done false
2227 Acked Read 3 Sent Done Done false
2228 Acked Read 9 Sent Done Done false
2229 Acked Read 9 Sent Done Done false
2230 Acked Read 2 Sent Done Done false
2231 Acked Write 9 Sent Done Done false
2232 Acked Read 8 Sent Done Done false
2233 Acked Write 3 Sent Done Done false
2234 Acked Write 4 Sent Done Done false
2235 Acked Read 4 Sent Done Done false
2236 Acked Flush 0 Sent Done Done false
2237 Acked Read 2 Sent Done Done false
2238 Acked Read 7 Sent Done Done false
2239 Acked Read 8 Sent Done Done false
2240 Acked Read 2 Sent Done Done false
2241 Acked Read 5 Sent Done Done false
2242 Acked Write 2 Sent Done Done false
2243 Acked Write 7 Sent Done Done false
2244 Acked Write 5 Sent Done Done false
2245 Acked Flush 0 Sent Done Done false
2246 Acked Write 1 Sent Done Done false
2247 Acked Read 3 Sent Done Done false
2248 Acked Read 4 Sent Done Done false
2249 Acked Read 9 Sent Done Done false
2250 Acked Read 5 Sent Done Done false
2251 Acked Read 1 Sent Done Done false
2252 Acked Read 9 Sent Done Done false
2253 Acked Write 7 Sent Done Done false
2254 Acked Read 9 Sent Done Done false
2255 Acked Write 7 Sent Done Done false
2256 Acked Write 5 Sent Done Done false
2257 Acked Read 5 Sent Done Done false
2258 Acked Write 7 Sent Done Done false
2259 Acked Read 8 Sent Done Done false
2260 Acked Write 3 Sent Done Done false
2261 Acked Read 2 Sent Done Done false
2262 Acked Read 5 Sent Done Done false
2263 Acked Read 7 Sent Done Done false
2264 Acked Read 4 Sent Done Done false
2265 Acked Read 2 Sent Done Done false
2266 Acked Write 8 Sent Done Done false
2267 Acked Write 7 Sent Done Done false
2268 Acked Read 3 Sent Done Done false
2269 Acked Flush 0 Sent Done Done false
2270 Acked Write 3 Sent Done Done false
2271 Acked Write 4 Sent Done Done false
2272 Acked Read 3 Sent Done Done false
2273 Acked Write 5 Sent Done Done false
2274 Acked Write 8 Sent Done Done false
2275 Acked Write 1 Sent Done Done false
2276 Acked Read 5 Sent Done Done false
2277 Acked Read 10 Sent Done Done false
2278 Acked Write 6 Sent Done Done false
2279 Acked Read 8 Sent Done Done false
2280 Acked Write 9 Sent Done Done false
2281 Acked Write 6 Sent Done Done false
2282 Acked Write 1 Sent Done Done false
2283 Acked Flush 0 Sent Done Done false
2284 Acked Read 6 Sent Done Done false
2285 Acked Write 4 Sent Done Done false
2286 Acked Write 6 Sent Done Done false
2287 Acked Read 3 Sent Done Done false
2288 Acked Read 4 Sent Done Done false
2289 Acked Read 3 Sent Done Done false
2290 Acked Flush 0 Sent Done Done false
2291 Acked Read 3 Sent Done Done false
2292 Acked Read 6 Sent Done Done false
2293 Acked Write 10 Sent Done Done false
2294 Acked Read 2 Sent Done Done false
2295 Acked Read 3 Sent Done Done false
2296 Acked Read 6 Sent Done Done false
2297 Acked Read 6 Sent Done Done false
2298 Acked Write 2 Sent Done Done false
2299 Acked Write 8 Sent Done Done false
2300 Acked Write 9 Sent Done Done false
2301 Acked Read 10 Sent Done Done false
2302 Acked Read 6 Sent Done Done false
2303 Acked Read 5 Sent Done Done false
2304 Acked Read 9 Sent Done Done false
2305 Acked Read 7 Sent Done Done false
2306 Acked Read 9 Sent Done Done false
2307 Acked Flush 0 Sent Done Done false
2308 Acked Write 5 Sent Done Done false
2309 Acked Flush 0 Sent Done Done false
2310 Acked Read 2 Sent Done Done false
2311 Acked Read 1 Sent Done Done false
2312 Acked Read 3 Sent Done Done false
2313 Acked Write 2 Sent Done Done false
2314 Acked Read 3 Sent Done Done false
2315 Acked Read 2 Sent Done Done false
2316 Acked Read 6 Sent Done Done false
2317 Acked Flush 0 Sent Done Done false
2318 Acked Write 5 Sent Done Done false
2319 Acked Write 4 Sent Done Done false
2320 Acked Write 10 Sent Done Done false
2321 Acked Write 1 Sent Done Done false
2322 Acked Read 2 Sent Done Done false
2323 Acked Write 8 Sent Done Done false
2324 Acked Write 9 Sent Done Done false
2325 Acked Flush 0 Sent Done Done false
2326 Acked Read 1 Sent Done Done false
2327 Acked Read 1 Sent Done Done false
2328 Acked Write 8 Sent Done Done false
2329 Acked Read 2 Sent Done Done false
2330 Acked Write 4 Sent Done Done false
2331 Acked Read 10 Sent Done Done false
2332 Acked Read 8 Sent Done Done false
2333 Acked Read 9 Sent Done Done false
2334 Acked Read 6 Sent Done Done false
2335 Acked Write 10 Sent Done Done false
2336 Acked Write 9 Sent Done Done false
2337 Acked Flush 0 Sent Done Done false
2338 Acked Read 4 Sent Done Done false
2339 Acked Read 10 Sent Done Done false
2340 Acked Write 10 Sent Done Done false
2341 Acked Write 3 Sent Done Done false
2342 Acked Write 3 Sent Done Done false
2343 Acked Write 2 Sent Done Done false
2344 Acked Write 3 Sent Done Done false
2345 Acked Flush 0 Sent Done Done false
2346 Acked Read 4 Sent Done Done false
2347 Acked Read 7 Sent Done Done false
2348 Acked Read 3 Sent Done Done false
2349 Acked Flush 0 Sent Done Done false
2350 Acked Write 8 Sent Done Done false
2351 Acked Read 5 Sent Done Done false
2352 Acked Read 3 Sent Done Done false
2353 Acked Write 2 Sent Done Done false
2354 Acked Read 6 Sent Done Done false
2355 Acked Read 7 Sent Done Done false
2356 Acked Read 2 Sent Done Done false
2357 Acked Read 1 Sent Done Done false
2358 Acked Read 6 Sent Done Done false
2359 Acked Write 5 Sent Done Done false
2360 Acked Write 4 Sent Done Done false
2361 Acked Read 7 Sent Done Done false
2362 Acked Read 4 Sent Done Done false
2363 Acked Write 5 Sent Done Done false
2364 Acked Write 9 Sent Done Done false
2365 Acked Read 4 Sent Done Done false
2366 Acked Read 1 Sent Done Done false
2367 Acked Read 5 Sent Done Done false
2368 Acked Read 1 Sent Done Done false
2369 Acked Read 10 Sent Done Done false
2370 Acked Read 4 Sent Done Done false
2371 Acked Read 5 Sent Done Done false
2372 Acked Read 3 Sent Done Done false
2373 Acked Flush 0 Sent Done Done false
2374 Acked Read 8 Sent Done Done false
2375 Acked Read 8 Sent Done Done false
2376 Acked Read 1 Sent Done Done false
2377 Acked Read 6 Sent Done Done false
2378 Acked Read 7 Sent Done Done false
2379 Acked Write 5 Sent Done Done false
2380 Acked Flush 0 Sent Done Done false
2381 Acked Read 7 Sent Done Done false
2382 Acked Flush 0 Sent Done Done false
2383 Acked Read 4 Sent Done Done false
2384 Acked Read 1 Sent Done Done false
2385 Acked Write 10 Sent Done Done false
2386 Acked Write 2 Sent Done Done false
2387 Acked Flush 0 Sent Done Done false
2388 Acked Flush 0 Sent Done Done false
2389 Acked Read 3 Sent Done Done false
2390 Acked Flush 0 Sent Done Done false
2391 Acked Read 5 Sent Done Done false
2392 Acked Read 10 Sent Done Done false
2393 Acked Write 4 Sent Done Done false
2394 Acked Write 3 Sent Done Done false
2395 Acked Read 1 Sent Done Done false
2396 Acked Read 6 Sent Done Done false
2397 Acked Read 9 Sent Done Done false
2398 Acked Write 5 Sent Done Done false
2399 Acked Read 8 Sent Done Done false
2400 Acked Read 1 Sent Done Done false
2401 Acked Read 1 Sent Done Done false
2402 Acked Read 7 Sent Done Done false
2403 Acked Read 6 Sent Done Done false
2404 Acked Read 4 Sent Done Done false
2405 Acked Read 5 Sent Done Done false
2406 Acked Write 3 Sent Done Done false
2407 Acked Write 7 Sent Done Done false
2408 Acked Read 6 Sent Done Done false
2409 Acked Write 7 Sent Done Done false
2410 Acked Read 10 Sent Done Done false
2411 Acked Read 3 Sent Done Done false
2412 Acked Read 5 Sent Done Done false
2413 Acked Write 4 Sent Done Done false
2414 Acked Write 5 Sent Done Done false
2415 Acked Write 10 Sent Done Done false
2416 Acked Write 7 Sent Done Done false
2417 Acked Read 2 Sent Done Done false
2418 Acked Read 2 Sent Done Done false
2419 Acked Write 6 Sent Done Done false
2420 Acked Read 3 Sent Done Done false
2421 Acked Read 6 Sent Done Done false
2422 Acked Write 8 Sent Done Done false
2423 Acked Read 10 Sent Done Done false
2424 Acked Write 4 Sent Done Done false
2425 Acked Write 3 Sent Done Done false
2426 Acked Read 4 Sent Done Done false
2427 Acked Write 2 Sent Done Done false
2428 Acked Write 4 Sent Done Done false
2429 Acked Write 3 Sent Done Done false
2430 Acked Read 10 Sent Done Done false
2431 Acked Read 9 Sent Done Done false
2432 Acked Write 8 Sent Done Done false
2433 Acked Write 1 Sent Done Done false
2434 Acked Read 6 Sent Done Done false
2435 Acked Write 9 Sent Done Done false
2436 Acked Write 3 Sent Done Done false
2437 Acked Write 9 Sent Done Done false
2438 Acked Write 2 Sent Done Done false
2439 Acked Write 9 Sent Done Done false
2440 Acked Write 1 Sent Done Done false
2441 Acked Write 6 Sent Done Done false
2442 Acked Read 4 Sent Done Done false
2443 Acked Flush 0 Sent Done Done false
2444 Acked Write 9 Sent Done Done false
2445 Acked Read 10 Sent Done Done false
2446 Acked Write 2 Sent Done Done false
2447 Acked Read 8 Sent Done Done false
2448 Acked Write 1 Sent Done Done false
2449 Acked Read 1 Sent Done Done false
2450 Acked Write 4 Sent Done Done false
2451 Acked Read 9 Sent Done Done false
2452 Acked Read 10 Sent Done Done false
2453 Acked Write 7 Sent Done Done false
2454 Acked Read 4 Sent Done Done false
2455 Acked Flush 0 Sent Done Done false
2456 Acked Read 9 Sent Done Done false
2457 Acked Write 4 Sent Done Done false
2458 Acked Read 8 Sent Done Done false
2459 Acked Write 5 Sent Done Done false
2460 Acked Flush 0 Sent Done Done false
2461 Acked Read 9 Sent Done Done false
2462 Acked Flush 0 Sent Done Done false
2463 Acked Write 2 Sent Done Done false
2464 Acked Write 7 Sent Done Done false
2465 Acked Write 3 Sent Done Done false
2466 Acked Read 7 Sent Done Done false
2467 Acked Flush 0 Sent Done Done false
2468 Acked Flush 0 Sent Done Done false
2469 Acked Write 5 Sent Done Done false
2470 Acked Write 1 Sent Done Done false
2471 Acked Write 2 Sent Done Done false
2472 Acked Read 7 Sent Done Done false
2473 Acked Read 7 Sent Done Done false
2474 Acked Write 10 Sent Done Done false
2475 Acked Read 8 Sent Done Done false
2476 Acked Write 1 Sent Done Done false
2477 Acked Flush 0 Sent Done Done false
2478 Acked Write 9 Sent Done Done false
2479 Acked Read 7 Sent Done Done false
2480 Acked Read 8 Sent Done Done false
2481 Acked Write 2 Sent Done Done false
2482 Acked Read 9 Sent Done Done false
2483 Acked Write 9 Sent Done Done false
2484 Acked Read 6 Sent Done Done false
2485 Acked Read 6 Sent Done Done false
2486 Acked Write 8 Sent Done Done false
2487 Acked Write 6 Sent Done Done false
2488 Acked Read 2 Sent Done Done false
2489 Acked Flush 0 Sent Done Done false
2490 Acked Read 4 Sent Done Done false
2491 Acked Write 1 Sent Done Done false
2492 Acked Flush 0 Sent Done Done false
2493 Acked Read 4 Sent Done Done false
2494 Acked Write 6 Sent Done Done false
2495 Acked Read 9 Sent Done Done false
2496 Acked Read 10 Sent Done Done false
2497 Acked Write 4 Sent Done Done false
2498 Acked Read 7 Sent Done Done false
2499 Acked Write 8 Sent Done Done false
2500 Acked Write 7 Sent Done Done false
2501 Acked Read 6 Sent Done Done false
2502 Acked Write 9 Sent Done Done false
2503 Acked Read 8 Sent Done Done false
2504 Acked Write 9 Sent Done Done false
2505 Acked Write 3 Sent Done Done false
2506 Acked Read 5 Sent Done Done false
2507 Acked Flush 0 Sent Done Done false
2508 Acked Write 9 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2207 2507 2507
Downstairs last five completed: 2207 2206 2205 2204 2203
Upstairs last five completed: 2508 2507 2506 2505 2504
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2208 Acked Read 4 Sent Done Done false
2209 Acked Flush 0 Sent Done Done false
2210 Acked Write 7 Sent Done Done false
2211 Acked Write 8 Sent Done Done false
2212 Acked Write 6 Sent Done Done false
2213 Acked Read 4 Sent Done Done false
2214 Acked Write 5 Sent Done Done false
2215 Acked Write 8 Sent Done Done false
2216 Acked Read 7 Sent Done Done false
2217 Acked Write 8 Sent Done Done false
2218 Acked Read 5 Sent Done Done false
2219 Acked Read 9 Sent Done Done false
2220 Acked Write 2 Sent Done Done false
2221 Acked Read 5 Sent Done Done false
2222 Acked Write 4 Sent Done Done false
2223 Acked Read 9 Sent Done Done false
2224 Acked Read 9 Sent Done Done false
2225 Acked Write 1 Sent Done Done false
2226 Acked Read 1 Sent Done Done false
2227 Acked Read 3 Sent Done Done false
2228 Acked Read 9 Sent Done Done false
2229 Acked Read 9 Sent Done Done false
2230 Acked Read 2 Sent Done Done false
2231 Acked Write 9 Sent Done Done false
2232 Acked Read 8 Sent Done Done false
2233 Acked Write 3 Sent Done Done false
2234 Acked Write 4 Sent Done Done false
2235 Acked Read 4 Sent Done Done false
2236 Acked Flush 0 Sent Done Done false
2237 Acked Read 2 Sent Done Done false
2238 Acked Read 7 Sent Done Done false
2239 Acked Read 8 Sent Done Done false
2240 Acked Read 2 Sent Done Done false
2241 Acked Read 5 Sent Done Done false
2242 Acked Write 2 Sent Done Done false
2243 Acked Write 7 Sent Done Done false
2244 Acked Write 5 Sent Done Done false
2245 Acked Flush 0 Sent Done Done false
2246 Acked Write 1 Sent Done Done false
2247 Acked Read 3 Sent Done Done false
2248 Acked Read 4 Sent Done Done false
2249 Acked Read 9 Sent Done Done false
2250 Acked Read 5 Sent Done Done false
2251 Acked Read 1 Sent Done Done false
2252 Acked Read 9 Sent Done Done false
2253 Acked Write 7 Sent Done Done false
2254 Acked Read 9 Sent Done Done false
2255 Acked Write 7 Sent Done Done false
2256 Acked Write 5 Sent Done Done false
2257 Acked Read 5 Sent Done Done false
2258 Acked Write 7 Sent Done Done false
2259 Acked Read 8 Sent Done Done false
2260 Acked Write 3 Sent Done Done false
2261 Acked Read 2 Sent Done Done false
2262 Acked Read 5 Sent Done Done false
2263 Acked Read 7 Sent Done Done false
2264 Acked Read 4 Sent Done Done false
2265 Acked Read 2 Sent Done Done false
2266 Acked Write 8 Sent Done Done false
2267 Acked Write 7 Sent Done Done false
2268 Acked Read 3 Sent Done Done false
2269 Acked Flush 0 Sent Done Done false
2270 Acked Write 3 Sent Done Done false
2271 Acked Write 4 Sent Done Done false
2272 Acked Read 3 Sent Done Done false
2273 Acked Write 5 Sent Done Done false
2274 Acked Write 8 Sent Done Done false
2275 Acked Write 1 Sent Done Done false
2276 Acked Read 5 Sent Done Done false
2277 Acked Read 10 Sent Done Done false
2278 Acked Write 6 Sent Done Done false
2279 Acked Read 8 Sent Done Done false
2280 Acked Write 9 Sent Done Done false
2281 Acked Write 6 Sent Done Done false
2282 Acked Write 1 Sent Done Done false
2283 Acked Flush 0 Sent Done Done false
2284 Acked Read 6 Sent Done Done false
2285 Acked Write 4 Sent Done Done false
2286 Acked Write 6 Sent Done Done false
2287 Acked Read 3 Sent Done Done false
2288 Acked Read 4 Sent Done Done false
2289 Acked Read 3 Sent Done Done false
2290 Acked Flush 0 Sent Done Done false
2291 Acked Read 3 Sent Done Done false
2292 Acked Read 6 Sent Done Done false
2293 Acked Write 10 Sent Done Done false
2294 Acked Read 2 Sent Done Done false
2295 Acked Read 3 Sent Done Done false
2296 Acked Read 6 Sent Done Done false
2297 Acked Read 6 Sent Done Done false
2298 Acked Write 2 Sent Done Done false
2299 Acked Write 8 Sent Done Done false
2300 Acked Write 9 Sent Done Done false
2301 Acked Read 10 Sent Done Done false
2302 Acked Read 6 Sent Done Done false
2303 Acked Read 5 Sent Done Done false
2304 Acked Read 9 Sent Done Done false
2305 Acked Read 7 Sent Done Done false
2306 Acked Read 9 Sent Done Done false
2307 Acked Flush 0 Sent Done Done false
2308 Acked Write 5 Sent Done Done false
2309 Acked Flush 0 Sent Done Done false
2310 Acked Read 2 Sent Done Done false
2311 Acked Read 1 Sent Done Done false
2312 Acked Read 3 Sent Done Done false
2313 Acked Write 2 Sent Done Done false
2314 Acked Read 3 Sent Done Done false
2315 Acked Read 2 Sent Done Done false
2316 Acked Read 6 Sent Done Done false
2317 Acked Flush 0 Sent Done Done false
2318 Acked Write 5 Sent Done Done false
2319 Acked Write 4 Sent Done Done false
2320 Acked Write 10 Sent Done Done false
2321 Acked Write 1 Sent Done Done false
2322 Acked Read 2 Sent Done Done false
2323 Acked Write 8 Sent Done Done false
2324 Acked Write 9 Sent Done Done false
2325 Acked Flush 0 Sent Done Done false
2326 Acked Read 1 Sent Done Done false
2327 Acked Read 1 Sent Done Done false
2328 Acked Write 8 Sent Done Done false
2329 Acked Read 2 Sent Done Done false
2330 Acked Write 4 Sent Done Done false
2331 Acked Read 10 Sent Done Done false
2332 Acked Read 8 Sent Done Done false
2333 Acked Read 9 Sent Done Done false
2334 Acked Read 6 Sent Done Done false
2335 Acked Write 10 Sent Done Done false
2336 Acked Write 9 Sent Done Done false
2337 Acked Flush 0 Sent Done Done false
2338 Acked Read 4 Sent Done Done false
2339 Acked Read 10 Sent Done Done false
2340 Acked Write 10 Sent Done Done false
2341 Acked Write 3 Sent Done Done false
2342 Acked Write 3 Sent Done Done false
2343 Acked Write 2 Sent Done Done false
2344 Acked Write 3 Sent Done Done false
2345 Acked Flush 0 Sent Done Done false
2346 Acked Read 4 Sent Done Done false
2347 Acked Read 7 Sent Done Done false
2348 Acked Read 3 Sent Done Done false
2349 Acked Flush 0 Sent Done Done false
2350 Acked Write 8 Sent Done Done false
2351 Acked Read 5 Sent Done Done false
2352 Acked Read 3 Sent Done Done false
2353 Acked Write 2 Sent Done Done false
2354 Acked Read 6 Sent Done Done false
2355 Acked Read 7 Sent Done Done false
2356 Acked Read 2 Sent Done Done false
2357 Acked Read 1 Sent Done Done false
2358 Acked Read 6 Sent Done Done false
2359 Acked Write 5 Sent Done Done false
2360 Acked Write 4 Sent Done Done false
2361 Acked Read 7 Sent Done Done false
2362 Acked Read 4 Sent Done Done false
2363 Acked Write 5 Sent Done Done false
2364 Acked Write 9 Sent Done Done false
2365 Acked Read 4 Sent Done Done false
2366 Acked Read 1 Sent Done Done false
2367 Acked Read 5 Sent Done Done false
2368 Acked Read 1 Sent Done Done false
2369 Acked Read 10 Sent Done Done false
2370 Acked Read 4 Sent Done Done false
2371 Acked Read 5 Sent Done Done false
2372 Acked Read 3 Sent Done Done false
2373 Acked Flush 0 Sent Done Done false
2374 Acked Read 8 Sent Done Done false
2375 Acked Read 8 Sent Done Done false
2376 Acked Read 1 Sent Done Done false
2377 Acked Read 6 Sent Done Done false
2378 Acked Read 7 Sent Done Done false
2379 Acked Write 5 Sent Done Done false
2380 Acked Flush 0 Sent Done Done false
2381 Acked Read 7 Sent Done Done false
2382 Acked Flush 0 Sent Done Done false
2383 Acked Read 4 Sent Done Done false
2384 Acked Read 1 Sent Done Done false
2385 Acked Write 10 Sent Done Done false
2386 Acked Write 2 Sent Done Done false
2387 Acked Flush 0 Sent Done Done false
2388 Acked Flush 0 Sent Done Done false
2389 Acked Read 3 Sent Done Done false
2390 Acked Flush 0 Sent Done Done false
2391 Acked Read 5 Sent Done Done false
2392 Acked Read 10 Sent Done Done false
2393 Acked Write 4 Sent Done Done false
2394 Acked Write 3 Sent Done Done false
2395 Acked Read 1 Sent Done Done false
2396 Acked Read 6 Sent Done Done false
2397 Acked Read 9 Sent Done Done false
2398 Acked Write 5 Sent Done Done false
2399 Acked Read 8 Sent Done Done false
2400 Acked Read 1 Sent Done Done false
2401 Acked Read 1 Sent Done Done false
2402 Acked Read 7 Sent Done Done false
2403 Acked Read 6 Sent Done Done false
2404 Acked Read 4 Sent Done Done false
2405 Acked Read 5 Sent Done Done false
2406 Acked Write 3 Sent Done Done false
2407 Acked Write 7 Sent Done Done false
2408 Acked Read 6 Sent Done Done false
2409 Acked Write 7 Sent Done Done false
2410 Acked Read 10 Sent Done Done false
2411 Acked Read 3 Sent Done Done false
2412 Acked Read 5 Sent Done Done false
2413 Acked Write 4 Sent Done Done false
2414 Acked Write 5 Sent Done Done false
2415 Acked Write 10 Sent Done Done false
2416 Acked Write 7 Sent Done Done false
2417 Acked Read 2 Sent Done Done false
2418 Acked Read 2 Sent Done Done false
2419 Acked Write 6 Sent Done Done false
2420 Acked Read 3 Sent Done Done false
2421 Acked Read 6 Sent Done Done false
2422 Acked Write 8 Sent Done Done false
2423 Acked Read 10 Sent Done Done false
2424 Acked Write 4 Sent Done Done false
2425 Acked Write 3 Sent Done Done false
2426 Acked Read 4 Sent Done Done false
2427 Acked Write 2 Sent Done Done false
2428 Acked Write 4 Sent Done Done false
2429 Acked Write 3 Sent Done Done false
2430 Acked Read 10 Sent Done Done false
2431 Acked Read 9 Sent Done Done false
2432 Acked Write 8 Sent Done Done false
2433 Acked Write 1 Sent Done Done false
2434 Acked Read 6 Sent Done Done false
2435 Acked Write 9 Sent Done Done false
2436 Acked Write 3 Sent Done Done false
2437 Acked Write 9 Sent Done Done false
2438 Acked Write 2 Sent Done Done false
2439 Acked Write 9 Sent Done Done false
2440 Acked Write 1 Sent Done Done false
2441 Acked Write 6 Sent Done Done false
2442 Acked Read 4 Sent Done Done false
2443 Acked Flush 0 Sent Done Done false
2444 Acked Write 9 Sent Done Done false
2445 Acked Read 10 Sent Done Done false
2446 Acked Write 2 Sent Done Done false
2447 Acked Read 8 Sent Done Done false
2448 Acked Write 1 Sent Done Done false
2449 Acked Read 1 Sent Done Done false
2450 Acked Write 4 Sent Done Done false
2451 Acked Read 9 Sent Done Done false
2452 Acked Read 10 Sent Done Done false
2453 Acked Write 7 Sent Done Done false
2454 Acked Read 4 Sent Done Done false
2455 Acked Flush 0 Sent Done Done false
2456 Acked Read 9 Sent Done Done false
2457 Acked Write 4 Sent Done Done false
2458 Acked Read 8 Sent Done Done false
2459 Acked Write 5 Sent Done Done false
2460 Acked Flush 0 Sent Done Done false
2461 Acked Read 9 Sent Done Done false
2462 Acked Flush 0 Sent Done Done false
2463 Acked Write 2 Sent Done Done false
2464 Acked Write 7 Sent Done Done false
2465 Acked Write 3 Sent Done Done false
2466 Acked Read 7 Sent Done Done false
2467 Acked Flush 0 Sent Done Done false
2468 Acked Flush 0 Sent Done Done false
2469 Acked Write 5 Sent Done Done false
2470 Acked Write 1 Sent Done Done false
2471 Acked Write 2 Sent Done Done false
2472 Acked Read 7 Sent Done Done false
2473 Acked Read 7 Sent Done Done false
2474 Acked Write 10 Sent Done Done false
2475 Acked Read 8 Sent Done Done false
2476 Acked Write 1 Sent Done Done false
2477 Acked Flush 0 Sent Done Done false
2478 Acked Write 9 Sent Done Done false
2479 Acked Read 7 Sent Done Done false
2480 Acked Read 8 Sent Done Done false
2481 Acked Write 2 Sent Done Done false
2482 Acked Read 9 Sent Done Done false
2483 Acked Write 9 Sent Done Done false
2484 Acked Read 6 Sent Done Done false
2485 Acked Read 6 Sent Done Done false
2486 Acked Write 8 Sent Done Done false
2487 Acked Write 6 Sent Done Done false
2488 Acked Read 2 Sent Done Done false
2489 Acked Flush 0 Sent Done Done false
2490 Acked Read 4 Sent Done Done false
2491 Acked Write 1 Sent Done Done false
2492 Acked Flush 0 Sent Done Done false
2493 Acked Read 4 Sent Done Done false
2494 Acked Write 6 Sent Done Done false
2495 Acked Read 9 Sent Done Done false
2496 Acked Read 10 Sent Done Done false
2497 Acked Write 4 Sent Done Done false
2498 Acked Read 7 Sent Done Done false
2499 Acked Write 8 Sent Done Done false
2500 Acked Write 7 Sent Done Done false
2501 Acked Read 6 Sent Done Done false
2502 Acked Write 9 Sent Done Done false
2503 Acked Read 8 Sent Done Done false
2504 Acked Write 9 Sent Done Done false
2505 Acked Write 3 Sent Done Done false
2506 Acked Read 5 Sent Done Done false
2507 Acked Flush 0 Sent Done Done false
2508 Acked Write 9 Sent Done Done false
2509 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2207 2509 2509
Downstairs last five completed: 2207 2206 2205 2204 2203
Upstairs last five completed: 2509 2508 2507 2506 2505
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.901246898Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.90149054Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.901519596Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.901539384Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.901628983Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1205)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2208 Acked Read 4 Sent Done Done false
2209 Acked Flush 0 Sent Done Done false
2210 Acked Write 7 Sent Done Done false
2211 Acked Write 8 Sent Done Done false
2212 Acked Write 6 Sent Done Done false
2213 Acked Read 4 Sent Done Done false
2214 Acked Write 5 Sent Done Done false
2215 Acked Write 8 Sent Done Done false
2216 Acked Read 7 Sent Done Done false
2217 Acked Write 8 Sent Done Done false
2218 Acked Read 5 Sent Done Done false
2219 Acked Read 9 Sent Done Done false
2220 Acked Write 2 Sent Done Done false
2221 Acked Read 5 Sent Done Done false
2222 Acked Write 4 Sent Done Done false
2223 Acked Read 9 Sent Done Done false
2224 Acked Read 9 Sent Done Done false
2225 Acked Write 1 Sent Done Done false
2226 Acked Read 1 Sent Done Done false
2227 Acked Read 3 Sent Done Done false
2228 Acked Read 9 Sent Done Done false
2229 Acked Read 9 Sent Done Done false
2230 Acked Read 2 Sent Done Done false
2231 Acked Write 9 Sent Done Done false
2232 Acked Read 8 Sent Done Done false
2233 Acked Write 3 Sent Done Done false
2234 Acked Write 4 Sent Done Done false
2235 Acked Read 4 Sent Done Done false
2236 Acked Flush 0 Sent Done Done false
2237 Acked Read 2 Sent Done Done false
2238 Acked Read 7 Sent Done Done false
2239 Acked Read 8 Sent Done Done false
2240 Acked Read 2 Sent Done Done false
2241 Acked Read 5 Sent Done Done false
2242 Acked Write 2 Sent Done Done false
2243 Acked Write 7 Sent Done Done false
2244 Acked Write 5 Sent Done Done false
2245 Acked Flush 0 Sent Done Done false
2246 Acked Write 1 Sent Done Done false
2247 Acked Read 3 Sent Done Done false
2248 Acked Read 4 Sent Done Done false
2249 Acked Read 9 Sent Done Done false
2250 Acked Read 5 Sent Done Done false
2251 Acked Read 1 Sent Done Done false
2252 Acked Read 9 Sent Done Done false
2253 Acked Write 7 Sent Done Done false
2254 Acked Read 9 Sent Done Done false
2255 Acked Write 7 Sent Done Done false
2256 Acked Write 5 Sent Done Done false
2257 Acked Read 5 Sent Done Done false
2258 Acked Write 7 Sent Done Done false
2259 Acked Read 8 Sent Done Done false
2260 Acked Write 3 Sent Done Done false
2261 Acked Read 2 Sent Done Done false
2262 Acked Read 5 Sent Done Done false
2263 Acked Read 7 Sent Done Done false
2264 Acked Read 4 Sent Done Done false
2265 Acked Read 2 Sent Done Done false
2266 Acked Write 8 Sent Done Done false
2267 Acked Write 7 Sent Done Done false
2268 Acked Read 3 Sent Done Done false
2269 Acked Flush 0 Sent Done Done false
2270 Acked Write 3 Sent Done Done false
2271 Acked Write 4 Sent Done Done false
2272 Acked Read 3 Sent Done Done false
2273 Acked Write 5 Sent Done Done false
2274 Acked Write 8 Sent Done Done false
2275 Acked Write 1 Sent Done Done false
2276 Acked Read 5 Sent Done Done false
2277 Acked Read 10 Sent Done Done false
2278 Acked Write 6 Sent Done Done false
2279 Acked Read 8 Sent Done Done false
2280 Acked Write 9 Sent Done Done false
2281 Acked Write 6 Sent Done Done false
2282 Acked Write 1 Sent Done Done false
2283 Acked Flush 0 Sent Done Done false
2284 Acked Read 6 Sent Done Done false
2285 Acked Write 4 Sent Done Done false
2286 Acked Write 6 Sent Done Done false
2287 Acked Read 3 Sent Done Done false
2288 Acked Read 4 Sent Done Done false
2289 Acked Read 3 Sent Done Done false
2290 Acked Flush 0 Sent Done Done false
2291 Acked Read 3 Sent Done Done false
2292 Acked Read 6 Sent Done Done false
2293 Acked Write 10 Sent Done Done false
2294 Acked Read 2 Sent Done Done false
2295 Acked Read 3 Sent Done Done false
2296 Acked Read 6 Sent Done Done false
2297 Acked Read 6 Sent Done Done false
2298 Acked Write 2 Sent Done Done false
2299 Acked Write 8 Sent Done Done false
2300 Acked Write 9 Sent Done Done false
2301 Acked Read 10 Sent Done Done false
2302 Acked Read 6 Sent Done Done false
2303 Acked Read 5 Sent Done Done false
2304 Acked Read 9 Sent Done Done false
2305 Acked Read 7 Sent Done Done false
2306 Acked Read 9 Sent Done Done false
2307 Acked Flush 0 Sent Done Done false
2308 Acked Write 5 Sent Done Done false
2309 Acked Flush 0 Sent Done Done false
2310 Acked Read 2 Sent Done Done false
2311 Acked Read 1 Sent Done Done false
2312 Acked Read 3 Sent Done Done false
2313 Acked Write 2 Sent Done Done false
2314 Acked Read 3 Sent Done Done false
2315 Acked Read 2 Sent Done Done false
2316 Acked Read 6 Sent Done Done false
2317 Acked Flush 0 Sent Done Done false
2318 Acked Write 5 Sent Done Done false
2319 Acked Write 4 Sent Done Done false
2320 Acked Write 10 Sent Done Done false
2321 Acked Write 1 Sent Done Done false
2322 Acked Read 2 Sent Done Done false
2323 Acked Write 8 Sent Done Done false
2324 Acked Write 9 Sent Done Done false
2325 Acked Flush 0 Sent Done Done false
2326 Acked Read 1 Sent Done Done false
2327 Acked Read 1 Sent Done Done false
2328 Acked Write 8 Sent Done Done false
2329 Acked Read 2 Sent Done Done false
2330 Acked Write 4 Sent Done Done false
2331 Acked Read 10 Sent Done Done false
2332 Acked Read 8 Sent Done Done false
2333 Acked Read 9 Sent Done Done false
2334 Acked Read 6 Sent Done Done false
2335 Acked Write 10 Sent Done Done false
2336 Acked Write 9 Sent Done Done false
2337 Acked Flush 0 Sent Done Done false
2338 Acked Read 4 Sent Done Done false
2339 Acked Read 10 Sent Done Done false
2340 Acked Write 10 Sent Done Done false
2341 Acked Write 3 Sent Done Done false
2342 Acked Write 3 Sent Done Done false
2343 Acked Write 2 Sent Done Done false
2344 Acked Write 3 Sent Done Done false
2345 Acked Flush 0 Sent Done Done false
2346 Acked Read 4 Sent Done Done false
2347 Acked Read 7 Sent Done Done false
2348 Acked Read 3 Sent Done Done false
2349 Acked Flush 0 Sent Done Done false
2350 Acked Write 8 Sent Done Done false
2351 Acked Read 5 Sent Done Done false
2352 Acked Read 3 Sent Done Done false
2353 Acked Write 2 Sent Done Done false
2354 Acked Read 6 Sent Done Done false
2355 Acked Read 7 Sent Done Done false
2356 Acked Read 2 Sent Done Done false
2357 Acked Read 1 Sent Done Done false
2358 Acked Read 6 Sent Done Done false
2359 Acked Write 5 Sent Done Done false
2360 Acked Write 4 Sent Done Done false
2361 Acked Read 7 Sent Done Done false
2362 Acked Read 4 Sent Done Done false
2363 Acked Write 5 Sent Done Done false
2364 Acked Write 9 Sent Done Done false
2365 Acked Read 4 Sent Done Done false
2366 Acked Read 1 Sent Done Done false
2367 Acked Read 5 Sent Done Done false
2368 Acked Read 1 Sent Done Done false
2369 Acked Read 10 Sent Done Done false
2370 Acked Read 4 Sent Done Done false
2371 Acked Read 5 Sent Done Done false
2372 Acked Read 3 Sent Done Done false
2373 Acked Flush 0 Sent Done Done false
2374 Acked Read 8 Sent Done Done false
2375 Acked Read 8 Sent Done Done false
2376 Acked Read 1 Sent Done Done false
2377 Acked Read 6 Sent Done Done false
2378 Acked Read 7 Sent Done Done false
2379 Acked Write 5 Sent Done Done false
2380 Acked Flush 0 Sent Done Done false
2381 Acked Read 7 Sent Done Done false
2382 Acked Flush 0 Sent Done Done false
2383 Acked Read 4 Sent Done Done false
2384 Acked Read 1 Sent Done Done false
2385 Acked Write 10 Sent Done Done false
2386 Acked Write 2 Sent Done Done false
2387 Acked Flush 0 Sent Done Done false
2388 Acked Flush 0 Sent Done Done false
2389 Acked Read 3 Sent Done Done false
2390 Acked Flush 0 Sent Done Done false
2391 Acked Read 5 Sent Done Done false
2392 Acked Read 10 Sent Done Done false
2393 Acked Write 4 Sent Done Done false
2394 Acked Write 3 Sent Done Done false
2395 Acked Read 1 Sent Done Done false
2396 Acked Read 6 Sent Done Done false
2397 Acked Read 9 Sent Done Done false
2398 Acked Write 5 Sent Done Done false
2399 Acked Read 8 Sent Done Done false
2400 Acked Read 1 Sent Done Done false
2401 Acked Read 1 Sent Done Done false
2402 Acked Read 7 Sent Done Done false
2403 Acked Read 6 Sent Done Done false
2404 Acked Read 4 Sent Done Done false
2405 Acked Read 5 Sent Done Done false
2406 Acked Write 3 Sent Done Done false
2407 Acked Write 7 Sent Done Done false
2408 Acked Read 6 Sent Done Done false
2409 Acked Write 7 Sent Done Done false
2410 Acked Read 10 Sent Done Done false
2411 Acked Read 3 Sent Done Done false
2412 Acked Read 5 Sent Done Done false
2413 Acked Write 4 Sent Done Done false
2414 Acked Write 5 Sent Done Done false
2415 Acked Write 10 Sent Done Done false
2416 Acked Write 7 Sent Done Done false
2417 Acked Read 2 Sent Done Done false
2418 Acked Read 2 Sent Done Done false
2419 Acked Write 6 Sent Done Done false
2420 Acked Read 3 Sent Done Done false
2421 Acked Read 6 Sent Done Done false
2422 Acked Write 8 Sent Done Done false
2423 Acked Read 10 Sent Done Done false
2424 Acked Write 4 Sent Done Done false
2425 Acked Write 3 Sent Done Done false
2426 Acked Read 4 Sent Done Done false
2427 Acked Write 2 Sent Done Done false
2428 Acked Write 4 Sent Done Done false
2429 Acked Write 3 Sent Done Done false
2430 Acked Read 10 Sent Done Done false
2431 Acked Read 9 Sent Done Done false
2432 Acked Write 8 Sent Done Done false
2433 Acked Write 1 Sent Done Done false
2434 Acked Read 6 Sent Done Done false
2435 Acked Write 9 Sent Done Done false
2436 Acked Write 3 Sent Done Done false
2437 Acked Write 9 Sent Done Done false
2438 Acked Write 2 Sent Done Done false
2439 Acked Write 9 Sent Done Done false
2440 Acked Write 1 Sent Done Done false
2441 Acked Write 6 Sent Done Done false
2442 Acked Read 4 Sent Done Done false
2443 Acked Flush 0 Sent Done Done false
2444 Acked Write 9 Sent Done Done false
2445 Acked Read 10 Sent Done Done false
2446 Acked Write 2 Sent Done Done false
2447 Acked Read 8 Sent Done Done false
2448 Acked Write 1 Sent Done Done false
2449 Acked Read 1 Sent Done Done false
2450 Acked Write 4 Sent Done Done false
2451 Acked Read 9 Sent Done Done false
2452 Acked Read 10 Sent Done Done false
2453 Acked Write 7 Sent Done Done false
2454 Acked Read 4 Sent Done Done false
2455 Acked Flush 0 Sent Done Done false
2456 Acked Read 9 Sent Done Done false
2457 Acked Write 4 Sent Done Done false
2458 Acked Read 8 Sent Done Done false
2459 Acked Write 5 Sent Done Done false
2460 Acked Flush 0 Sent Done Done false
2461 Acked Read 9 Sent Done Done false
2462 Acked Flush 0 Sent Done Done false
2463 Acked Write 2 Sent Done Done false
2464 Acked Write 7 Sent Done Done false
2465 Acked Write 3 Sent Done Done false
2466 Acked Read 7 Sent Done Done false
2467 Acked Flush 0 Sent Done Done false
2468 Acked Flush 0 Sent Done Done false
2469 Acked Write 5 Sent Done Done false
2470 Acked Write 1 Sent Done Done false
2471 Acked Write 2 Sent Done Done false
2472 Acked Read 7 Sent Done Done false
2473 Acked Read 7 Sent Done Done false
2474 Acked Write 10 Sent Done Done false
2475 Acked Read 8 Sent Done Done false
2476 Acked Write 1 Sent Done Done false
2477 Acked Flush 0 Sent Done Done false
2478 Acked Write 9 Sent Done Done false
2479 Acked Read 7 Sent Done Done false
2480 Acked Read 8 Sent Done Done false
2481 Acked Write 2 Sent Done Done false
2482 Acked Read 9 Sent Done Done false
2483 Acked Write 9 Sent Done Done false
2484 Acked Read 6 Sent Done Done false
2485 Acked Read 6 Sent Done Done false
2486 Acked Write 8 Sent Done Done false
2487 Acked Write 6 Sent Done Done false
2488 Acked Read 2 Sent Done Done false
2489 Acked Flush 0 Sent Done Done false
2490 Acked Read 4 Sent Done Done false
2491 Acked Write 1 Sent Done Done false
2492 Acked Flush 0 Sent Done Done false
2493 Acked Read 4 Sent Done Done false
2494 Acked Write 6 Sent Done Done false
2495 Acked Read 9 Sent Done Done false
2496 Acked Read 10 Sent Done Done false
2497 Acked Write 4 Sent Done Done false
2498 Acked Read 7 Sent Done Done false
2499 Acked Write 8 Sent Done Done false
2500 Acked Write 7 Sent Done Done false
2501 Acked Read 6 Sent Done Done false
2502 Acked Write 9 Sent Done Done false
2503 Acked Read 8 Sent Done Done false
2504 Acked Write 9 Sent Done Done false
2505 Acked Write 3 Sent Done Done false
2506 Acked Read 5 Sent Done Done false
2507 Acked Flush 0 Sent Done Done false
2508 Acked Write 9 Sent Done Done false
2509 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2207 2509 2509
Downstairs last five completed: 2207 2206 2205 2204 2203
Upstairs last five completed: 2509 2508 2507 2506 2505
CLIENT: Up:0 ds:302 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2208 Acked Read 4 Sent Done Done false
2209 Acked Flush 0 Sent Done Done false
2210 Acked Write 7 Sent Done Done false
2211 Acked Write 8 Sent Done Done false
2212 Acked Write 6 Sent Done Done false
2213 Acked Read 4 Sent Done Done false
2214 Acked Write 5 Sent Done Done false
2215 Acked Write 8 Sent Done Done false
2216 Acked Read 7 Sent Done Done false
2217 Acked Write 8 Sent Done Done false
2218 Acked Read 5 Sent Done Done false
2219 Acked Read 9 Sent Done Done false
2220 Acked Write 2 Sent Done Done false
2221 Acked Read 5 Sent Done Done false
2222 Acked Write 4 Sent Done Done false
2223 Acked Read 9 Sent Done Done false
2224 Acked Read 9 Sent Done Done false
2225 Acked Write 1 Sent Done Done false
2226 Acked Read 1 Sent Done Done false
2227 Acked Read 3 Sent Done Done false
2228 Acked Read 9 Sent Done Done false
2229 Acked Read 9 Sent Done Done false
2230 Acked Read 2 Sent Done Done false
2231 Acked Write 9 Sent Done Done false
2232 Acked Read 8 Sent Done Done false
2233 Acked Write 3 Sent Done Done false
2234 Acked Write 4 Sent Done Done false
2235 Acked Read 4 Sent Done Done false
2236 Acked Flush 0 Sent Done Done false
2237 Acked Read 2 Sent Done Done false
2238 Acked Read 7 Sent Done Done false
2239 Acked Read 8 Sent Done Done false
2240 Acked Read 2 Sent Done Done false
2241 Acked Read 5 Sent Done Done false
2242 Acked Write 2 Sent Done Done false
2243 Acked Write 7 Sent Done Done false
2244 Acked Write 5 Sent Done Done false
2245 Acked Flush 0 Sent Done Done false
2246 Acked Write 1 Sent Done Done false
2247 Acked Read 3 Sent Done Done false
2248 Acked Read 4 Sent Done Done false
2249 Acked Read 9 Sent Done Done false
2250 Acked Read 5 Sent Done Done false
2251 Acked Read 1 Sent Done Done false
2252 Acked Read 9 Sent Done Done false
2253 Acked Write 7 Sent Done Done false
2254 Acked Read 9 Sent Done Done false
2255 Acked Write 7 Sent Done Done false
2256 Acked Write 5 Sent Done Done false
2257 Acked Read 5 Sent Done Done false
2258 Acked Write 7 Sent Done Done false
2259 Acked Read 8 Sent Done Done false
2260 Acked Write 3 Sent Done Done false
2261 Acked Read 2 Sent Done Done false
2262 Acked Read 5 Sent Done Done false
2263 Acked Read 7 Sent Done Done false
2264 Acked Read 4 Sent Done Done false
2265 Acked Read 2 Sent Done Done false
2266 Acked Write 8 Sent Done Done false
2267 Acked Write 7 Sent Done Done false
2268 Acked Read 3 Sent Done Done false
2269 Acked Flush 0 Sent Done Done false
2270 Acked Write 3 Sent Done Done false
2271 Acked Write 4 Sent Done Done false
2272 Acked Read 3 Sent Done Done false
2273 Acked Write 5 Sent Done Done false
2274 Acked Write 8 Sent Done Done false
2275 Acked Write 1 Sent Done Done false
2276 Acked Read 5 Sent Done Done false
2277 Acked Read 10 Sent Done Done false
2278 Acked Write 6 Sent Done Done false
2279 Acked Read 8 Sent Done Done false
2280 Acked Write 9 Sent Done Done false
2281 Acked Write 6 Sent Done Done false
2282 Acked Write 1 Sent Done Done false
2283 Acked Flush 0 Sent Done Done false
2284 Acked Read 6 Sent Done Done false
2285 Acked Write 4 Sent Done Done false
2286 Acked Write 6 Sent Done Done false
2287 Acked Read 3 Sent Done Done false
2288 Acked Read 4 Sent Done Done false
2289 Acked Read 3 Sent Done Done false
2290 Acked Flush 0 Sent Done Done false
2291 Acked Read 3 Sent Done Done false
2292 Acked Read 6 Sent Done Done false
2293 Acked Write 10 Sent Done Done false
2294 Acked Read 2 Sent Done Done false
2295 Acked Read 3 Sent Done Done false
2296 Acked Read 6 Sent Done Done false
2297 Acked Read 6 Sent Done Done false
2298 Acked Write 2 Sent Done Done false
2299 Acked Write 8 Sent Done Done false
2300 Acked Write 9 Sent Done Done false
2301 Acked Read 10 Sent Done Done false
2302 Acked Read 6 Sent Done Done false
2303 Acked Read 5 Sent Done Done false
2304 Acked Read 9 Sent Done Done false
2305 Acked Read 7 Sent Done Done false
2306 Acked Read 9 Sent Done Done false
2307 Acked Flush 0 Sent Done Done false
2308 Acked Write 5 Sent Done Done false
2309 Acked Flush 0 Sent Done Done false
2310 Acked Read 2 Sent Done Done false
2311 Acked Read 1 Sent Done Done false
2312 Acked Read 3 Sent Done Done false
2313 Acked Write 2 Sent Done Done false
2314 Acked Read 3 Sent Done Done false
2315 Acked Read 2 Sent Done Done false
2316 Acked Read 6 Sent Done Done false
2317 Acked Flush 0 Sent Done Done false
2318 Acked Write 5 Sent Done Done false
2319 Acked Write 4 Sent Done Done false
2320 Acked Write 10 Sent Done Done false
2321 Acked Write 1 Sent Done Done false
2322 Acked Read 2 Sent Done Done false
2323 Acked Write 8 Sent Done Done false
2324 Acked Write 9 Sent Done Done false
2325 Acked Flush 0 Sent Done Done false
2326 Acked Read 1 Sent Done Done false
2327 Acked Read 1 Sent Done Done false
2328 Acked Write 8 Sent Done Done false
2329 Acked Read 2 Sent Done Done false
2330 Acked Write 4 Sent Done Done false
2331 Acked Read 10 Sent Done Done false
2332 Acked Read 8 Sent Done Done false
2333 Acked Read 9 Sent Done Done false
2334 Acked Read 6 Sent Done Done false
2335 Acked Write 10 Sent Done Done false
2336 Acked Write 9 Sent Done Done false
2337 Acked Flush 0 Sent Done Done false
2338 Acked Read 4 Sent Done Done false
2339 Acked Read 10 Sent Done Done false
2340 Acked Write 10 Sent Done Done false
2341 Acked Write 3 Sent Done Done false
2342 Acked Write 3 Sent Done Done false
2343 Acked Write 2 Sent Done Done false
2344 Acked Write 3 Sent Done Done false
2345 Acked Flush 0 Sent Done Done false
2346 Acked Read 4 Sent Done Done false
2347 Acked Read 7 Sent Done Done false
2348 Acked Read 3 Sent Done Done false
2349 Acked Flush 0 Sent Done Done false
2350 Acked Write 8 Sent Done Done false
2351 Acked Read 5 Sent Done Done false
2352 Acked Read 3 Sent Done Done false
2353 Acked Write 2 Sent Done Done false
2354 Acked Read 6 Sent Done Done false
2355 Acked Read 7 Sent Done Done false
2356 Acked Read 2 Sent Done Done false
2357 Acked Read 1 Sent Done Done false
2358 Acked Read 6 Sent Done Done false
2359 Acked Write 5 Sent Done Done false
2360 Acked Write 4 Sent Done Done false
2361 Acked Read 7 Sent Done Done false
2362 Acked Read 4 Sent Done Done false
2363 Acked Write 5 Sent Done Done false
2364 Acked Write 9 Sent Done Done false
2365 Acked Read 4 Sent Done Done false
2366 Acked Read 1 Sent Done Done false
2367 Acked Read 5 Sent Done Done false
2368 Acked Read 1 Sent Done Done false
2369 Acked Read 10 Sent Done Done false
2370 Acked Read 4 Sent Done Done false
2371 Acked Read 5 Sent Done Done false
2372 Acked Read 3 Sent Done Done false
2373 Acked Flush 0 Sent Done Done false
2374 Acked Read 8 Sent Done Done false
2375 Acked Read 8 Sent Done Done false
2376 Acked Read 1 Sent Done Done false
2377 Acked Read 6 Sent Done Done false
2378 Acked Read 7 Sent Done Done false
2379 Acked Write 5 Sent Done Done false
2380 Acked Flush 0 Sent Done Done false
2381 Acked Read 7 Sent Done Done false
2382 Acked Flush 0 Sent Done Done false
2383 Acked Read 4 Sent Done Done false
2384 Acked Read 1 Sent Done Done false
2385 Acked Write 10 Sent Done Done false
2386 Acked Write 2 Sent Done Done false
2387 Acked Flush 0 Sent Done Done false
2388 Acked Flush 0 Sent Done Done false
2389 Acked Read 3 Sent Done Done false
2390 Acked Flush 0 Sent Done Done false
2391 Acked Read 5 Sent Done Done false
2392 Acked Read 10 Sent Done Done false
2393 Acked Write 4 Sent Done Done false
2394 Acked Write 3 Sent Done Done false
2395 Acked Read 1 Sent Done Done false
2396 Acked Read 6 Sent Done Done false
2397 Acked Read 9 Sent Done Done false
2398 Acked Write 5 Sent Done Done false
2399 Acked Read 8 Sent Done Done false
2400 Acked Read 1 Sent Done Done false
2401 Acked Read 1 Sent Done Done false
2402 Acked Read 7 Sent Done Done false
2403 Acked Read 6 Sent Done Done false
2404 Acked Read 4 Sent Done Done false
2405 Acked Read 5 Sent Done Done false
2406 Acked Write 3 Sent Done Done false
2407 Acked Write 7 Sent Done Done false
2408 Acked Read 6 Sent Done Done false
2409 Acked Write 7 Sent Done Done false
2410 Acked Read 10 Sent Done Done false
2411 Acked Read 3 Sent Done Done false
2412 Acked Read 5 Sent Done Done false
2413 Acked Write 4 Sent Done Done false
2414 Acked Write 5 Sent Done Done false
2415 Acked Write 10 Sent Done Done false
2416 Acked Write 7 Sent Done Done false
2417 Acked Read 2 Sent Done Done false
2418 Acked Read 2 Sent Done Done false
2419 Acked Write 6 Sent Done Done false
2420 Acked Read 3 Sent Done Done false
2421 Acked Read 6 Sent Done Done false
2422 Acked Write 8 Sent Done Done false
2423 Acked Read 10 Sent Done Done false
2424 Acked Write 4 Sent Done Done false
2425 Acked Write 3 Sent Done Done false
2426 Acked Read 4 Sent Done Done false
2427 Acked Write 2 Sent Done Done false
2428 Acked Write 4 Sent Done Done false
2429 Acked Write 3 Sent Done Done false
2430 Acked Read 10 Sent Done Done false
2431 Acked Read 9 Sent Done Done false
2432 Acked Write 8 Sent Done Done false
2433 Acked Write 1 Sent Done Done false
2434 Acked Read 6 Sent Done Done false
2435 Acked Write 9 Sent Done Done false
2436 Acked Write 3 Sent Done Done false
2437 Acked Write 9 Sent Done Done false
2438 Acked Write 2 Sent Done Done false
2439 Acked Write 9 Sent Done Done false
2440 Acked Write 1 Sent Done Done false
2441 Acked Write 6 Sent Done Done false
2442 Acked Read 4 Sent Done Done false
2443 Acked Flush 0 Sent Done Done false
2444 Acked Write 9 Sent Done Done false
2445 Acked Read 10 Sent Done Done false
2446 Acked Write 2 Sent Done Done false
2447 Acked Read 8 Sent Done Done false
2448 Acked Write 1 Sent Done Done false
2449 Acked Read 1 Sent Done Done false
2450 Acked Write 4 Sent Done Done false
2451 Acked Read 9 Sent Done Done false
2452 Acked Read 10 Sent Done Done false
2453 Acked Write 7 Sent Done Done false
2454 Acked Read 4 Sent Done Done false
2455 Acked Flush 0 Sent Done Done false
2456 Acked Read 9 Sent Done Done false
2457 Acked Write 4 Sent Done Done false
2458 Acked Read 8 Sent Done Done false
2459 Acked Write 5 Sent Done Done false
2460 Acked Flush 0 Sent Done Done false
2461 Acked Read 9 Sent Done Done false
2462 Acked Flush 0 Sent Done Done false
2463 Acked Write 2 Sent Done Done false
2464 Acked Write 7 Sent Done Done false
2465 Acked Write 3 Sent Done Done false
2466 Acked Read 7 Sent Done Done false
2467 Acked Flush 0 Sent Done Done false
2468 Acked Flush 0 Sent Done Done false
2469 Acked Write 5 Sent Done Done false
2470 Acked Write 1 Sent Done Done false
2471 Acked Write 2 Sent Done Done false
2472 Acked Read 7 Sent Done Done false
2473 Acked Read 7 Sent Done Done false
2474 Acked Write 10 Sent Done Done false
2475 Acked Read 8 Sent Done Done false
2476 Acked Write 1 Sent Done Done false
2477 Acked Flush 0 Sent Done Done false
2478 Acked Write 9 Sent Done Done false
2479 Acked Read 7 Sent Done Done false
2480 Acked Read 8 Sent Done Done false
2481 Acked Write 2 Sent Done Done false
2482 Acked Read 9 Sent Done Done false
2483 Acked Write 9 Sent Done Done false
2484 Acked Read 6 Sent Done Done false
2485 Acked Read 6 Sent Done Done false
2486 Acked Write 8 Sent Done Done false
2487 Acked Write 6 Sent Done Done false
2488 Acked Read 2 Sent Done Done false
2489 Acked Flush 0 Sent Done Done false
2490 Acked Read 4 Sent Done Done false
2491 Acked Write 1 Sent Done Done false
2492 Acked Flush 0 Sent Done Done false
2493 Acked Read 4 Sent Done Done false
2494 Acked Write 6 Sent Done Done false
2495 Acked Read 9 Sent Done Done false
2496 Acked Read 10 Sent Done Done false
2497 Acked Write 4 Sent Done Done false
2498 Acked Read 7 Sent Done Done false
2499 Acked Write 8 Sent Done Done false
2500 Acked Write 7 Sent Done Done false
2501 Acked Read 6 Sent Done Done false
2502 Acked Write 9 Sent Done Done false
2503 Acked Read 8 Sent Done Done false
2504 Acked Write 9 Sent Done Done false
2505 Acked Write 3 Sent Done Done false
2506 Acked Read 5 Sent Done Done false
2507 Acked Flush 0 Sent Done Done false
2508 Acked Write 9 Sent Done Done false
2509 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2207 2509 2509
Downstairs last five completed: 2207 2206 2205 2204 2203
Upstairs last five completed: 2509 2508 2507 2506 2505
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.903006599Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43186)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.903126345Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.904205289Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.904792521Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.904812418Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 2207","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.904826037Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 2207","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.905074157Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 2207","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.905098795Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 2207","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.905328458Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2509 2508 2507 2506 2505
Upstairs last five completed: 2509 2508 2507 2506 2505
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:37:34.152 INFO [8810] Got stop action so:false kr:false
request completed, latency_us: 354, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 909c229c-967c-46e0-b904-2fb5fdfd1032, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:34.153489631Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:34.153539095Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:34.153574551Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:34.153639733Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:37:34.153 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/0, method: GET, req_id: de9fe33f-328d-45bc-b2cb-6edfad43c2d2, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:37:38.156 INFO request completed, latency_us: 310, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9b102583-1124-4240-a2db-594a0272657b, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 4566 len 24576
002/300 Read block 1011 len 12288
003/300 Read block 357 len 16384
004/300 Write block 4790 len 20480 data: 4 4 4 4 4
005/300 Read block 609 len 36864
006/300 Write block 3036 len 4096 data: 2
007/300 Read block 4686 len 20480
008/300 Read block 3167 len 20480
009/300 Read block 251 len 28672
010/300 Read block 1654 len 32768
011/300 Read block 4784 len 36864
012/300 Flush
013/300 Write block 4523 len 8192 data: 3 3
014/300 Read block 4292 len 28672
015/300 Read block 2876 len 40960
016/300 Write block 4993 len 24576 data: 2 2 3 3 3 3
017/300 Read block 4375 len 20480
018/300 Read block 3263 len 24576
019/300 Read block 741 len 24576
020/300 Read block 3609 len 24576
021/300 Read block 3833 len 40960
022/300 Read block 3383 len 24576
023/300 Flush
024/300 Read block 1393 len 28672
025/300 Read block 1885 len 4096
026/300 Write block 4512 len 12288 data: 3 3 4
027/300 Read block 1933 len 12288
028/300 Write block 4207 len 16384 data: 2 2 2 2
029/300 Read block 1688 len 20480
030/300 Read block 3401 len 40960
031/300 Read block 43 len 40960
032/300 Write block 443 len 4096 data: 3
033/300 Write block 2574 len 28672 data: 2 2 3 3 3 3 3
034/300 Read block 4270 len 40960
035/300 Read block 273 len 12288
036/300 Write block 3046 len 28672 data: 4 3 3 3 3 3 3
037/300 Read block 4105 len 28672
038/300 Write block 2353 len 16384 data: 4 4 3 3
039/300 Read block 833 len 8192
040/300 Read block 249 len 4096
041/300 Write block 167 len 24576 data: 3 3 3 3 3 2
042/300 Read block 3649 len 36864
043/300 Write block 2470 len 28672 data: 3 2 2 3 3 3 3
044/300 Read block 2906 len 36864
045/300 Read block 422 len 28672
046/300 Read block 3343 len 20480
047/300 Flush
048/300 Write block 3359 len 24576 data: 2 2 2 2 2 3
049/300 Write block 351 len 24576 data: 3 3 3 3 3 3
050/300 Write block 3367 len 32768 data: 3 3 3 3 3 4 3 3
051/300 Write block 327 len 40960 data: 2 2 2 2 2 2 2 2 2 2
052/300 Write block 1597 len 40960 data: 3 3 3 3 3 3 3 2 2 2
053/300 Read block 3625 len 16384
054/300 Flush
055/300 Write block 4453 len 4096 data: 2
056/300 Write block 4960 len 12288 data: 4 4 4
057/300 Flush
058/300 Flush
059/300 Write block 4224 len 12288 data: 2 2 2
060/300 Write block 4009 len 20480 data: 3 3 2 2 3
061/300 Read block 1215 len 36864
062/300 Read block 4930 len 36864
063/300 Read block 4817 len 24576
064/300 Write block 2832 len 8192 data: 3 3
065/300 Write block 2400 len 8192 data: 3 3
066/300 Read block 2223 len 16384
067/300 Read block 4039 len 28672
068/300 Read block 3448 len 20480
069/300 Write block 3735 len 24576 data: 3 3 3 2 2 2
070/300 Write block 2751 len 4096 data: 3
071/300 Write block 68 len 24576 data: 3 3 3 2 2 2
072/300 Read block 3140 len 20480
073/300 Write block 4427 len 32768 data: 2 2 2 2 2 2 2 2
074/300 Flush
075/300 Read block 1537 len 40960
076/300 Write block 2885 len 36864 data: 2 2 2 2 2 2 2 2 2
077/300 Flush
078/300 Read block 2809 len 36864
079/300 Read block 2862 len 4096
080/300 Write block 432 len 4096 data: 2
081/300 Flush
082/300 Read block 308 len 20480
083/300 Write block 2988 len 36864 data: 2 2 2 2 2 2 2 2 2
084/300 Write block 638 len 20480 data: 2 2 2 2 2
085/300 Read block 4683 len 16384
086/300 Flush
087/300 Write block 3974 len 4096 data: 2
088/300 Flush
089/300 Read block 4530 len 32768
090/300 Write block 3519 len 36864 data: 3 3 3 3 3 3 3 3 2
091/300 Write block 1470 len 4096 data: 4
092/300 Write block 1886 len 40960 data: 2 2 2 2 2 2 2 2 2 2
093/300 Write block 2936 len 28672 data: 2 2 2 2 2 2 2
094/300 Read block 3766 len 20480
095/300 Read block 2066 len 16384
096/300 Flush
097/300 Flush
098/300 Write block 873 len 4096 data: 4
099/300 Write block 2173 len 32768 data: 2 2 2 2 2 2 2 2
100/300 Read block 1977 len 4096
101/300 Read block 2127 len 8192
102/300 Read block 2267 len 8192
103/300 Write block 3488 len 32768 data: 2 2 2 2 2 2 2 2
104/300 Flush
105/300 Read block 349 len 32768
106/300 Write block 1471 len 4096 data: 4
107/300 Write block 2369 len 8192 data: 3 3
108/300 Write block 655 len 4096 data: 3
109/300 Read block 1225 len 24576
110/300 Write block 2412 len 36864 data: 3 3 3 3 3 2 2 2 2
111/300 Write block 1862 len 20480 data: 3 2 2 2 2
112/300 Read block 2355 len 4096
113/300 Read block 3368 len 8192
114/300 Read block 1926 len 4096
115/300 Flush
116/300 Write block 4215 len 8192 data: 2 2
117/300 Write block 4156 len 24576 data: 2 2 2 2 3 3
118/300 Read block 4178 len 16384
119/300 Write block 3527 len 8192 data: 3 2
120/300 Flush
121/300 Read block 2289 len 16384
122/300 Read block 1719 len 24576
123/300 Read block 1638 len 20480
124/300 Flush
125/300 Write block 314 len 32768 data: 3 3 3 4 4 3 4 4
126/300 Read block 2610 len 32768
127/300 Read block 4175 len 28672
128/300 Read block 4382 len 16384
129/300 Read block 1875 len 36864
130/300 Write block 497 len 4096 data: 3
131/300 Write block 1591 len 32768 data: 2 2 2 2 2 2 4 4
132/300 Write block 1129 len 16384 data: 3 3 3 3
133/300 Read block 4185 len 4096
134/300 Read block 757 len 24576
135/300 Write block 4755 len 4096 data: 2
136/300 Read block 87 len 16384
137/300 Write block 574 len 40960 data: 2 3 3 3 3 3 3 3 3 3
138/300 Read block 3352 len 16384
139/300 Read block 3963 len 24576
140/300 Flush
141/300 Read block 453 len 8192
142/300 Read block 4231 len 4096
143/300 Flush
144/300 Read block 2019 len 32768
145/300 Read block 1274 len 12288
146/300 Read block 3763 len 16384
147/300 Read block 970 len 16384
148/300 Read block 822 len 12288
149/300 Read block 2574 len 24576
150/300 Read block 3302 len 28672
151/300 Read block 4812 len 40960
152/300 Read block 124 len 40960
153/300 Read block 3434 len 40960
154/300 Write block 1090 len 8192 data: 2 2
155/300 Write block 1708 len 4096 data: 2
156/300 Read block 1484 len 4096
157/300 Write block 3836 len 28672 data: 2 2 2 2 2 2 2
158/300 Flush
159/300 Read block 1724 len 16384
160/300 Read block 4472 len 20480
161/300 Write block 3197 len 24576 data: 2 2 2 2 2 2
162/300 Read block 2306 len 12288
163/300 Flush
164/300 Read block 1188 len 32768
165/300 Read block 346 len 20480
166/300 Write block 559 len 20480 data: 3 3 3 3 3
167/300 Write block 2190 len 40960 data: 2 2 2 3 3 3 3 3 3 3
168/300 Write block 3242 len 8192 data: 3 3
169/300 Flush
170/300 Flush
171/300 Write block 2175 len 32768 data: 3 3 3 3 3 3 3 3
172/300 Read block 3265 len 4096
173/300 Read block 987 len 28672
174/300 Flush
175/300 Write block 3841 len 12288 data: 3 3 2
176/300 Read block 4509 len 8192
177/300 Read block 1442 len 8192
178/300 Write block 1111 len 32768 data: 3 2 2 2 2 2 2 2
179/300 Write block 1627 len 12288 data: 3 3 3
180/300 Read block 3298 len 16384
181/300 Read block 115 len 36864
182/300 Read block 3063 len 32768
183/300 Write block 2512 len 24576 data: 2 2 2 2 2 2
184/300 Write block 1222 len 12288 data: 4 5 5
185/300 Read block 47 len 12288
186/300 Read block 3936 len 24576
187/300 Write block 2099 len 16384 data: 2 2 2 3
188/300 Flush
189/300 Read block 4997 len 12288
190/300 Read block 1454 len 36864
191/300 Read block 221 len 32768
192/300 Read block 3283 len 24576
193/300 Write block 128 len 24576 data: 3 2 2 3 2 2
194/300 Flush
195/300 Write block 928 len 28672 data: 2 2 3 3 3 3 2
196/300 Write block 4027 len 4096 data: 2
197/300 Write block 4486 len 12288 data: 2 2 2
198/300 Read block 2891 len 8192
199/300 Read block 797 len 24576
200/300 Flush
201/300 Write block 1870 len 40960 data: 2 2 2 2 2 2 2 2 2 2
202/300 Flush
203/300 Flush
204/300 Flush
205/300 Read block 1341 len 12288
206/300 Write block 2710 len 12288 data: 3 2 2
207/300 Read block 2126 len 12288
208/300 Read block 1070 len 28672
209/300 Read block 1267 len 36864
210/300 Read block 576 len 12288
211/300 Read block 2695 len 4096
212/300 Read block 1087 len 24576
213/300 Read block 1700 len 32768
214/300 Write block 1475 len 32768 data: 3 3 2 3 3 4 4 4
215/300 Read block 2154 len 28672
216/300 Write block 4010 len 16384 data: 4 3 3 4
217/300 Read block 4264 len 8192
218/300 Write block 3893 len 40960 data: 2 2 2 3 3 3 3 2 2 3
219/300 Read block 4672 len 12288
220/300 Read block 1006 len 24576
221/300 Read block 279 len 32768
222/300 Write block 4059 len 4096 data: 2
223/300 Write block 689 len 4096 data: 2
224/300 Read block 820 len 28672
225/300 Write block 4922 len 28672 data: 4 4 3 3 3 3 3
226/300 Read block 2132 len 24576
227/300 Write block 2506 len 16384 data: 2 2 2 2
228/300 Read block 4119 len 40960
229/300 Read block 742 len 4096
230/300 Write block 3956 len 32768 data: 3 3 3 3 2 2 2 2
231/300 Read block 2981 len 20480
232/300 Write block 1803 len 32768 data: 2 2 2 2 2 2 2 4
233/300 Write block 3708 len 16384 data: 3 3 3 3
234/300 Write block 2331 len 40960 data: 2 2 2 2 3 3 3 3 4 4
235/300 Write block 86 len 20480 data: 2 2 3 2 2
236/300 Flush
237/300 Write block 2375 len 24576 data: 3 3 3 4 3 3
238/300 Read block 4392 len 28672
239/300 Write block 1298 len 40960 data: 2 2 2 3 3 4 4 3 4 4
240/300 Read block 1160 len 20480
241/300 Read block 2652 len 24576
242/300 Write block 2934 len 16384 data: 2 2 3 3
243/300 Write block 2154 len 40960 data: 2 2 2 2 2 2 2 3 4 4
244/300 Write block 1557 len 12288 data: 3 3 3
245/300 Read block 2442 len 4096
246/300 Read block 3714 len 40960
247/300 Write block 817 len 36864 data: 2 2 2 3 2 2 2 2 2
248/300 Flush
249/300 Read block 292 len 36864
250/300 Write block 3518 len 20480 data: 3 4 4 4 4
251/300 Read block 2584 len 32768
252/300 Read block 418 len 8192
253/300 Write block 583 len 4096 data: 4
254/300 Write block 4615 len 20480 data: 2 2 2 2 2
255/300 Read block 2497 len 4096
256/300 Read block 1140 len 40960
257/300 Read block 825 len 8192
258/300 Read block 2274 len 40960
259/300 Write block 2414 len 4096 data: 4
260/300 Write block 4867 len 24576 data: 2 2 2 2 2 2
261/300 Read block 3559 len 8192
262/300 Write block 2629 len 4096 data: 2
263/300 Write block 1569 len 20480 data: 2 2 3 3 3
264/300 Read block 2498 len 36864
265/300 Write block 3633 len 16384 data: 2 2 3 3
266/300 Write block 3129 len 28672 data: 3 4 4 4 4 3 4
267/300 Write block 708 len 4096 data: 2
268/300 Read block 2805 len 20480
269/300 Write block 3568 len 16384 data: 2 3 3 3
270/300 Write block 46 len 8192 data: 3 3
271/300 Read block 2478 len 16384
272/300 Read block 3669 len 12288
273/300 Read block 2989 len 24576
274/300 Write block 681 len 16384 data: 2 2 2 2
275/300 Read block 1484 len 32768
276/300 Read block 680 len 32768
277/300 Read block 4111 len 24576
278/300 Read block 3508 len 36864
279/300 Read block 2447 len 28672
280/300 Read block 1041 len 32768
281/300 Flush
282/300 Write block 173 len 20480 data: 2 2 2 2 2
283/300 Write block 2944 len 12288 data: 2 2 2
284/300 Read block 2913 len 16384
285/300 Write block 2979 len 40960 data: 5 5 5 4 4 3 3 3 3 3
286/300 Read block 2607 len 8192
287/300 Write block 1754 len 36864 data: 2 2 2 2 2 2 2 2 2
288/300 Read block 2064 len 24576
289/300 Read block 4079 len 36864
290/300 Write block 3836 len 20480 data: 3 3 3 3 3
291/300 Read block 2321 len 20480
292/300 Write block 4838 len 36864 data: 2 2 2 2 2 2 2 2 2
293/300 Write block 2679 len 24576 data: 4 4 4 4 4 4
294/300 Read block 932 len 16384
295/300 Flush
296/300 Write block 4065 len 36864 data: 2 2 2 2 2 2 2 2 2
297/300 Read block 4503 len 40960
298/300 Write block 3280 len 16384 data: 2 2 2 2
299/300 Read block 317 len 36864
300/300 Write block 4265 len 28672 data: 3 3 2 2 2 2 4
301/300 Flush
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:37:38.394 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/0, method: POST, req_id: 781ce18e-daaf-485b-96cf-83f7856313c6, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2510 Acked Read 6 Sent Done Done false
2511 Acked Read 3 Sent Done Done false
2512 Acked Read 4 Sent Done Done false
2513 Acked Write 5 Sent Done Done false
2514 Acked Read 9 Sent Done Done false
2515 Acked Write 1 Sent Done Done false
2516 Acked Read 5 Sent Done Done false
2517 Acked Read 5 Sent Done Done false
2518 Acked Read 7 Sent Done Done false
2519 Acked Read 8 Sent Done Done false
2520 Acked Read 9 Sent Done Done false
2521 Acked Flush 0 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Read 7 Sent Done Done false
2524 Acked Read 10 Sent Done Done false
2525 Acked Write 6 Sent Done Done false
2526 Acked Read 5 Sent Done Done false
2527 Acked Read 6 Sent Done Done false
2528 Acked Read 6 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Read 10 Sent Done Done false
2531 Acked Read 6 Sent Done Done false
2532 Acked Flush 0 Sent Done Done false
2533 Acked Read 7 Sent Done Done false
2534 Acked Read 1 Sent Done Done false
2535 Acked Write 3 Sent Done Done false
2536 Acked Read 3 Sent Done Done false
2537 Acked Write 4 Sent Done Done false
2538 Acked Read 5 Sent Done Done false
2539 Acked Read 10 Sent Done Done false
2540 Acked Read 10 Sent Done Done false
2541 Acked Write 1 Sent Done Done false
2542 Acked Write 7 Sent Done Done false
2543 Acked Read 10 Sent Done Done false
2544 Acked Read 3 Sent Done Done false
2545 Acked Write 7 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 4 Sent Done Done false
2548 Acked Read 2 Sent Done Done false
2549 Acked Read 1 Sent Done Done false
2550 Acked Write 6 Sent Done Done false
2551 Acked Read 9 Sent Done Done false
2552 Acked Write 7 Sent Done Done false
2553 Acked Read 9 Sent Done Done false
2554 Acked Read 7 Sent Done Done false
2555 Acked Read 5 Sent Done Done false
2556 Acked Flush 0 Sent Done Done false
2557 Acked Write 6 Sent Done Done false
2558 Acked Write 6 Sent Done Done false
2559 Acked Write 8 Sent Done Done false
2560 Acked Write 10 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Read 4 Sent Done Done false
2563 Acked Flush 0 Sent Done Done false
2564 Acked Write 1 Sent Done Done false
2565 Acked Write 3 Sent Done Done false
2566 Acked Flush 0 Sent Done Done false
2567 Acked Flush 0 Sent Done Done false
2568 Acked Write 3 Sent Done Done false
2569 Acked Write 5 Sent Done Done false
2570 Acked Read 9 Sent Done Done false
2571 Acked Read 9 Sent Done Done false
2572 Acked Read 6 Sent Done Done false
2573 Acked Write 2 Sent Done Done false
2574 Acked Write 2 Sent Done Done false
2575 Acked Read 4 Sent Done Done false
2576 Acked Read 7 Sent Done Done false
2577 Acked Read 5 Sent Done Done false
2578 Acked Write 6 Sent Done Done false
2579 Acked Write 1 Sent Done Done false
2580 Acked Write 6 Sent Done Done false
2581 Acked Read 5 Sent Done Done false
2582 Acked Write 8 Sent Done Done false
2583 Acked Flush 0 Sent Done Done false
2584 Acked Read 10 Sent Done Done false
2585 Acked Write 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Read 9 Sent Done Done false
2588 Acked Read 1 Sent Done Done false
2589 Acked Write 1 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Read 5 Sent Done Done false
2592 Acked Write 9 Sent Done Done false
2593 Acked Write 5 Sent Done Done false
2594 Acked Read 4 Sent Done Done false
2595 Acked Flush 0 Sent Done Done false
2596 Acked Write 1 Sent Done Done false
2597 Acked Flush 0 Sent Done Done false
2598 Acked Read 8 Sent Done Done false
2599 Acked Write 9 Sent Done Done false
2600 Acked Write 1 Sent Done Done false
2601 Acked Write 10 Sent Done Done false
2602 Acked Write 7 Sent Done Done false
2603 Acked Read 5 Sent Done Done false
2604 Acked Read 4 Sent Done Done false
2605 Acked Flush 0 Sent Done Done false
2606 Acked Flush 0 Sent Done Done false
2607 Acked Write 1 Sent Done Done false
2608 Acked Write 8 Sent Done Done false
2609 Acked Read 1 Sent Done Done false
2610 Acked Read 2 Sent Done Done false
2611 Acked Read 2 Sent Done Done false
2612 Acked Write 8 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Read 8 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Write 2 Sent Done Done false
2617 Acked Write 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Write 9 Sent Done Done false
2620 Acked Write 5 Sent Done Done false
2621 Acked Read 1 Sent Done Done false
2622 Acked Read 2 Sent Done Done false
2623 Acked Read 1 Sent Done Done false
2624 Acked Flush 0 Sent Done Done false
2625 Acked Write 2 Sent Done Done false
2626 Acked Write 6 Sent Done Done false
2627 Acked Read 4 Sent Done Done false
2628 Acked Write 2 Sent Done Done false
2629 Acked Flush 0 Sent Done Done false
2630 Acked Read 4 Sent Done Done false
2631 Acked Read 6 Sent Done Done false
2632 Acked Read 5 Sent Done Done false
2633 Acked Flush 0 Sent Done Done false
2634 Acked Write 8 Sent Done Done false
2635 Acked Read 8 Sent Done Done false
2636 Acked Read 7 Sent Done Done false
2637 Acked Read 4 Sent Done Done false
2638 Acked Read 9 Sent Done Done false
2639 Acked Write 1 Sent Done Done false
2640 Acked Write 8 Sent Done Done false
2641 Acked Write 4 Sent Done Done false
2642 Acked Read 1 Sent Done Done false
2643 Acked Read 6 Sent Done Done false
2644 Acked Write 1 Sent Done Done false
2645 Acked Read 4 Sent Done Done false
2646 Acked Write 10 Sent Done Done false
2647 Acked Read 4 Sent Done Done false
2648 Acked Read 6 Sent Done Done false
2649 Acked Flush 0 Sent Done Done false
2650 Acked Read 2 Sent Done Done false
2651 Acked Read 1 Sent Done Done false
2652 Acked Flush 0 Sent Done Done false
2653 Acked Read 8 Sent Done Done false
2654 Acked Read 3 Sent Done Done false
2655 Acked Read 4 Sent Done Done false
2656 Acked Read 4 Sent Done Done false
2657 Acked Read 3 Sent Done Done false
2658 Acked Read 6 Sent Done Done false
2659 Acked Read 7 Sent Done Done false
2660 Acked Read 10 Sent Done Done false
2661 Acked Read 10 Sent Done Done false
2662 Acked Read 10 Sent Done Done false
2663 Acked Write 2 Sent Done Done false
2664 Acked Write 1 Sent Done Done false
2665 Acked Read 1 Sent Done Done false
2666 Acked Write 7 Sent Done Done false
2667 Acked Flush 0 Sent Done Done false
2668 Acked Read 4 Sent Done Done false
2669 Acked Read 5 Sent Done Done false
2670 Acked Write 6 Sent Done Done false
2671 Acked Read 3 Sent Done Done false
2672 Acked Flush 0 Sent Done Done false
2673 Acked Read 8 Sent Done Done false
2674 Acked Read 5 Sent Done Done false
2675 Acked Write 5 Sent Done Done false
2676 Acked Write 10 Sent Done Done false
2677 Acked Write 2 Sent Done Done false
2678 Acked Flush 0 Sent Done Done false
2679 Acked Flush 0 Sent Done Done false
2680 Acked Write 8 Sent Done Done false
2681 Acked Read 1 Sent Done Done false
2682 Acked Read 7 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 3 Sent Done Done false
2685 Acked Read 2 Sent Done Done false
2686 Acked Read 2 Sent Done Done false
2687 Acked Write 8 Sent Done Done false
2688 Acked Write 3 Sent Done Done false
2689 Acked Read 4 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 8 Sent Done Done false
2692 Acked Write 6 Sent Done Done false
2693 Acked Write 3 Sent Done Done false
2694 Acked Read 3 Sent Done Done false
2695 Acked Read 6 Sent Done Done false
2696 Acked Write 4 Sent Done Done false
2697 Acked Flush 0 Sent Done Done false
2698 Acked Read 3 Sent Done Done false
2699 Acked Read 9 Sent Done Done false
2700 Acked Read 8 Sent Done Done false
2701 Acked Read 6 Sent Done Done false
2702 Acked Write 6 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Write 7 Sent Done Done false
2705 Acked Write 1 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Read 2 Sent Done Done false
2708 Acked Read 6 Sent Done Done false
2709 Acked Flush 0 Sent Done Done false
2710 Acked Write 10 Sent Done Done false
2711 Acked Flush 0 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Flush 0 Sent Done Done false
2714 Acked Read 3 Sent Done Done false
2715 Acked Write 3 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 7 Sent Done Done false
2718 Acked Read 9 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Read 6 Sent Done Done false
2722 Acked Read 8 Sent Done Done false
2723 Acked Write 8 Sent Done Done false
2724 Acked Read 7 Sent Done Done false
2725 Acked Write 4 Sent Done Done false
2726 Acked Read 2 Sent Done Done false
2727 Acked Write 10 Sent Done Done false
2728 Acked Read 3 Sent Done Done false
2729 Acked Read 6 Sent Done Done false
2730 Acked Read 8 Sent Done Done false
2731 Acked Write 1 Sent Done Done false
2732 Acked Write 1 Sent Done Done false
2733 Acked Read 7 Sent Done Done false
2734 Acked Write 7 Sent Done Done false
2735 Acked Read 6 Sent Done Done false
2736 Acked Write 4 Sent Done Done false
2737 Acked Read 10 Sent Done Done false
2738 Acked Read 1 Sent Done Done false
2739 Acked Write 8 Sent Done Done false
2740 Acked Read 5 Sent Done Done false
2741 Acked Write 8 Sent Done Done false
2742 Acked Write 4 Sent Done Done false
2743 Acked Write 10 Sent Done Done false
2744 Acked Write 5 Sent Done Done false
2745 Acked Flush 0 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Read 7 Sent Done Done false
2748 Acked Write 10 Sent Done Done false
2749 Acked Read 5 Sent Done Done false
2750 Acked Read 6 Sent Done Done false
2751 Acked Write 4 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 3 Sent Done Done false
2754 Acked Read 1 Sent Done Done false
2755 Acked Read 10 Sent Done Done false
2756 Acked Write 9 Sent Done Done false
2757 Acked Flush 0 Sent Done Done false
2758 Acked Read 9 Sent Done Done false
2759 Acked Write 5 Sent Done Done false
2760 Acked Read 8 Sent Done Done false
2761 Acked Read 2 Sent Done Done false
2762 Acked Write 1 Sent Done Done false
2763 Acked Write 5 Sent Done Done false
2764 Acked Read 1 Sent Done Done false
2765 Acked Read 10 Sent Done Done false
2766 Acked Read 2 Sent Done Done false
2767 Acked Read 10 Sent Done Done false
2768 Acked Write 1 Sent Done Done false
2769 Acked Write 6 Sent Done Done false
2770 Acked Read 2 Sent Done Done false
2771 Acked Write 1 Sent Done Done false
2772 Acked Write 5 Sent Done Done false
2773 Acked Read 9 Sent Done Done false
2774 Acked Write 4 Sent Done Done false
2775 Acked Write 7 Sent Done Done false
2776 Acked Write 1 Sent Done Done false
2777 Acked Read 5 Sent Done Done false
2778 Acked Write 4 Sent Done Done false
2779 Acked Write 2 Sent Done Done false
2780 Acked Read 4 Sent Done Done false
2781 Acked Read 3 Sent Done Done false
2782 Acked Read 6 Sent Done Done false
2783 Acked Write 4 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Read 8 Sent Done Done false
2786 Acked Read 6 Sent Done Done false
2787 Acked Read 9 Sent Done Done false
2788 Acked Read 7 Sent Done Done false
2789 Acked Read 8 Sent Done Done false
2790 Acked Flush 0 Sent Done Done false
2791 Acked Write 5 Sent Done Done false
2792 Acked Write 3 Sent Done Done false
2793 Acked Read 4 Sent Done Done false
2794 Acked Write 10 Sent Done Done false
2795 Acked Read 2 Sent Done Done false
2796 Acked Write 9 Sent Done Done false
2797 Acked Read 6 Sent Done Done false
2798 Acked Read 9 Sent Done Done false
2799 Acked Write 5 Sent Done Done false
2800 Acked Read 5 Sent Done Done false
2801 Acked Write 9 Sent Done Done false
2802 Acked Write 6 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Flush 0 Sent Done Done false
2805 Acked Write 9 Sent Done Done false
2806 Acked Read 10 Sent Done Done false
2807 Acked Write 4 Sent Done Done false
2808 Acked Read 9 Sent Done Done false
2809 Acked Write 7 Sent Done Done false
2810 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2509 2810 2810
Downstairs last five completed: 2509 2508 2507 2506 2505
Upstairs last five completed: 2810 2809 2808 2807 2806
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1206)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2510 Acked Read 6 Sent Done Done false
2511 Acked Read 3 Sent Done Done false
2512 Acked Read 4 Sent Done Done false
2513 Acked Write 5 Sent Done Done false
2514 Acked Read 9 Sent Done Done false
2515 Acked Write 1 Sent Done Done false
2516 Acked Read 5 Sent Done Done false
2517 Acked Read 5 Sent Done Done false
2518 Acked Read 7 Sent Done Done false
2519 Acked Read 8 Sent Done Done false
2520 Acked Read 9 Sent Done Done false
2521 Acked Flush 0 Sent Done Done false
2522 Acked Write 2 Sent Done Done false
2523 Acked Read 7 Sent Done Done false
2524 Acked Read 10 Sent Done Done false
2525 Acked Write 6 Sent Done Done false
2526 Acked Read 5 Sent Done Done false
2527 Acked Read 6 Sent Done Done false
2528 Acked Read 6 Sent Done Done false
2529 Acked Read 6 Sent Done Done false
2530 Acked Read 10 Sent Done Done false
2531 Acked Read 6 Sent Done Done false
2532 Acked Flush 0 Sent Done Done false
2533 Acked Read 7 Sent Done Done false
2534 Acked Read 1 Sent Done Done false
2535 Acked Write 3 Sent Done Done false
2536 Acked Read 3 Sent Done Done false
2537 Acked Write 4 Sent Done Done false
2538 Acked Read 5 Sent Done Done false
2539 Acked Read 10 Sent Done Done false
2540 Acked Read 10 Sent Done Done false
2541 Acked Write 1 Sent Done Done false
2542 Acked Write 7 Sent Done Done false
2543 Acked Read 10 Sent Done Done false
2544 Acked Read 3 Sent Done Done false
2545 Acked Write 7 Sent Done Done false
2546 Acked Read 7 Sent Done Done false
2547 Acked Write 4 Sent Done Done false
2548 Acked Read 2 Sent Done Done false
2549 Acked Read 1 Sent Done Done false
2550 Acked Write 6 Sent Done Done false
2551 Acked Read 9 Sent Done Done false
2552 Acked Write 7 Sent Done Done false
2553 Acked Read 9 Sent Done Done false
2554 Acked Read 7 Sent Done Done false
2555 Acked Read 5 Sent Done Done false
2556 Acked Flush 0 Sent Done Done false
2557 Acked Write 6 Sent Done Done false
2558 Acked Write 6 Sent Done Done false
2559 Acked Write 8 Sent Done Done false
2560 Acked Write 10 Sent Done Done false
2561 Acked Write 10 Sent Done Done false
2562 Acked Read 4 Sent Done Done false
2563 Acked Flush 0 Sent Done Done false
2564 Acked Write 1 Sent Done Done false
2565 Acked Write 3 Sent Done Done false
2566 Acked Flush 0 Sent Done Done false
2567 Acked Flush 0 Sent Done Done false
2568 Acked Write 3 Sent Done Done false
2569 Acked Write 5 Sent Done Done false
2570 Acked Read 9 Sent Done Done false
2571 Acked Read 9 Sent Done Done false
2572 Acked Read 6 Sent Done Done false
2573 Acked Write 2 Sent Done Done false
2574 Acked Write 2 Sent Done Done false
2575 Acked Read 4 Sent Done Done false
2576 Acked Read 7 Sent Done Done false
2577 Acked Read 5 Sent Done Done false
2578 Acked Write 6 Sent Done Done false
2579 Acked Write 1 Sent Done Done false
2580 Acked Write 6 Sent Done Done false
2581 Acked Read 5 Sent Done Done false
2582 Acked Write 8 Sent Done Done false
2583 Acked Flush 0 Sent Done Done false
2584 Acked Read 10 Sent Done Done false
2585 Acked Write 9 Sent Done Done false
2586 Acked Flush 0 Sent Done Done false
2587 Acked Read 9 Sent Done Done false
2588 Acked Read 1 Sent Done Done false
2589 Acked Write 1 Sent Done Done false
2590 Acked Flush 0 Sent Done Done false
2591 Acked Read 5 Sent Done Done false
2592 Acked Write 9 Sent Done Done false
2593 Acked Write 5 Sent Done Done false
2594 Acked Read 4 Sent Done Done false
2595 Acked Flush 0 Sent Done Done false
2596 Acked Write 1 Sent Done Done false
2597 Acked Flush 0 Sent Done Done false
2598 Acked Read 8 Sent Done Done false
2599 Acked Write 9 Sent Done Done false
2600 Acked Write 1 Sent Done Done false
2601 Acked Write 10 Sent Done Done false
2602 Acked Write 7 Sent Done Done false
2603 Acked Read 5 Sent Done Done false
2604 Acked Read 4 Sent Done Done false
2605 Acked Flush 0 Sent Done Done false
2606 Acked Flush 0 Sent Done Done false
2607 Acked Write 1 Sent Done Done false
2608 Acked Write 8 Sent Done Done false
2609 Acked Read 1 Sent Done Done false
2610 Acked Read 2 Sent Done Done false
2611 Acked Read 2 Sent Done Done false
2612 Acked Write 8 Sent Done Done false
2613 Acked Flush 0 Sent Done Done false
2614 Acked Read 8 Sent Done Done false
2615 Acked Write 1 Sent Done Done false
2616 Acked Write 2 Sent Done Done false
2617 Acked Write 1 Sent Done Done false
2618 Acked Read 6 Sent Done Done false
2619 Acked Write 9 Sent Done Done false
2620 Acked Write 5 Sent Done Done false
2621 Acked Read 1 Sent Done Done false
2622 Acked Read 2 Sent Done Done false
2623 Acked Read 1 Sent Done Done false
2624 Acked Flush 0 Sent Done Done false
2625 Acked Write 2 Sent Done Done false
2626 Acked Write 6 Sent Done Done false
2627 Acked Read 4 Sent Done Done false
2628 Acked Write 2 Sent Done Done false
2629 Acked Flush 0 Sent Done Done false
2630 Acked Read 4 Sent Done Done false
2631 Acked Read 6 Sent Done Done false
2632 Acked Read 5 Sent Done Done false
2633 Acked Flush 0 Sent Done Done false
2634 Acked Write 8 Sent Done Done false
2635 Acked Read 8 Sent Done Done false
2636 Acked Read 7 Sent Done Done false
2637 Acked Read 4 Sent Done Done false
2638 Acked Read 9 Sent Done Done false
2639 Acked Write 1 Sent Done Done false
2640 Acked Write 8 Sent Done Done false
2641 Acked Write 4 Sent Done Done false
2642 Acked Read 1 Sent Done Done false
2643 Acked Read 6 Sent Done Done false
2644 Acked Write 1 Sent Done Done false
2645 Acked Read 4 Sent Done Done false
2646 Acked Write 10 Sent Done Done false
2647 Acked Read 4 Sent Done Done false
2648 Acked Read 6 Sent Done Done false
2649 Acked Flush 0 Sent Done Done false
2650 Acked Read 2 Sent Done Done false
2651 Acked Read 1 Sent Done Done false
2652 Acked Flush 0 Sent Done Done false
2653 Acked Read 8 Sent Done Done false
2654 Acked Read 3 Sent Done Done false
2655 Acked Read 4 Sent Done Done false
2656 Acked Read 4 Sent Done Done false
2657 Acked Read 3 Sent Done Done false
2658 Acked Read 6 Sent Done Done false
2659 Acked Read 7 Sent Done Done false
2660 Acked Read 10 Sent Done Done false
2661 Acked Read 10 Sent Done Done false
2662 Acked Read 10 Sent Done Done false
2663 Acked Write 2 Sent Done Done false
2664 Acked Write 1 Sent Done Done false
2665 Acked Read 1 Sent Done Done false
2666 Acked Write 7 Sent Done Done false
2667 Acked Flush 0 Sent Done Done false
2668 Acked Read 4 Sent Done Done false
2669 Acked Read 5 Sent Done Done false
2670 Acked Write 6 Sent Done Done false
2671 Acked Read 3 Sent Done Done false
2672 Acked Flush 0 Sent Done Done false
2673 Acked Read 8 Sent Done Done false
2674 Acked Read 5 Sent Done Done false
2675 Acked Write 5 Sent Done Done false
2676 Acked Write 10 Sent Done Done false
2677 Acked Write 2 Sent Done Done false
2678 Acked Flush 0 Sent Done Done false
2679 Acked Flush 0 Sent Done Done false
2680 Acked Write 8 Sent Done Done false
2681 Acked Read 1 Sent Done Done false
2682 Acked Read 7 Sent Done Done false
2683 Acked Flush 0 Sent Done Done false
2684 Acked Write 3 Sent Done Done false
2685 Acked Read 2 Sent Done Done false
2686 Acked Read 2 Sent Done Done false
2687 Acked Write 8 Sent Done Done false
2688 Acked Write 3 Sent Done Done false
2689 Acked Read 4 Sent Done Done false
2690 Acked Read 9 Sent Done Done false
2691 Acked Read 8 Sent Done Done false
2692 Acked Write 6 Sent Done Done false
2693 Acked Write 3 Sent Done Done false
2694 Acked Read 3 Sent Done Done false
2695 Acked Read 6 Sent Done Done false
2696 Acked Write 4 Sent Done Done false
2697 Acked Flush 0 Sent Done Done false
2698 Acked Read 3 Sent Done Done false
2699 Acked Read 9 Sent Done Done false
2700 Acked Read 8 Sent Done Done false
2701 Acked Read 6 Sent Done Done false
2702 Acked Write 6 Sent Done Done false
2703 Acked Flush 0 Sent Done Done false
2704 Acked Write 7 Sent Done Done false
2705 Acked Write 1 Sent Done Done false
2706 Acked Write 3 Sent Done Done false
2707 Acked Read 2 Sent Done Done false
2708 Acked Read 6 Sent Done Done false
2709 Acked Flush 0 Sent Done Done false
2710 Acked Write 10 Sent Done Done false
2711 Acked Flush 0 Sent Done Done false
2712 Acked Flush 0 Sent Done Done false
2713 Acked Flush 0 Sent Done Done false
2714 Acked Read 3 Sent Done Done false
2715 Acked Write 3 Sent Done Done false
2716 Acked Read 3 Sent Done Done false
2717 Acked Read 7 Sent Done Done false
2718 Acked Read 9 Sent Done Done false
2719 Acked Read 3 Sent Done Done false
2720 Acked Read 1 Sent Done Done false
2721 Acked Read 6 Sent Done Done false
2722 Acked Read 8 Sent Done Done false
2723 Acked Write 8 Sent Done Done false
2724 Acked Read 7 Sent Done Done false
2725 Acked Write 4 Sent Done Done false
2726 Acked Read 2 Sent Done Done false
2727 Acked Write 10 Sent Done Done false
2728 Acked Read 3 Sent Done Done false
2729 Acked Read 6 Sent Done Done false
2730 Acked Read 8 Sent Done Done false
2731 Acked Write 1 Sent Done Done false
2732 Acked Write 1 Sent Done Done false
2733 Acked Read 7 Sent Done Done false
2734 Acked Write 7 Sent Done Done false
2735 Acked Read 6 Sent Done Done false
2736 Acked Write 4 Sent Done Done false
2737 Acked Read 10 Sent Done Done false
2738 Acked Read 1 Sent Done Done false
2739 Acked Write 8 Sent Done Done false
2740 Acked Read 5 Sent Done Done false
2741 Acked Write 8 Sent Done Done false
2742 Acked Write 4 Sent Done Done false
2743 Acked Write 10 Sent Done Done false
2744 Acked Write 5 Sent Done Done false
2745 Acked Flush 0 Sent Done Done false
2746 Acked Write 6 Sent Done Done false
2747 Acked Read 7 Sent Done Done false
2748 Acked Write 10 Sent Done Done false
2749 Acked Read 5 Sent Done Done false
2750 Acked Read 6 Sent Done Done false
2751 Acked Write 4 Sent Done Done false
2752 Acked Write 10 Sent Done Done false
2753 Acked Write 3 Sent Done Done false
2754 Acked Read 1 Sent Done Done false
2755 Acked Read 10 Sent Done Done false
2756 Acked Write 9 Sent Done Done false
2757 Acked Flush 0 Sent Done Done false
2758 Acked Read 9 Sent Done Done false
2759 Acked Write 5 Sent Done Done false
2760 Acked Read 8 Sent Done Done false
2761 Acked Read 2 Sent Done Done false
2762 Acked Write 1 Sent Done Done false
2763 Acked Write 5 Sent Done Done false
2764 Acked Read 1 Sent Done Done false
2765 Acked Read 10 Sent Done Done false
2766 Acked Read 2 Sent Done Done false
2767 Acked Read 10 Sent Done Done false
2768 Acked Write 1 Sent Done Done false
2769 Acked Write 6 Sent Done Done false
2770 Acked Read 2 Sent Done Done false
2771 Acked Write 1 Sent Done Done false
2772 Acked Write 5 Sent Done Done false
2773 Acked Read 9 Sent Done Done false
2774 Acked Write 4 Sent Done Done false
2775 Acked Write 7 Sent Done Done false
2776 Acked Write 1 Sent Done Done false
2777 Acked Read 5 Sent Done Done false
2778 Acked Write 4 Sent Done Done false
2779 Acked Write 2 Sent Done Done false
2780 Acked Read 4 Sent Done Done false
2781 Acked Read 3 Sent Done Done false
2782 Acked Read 6 Sent Done Done false
2783 Acked Write 4 Sent Done Done false
2784 Acked Read 8 Sent Done Done false
2785 Acked Read 8 Sent Done Done false
2786 Acked Read 6 Sent Done Done false
2787 Acked Read 9 Sent Done Done false
2788 Acked Read 7 Sent Done Done false
2789 Acked Read 8 Sent Done Done false
2790 Acked Flush 0 Sent Done Done false
2791 Acked Write 5 Sent Done Done false
2792 Acked Write 3 Sent Done Done false
2793 Acked Read 4 Sent Done Done false
2794 Acked Write 10 Sent Done Done false
2795 Acked Read 2 Sent Done Done false
2796 Acked Write 9 Sent Done Done false
2797 Acked Read 6 Sent Done Done false
2798 Acked Read 9 Sent Done Done false
2799 Acked Write 5 Sent Done Done false
2800 Acked Read 5 Sent Done Done false
2801 Acked Write 9 Sent Done Done false
2802 Acked Write 6 Sent Done Done false
2803 Acked Read 4 Sent Done Done false
2804 Acked Flush 0 Sent Done Done false
2805 Acked Write 9 Sent Done Done false
2806 Acked Read 10 Sent Done Done false
2807 Acked Write 4 Sent Done Done false
2808 Acked Read 9 Sent Done Done false
2809 Acked Write 7 Sent Done Done false
2810 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2509 2810 2810
Downstairs last five completed: 2509 2508 2507 2506 2505
Upstairs last five completed: 2810 2809 2808 2807 2806
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.154341277Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43241)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.154501489Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.155638966Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156235526Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156256334Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 2509","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156270402Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 2509","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156520003Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 301 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156539261Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 301 jobs since flush: 2509","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:44.156782922Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2810 2809 2808 2807 2806
Upstairs last five completed: 2810 2809 2808 2807 2806
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:37:46.400 INFO request completed[8810] Got stop action so:false kr:false
, latency_us: 317, response_code: 204, uri: /stop/cid/0, method: POST, req_id: dc48ee5a-bc75-43ad-96ac-026b8780f1dd, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.400830263Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.400881486Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.400904314Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:37:46.400 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9cd0afa2-f6ea-459f-827c-e293df9ee632, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.400985954Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:37:50.402 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/0, method: GET, req_id: 970aa308-e7f3-421d-aba5-f8b4dc4ae539, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2437 len 40960 data: 4 4 4 4 4 4 4 4 4 4
002/300 Read block 4351 len 20480
003/300 Write block 4785 len 28672 data: 2 2 2 3 3 5 5
004/300 Read block 2364 len 20480
005/300 Write block 1764 len 8192 data: 2 2
006/300 Read block 515 len 8192
007/300 Read block 3804 len 8192
008/300 Read block 2697 len 24576
009/300 Read block 2699 len 40960
010/300 Read block 3418 len 20480
011/300 Write block 4561 len 24576 data: 3 3 4 4 4 3
012/300 Read block 3408 len 12288
013/300 Write block 3742 len 36864 data: 2 2 2 2 2 3 3 3 3
014/300 Read block 4944 len 24576
015/300 Write block 1950 len 40960 data: 2 2 3 3 3 3 3 3 3 3
016/300 Write block 2796 len 4096 data: 3
017/300 Flush
018/300 Write block 2472 len 24576 data: 3 4 4 4 4 3
019/300 Write block 4480 len 16384 data: 2 2 2 2
020/300 Write block 1591 len 12288 data: 3 3 3
021/300 Read block 1310 len 40960
022/300 Flush
023/300 Write block 3137 len 28672 data: 4 4 4 4 4 4 4
024/300 Write block 4036 len 32768 data: 4 4 2 2 2 2 3 3
025/300 Read block 721 len 24576
026/300 Read block 4356 len 36864
027/300 Write block 4168 len 8192 data: 3 3
028/300 Write block 868 len 24576 data: 4 3 3 4 4 5
029/300 Read block 2128 len 8192
030/300 Write block 1805 len 28672 data: 3 3 3 3 3 5 5
031/300 Flush
032/300 Read block 2725 len 32768
033/300 Write block 3277 len 32768 data: 3 3 2 3 3 3 3 2
034/300 Write block 2428 len 8192 data: 3 3
035/300 Write block 226 len 4096 data: 4
036/300 Flush
037/300 Read block 4231 len 40960
038/300 Read block 953 len 16384
039/300 Read block 976 len 40960
040/300 Write block 802 len 8192 data: 2 2
041/300 Read block 3961 len 32768
042/300 Write block 2692 len 8192 data: 2 2
043/300 Read block 3369 len 28672
044/300 Write block 1531 len 12288 data: 2 2 2
045/300 Read block 4003 len 36864
046/300 Read block 1278 len 8192
047/300 Write block 2565 len 8192 data: 5 5
048/300 Read block 4563 len 16384
049/300 Write block 2757 len 20480 data: 2 2 2 2 2
050/300 Read block 4212 len 32768
051/300 Read block 2473 len 20480
052/300 Flush
053/300 Write block 1904 len 36864 data: 2 2 2 2 2 2 2 2 2
054/300 Read block 1005 len 24576
055/300 Write block 2254 len 40960 data: 3 5 4 3 3 2 2 2 2 2
056/300 Read block 1868 len 40960
057/300 Read block 2899 len 4096
058/300 Write block 956 len 12288 data: 3 3 3
059/300 Read block 1844 len 40960
060/300 Read block 3935 len 28672
061/300 Read block 1887 len 40960
062/300 Read block 2820 len 24576
063/300 Write block 2013 len 12288 data: 2 2 2
064/300 Read block 1157 len 32768
065/300 Read block 3843 len 20480
066/300 Read block 4107 len 16384
067/300 Write block 1754 len 4096 data: 3
068/300 Read block 4204 len 24576
069/300 Read block 175 len 20480
070/300 Write block 4656 len 16384 data: 2 2 2 2
071/300 Read block 806 len 12288
072/300 Read block 1425 len 40960
073/300 Write block 4706 len 12288 data: 3 3 4
074/300 Read block 2458 len 12288
075/300 Read block 3624 len 24576
076/300 Write block 4279 len 40960 data: 2 2 2 2 2 3 3 3 2 2
077/300 Read block 4900 len 24576
078/300 Read block 4231 len 12288
079/300 Read block 1989 len 28672
080/300 Write block 542 len 32768 data: 2 2 2 3 3 3 3 3
081/300 Read block 1517 len 8192
082/300 Flush
083/300 Read block 4892 len 36864
084/300 Write block 154 len 28672 data: 2 2 2 2 2 2 2
085/300 Read block 4041 len 20480
086/300 Write block 798 len 4096 data: 2
087/300 Write block 3751 len 40960 data: 2 3 3 3 3 3 3 3 3 3
088/300 Write block 3140 len 8192 data: 5 5
089/300 Read block 2449 len 12288
090/300 Read block 833 len 32768
091/300 Write block 2719 len 4096 data: 4
092/300 Write block 3187 len 40960 data: 2 2 2 2 2 2 2 2 2 2
093/300 Flush
094/300 Flush
095/300 Read block 4528 len 8192
096/300 Read block 617 len 8192
097/300 Write block 3842 len 40960 data: 4 3 2 3 3 3 3 3 2 2
098/300 Write block 4372 len 28672 data: 2 2 2 3 3 3 3
099/300 Read block 3673 len 28672
100/300 Read block 2316 len 40960
101/300 Flush
102/300 Read block 4886 len 36864
103/300 Read block 2172 len 8192
104/300 Read block 2807 len 20480
105/300 Write block 205 len 28672 data: 2 3 3 3 3 3 3
106/300 Read block 1925 len 36864
107/300 Read block 3618 len 12288
108/300 Read block 4240 len 8192
109/300 Read block 2134 len 16384
110/300 Read block 171 len 4096
111/300 Write block 3965 len 40960 data: 2 2 2 2 2 2 2 2 2 3
112/300 Write block 3605 len 4096 data: 2
113/300 Read block 3486 len 12288
114/300 Write block 64 len 12288 data: 3 3 3
115/300 Read block 2361 len 8192
116/300 Read block 1441 len 4096
117/300 Read block 3621 len 32768
118/300 Write block 4188 len 40960 data: 2 4 6 6 6 6 6 6 7 6
119/300 Read block 4018 len 12288
120/300 Write block 1145 len 20480 data: 2 2 2 2 2
121/300 Write block 2524 len 32768 data: 3 2 2 2 2 2 2 2
122/300 Write block 1314 len 16384 data: 3 3 3 3
123/300 Read block 2567 len 8192
124/300 Read block 2235 len 20480
125/300 Read block 799 len 4096
126/300 Write block 2534 len 24576 data: 2 2 2 2 2 2
127/300 Write block 3449 len 12288 data: 3 3 3
128/300 Read block 3553 len 12288
129/300 Read block 2787 len 40960
130/300 Read block 3631 len 12288
131/300 Flush
132/300 Read block 1618 len 36864
133/300 Read block 4410 len 24576
134/300 Read block 1998 len 40960
135/300 Read block 932 len 16384
136/300 Flush
137/300 Flush
138/300 Read block 2885 len 32768
139/300 Read block 4967 len 36864
140/300 Read block 938 len 8192
141/300 Read block 3103 len 20480
142/300 Write block 1032 len 24576 data: 5 5 5 4 4 3
143/300 Write block 4579 len 12288 data: 2 2 2
144/300 Write block 3559 len 40960 data: 4 4 4 4 4 4 4 2 2 3
145/300 Write block 498 len 24576 data: 4 4 3 3 3 3
146/300 Read block 4241 len 40960
147/300 Read block 4718 len 24576
148/300 Write block 129 len 16384 data: 3 3 4 3
149/300 Read block 553 len 40960
150/300 Flush
151/300 Write block 278 len 24576 data: 4 4 6 6 5 4
152/300 Write block 1105 len 4096 data: 3
153/300 Write block 1411 len 28672 data: 3 4 4 4 4 4 5
154/300 Write block 3464 len 28672 data: 3 3 3 3 2 2 2
155/300 Write block 4435 len 32768 data: 2 2 3 3 4 4 4 4
156/300 Read block 3956 len 28672
157/300 Read block 1511 len 20480
158/300 Read block 1204 len 36864
159/300 Read block 4554 len 16384
160/300 Write block 2444 len 32768 data: 5 5 5 2 3 3 3 3
161/300 Read block 2711 len 16384
162/300 Read block 2344 len 28672
163/300 Read block 3016 len 12288
164/300 Write block 4093 len 16384 data: 2 2 2 2
165/300 Read block 2916 len 12288
166/300 Read block 502 len 20480
167/300 Flush
168/300 Write block 4492 len 16384 data: 3 3 3 3
169/300 Read block 548 len 24576
170/300 Read block 1292 len 40960
171/300 Write block 234 len 16384 data: 2 2 2 2
172/300 Read block 304 len 4096
173/300 Write block 3107 len 8192 data: 2 2
174/300 Write block 3635 len 16384 data: 4 4 3 3
175/300 Read block 1667 len 4096
176/300 Write block 1196 len 12288 data: 4 2 2
177/300 Read block 4382 len 24576
178/300 Write block 4654 len 24576 data: 2 2 3 3 3 3
179/300 Read block 3331 len 16384
180/300 Write block 1231 len 40960 data: 2 2 3 4 4 4 4 4 4 3
181/300 Read block 4277 len 8192
182/300 Write block 3841 len 36864 data: 4 5 4 3 4 4 4 4 4
183/300 Read block 4168 len 12288
184/300 Read block 2784 len 8192
185/300 Read block 1327 len 28672
186/300 Write block 2529 len 24576 data: 3 3 3 2 2 3
187/300 Read block 4295 len 24576
188/300 Write block 746 len 12288 data: 3 3 3
189/300 Flush
190/300 Write block 3217 len 20480 data: 2 3 3 3 3
191/300 Read block 1002 len 28672
192/300 Read block 1417 len 36864
193/300 Write block 3937 len 20480 data: 4 3 3 3 3
194/300 Read block 2860 len 20480
195/300 Read block 1420 len 12288
196/300 Read block 132 len 12288
197/300 Write block 4897 len 16384 data: 3 3 3 2
198/300 Write block 2398 len 12288 data: 2 2 4
199/300 Write block 2621 len 16384 data: 4 4 4 4
200/300 Flush
201/300 Flush
202/300 Read block 3101 len 32768
203/300 Read block 725 len 36864
204/300 Write block 4683 len 4096 data: 3
205/300 Write block 4982 len 36864 data: 2 2 2 3 3 3 3 3 2
206/300 Read block 2419 len 40960
207/300 Read block 425 len 40960
208/300 Read block 1498 len 24576
209/300 Flush
210/300 Read block 4688 len 8192
211/300 Read block 111 len 12288
212/300 Read block 57 len 4096
213/300 Read block 1196 len 24576
214/300 Write block 3791 len 12288 data: 2 2 2
215/300 Read block 3896 len 16384
216/300 Write block 4446 len 40960 data: 3 2 2 2 2 2 2 3 2 2
217/300 Flush
218/300 Read block 1691 len 24576
219/300 Read block 2790 len 20480
220/300 Write block 3397 len 36864 data: 4 4 3 3 3 3 4 3 3
221/300 Write block 4900 len 12288 data: 3 2 3
222/300 Write block 2600 len 40960 data: 3 2 2 2 2 2 2 2 2 2
223/300 Read block 882 len 16384
224/300 Read block 3580 len 36864
225/300 Write block 2035 len 4096 data: 2
226/300 Flush
227/300 Read block 1159 len 16384
228/300 Write block 223 len 8192 data: 3 3
229/300 Read block 951 len 20480
230/300 Write block 719 len 24576 data: 3 4 4 4 4 4
231/300 Read block 1930 len 4096
232/300 Write block 711 len 16384 data: 3 3 7 7
233/300 Write block 705 len 16384 data: 2 2 2 3
234/300 Flush
235/300 Write block 437 len 16384 data: 2 2 3 3
236/300 Write block 4883 len 32768 data: 3 3 2 2 3 3 2 2
237/300 Read block 4982 len 16384
238/300 Read block 4118 len 20480
239/300 Read block 127 len 40960
240/300 Read block 2390 len 28672
241/300 Read block 3881 len 8192
242/300 Read block 2268 len 8192
243/300 Read block 4643 len 28672
244/300 Flush
245/300 Read block 4224 len 8192
246/300 Read block 3756 len 28672
247/300 Read block 4337 len 8192
248/300 Read block 4190 len 4096
249/300 Read block 4595 len 32768
250/300 Write block 239 len 16384 data: 3 3 3 3
251/300 Read block 3145 len 40960
252/300 Flush
253/300 Write block 400 len 20480 data: 3 4 4 4 4
254/300 Write block 956 len 40960 data: 4 4 4 4 4 4 3 3 3 2
255/300 Flush
256/300 Read block 432 len 8192
257/300 Read block 3282 len 32768
258/300 Write block 1458 len 8192 data: 4 3
259/300 Flush
260/300 Read block 4497 len 40960
261/300 Read block 3874 len 12288
262/300 Read block 1700 len 12288
263/300 Write block 4857 len 20480 data: 2 2 2 2 2
264/300 Write block 2830 len 20480 data: 3 3 4 4 3
265/300 Read block 4374 len 20480
266/300 Read block 3293 len 16384
267/300 Write block 3519 len 32768 data: 5 5 5 5 4 4 4 4
268/300 Read block 2054 len 12288
269/300 Read block 2738 len 24576
270/300 Read block 1021 len 8192
271/300 Read block 789 len 28672
272/300 Write block 1623 len 20480 data: 2 2 2 2 4
273/300 Read block 1077 len 32768
274/300 Read block 3003 len 4096
275/300 Write block 3721 len 40960 data: 2 2 2 2 3 3 4 4 4 3
276/300 Flush
277/300 Write block 3492 len 40960 data: 3 3 3 3 2 2 2 3 3 2
278/300 Write block 4628 len 28672 data: 2 2 2 2 2 3 3
279/300 Read block 387 len 40960
280/300 Read block 3240 len 24576
281/300 Write block 942 len 4096 data: 3
282/300 Write block 994 len 8192 data: 2 2
283/300 Read block 2493 len 16384
284/300 Flush
285/300 Read block 551 len 4096
286/300 Read block 2670 len 32768
287/300 Write block 2010 len 36864 data: 2 3 2 3 3 3 2 2 3
288/300 Read block 3235 len 24576
289/300 Write block 2482 len 12288 data: 2 3 3
290/300 Write block 1333 len 16384 data: 4 3 3 3
291/300 Read block 4897 len 20480
292/300 Read block 2124 len 24576
293/300 Read block 2397 len 20480
294/300 Write block 3604 len 32768 data: 2 3 2 2 2 2 2 2
295/300 Write block 4562 len 12288 data: 4 5 5
296/300 Write block 4651 len 16384 data: 2 2 2 3
297/300 Read block 1888 len 24576
298/300 Write block 3309 len 36864 data: 2 2 2 2 2 3 3 3 3
299/300 Write block 3603 len 24576 data: 2 3 4 3 3 3
300/300 Read block 1928 len 24576
301/300 Flush
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:37:50.644 INFO request completed, latency_us: 167, response_code: 204, uri: /start/cid/0, method: POST, req_id: 84d9516a-2cd9-4253-b134-ea60c908dbeb, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2811 Acked Write 10 Sent Done Done false
2812 Acked Read 5 Sent Done Done false
2813 Acked Write 7 Sent Done Done false
2814 Acked Read 5 Sent Done Done false
2815 Acked Write 2 Sent Done Done false
2816 Acked Read 2 Sent Done Done false
2817 Acked Read 2 Sent Done Done false
2818 Acked Read 6 Sent Done Done false
2819 Acked Read 10 Sent Done Done false
2820 Acked Read 5 Sent Done Done false
2821 Acked Write 6 Sent Done Done false
2822 Acked Read 3 Sent Done Done false
2823 Acked Write 9 Sent Done Done false
2824 Acked Read 6 Sent Done Done false
2825 Acked Write 10 Sent Done Done false
2826 Acked Write 1 Sent Done Done false
2827 Acked Flush 0 Sent Done Done false
2828 Acked Write 6 Sent Done Done false
2829 Acked Write 4 Sent Done Done false
2830 Acked Write 3 Sent Done Done false
2831 Acked Read 10 Sent Done Done false
2832 Acked Flush 0 Sent Done Done false
2833 Acked Write 7 Sent Done Done false
2834 Acked Write 8 Sent Done Done false
2835 Acked Read 6 Sent Done Done false
2836 Acked Read 9 Sent Done Done false
2837 Acked Write 2 Sent Done Done false
2838 Acked Write 6 Sent Done Done false
2839 Acked Read 2 Sent Done Done false
2840 Acked Write 7 Sent Done Done false
2841 Acked Flush 0 Sent Done Done false
2842 Acked Read 8 Sent Done Done false
2843 Acked Write 8 Sent Done Done false
2844 Acked Write 2 Sent Done Done false
2845 Acked Write 1 Sent Done Done false
2846 Acked Flush 0 Sent Done Done false
2847 Acked Read 10 Sent Done Done false
2848 Acked Read 4 Sent Done Done false
2849 Acked Read 10 Sent Done Done false
2850 Acked Write 2 Sent Done Done false
2851 Acked Read 8 Sent Done Done false
2852 Acked Write 2 Sent Done Done false
2853 Acked Read 7 Sent Done Done false
2854 Acked Write 3 Sent Done Done false
2855 Acked Read 9 Sent Done Done false
2856 Acked Read 2 Sent Done Done false
2857 Acked Write 2 Sent Done Done false
2858 Acked Read 4 Sent Done Done false
2859 Acked Write 5 Sent Done Done false
2860 Acked Read 8 Sent Done Done false
2861 Acked Read 5 Sent Done Done false
2862 Acked Flush 0 Sent Done Done false
2863 Acked Write 9 Sent Done Done false
2864 Acked Read 6 Sent Done Done false
2865 Acked Write 10 Sent Done Done false
2866 Acked Read 10 Sent Done Done false
2867 Acked Read 1 Sent Done Done false
2868 Acked Write 3 Sent Done Done false
2869 Acked Read 10 Sent Done Done false
2870 Acked Read 7 Sent Done Done false
2871 Acked Read 10 Sent Done Done false
2872 Acked Read 6 Sent Done Done false
2873 Acked Write 3 Sent Done Done false
2874 Acked Read 8 Sent Done Done false
2875 Acked Read 5 Sent Done Done false
2876 Acked Read 4 Sent Done Done false
2877 Acked Write 1 Sent Done Done false
2878 Acked Read 6 Sent Done Done false
2879 Acked Read 5 Sent Done Done false
2880 Acked Write 4 Sent Done Done false
2881 Acked Read 3 Sent Done Done false
2882 Acked Read 10 Sent Done Done false
2883 Acked Write 3 Sent Done Done false
2884 Acked Read 3 Sent Done Done false
2885 Acked Read 6 Sent Done Done false
2886 Acked Write 10 Sent Done Done false
2887 Acked Read 6 Sent Done Done false
2888 Acked Read 3 Sent Done Done false
2889 Acked Read 7 Sent Done Done false
2890 Acked Write 8 Sent Done Done false
2891 Acked Read 2 Sent Done Done false
2892 Acked Flush 0 Sent Done Done false
2893 Acked Read 9 Sent Done Done false
2894 Acked Write 7 Sent Done Done false
2895 Acked Read 5 Sent Done Done false
2896 Acked Write 1 Sent Done Done false
2897 Acked Write 10 Sent Done Done false
2898 Acked Write 2 Sent Done Done false
2899 Acked Read 3 Sent Done Done false
2900 Acked Read 8 Sent Done Done false
2901 Acked Write 1 Sent Done Done false
2902 Acked Write 10 Sent Done Done false
2903 Acked Flush 0 Sent Done Done false
2904 Acked Flush 0 Sent Done Done false
2905 Acked Read 2 Sent Done Done false
2906 Acked Read 2 Sent Done Done false
2907 Acked Write 10 Sent Done Done false
2908 Acked Write 7 Sent Done Done false
2909 Acked Read 7 Sent Done Done false
2910 Acked Read 10 Sent Done Done false
2911 Acked Flush 0 Sent Done Done false
2912 Acked Read 9 Sent Done Done false
2913 Acked Read 2 Sent Done Done false
2914 Acked Read 5 Sent Done Done false
2915 Acked Write 7 Sent Done Done false
2916 Acked Read 9 Sent Done Done false
2917 Acked Read 3 Sent Done Done false
2918 Acked Read 2 Sent Done Done false
2919 Acked Read 4 Sent Done Done false
2920 Acked Read 1 Sent Done Done false
2921 Acked Write 10 Sent Done Done false
2922 Acked Write 1 Sent Done Done false
2923 Acked Read 3 Sent Done Done false
2924 Acked Write 3 Sent Done Done false
2925 Acked Read 2 Sent Done Done false
2926 Acked Read 1 Sent Done Done false
2927 Acked Read 8 Sent Done Done false
2928 Acked Write 10 Sent Done Done false
2929 Acked Read 3 Sent Done Done false
2930 Acked Write 5 Sent Done Done false
2931 Acked Write 8 Sent Done Done false
2932 Acked Write 4 Sent Done Done false
2933 Acked Read 2 Sent Done Done false
2934 Acked Read 5 Sent Done Done false
2935 Acked Read 1 Sent Done Done false
2936 Acked Write 6 Sent Done Done false
2937 Acked Write 3 Sent Done Done false
2938 Acked Read 3 Sent Done Done false
2939 Acked Read 10 Sent Done Done false
2940 Acked Read 3 Sent Done Done false
2941 Acked Flush 0 Sent Done Done false
2942 Acked Read 9 Sent Done Done false
2943 Acked Read 6 Sent Done Done false
2944 Acked Read 10 Sent Done Done false
2945 Acked Read 4 Sent Done Done false
2946 Acked Flush 0 Sent Done Done false
2947 Acked Flush 0 Sent Done Done false
2948 Acked Read 8 Sent Done Done false
2949 Acked Read 9 Sent Done Done false
2950 Acked Read 2 Sent Done Done false
2951 Acked Read 5 Sent Done Done false
2952 Acked Write 6 Sent Done Done false
2953 Acked Write 3 Sent Done Done false
2954 Acked Write 10 Sent Done Done false
2955 Acked Write 6 Sent Done Done false
2956 Acked Read 10 Sent Done Done false
2957 Acked Read 6 Sent Done Done false
2958 Acked Write 4 Sent Done Done false
2959 Acked Read 10 Sent Done Done false
2960 Acked Flush 0 Sent Done Done false
2961 Acked Write 6 Sent Done Done false
2962 Acked Write 1 Sent Done Done false
2963 Acked Write 7 Sent Done Done false
2964 Acked Write 7 Sent Done Done false
2965 Acked Write 8 Sent Done Done false
2966 Acked Read 7 Sent Done Done false
2967 Acked Read 5 Sent Done Done false
2968 Acked Read 9 Sent Done Done false
2969 Acked Read 4 Sent Done Done false
2970 Acked Write 8 Sent Done Done false
2971 Acked Read 4 Sent Done Done false
2972 Acked Read 7 Sent Done Done false
2973 Acked Read 3 Sent Done Done false
2974 Acked Write 4 Sent Done Done false
2975 Acked Read 3 Sent Done Done false
2976 Acked Read 5 Sent Done Done false
2977 Acked Flush 0 Sent Done Done false
2978 Acked Write 4 Sent Done Done false
2979 Acked Read 6 Sent Done Done false
2980 Acked Read 10 Sent Done Done false
2981 Acked Write 4 Sent Done Done false
2982 Acked Read 1 Sent Done Done false
2983 Acked Write 2 Sent Done Done false
2984 Acked Write 4 Sent Done Done false
2985 Acked Read 1 Sent Done Done false
2986 Acked Write 3 Sent Done Done false
2987 Acked Read 6 Sent Done Done false
2988 Acked Write 6 Sent Done Done false
2989 Acked Read 4 Sent Done Done false
2990 Acked Write 10 Sent Done Done false
2991 Acked Read 2 Sent Done Done false
2992 Acked Write 9 Sent Done Done false
2993 Acked Read 3 Sent Done Done false
2994 Acked Read 2 Sent Done Done false
2995 Acked Read 7 Sent Done Done false
2996 Acked Write 6 Sent Done Done false
2997 Acked Read 6 Sent Done Done false
2998 Acked Write 3 Sent Done Done false
2999 Acked Flush 0 Sent Done Done false
3000 Acked Write 5 Sent Done Done false
3001 Acked Read 7 Sent Done Done false
3002 Acked Read 9 Sent Done Done false
3003 Acked Write 5 Sent Done Done false
3004 Acked Read 5 Sent Done Done false
3005 Acked Read 3 Sent Done Done false
3006 Acked Read 3 Sent Done Done false
3007 Acked Write 4 Sent Done Done false
3008 Acked Write 3 Sent Done Done false
3009 Acked Write 4 Sent Done Done false
3010 Acked Flush 0 Sent Done Done false
3011 Acked Flush 0 Sent Done Done false
3012 Acked Read 8 Sent Done Done false
3013 Acked Read 9 Sent Done Done false
3014 Acked Write 1 Sent Done Done false
3015 Acked Write 9 Sent Done Done false
3016 Acked Read 10 Sent Done Done false
3017 Acked Read 10 Sent Done Done false
3018 Acked Read 6 Sent Done Done false
3019 Acked Flush 0 Sent Done Done false
3020 Acked Read 2 Sent Done Done false
3021 Acked Read 3 Sent Done Done false
3022 Acked Read 1 Sent Done Done false
3023 Acked Read 6 Sent Done Done false
3024 Acked Write 3 Sent Done Done false
3025 Acked Read 4 Sent Done Done false
3026 Acked Write 10 Sent Done Done false
3027 Acked Flush 0 Sent Done Done false
3028 Acked Read 6 Sent Done Done false
3029 Acked Read 5 Sent Done Done false
3030 Acked Write 9 Sent Done Done false
3031 Acked Write 3 Sent Done Done false
3032 Acked Write 10 Sent Done Done false
3033 Acked Read 4 Sent Done Done false
3034 Acked Read 9 Sent Done Done false
3035 Acked Write 1 Sent Done Done false
3036 Acked Flush 0 Sent Done Done false
3037 Acked Read 4 Sent Done Done false
3038 Acked Write 2 Sent Done Done false
3039 Acked Read 5 Sent Done Done false
3040 Acked Write 6 Sent Done Done false
3041 Acked Read 1 Sent Done Done false
3042 Acked Write 4 Sent Done Done false
3043 Acked Write 4 Sent Done Done false
3044 Acked Flush 0 Sent Done Done false
3045 Acked Write 4 Sent Done Done false
3046 Acked Write 8 Sent Done Done false
3047 Acked Read 4 Sent Done Done false
3048 Acked Read 5 Sent Done Done false
3049 Acked Read 10 Sent Done Done false
3050 Acked Read 7 Sent Done Done false
3051 Acked Read 2 Sent Done Done false
3052 Acked Read 2 Sent Done Done false
3053 Acked Read 7 Sent Done Done false
3054 Acked Flush 0 Sent Done Done false
3055 Acked Read 2 Sent Done Done false
3056 Acked Read 7 Sent Done Done false
3057 Acked Read 2 Sent Done Done false
3058 Acked Read 1 Sent Done Done false
3059 Acked Read 8 Sent Done Done false
3060 Acked Write 4 Sent Done Done false
3061 Acked Read 10 Sent Done Done false
3062 Acked Flush 0 Sent Done Done false
3063 Acked Write 5 Sent Done Done false
3064 Acked Write 10 Sent Done Done false
3065 Acked Flush 0 Sent Done Done false
3066 Acked Read 2 Sent Done Done false
3067 Acked Read 8 Sent Done Done false
3068 Acked Write 2 Sent Done Done false
3069 Acked Flush 0 Sent Done Done false
3070 Acked Read 10 Sent Done Done false
3071 Acked Read 3 Sent Done Done false
3072 Acked Read 3 Sent Done Done false
3073 Acked Write 5 Sent Done Done false
3074 Acked Write 5 Sent Done Done false
3075 Acked Read 5 Sent Done Done false
3076 Acked Read 4 Sent Done Done false
3077 Acked Write 8 Sent Done Done false
3078 Acked Read 3 Sent Done Done false
3079 Acked Read 6 Sent Done Done false
3080 Acked Read 2 Sent Done Done false
3081 Acked Read 7 Sent Done Done false
3082 Acked Write 5 Sent Done Done false
3083 Acked Read 8 Sent Done Done false
3084 Acked Read 1 Sent Done Done false
3085 Acked Write 10 Sent Done Done false
3086 Acked Flush 0 Sent Done Done false
3087 Acked Write 10 Sent Done Done false
3088 Acked Write 7 Sent Done Done false
3089 Acked Read 10 Sent Done Done false
3090 Acked Read 6 Sent Done Done false
3091 Acked Write 1 Sent Done Done false
3092 Acked Write 2 Sent Done Done false
3093 Acked Read 4 Sent Done Done false
3094 Acked Flush 0 Sent Done Done false
3095 Acked Read 1 Sent Done Done false
3096 Acked Read 8 Sent Done Done false
3097 Acked Write 9 Sent Done Done false
3098 Acked Read 6 Sent Done Done false
3099 Acked Write 3 Sent Done Done false
3100 Acked Write 4 Sent Done Done false
3101 Acked Read 5 Sent Done Done false
3102 Acked Read 6 Sent Done Done false
3103 Acked Read 5 Sent Done Done false
3104 Acked Write 8 Sent Done Done false
3105 Acked Write 3 Sent Done Done false
3106 Acked Write 4 Sent Done Done false
3107 Acked Read 6 Sent Done Done false
3108 Acked Write 9 Sent Done Done false
3109 Acked Write 6 Sent Done Done false
3110 Acked Read 6 Sent Done Done false
3111 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2810 3111 3111
Downstairs last five completed: 2810 2809 2808 2807 2806
Upstairs last five completed: 3111 3110 3109 3108 3107
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1207)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2811 Acked Write 10 Sent Done Done false
2812 Acked Read 5 Sent Done Done false
2813 Acked Write 7 Sent Done Done false
2814 Acked Read 5 Sent Done Done false
2815 Acked Write 2 Sent Done Done false
2816 Acked Read 2 Sent Done Done false
2817 Acked Read 2 Sent Done Done false
2818 Acked Read 6 Sent Done Done false
2819 Acked Read 10 Sent Done Done false
2820 Acked Read 5 Sent Done Done false
2821 Acked Write 6 Sent Done Done false
2822 Acked Read 3 Sent Done Done false
2823 Acked Write 9 Sent Done Done false
2824 Acked Read 6 Sent Done Done false
2825 Acked Write 10 Sent Done Done false
2826 Acked Write 1 Sent Done Done false
2827 Acked Flush 0 Sent Done Done false
2828 Acked Write 6 Sent Done Done false
2829 Acked Write 4 Sent Done Done false
2830 Acked Write 3 Sent Done Done false
2831 Acked Read 10 Sent Done Done false
2832 Acked Flush 0 Sent Done Done false
2833 Acked Write 7 Sent Done Done false
2834 Acked Write 8 Sent Done Done false
2835 Acked Read 6 Sent Done Done false
2836 Acked Read 9 Sent Done Done false
2837 Acked Write 2 Sent Done Done false
2838 Acked Write 6 Sent Done Done false
2839 Acked Read 2 Sent Done Done false
2840 Acked Write 7 Sent Done Done false
2841 Acked Flush 0 Sent Done Done false
2842 Acked Read 8 Sent Done Done false
2843 Acked Write 8 Sent Done Done false
2844 Acked Write 2 Sent Done Done false
2845 Acked Write 1 Sent Done Done false
2846 Acked Flush 0 Sent Done Done false
2847 Acked Read 10 Sent Done Done false
2848 Acked Read 4 Sent Done Done false
2849 Acked Read 10 Sent Done Done false
2850 Acked Write 2 Sent Done Done false
2851 Acked Read 8 Sent Done Done false
2852 Acked Write 2 Sent Done Done false
2853 Acked Read 7 Sent Done Done false
2854 Acked Write 3 Sent Done Done false
2855 Acked Read 9 Sent Done Done false
2856 Acked Read 2 Sent Done Done false
2857 Acked Write 2 Sent Done Done false
2858 Acked Read 4 Sent Done Done false
2859 Acked Write 5 Sent Done Done false
2860 Acked Read 8 Sent Done Done false
2861 Acked Read 5 Sent Done Done false
2862 Acked Flush 0 Sent Done Done false
2863 Acked Write 9 Sent Done Done false
2864 Acked Read 6 Sent Done Done false
2865 Acked Write 10 Sent Done Done false
2866 Acked Read 10 Sent Done Done false
2867 Acked Read 1 Sent Done Done false
2868 Acked Write 3 Sent Done Done false
2869 Acked Read 10 Sent Done Done false
2870 Acked Read 7 Sent Done Done false
2871 Acked Read 10 Sent Done Done false
2872 Acked Read 6 Sent Done Done false
2873 Acked Write 3 Sent Done Done false
2874 Acked Read 8 Sent Done Done false
2875 Acked Read 5 Sent Done Done false
2876 Acked Read 4 Sent Done Done false
2877 Acked Write 1 Sent Done Done false
2878 Acked Read 6 Sent Done Done false
2879 Acked Read 5 Sent Done Done false
2880 Acked Write 4 Sent Done Done false
2881 Acked Read 3 Sent Done Done false
2882 Acked Read 10 Sent Done Done false
2883 Acked Write 3 Sent Done Done false
2884 Acked Read 3 Sent Done Done false
2885 Acked Read 6 Sent Done Done false
2886 Acked Write 10 Sent Done Done false
2887 Acked Read 6 Sent Done Done false
2888 Acked Read 3 Sent Done Done false
2889 Acked Read 7 Sent Done Done false
2890 Acked Write 8 Sent Done Done false
2891 Acked Read 2 Sent Done Done false
2892 Acked Flush 0 Sent Done Done false
2893 Acked Read 9 Sent Done Done false
2894 Acked Write 7 Sent Done Done false
2895 Acked Read 5 Sent Done Done false
2896 Acked Write 1 Sent Done Done false
2897 Acked Write 10 Sent Done Done false
2898 Acked Write 2 Sent Done Done false
2899 Acked Read 3 Sent Done Done false
2900 Acked Read 8 Sent Done Done false
2901 Acked Write 1 Sent Done Done false
2902 Acked Write 10 Sent Done Done false
2903 Acked Flush 0 Sent Done Done false
2904 Acked Flush 0 Sent Done Done false
2905 Acked Read 2 Sent Done Done false
2906 Acked Read 2 Sent Done Done false
2907 Acked Write 10 Sent Done Done false
2908 Acked Write 7 Sent Done Done false
2909 Acked Read 7 Sent Done Done false
2910 Acked Read 10 Sent Done Done false
2911 Acked Flush 0 Sent Done Done false
2912 Acked Read 9 Sent Done Done false
2913 Acked Read 2 Sent Done Done false
2914 Acked Read 5 Sent Done Done false
2915 Acked Write 7 Sent Done Done false
2916 Acked Read 9 Sent Done Done false
2917 Acked Read 3 Sent Done Done false
2918 Acked Read 2 Sent Done Done false
2919 Acked Read 4 Sent Done Done false
2920 Acked Read 1 Sent Done Done false
2921 Acked Write 10 Sent Done Done false
2922 Acked Write 1 Sent Done Done false
2923 Acked Read 3 Sent Done Done false
2924 Acked Write 3 Sent Done Done false
2925 Acked Read 2 Sent Done Done false
2926 Acked Read 1 Sent Done Done false
2927 Acked Read 8 Sent Done Done false
2928 Acked Write 10 Sent Done Done false
2929 Acked Read 3 Sent Done Done false
2930 Acked Write 5 Sent Done Done false
2931 Acked Write 8 Sent Done Done false
2932 Acked Write 4 Sent Done Done false
2933 Acked Read 2 Sent Done Done false
2934 Acked Read 5 Sent Done Done false
2935 Acked Read 1 Sent Done Done false
2936 Acked Write 6 Sent Done Done false
2937 Acked Write 3 Sent Done Done false
2938 Acked Read 3 Sent Done Done false
2939 Acked Read 10 Sent Done Done false
2940 Acked Read 3 Sent Done Done false
2941 Acked Flush 0 Sent Done Done false
2942 Acked Read 9 Sent Done Done false
2943 Acked Read 6 Sent Done Done false
2944 Acked Read 10 Sent Done Done false
2945 Acked Read 4 Sent Done Done false
2946 Acked Flush 0 Sent Done Done false
2947 Acked Flush 0 Sent Done Done false
2948 Acked Read 8 Sent Done Done false
2949 Acked Read 9 Sent Done Done false
2950 Acked Read 2 Sent Done Done false
2951 Acked Read 5 Sent Done Done false
2952 Acked Write 6 Sent Done Done false
2953 Acked Write 3 Sent Done Done false
2954 Acked Write 10 Sent Done Done false
2955 Acked Write 6 Sent Done Done false
2956 Acked Read 10 Sent Done Done false
2957 Acked Read 6 Sent Done Done false
2958 Acked Write 4 Sent Done Done false
2959 Acked Read 10 Sent Done Done false
2960 Acked Flush 0 Sent Done Done false
2961 Acked Write 6 Sent Done Done false
2962 Acked Write 1 Sent Done Done false
2963 Acked Write 7 Sent Done Done false
2964 Acked Write 7 Sent Done Done false
2965 Acked Write 8 Sent Done Done false
2966 Acked Read 7 Sent Done Done false
2967 Acked Read 5 Sent Done Done false
2968 Acked Read 9 Sent Done Done false
2969 Acked Read 4 Sent Done Done false
2970 Acked Write 8 Sent Done Done false
2971 Acked Read 4 Sent Done Done false
2972 Acked Read 7 Sent Done Done false
2973 Acked Read 3 Sent Done Done false
2974 Acked Write 4 Sent Done Done false
2975 Acked Read 3 Sent Done Done false
2976 Acked Read 5 Sent Done Done false
2977 Acked Flush 0 Sent Done Done false
2978 Acked Write 4 Sent Done Done false
2979 Acked Read 6 Sent Done Done false
2980 Acked Read 10 Sent Done Done false
2981 Acked Write 4 Sent Done Done false
2982 Acked Read 1 Sent Done Done false
2983 Acked Write 2 Sent Done Done false
2984 Acked Write 4 Sent Done Done false
2985 Acked Read 1 Sent Done Done false
2986 Acked Write 3 Sent Done Done false
2987 Acked Read 6 Sent Done Done false
2988 Acked Write 6 Sent Done Done false
2989 Acked Read 4 Sent Done Done false
2990 Acked Write 10 Sent Done Done false
2991 Acked Read 2 Sent Done Done false
2992 Acked Write 9 Sent Done Done false
2993 Acked Read 3 Sent Done Done false
2994 Acked Read 2 Sent Done Done false
2995 Acked Read 7 Sent Done Done false
2996 Acked Write 6 Sent Done Done false
2997 Acked Read 6 Sent Done Done false
2998 Acked Write 3 Sent Done Done false
2999 Acked Flush 0 Sent Done Done false
3000 Acked Write 5 Sent Done Done false
3001 Acked Read 7 Sent Done Done false
3002 Acked Read 9 Sent Done Done false
3003 Acked Write 5 Sent Done Done false
3004 Acked Read 5 Sent Done Done false
3005 Acked Read 3 Sent Done Done false
3006 Acked Read 3 Sent Done Done false
3007 Acked Write 4 Sent Done Done false
3008 Acked Write 3 Sent Done Done false
3009 Acked Write 4 Sent Done Done false
3010 Acked Flush 0 Sent Done Done false
3011 Acked Flush 0 Sent Done Done false
3012 Acked Read 8 Sent Done Done false
3013 Acked Read 9 Sent Done Done false
3014 Acked Write 1 Sent Done Done false
3015 Acked Write 9 Sent Done Done false
3016 Acked Read 10 Sent Done Done false
3017 Acked Read 10 Sent Done Done false
3018 Acked Read 6 Sent Done Done false
3019 Acked Flush 0 Sent Done Done false
3020 Acked Read 2 Sent Done Done false
3021 Acked Read 3 Sent Done Done false
3022 Acked Read 1 Sent Done Done false
3023 Acked Read 6 Sent Done Done false
3024 Acked Write 3 Sent Done Done false
3025 Acked Read 4 Sent Done Done false
3026 Acked Write 10 Sent Done Done false
3027 Acked Flush 0 Sent Done Done false
3028 Acked Read 6 Sent Done Done false
3029 Acked Read 5 Sent Done Done false
3030 Acked Write 9 Sent Done Done false
3031 Acked Write 3 Sent Done Done false
3032 Acked Write 10 Sent Done Done false
3033 Acked Read 4 Sent Done Done false
3034 Acked Read 9 Sent Done Done false
3035 Acked Write 1 Sent Done Done false
3036 Acked Flush 0 Sent Done Done false
3037 Acked Read 4 Sent Done Done false
3038 Acked Write 2 Sent Done Done false
3039 Acked Read 5 Sent Done Done false
3040 Acked Write 6 Sent Done Done false
3041 Acked Read 1 Sent Done Done false
3042 Acked Write 4 Sent Done Done false
3043 Acked Write 4 Sent Done Done false
3044 Acked Flush 0 Sent Done Done false
3045 Acked Write 4 Sent Done Done false
3046 Acked Write 8 Sent Done Done false
3047 Acked Read 4 Sent Done Done false
3048 Acked Read 5 Sent Done Done false
3049 Acked Read 10 Sent Done Done false
3050 Acked Read 7 Sent Done Done false
3051 Acked Read 2 Sent Done Done false
3052 Acked Read 2 Sent Done Done false
3053 Acked Read 7 Sent Done Done false
3054 Acked Flush 0 Sent Done Done false
3055 Acked Read 2 Sent Done Done false
3056 Acked Read 7 Sent Done Done false
3057 Acked Read 2 Sent Done Done false
3058 Acked Read 1 Sent Done Done false
3059 Acked Read 8 Sent Done Done false
3060 Acked Write 4 Sent Done Done false
3061 Acked Read 10 Sent Done Done false
3062 Acked Flush 0 Sent Done Done false
3063 Acked Write 5 Sent Done Done false
3064 Acked Write 10 Sent Done Done false
3065 Acked Flush 0 Sent Done Done false
3066 Acked Read 2 Sent Done Done false
3067 Acked Read 8 Sent Done Done false
3068 Acked Write 2 Sent Done Done false
3069 Acked Flush 0 Sent Done Done false
3070 Acked Read 10 Sent Done Done false
3071 Acked Read 3 Sent Done Done false
3072 Acked Read 3 Sent Done Done false
3073 Acked Write 5 Sent Done Done false
3074 Acked Write 5 Sent Done Done false
3075 Acked Read 5 Sent Done Done false
3076 Acked Read 4 Sent Done Done false
3077 Acked Write 8 Sent Done Done false
3078 Acked Read 3 Sent Done Done false
3079 Acked Read 6 Sent Done Done false
3080 Acked Read 2 Sent Done Done false
3081 Acked Read 7 Sent Done Done false
3082 Acked Write 5 Sent Done Done false
3083 Acked Read 8 Sent Done Done false
3084 Acked Read 1 Sent Done Done false
3085 Acked Write 10 Sent Done Done false
3086 Acked Flush 0 Sent Done Done false
3087 Acked Write 10 Sent Done Done false
3088 Acked Write 7 Sent Done Done false
3089 Acked Read 10 Sent Done Done false
3090 Acked Read 6 Sent Done Done false
3091 Acked Write 1 Sent Done Done false
3092 Acked Write 2 Sent Done Done false
3093 Acked Read 4 Sent Done Done false
3094 Acked Flush 0 Sent Done Done false
3095 Acked Read 1 Sent Done Done false
3096 Acked Read 8 Sent Done Done false
3097 Acked Write 9 Sent Done Done false
3098 Acked Read 6 Sent Done Done false
3099 Acked Write 3 Sent Done Done false
3100 Acked Write 4 Sent Done Done false
3101 Acked Read 5 Sent Done Done false
3102 Acked Read 6 Sent Done Done false
3103 Acked Read 5 Sent Done Done false
3104 Acked Write 8 Sent Done Done false
3105 Acked Write 3 Sent Done Done false
3106 Acked Write 4 Sent Done Done false
3107 Acked Read 6 Sent Done Done false
3108 Acked Write 9 Sent Done Done false
3109 Acked Write 6 Sent Done Done false
3110 Acked Read 6 Sent Done Done false
3111 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2810 3111 3111
Downstairs last five completed: 2810 2809 2808 2807 2806
Upstairs last five completed: 3111 3110 3109 3108 3107
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.402175022Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36058)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.402356841Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.403369783Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.403947595Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.403983401Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 2810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.404021466Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 2810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.404204755Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 301 jobs since flush 2810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.404230022Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 301 jobs since flush: 2810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:56.404483922Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3111 3110 3109 3108 3107
Upstairs last five completed: 3111 3110 3109 3108 3107
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:37:58.650[8830] Got stop action so:false kr:false
INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a1769050-cc4e-45d7-8611-7203ddc2db3f, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:58.651065147Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:58.651213909Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:58.651243126Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:37:58.651 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/2, method: GET, req_id: b7c7087b-a422-4035-9ab5-218bdfb28bd7, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:58.651383539Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:38:02.653 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: cf50052b-bc81-40c2-ba21-b3be849d8f51, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 4628 len 16384
003/300 Write block 3211 len 12288 data: 2 2 2
004/300 Write block 2390 len 8192 data: 3 4
005/300 Read block 3152 len 4096
006/300 Write block 1669 len 24576 data: 2 2 2 3 3 4
007/300 Read block 2496 len 32768
008/300 Read block 2204 len 20480
009/300 Write block 3687 len 32768 data: 2 3 3 3 3 3 3 3
010/300 Write block 4475 len 16384 data: 2 2 2 2
011/300 Flush
012/300 Flush
013/300 Flush
014/300 Read block 3150 len 24576
015/300 Write block 2951 len 28672 data: 3 3 3 3 3 3 2
016/300 Read block 350 len 4096
017/300 Write block 1522 len 16384 data: 2 2 2 2
018/300 Read block 1356 len 12288
019/300 Read block 4521 len 12288
020/300 Read block 1467 len 40960
021/300 Read block 216 len 4096
022/300 Read block 1786 len 8192
023/300 Flush
024/300 Flush
025/300 Read block 4131 len 36864
026/300 Read block 3629 len 8192
027/300 Read block 3488 len 12288
028/300 Write block 2276 len 12288 data: 2 2 2
029/300 Write block 4344 len 36864 data: 2 2 2 2 2 2 2 2 2
030/300 Write block 2231 len 4096 data: 3
031/300 Read block 723 len 40960
032/300 Write block 352 len 28672 data: 4 4 4 4 4 3 3
033/300 Read block 338 len 8192
034/300 Write block 3434 len 32768 data: 2 2 2 2 2 2 2 2
035/300 Write block 4658 len 4096 data: 4
036/300 Write block 1384 len 20480 data: 3 2 2 2 2
037/300 Write block 980 len 4096 data: 3
038/300 Read block 970 len 36864
039/300 Read block 3776 len 4096
040/300 Read block 4737 len 36864
041/300 Write block 3290 len 28672 data: 2 2 3 4 4 4 4
042/300 Read block 1075 len 4096
043/300 Write block 2167 len 20480 data: 4 4 4 3 2
044/300 Read block 1240 len 36864
045/300 Write block 4139 len 24576 data: 3 3 4 4 4 4
046/300 Read block 3991 len 4096
047/300 Read block 320 len 8192
048/300 Read block 731 len 8192
049/300 Read block 369 len 36864
050/300 Write block 3950 len 12288 data: 2 2 3
051/300 Write block 861 len 32768 data: 3 3 2 2 2 4 4 5
052/300 Write block 1389 len 24576 data: 2 2 3 3 3 3
053/300 Read block 4739 len 8192
054/300 Read block 4310 len 24576
055/300 Read block 564 len 12288
056/300 Write block 58 len 12288 data: 2 2 2
057/300 Read block 475 len 12288
058/300 Write block 2340 len 8192 data: 5 4
059/300 Write block 2234 len 32768 data: 2 2 2 2 2 3 3 3
060/300 Read block 12 len 32768
061/300 Write block 4810 len 28672 data: 5 4 2 2 2 2 2
062/300 Read block 3604 len 12288
063/300 Write block 2983 len 20480 data: 5 4 4 4 4
064/300 Read block 706 len 8192
065/300 Write block 4245 len 36864 data: 5 6 6 6 6 6 4 3 3
066/300 Write block 3759 len 28672 data: 4 4 3 3 2 2 2
067/300 Flush
068/300 Write block 279 len 16384 data: 5 7 7 6
069/300 Flush
070/300 Read block 2436 len 24576
071/300 Read block 1931 len 28672
072/300 Write block 1148 len 8192 data: 3 3
073/300 Read block 4492 len 16384
074/300 Read block 2954 len 4096
075/300 Read block 3821 len 8192
076/300 Read block 1193 len 36864
077/300 Write block 2152 len 40960 data: 2 2 3 3 3 3 3 3 3 4
078/300 Write block 259 len 24576 data: 2 2 2 2 2 2
079/300 Read block 4055 len 24576
080/300 Read block 2752 len 28672
081/300 Read block 448 len 16384
082/300 Read block 2151 len 16384
083/300 Read block 3785 len 16384
084/300 Write block 2097 len 12288 data: 2 2 3
085/300 Write block 1318 len 24576 data: 3 3 3 3 2 2
086/300 Read block 2724 len 40960
087/300 Read block 2884 len 36864
088/300 Write block 2491 len 36864 data: 2 2 3 3 2 2 2 2 2
089/300 Read block 4157 len 20480
090/300 Read block 3779 len 24576
091/300 Write block 4406 len 32768 data: 3 3 3 2 2 3 3 4
092/300 Write block 1463 len 20480 data: 2 3 3 3 3
093/300 Read block 3786 len 16384
094/300 Write block 1167 len 28672 data: 3 3 3 3 3 3 2
095/300 Write block 564 len 32768 data: 4 3 2 2 3 3 3 3
096/300 Read block 3717 len 24576
097/300 Flush
098/300 Write block 297 len 16384 data: 3 3 3 4
099/300 Write block 1636 len 28672 data: 2 2 2 2 2 3 3
100/300 Write block 3346 len 20480 data: 2 2 2 2 2
101/300 Read block 3775 len 40960
102/300 Read block 4277 len 4096
103/300 Read block 4449 len 40960
104/300 Flush
105/300 Write block 1128 len 28672 data: 3 4 4 4 4 3 3
106/300 Write block 3707 len 36864 data: 2 4 4 4 4 3 3 3 3
107/300 Write block 22 len 16384 data: 2 2 2 2
108/300 Flush
109/300 Write block 2844 len 20480 data: 4 4 4 2 3
110/300 Read block 1979 len 28672
111/300 Read block 3610 len 36864
112/300 Write block 4164 len 28672 data: 3 4 3 3 4 4 4
113/300 Write block 967 len 40960 data: 2 2 2 2 2 2 2 3 3 3
114/300 Write block 1221 len 4096 data: 4
115/300 Read block 1751 len 24576
116/300 Write block 781 len 16384 data: 2 2 2 2
117/300 Read block 3922 len 24576
118/300 Write block 1726 len 16384 data: 3 3 2 2
119/300 Write block 3096 len 36864 data: 2 2 2 2 2 2 2 2 2
120/300 Write block 2962 len 32768 data: 2 2 2 2 2 2 2 3
121/300 Write block 3091 len 12288 data: 3 3 3
122/300 Read block 1625 len 24576
123/300 Read block 2572 len 28672
124/300 Write block 4534 len 8192 data: 3 3
125/300 Write block 3761 len 32768 data: 4 4 3 3 3 3 3 3
126/300 Write block 715 len 8192 data: 6 5
127/300 Read block 4018 len 8192
128/300 Write block 4842 len 20480 data: 3 3 3 3 3
129/300 Read block 1531 len 28672
130/300 Write block 3479 len 16384 data: 2 2 2 2
131/300 Write block 2171 len 36864 data: 3 2 3 3 4 4 4 4 4
132/300 Write block 3873 len 20480 data: 3 3 3 3 3
133/300 Flush
134/300 Read block 4432 len 36864
135/300 Read block 2921 len 4096
136/300 Read block 2923 len 16384
137/300 Read block 2768 len 20480
138/300 Read block 4689 len 24576
139/300 Write block 1438 len 20480 data: 3 3 3 3 3
140/300 Read block 79 len 20480
141/300 Read block 1266 len 28672
142/300 Write block 1004 len 24576 data: 2 2 2 2 2 3
143/300 Write block 1654 len 40960 data: 2 2 2 2 2 2 2 2 2 2
144/300 Write block 1394 len 40960 data: 4 3 2 2 4 4 4 4 3 3
145/300 Write block 9 len 36864 data: 2 2 2 2 3 3 3 4 4
146/300 Read block 4750 len 28672
147/300 Read block 47 len 16384
148/300 Write block 2441 len 12288 data: 5 5 5
149/300 Write block 143 len 24576 data: 2 2 2 2 2 3
150/300 Write block 138 len 8192 data: 2 2
151/300 Write block 3721 len 28672 data: 3 3 3 3 4 4 5
152/300 Flush
153/300 Read block 1187 len 20480
154/300 Read block 3744 len 4096
155/300 Read block 293 len 40960
156/300 Read block 3424 len 8192
157/300 Write block 4671 len 24576 data: 3 3 3 2 2 2
158/300 Read block 1355 len 16384
159/300 Read block 4847 len 32768
160/300 Flush
161/300 Flush
162/300 Write block 1079 len 16384 data: 3 3 2 2
163/300 Read block 719 len 12288
164/300 Read block 4944 len 12288
165/300 Read block 4036 len 36864
166/300 Flush
167/300 Read block 1790 len 20480
168/300 Read block 4934 len 36864
169/300 Write block 1666 len 4096 data: 2
170/300 Write block 1136 len 8192 data: 2 2
171/300 Read block 233 len 20480
172/300 Read block 3521 len 16384
173/300 Write block 3239 len 36864 data: 3 3 3 4 4 2 2 2 2
174/300 Write block 40 len 4096 data: 3
175/300 Write block 3414 len 12288 data: 2 2 2
176/300 Read block 3048 len 32768
177/300 Write block 713 len 4096 data: 8
178/300 Flush
179/300 Read block 2389 len 20480
180/300 Read block 2422 len 28672
181/300 Read block 1679 len 4096
182/300 Write block 2857 len 40960 data: 3 3 3 3 3 3 3 3 3 3
183/300 Read block 574 len 8192
184/300 Write block 2383 len 40960 data: 2 2 3 3 3 3 3 4 5 3
185/300 Flush
186/300 Write block 4009 len 8192 data: 4 5
187/300 Read block 431 len 8192
188/300 Write block 3671 len 4096 data: 4
189/300 Read block 1278 len 20480
190/300 Read block 4418 len 32768
191/300 Write block 2141 len 16384 data: 2 2 2 3
192/300 Read block 1927 len 36864
193/300 Read block 1205 len 28672
194/300 Read block 1206 len 28672
195/300 Read block 4478 len 28672
196/300 Write block 3867 len 16384 data: 3 3 2 2
197/300 Write block 3172 len 8192 data: 2 2
198/300 Write block 460 len 24576 data: 2 2 2 2 2 2
199/300 Read block 2637 len 28672
200/300 Read block 409 len 4096
201/300 Write block 4962 len 4096 data: 5
202/300 Write block 3138 len 36864 data: 5 5 6 6 5 5 3 3 3
203/300 Write block 371 len 28672 data: 3 4 4 4 3 3 3
204/300 Read block 798 len 4096
205/300 Read block 2175 len 8192
206/300 Read block 4632 len 36864
207/300 Write block 1292 len 40960 data: 2 2 2 2 2 3 3 3 3 4
208/300 Write block 478 len 4096 data: 3
209/300 Read block 3085 len 8192
210/300 Read block 4592 len 28672
211/300 Write block 2651 len 36864 data: 2 2 2 2 2 2 2 3 3
212/300 Read block 4083 len 32768
213/300 Read block 1152 len 8192
214/300 Write block 4869 len 4096 data: 3
215/300 Write block 3329 len 8192 data: 3 3
216/300 Read block 4438 len 36864
217/300 Flush
218/300 Write block 2849 len 40960 data: 3 3 4 4 4 3 3 3 4 4
219/300 Write block 4784 len 16384 data: 2 3 3 3
220/300 Read block 3990 len 8192
221/300 Write block 176 len 40960 data: 3 3 2 2 2 2 2 2 3 3
222/300 Read block 3779 len 36864
223/300 Read block 169 len 20480
224/300 Read block 368 len 12288
225/300 Write block 4424 len 24576 data: 2 2 2 3 3 3
226/300 Read block 4798 len 8192
227/300 Write block 846 len 24576 data: 2 2 2 2 2 2
228/300 Read block 117 len 28672
229/300 Write block 4959 len 24576 data: 4 5 5 6 4 4
230/300 Write block 463 len 36864 data: 3 3 3 2 2 2 2 2 2
231/300 Read block 3241 len 20480
232/300 Flush
233/300 Flush
234/300 Read block 3140 len 16384
235/300 Read block 2830 len 20480
236/300 Write block 4435 len 4096 data: 3
237/300 Write block 956 len 8192 data: 5 5
238/300 Write block 199 len 24576 data: 4 4 3 2 2 2
239/300 Read block 4451 len 20480
240/300 Write block 205 len 20480 data: 3 4 4 4 4
241/300 Write block 601 len 4096 data: 3
242/300 Read block 3471 len 32768
243/300 Read block 3126 len 24576
244/300 Write block 3878 len 40960 data: 3 2 3 3 3 3 3 3 3 3
245/300 Read block 409 len 36864
246/300 Read block 938 len 4096
247/300 Read block 1984 len 16384
248/300 Read block 197 len 36864
249/300 Read block 931 len 4096
250/300 Write block 1138 len 24576 data: 2 2 2 2 2 2
251/300 Write block 2649 len 40960 data: 2 2 3 3 3 3 3 3 3 4
252/300 Write block 4613 len 32768 data: 2 2 3 3 3 3 3 2
253/300 Read block 632 len 12288
254/300 Flush
255/300 Write block 3772 len 24576 data: 3 5 5 5 5 5
256/300 Write block 1358 len 40960 data: 4 4 4 5 4 4 4 4 4 4
257/300 Write block 279 len 12288 data: 6 8 8
258/300 Write block 294 len 28672 data: 3 3 3 4 4 4 5
259/300 Flush
260/300 Write block 4655 len 12288 data: 3 4 4
261/300 Read block 4144 len 20480
262/300 Read block 1945 len 32768
263/300 Read block 2930 len 32768
264/300 Read block 1974 len 16384
265/300 Write block 537 len 12288 data: 3 3 2
266/300 Flush
267/300 Write block 4335 len 20480 data: 4 5 5 5 4
268/300 Read block 899 len 20480
269/300 Flush
270/300 Read block 4042 len 4096
271/300 Write block 475 len 32768 data: 3 3 3 4 3 3 3 3
272/300 Write block 138 len 16384 data: 3 3 2 2
273/300 Read block 21 len 20480
274/300 Write block 3105 len 8192 data: 2 2
275/300 Write block 1879 len 32768 data: 3 2 2 2 2 2 2 3
276/300 Read block 3325 len 20480
277/300 Read block 774 len 12288
278/300 Flush
279/300 Read block 4874 len 24576
280/300 Read block 2431 len 12288
281/300 Read block 1105 len 40960
282/300 Read block 1752 len 20480
283/300 Read block 1659 len 8192
284/300 Write block 3354 len 36864 data: 2 2 2 2 2 3 3 3 3
285/300 Write block 1662 len 12288 data: 3 3 2
286/300 Write block 3246 len 4096 data: 3
287/300 Read block 1050 len 20480
288/300 Read block 3493 len 32768
289/300 Read block 1652 len 12288
290/300 Write block 1405 len 12288 data: 3 3 2
291/300 Write block 1811 len 8192 data: 6 5
292/300 Write block 924 len 32768 data: 3 3 3 2 3 3 4 4
293/300 Write block 832 len 20480 data: 4 3 3 3 2
294/300 Read block 810 len 32768
295/300 Write block 1059 len 8192 data: 3 3
296/300 Write block 2608 len 4096 data: 3
297/300 Read block 1446 len 32768
298/300 Read block 2396 len 20480
299/300 Write block 1353 len 36864 data: 3 3 3 3 3 5 5 5 6
300/300 Read block 903 len 40960
301/300 Read block 4299 len 36864
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:38:02.881 INFO request completed, latency_us: 187, response_code: 204, uri: /start/cid/2, method: POST, req_id: 71bf678c-629e-4503-85e5-7859c18af32d, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3112 Acked Flush 0 Done Done Sent false
3113 Acked Read 4 Done Done Sent false
3114 Acked Write 3 Done Done Sent false
3115 Acked Write 2 Done Done Sent false
3116 Acked Read 1 Done Done Sent false
3117 Acked Write 6 Done Done Sent false
3118 Acked Read 8 Done Done Sent false
3119 Acked Read 5 Done Done Sent false
3120 Acked Write 8 Done Done Sent false
3121 Acked Write 4 Done Done Sent false
3122 Acked Flush 0 Done Done Sent false
3123 Acked Flush 0 Done Done Sent false
3124 Acked Flush 0 Done Done Sent false
3125 Acked Read 6 Done Done Sent false
3126 Acked Write 7 Done Done Sent false
3127 Acked Read 1 Done Done Sent false
3128 Acked Write 4 Done Done Sent false
3129 Acked Read 3 Done Done Sent false
3130 Acked Read 3 Done Done Sent false
3131 Acked Read 10 Done Done Sent false
3132 Acked Read 1 Done Done Sent false
3133 Acked Read 2 Done Done Sent false
3134 Acked Flush 0 Done Done Sent false
3135 Acked Flush 0 Done Done Sent false
3136 Acked Read 9 Done Done Sent false
3137 Acked Read 2 Done Done Sent false
3138 Acked Read 3 Done Done Sent false
3139 Acked Write 3 Done Done Sent false
3140 Acked Write 9 Done Done Sent false
3141 Acked Write 1 Done Done Sent false
3142 Acked Read 10 Done Done Sent false
3143 Acked Write 7 Done Done Sent false
3144 Acked Read 2 Done Done Sent false
3145 Acked Write 8 Done Done Sent false
3146 Acked Write 1 Done Done Sent false
3147 Acked Write 5 Done Done Sent false
3148 Acked Write 1 Done Done Sent false
3149 Acked Read 9 Done Done Sent false
3150 Acked Read 1 Done Done Sent false
3151 Acked Read 9 Done Done Sent false
3152 Acked Write 7 Done Done Sent false
3153 Acked Read 1 Done Done Sent false
3154 Acked Write 5 Done Done Sent false
3155 Acked Read 9 Done Done Sent false
3156 Acked Write 6 Done Done Sent false
3157 Acked Read 1 Done Done Sent false
3158 Acked Read 2 Done Done Sent false
3159 Acked Read 2 Done Done Sent false
3160 Acked Read 9 Done Done Sent false
3161 Acked Write 3 Done Done Sent false
3162 Acked Write 8 Done Done Sent false
3163 Acked Write 6 Done Done Sent false
3164 Acked Read 2 Done Done Sent false
3165 Acked Read 6 Done Done Sent false
3166 Acked Read 3 Done Done Sent false
3167 Acked Write 3 Done Done Sent false
3168 Acked Read 3 Done Done Sent false
3169 Acked Write 2 Done Done Sent false
3170 Acked Write 8 Done Done Sent false
3171 Acked Read 8 Done Done Sent false
3172 Acked Write 7 Done Done Sent false
3173 Acked Read 3 Done Done Sent false
3174 Acked Write 5 Done Done Sent false
3175 Acked Read 2 Done Done Sent false
3176 Acked Write 9 Done Done Sent false
3177 Acked Write 7 Done Done Sent false
3178 Acked Flush 0 Done Done Sent false
3179 Acked Write 4 Done Done Sent false
3180 Acked Flush 0 Done Done Sent false
3181 Acked Read 6 Done Done Sent false
3182 Acked Read 7 Done Done Sent false
3183 Acked Write 2 Done Done Sent false
3184 Acked Read 4 Done Done Sent false
3185 Acked Read 1 Done Done Sent false
3186 Acked Read 2 Done Done Sent false
3187 Acked Read 9 Done Done Sent false
3188 Acked Write 10 Done Done Sent false
3189 Acked Write 6 Done Done Sent false
3190 Acked Read 6 Done Done Sent false
3191 Acked Read 7 Done Done Sent false
3192 Acked Read 4 Done Done Sent false
3193 Acked Read 4 Done Done Sent false
3194 Acked Read 4 Done Done Sent false
3195 Acked Write 3 Done Done Sent false
3196 Acked Write 6 Done Done Sent false
3197 Acked Read 10 Done Done Sent false
3198 Acked Read 9 Done Done Sent false
3199 Acked Write 9 Done Done Sent false
3200 Acked Read 5 Done Done Sent false
3201 Acked Read 6 Done Done Sent false
3202 Acked Write 8 Done Done Sent false
3203 Acked Write 5 Done Done Sent false
3204 Acked Read 4 Done Done Sent false
3205 Acked Write 7 Done Done Sent false
3206 Acked Write 8 Done Done Sent false
3207 Acked Read 6 Done Done Sent false
3208 Acked Flush 0 Done Done Sent false
3209 Acked Write 4 Done Done Sent false
3210 Acked Write 7 Done Done Sent false
3211 Acked Write 5 Done Done Sent false
3212 Acked Read 10 Done Done Sent false
3213 Acked Read 1 Done Done Sent false
3214 Acked Read 10 Done Done Sent false
3215 Acked Flush 0 Done Done Sent false
3216 Acked Write 7 Done Done Sent false
3217 Acked Write 9 Done Done Sent false
3218 Acked Write 4 Done Done Sent false
3219 Acked Flush 0 Done Done Sent false
3220 Acked Write 5 Done Done Sent false
3221 Acked Read 7 Done Done Sent false
3222 Acked Read 9 Done Done Sent false
3223 Acked Write 7 Done Done Sent false
3224 Acked Write 10 Done Done Sent false
3225 Acked Write 1 Done Done Sent false
3226 Acked Read 6 Done Done Sent false
3227 Acked Write 4 Done Done Sent false
3228 Acked Read 6 Done Done Sent false
3229 Acked Write 4 Done Done Sent false
3230 Acked Write 9 Done Done Sent false
3231 Acked Write 8 Done Done Sent false
3232 Acked Write 3 Done Done Sent false
3233 Acked Read 6 Done Done Sent false
3234 Acked Read 7 Done Done Sent false
3235 Acked Write 2 Done Done Sent false
3236 Acked Write 8 Done Done Sent false
3237 Acked Write 2 Done Done Sent false
3238 Acked Read 2 Done Done Sent false
3239 Acked Write 5 Done Done Sent false
3240 Acked Read 7 Done Done Sent false
3241 Acked Write 4 Done Done Sent false
3242 Acked Write 9 Done Done Sent false
3243 Acked Write 5 Done Done Sent false
3244 Acked Flush 0 Done Done Sent false
3245 Acked Read 9 Done Done Sent false
3246 Acked Read 1 Done Done Sent false
3247 Acked Read 4 Done Done Sent false
3248 Acked Read 5 Done Done Sent false
3249 Acked Read 6 Done Done Sent false
3250 Acked Write 5 Done Done Sent false
3251 Acked Read 5 Done Done Sent false
3252 Acked Read 7 Done Done Sent false
3253 Acked Write 6 Done Done Sent false
3254 Acked Write 10 Done Done Sent false
3255 Acked Write 10 Done Done Sent false
3256 Acked Write 9 Done Done Sent false
3257 Acked Read 7 Done Done Sent false
3258 Acked Read 4 Done Done Sent false
3259 Acked Write 3 Done Done Sent false
3260 Acked Write 6 Done Done Sent false
3261 Acked Write 2 Done Done Sent false
3262 Acked Write 7 Done Done Sent false
3263 Acked Flush 0 Done Done Sent false
3264 Acked Read 5 Done Done Sent false
3265 Acked Read 1 Done Done Sent false
3266 Acked Read 10 Done Done Sent false
3267 Acked Read 2 Done Done Sent false
3268 Acked Write 6 Done Done Sent false
3269 Acked Read 4 Done Done Sent false
3270 Acked Read 8 Done Done Sent false
3271 Acked Flush 0 Done Done Sent false
3272 Acked Flush 0 Done Done Sent false
3273 Acked Write 4 Done Done Sent false
3274 Acked Read 3 Done Done Sent false
3275 Acked Read 3 Done Done Sent false
3276 Acked Read 9 Done Done Sent false
3277 Acked Flush 0 Done Done Sent false
3278 Acked Read 5 Done Done Sent false
3279 Acked Read 9 Done Done Sent false
3280 Acked Write 1 Done Done Sent false
3281 Acked Write 2 Done Done Sent false
3282 Acked Read 5 Done Done Sent false
3283 Acked Read 4 Done Done Sent false
3284 Acked Write 9 Done Done Sent false
3285 Acked Write 1 Done Done Sent false
3286 Acked Write 3 Done Done Sent false
3287 Acked Read 8 Done Done Sent false
3288 Acked Write 1 Done Done Sent false
3289 Acked Flush 0 Done Done Sent false
3290 Acked Read 5 Done Done Sent false
3291 Acked Read 7 Done Done Sent false
3292 Acked Read 1 Done Done Sent false
3293 Acked Write 10 Done Done Sent false
3294 Acked Read 2 Done Done Sent false
3295 Acked Write 10 Done Done Sent false
3296 Acked Flush 0 Done Done Sent false
3297 Acked Write 2 Done Done Sent false
3298 Acked Read 2 Done Done Sent false
3299 Acked Write 1 Done Done Sent false
3300 Acked Read 5 Done Done Sent false
3301 Acked Read 8 Done Done Sent false
3302 Acked Write 4 Done Done Sent false
3303 Acked Read 9 Done Done Sent false
3304 Acked Read 7 Done Done Sent false
3305 Acked Read 7 Done Done Sent false
3306 Acked Read 7 Done Done Sent false
3307 Acked Write 4 Done Done Sent false
3308 Acked Write 2 Done Done Sent false
3309 Acked Write 6 Done Done Sent false
3310 Acked Read 7 Done Done Sent false
3311 Acked Read 1 Done Done Sent false
3312 Acked Write 1 Done Done Sent false
3313 Acked Write 9 Done Done Sent false
3314 Acked Write 7 Done Done Sent false
3315 Acked Read 1 Done Done Sent false
3316 Acked Read 2 Done Done Sent false
3317 Acked Read 9 Done Done Sent false
3318 Acked Write 10 Done Done Sent false
3319 Acked Write 1 Done Done Sent false
3320 Acked Read 2 Done Done Sent false
3321 Acked Read 7 Done Done Sent false
3322 Acked Write 9 Done Done Sent false
3323 Acked Read 8 Done Done Sent false
3324 Acked Read 2 Done Done Sent false
3325 Acked Write 1 Done Done Sent false
3326 Acked Write 2 Done Done Sent false
3327 Acked Read 9 Done Done Sent false
3328 Acked Flush 0 Done Done Sent false
3329 Acked Write 10 Done Done Sent false
3330 Acked Write 4 Done Done Sent false
3331 Acked Read 2 Done Done Sent false
3332 Acked Write 10 Done Done Sent false
3333 Acked Read 9 Done Done Sent false
3334 Acked Read 5 Done Done Sent false
3335 Acked Read 3 Done Done Sent false
3336 Acked Write 6 Done Done Sent false
3337 Acked Read 2 Done Done Sent false
3338 Acked Write 6 Done Done Sent false
3339 Acked Read 7 Done Done Sent false
3340 Acked Write 6 Done Done Sent false
3341 Acked Write 9 Done Done Sent false
3342 Acked Read 5 Done Done Sent false
3343 Acked Flush 0 Done Done Sent false
3344 Acked Flush 0 Done Done Sent false
3345 Acked Read 4 Done Done Sent false
3346 Acked Read 5 Done Done Sent false
3347 Acked Write 1 Done Done Sent false
3348 Acked Write 2 Done Done Sent false
3349 Acked Write 6 Done Done Sent false
3350 Acked Read 5 Done Done Sent false
3351 Acked Write 5 Done Done Sent false
3352 Acked Write 1 Done Done Sent false
3353 Acked Read 8 Done Done Sent false
3354 Acked Read 6 Done Done Sent false
3355 Acked Write 10 Done Done Sent false
3356 Acked Read 9 Done Done Sent false
3357 Acked Read 1 Done Done Sent false
3358 Acked Read 4 Done Done Sent false
3359 Acked Read 9 Done Done Sent false
3360 Acked Read 1 Done Done Sent false
3361 Acked Write 6 Done Done Sent false
3362 Acked Write 10 Done Done Sent false
3363 Acked Write 8 Done Done Sent false
3364 Acked Read 3 Done Done Sent false
3365 Acked Flush 0 Done Done Sent false
3366 Acked Write 6 Done Done Sent false
3367 Acked Write 10 Done Done Sent false
3368 Acked Write 3 Done Done Sent false
3369 Acked Write 7 Done Done Sent false
3370 Acked Flush 0 Done Done Sent false
3371 Acked Write 3 Done Done Sent false
3372 Acked Read 5 Done Done Sent false
3373 Acked Read 8 Done Done Sent false
3374 Acked Read 8 Done Done Sent false
3375 Acked Read 4 Done Done Sent false
3376 Acked Write 3 Done Done Sent false
3377 Acked Flush 0 Done Done Sent false
3378 Acked Write 5 Done Done Sent false
3379 Acked Read 5 Done Done Sent false
3380 Acked Flush 0 Done Done Sent false
3381 Acked Read 1 Done Done Sent false
3382 Acked Write 8 Done Done Sent false
3383 Acked Write 4 Done Done Sent false
3384 Acked Read 5 Done Done Sent false
3385 Acked Write 2 Done Done Sent false
3386 Acked Write 8 Done Done Sent false
3387 Acked Read 5 Done Done Sent false
3388 Acked Read 3 Done Done Sent false
3389 Acked Flush 0 Done Done Sent false
3390 Acked Read 6 Done Done Sent false
3391 Acked Read 3 Done Done Sent false
3392 Acked Read 10 Done Done Sent false
3393 Acked Read 5 Done Done Sent false
3394 Acked Read 2 Done Done Sent false
3395 Acked Write 9 Done Done Sent false
3396 Acked Write 3 Done Done Sent false
3397 Acked Write 1 Done Done Sent false
3398 Acked Read 5 Done Done Sent false
3399 Acked Read 8 Done Done Sent false
3400 Acked Read 3 Done Done Sent false
3401 Acked Write 3 Done Done Sent false
3402 Acked Write 2 Done Done Sent false
3403 Acked Write 8 Done Done Sent false
3404 Acked Write 5 Done Done Sent false
3405 Acked Read 8 Done Done Sent false
3406 Acked Write 2 Done Done Sent false
3407 Acked Write 1 Done Done Sent false
3408 Acked Read 8 Done Done Sent false
3409 Acked Read 5 Done Done Sent false
3410 Acked Write 9 Done Done Sent false
3411 Acked Read 10 Done Done Sent false
3412 Acked Read 9 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3389 3389 3111
Downstairs last five completed: 3111 3110 3109 3108 3107
Upstairs last five completed: 3412 3411 3410 3409 3408
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1208)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3112 Acked Flush 0 Done Done Sent false
3113 Acked Read 4 Done Done Sent false
3114 Acked Write 3 Done Done Sent false
3115 Acked Write 2 Done Done Sent false
3116 Acked Read 1 Done Done Sent false
3117 Acked Write 6 Done Done Sent false
3118 Acked Read 8 Done Done Sent false
3119 Acked Read 5 Done Done Sent false
3120 Acked Write 8 Done Done Sent false
3121 Acked Write 4 Done Done Sent false
3122 Acked Flush 0 Done Done Sent false
3123 Acked Flush 0 Done Done Sent false
3124 Acked Flush 0 Done Done Sent false
3125 Acked Read 6 Done Done Sent false
3126 Acked Write 7 Done Done Sent false
3127 Acked Read 1 Done Done Sent false
3128 Acked Write 4 Done Done Sent false
3129 Acked Read 3 Done Done Sent false
3130 Acked Read 3 Done Done Sent false
3131 Acked Read 10 Done Done Sent false
3132 Acked Read 1 Done Done Sent false
3133 Acked Read 2 Done Done Sent false
3134 Acked Flush 0 Done Done Sent false
3135 Acked Flush 0 Done Done Sent false
3136 Acked Read 9 Done Done Sent false
3137 Acked Read 2 Done Done Sent false
3138 Acked Read 3 Done Done Sent false
3139 Acked Write 3 Done Done Sent false
3140 Acked Write 9 Done Done Sent false
3141 Acked Write 1 Done Done Sent false
3142 Acked Read 10 Done Done Sent false
3143 Acked Write 7 Done Done Sent false
3144 Acked Read 2 Done Done Sent false
3145 Acked Write 8 Done Done Sent false
3146 Acked Write 1 Done Done Sent false
3147 Acked Write 5 Done Done Sent false
3148 Acked Write 1 Done Done Sent false
3149 Acked Read 9 Done Done Sent false
3150 Acked Read 1 Done Done Sent false
3151 Acked Read 9 Done Done Sent false
3152 Acked Write 7 Done Done Sent false
3153 Acked Read 1 Done Done Sent false
3154 Acked Write 5 Done Done Sent false
3155 Acked Read 9 Done Done Sent false
3156 Acked Write 6 Done Done Sent false
3157 Acked Read 1 Done Done Sent false
3158 Acked Read 2 Done Done Sent false
3159 Acked Read 2 Done Done Sent false
3160 Acked Read 9 Done Done Sent false
3161 Acked Write 3 Done Done Sent false
3162 Acked Write 8 Done Done Sent false
3163 Acked Write 6 Done Done Sent false
3164 Acked Read 2 Done Done Sent false
3165 Acked Read 6 Done Done Sent false
3166 Acked Read 3 Done Done Sent false
3167 Acked Write 3 Done Done Sent false
3168 Acked Read 3 Done Done Sent false
3169 Acked Write 2 Done Done Sent false
3170 Acked Write 8 Done Done Sent false
3171 Acked Read 8 Done Done Sent false
3172 Acked Write 7 Done Done Sent false
3173 Acked Read 3 Done Done Sent false
3174 Acked Write 5 Done Done Sent false
3175 Acked Read 2 Done Done Sent false
3176 Acked Write 9 Done Done Sent false
3177 Acked Write 7 Done Done Sent false
3178 Acked Flush 0 Done Done Sent false
3179 Acked Write 4 Done Done Sent false
3180 Acked Flush 0 Done Done Sent false
3181 Acked Read 6 Done Done Sent false
3182 Acked Read 7 Done Done Sent false
3183 Acked Write 2 Done Done Sent false
3184 Acked Read 4 Done Done Sent false
3185 Acked Read 1 Done Done Sent false
3186 Acked Read 2 Done Done Sent false
3187 Acked Read 9 Done Done Sent false
3188 Acked Write 10 Done Done Sent false
3189 Acked Write 6 Done Done Sent false
3190 Acked Read 6 Done Done Sent false
3191 Acked Read 7 Done Done Sent false
3192 Acked Read 4 Done Done Sent false
3193 Acked Read 4 Done Done Sent false
3194 Acked Read 4 Done Done Sent false
3195 Acked Write 3 Done Done Sent false
3196 Acked Write 6 Done Done Sent false
3197 Acked Read 10 Done Done Sent false
3198 Acked Read 9 Done Done Sent false
3199 Acked Write 9 Done Done Sent false
3200 Acked Read 5 Done Done Sent false
3201 Acked Read 6 Done Done Sent false
3202 Acked Write 8 Done Done Sent false
3203 Acked Write 5 Done Done Sent false
3204 Acked Read 4 Done Done Sent false
3205 Acked Write 7 Done Done Sent false
3206 Acked Write 8 Done Done Sent false
3207 Acked Read 6 Done Done Sent false
3208 Acked Flush 0 Done Done Sent false
3209 Acked Write 4 Done Done Sent false
3210 Acked Write 7 Done Done Sent false
3211 Acked Write 5 Done Done Sent false
3212 Acked Read 10 Done Done Sent false
3213 Acked Read 1 Done Done Sent false
3214 Acked Read 10 Done Done Sent false
3215 Acked Flush 0 Done Done Sent false
3216 Acked Write 7 Done Done Sent false
3217 Acked Write 9 Done Done Sent false
3218 Acked Write 4 Done Done Sent false
3219 Acked Flush 0 Done Done Sent false
3220 Acked Write 5 Done Done Sent false
3221 Acked Read 7 Done Done Sent false
3222 Acked Read 9 Done Done Sent false
3223 Acked Write 7 Done Done Sent false
3224 Acked Write 10 Done Done Sent false
3225 Acked Write 1 Done Done Sent false
3226 Acked Read 6 Done Done Sent false
3227 Acked Write 4 Done Done Sent false
3228 Acked Read 6 Done Done Sent false
3229 Acked Write 4 Done Done Sent false
3230 Acked Write 9 Done Done Sent false
3231 Acked Write 8 Done Done Sent false
3232 Acked Write 3 Done Done Sent false
3233 Acked Read 6 Done Done Sent false
3234 Acked Read 7 Done Done Sent false
3235 Acked Write 2 Done Done Sent false
3236 Acked Write 8 Done Done Sent false
3237 Acked Write 2 Done Done Sent false
3238 Acked Read 2 Done Done Sent false
3239 Acked Write 5 Done Done Sent false
3240 Acked Read 7 Done Done Sent false
3241 Acked Write 4 Done Done Sent false
3242 Acked Write 9 Done Done Sent false
3243 Acked Write 5 Done Done Sent false
3244 Acked Flush 0 Done Done Sent false
3245 Acked Read 9 Done Done Sent false
3246 Acked Read 1 Done Done Sent false
3247 Acked Read 4 Done Done Sent false
3248 Acked Read 5 Done Done Sent false
3249 Acked Read 6 Done Done Sent false
3250 Acked Write 5 Done Done Sent false
3251 Acked Read 5 Done Done Sent false
3252 Acked Read 7 Done Done Sent false
3253 Acked Write 6 Done Done Sent false
3254 Acked Write 10 Done Done Sent false
3255 Acked Write 10 Done Done Sent false
3256 Acked Write 9 Done Done Sent false
3257 Acked Read 7 Done Done Sent false
3258 Acked Read 4 Done Done Sent false
3259 Acked Write 3 Done Done Sent false
3260 Acked Write 6 Done Done Sent false
3261 Acked Write 2 Done Done Sent false
3262 Acked Write 7 Done Done Sent false
3263 Acked Flush 0 Done Done Sent false
3264 Acked Read 5 Done Done Sent false
3265 Acked Read 1 Done Done Sent false
3266 Acked Read 10 Done Done Sent false
3267 Acked Read 2 Done Done Sent false
3268 Acked Write 6 Done Done Sent false
3269 Acked Read 4 Done Done Sent false
3270 Acked Read 8 Done Done Sent false
3271 Acked Flush 0 Done Done Sent false
3272 Acked Flush 0 Done Done Sent false
3273 Acked Write 4 Done Done Sent false
3274 Acked Read 3 Done Done Sent false
3275 Acked Read 3 Done Done Sent false
3276 Acked Read 9 Done Done Sent false
3277 Acked Flush 0 Done Done Sent false
3278 Acked Read 5 Done Done Sent false
3279 Acked Read 9 Done Done Sent false
3280 Acked Write 1 Done Done Sent false
3281 Acked Write 2 Done Done Sent false
3282 Acked Read 5 Done Done Sent false
3283 Acked Read 4 Done Done Sent false
3284 Acked Write 9 Done Done Sent false
3285 Acked Write 1 Done Done Sent false
3286 Acked Write 3 Done Done Sent false
3287 Acked Read 8 Done Done Sent false
3288 Acked Write 1 Done Done Sent false
3289 Acked Flush 0 Done Done Sent false
3290 Acked Read 5 Done Done Sent false
3291 Acked Read 7 Done Done Sent false
3292 Acked Read 1 Done Done Sent false
3293 Acked Write 10 Done Done Sent false
3294 Acked Read 2 Done Done Sent false
3295 Acked Write 10 Done Done Sent false
3296 Acked Flush 0 Done Done Sent false
3297 Acked Write 2 Done Done Sent false
3298 Acked Read 2 Done Done Sent false
3299 Acked Write 1 Done Done Sent false
3300 Acked Read 5 Done Done Sent false
3301 Acked Read 8 Done Done Sent false
3302 Acked Write 4 Done Done Sent false
3303 Acked Read 9 Done Done Sent false
3304 Acked Read 7 Done Done Sent false
3305 Acked Read 7 Done Done Sent false
3306 Acked Read 7 Done Done Sent false
3307 Acked Write 4 Done Done Sent false
3308 Acked Write 2 Done Done Sent false
3309 Acked Write 6 Done Done Sent false
3310 Acked Read 7 Done Done Sent false
3311 Acked Read 1 Done Done Sent false
3312 Acked Write 1 Done Done Sent false
3313 Acked Write 9 Done Done Sent false
3314 Acked Write 7 Done Done Sent false
3315 Acked Read 1 Done Done Sent false
3316 Acked Read 2 Done Done Sent false
3317 Acked Read 9 Done Done Sent false
3318 Acked Write 10 Done Done Sent false
3319 Acked Write 1 Done Done Sent false
3320 Acked Read 2 Done Done Sent false
3321 Acked Read 7 Done Done Sent false
3322 Acked Write 9 Done Done Sent false
3323 Acked Read 8 Done Done Sent false
3324 Acked Read 2 Done Done Sent false
3325 Acked Write 1 Done Done Sent false
3326 Acked Write 2 Done Done Sent false
3327 Acked Read 9 Done Done Sent false
3328 Acked Flush 0 Done Done Sent false
3329 Acked Write 10 Done Done Sent false
3330 Acked Write 4 Done Done Sent false
3331 Acked Read 2 Done Done Sent false
3332 Acked Write 10 Done Done Sent false
3333 Acked Read 9 Done Done Sent false
3334 Acked Read 5 Done Done Sent false
3335 Acked Read 3 Done Done Sent false
3336 Acked Write 6 Done Done Sent false
3337 Acked Read 2 Done Done Sent false
3338 Acked Write 6 Done Done Sent false
3339 Acked Read 7 Done Done Sent false
3340 Acked Write 6 Done Done Sent false
3341 Acked Write 9 Done Done Sent false
3342 Acked Read 5 Done Done Sent false
3343 Acked Flush 0 Done Done Sent false
3344 Acked Flush 0 Done Done Sent false
3345 Acked Read 4 Done Done Sent false
3346 Acked Read 5 Done Done Sent false
3347 Acked Write 1 Done Done Sent false
3348 Acked Write 2 Done Done Sent false
3349 Acked Write 6 Done Done Sent false
3350 Acked Read 5 Done Done Sent false
3351 Acked Write 5 Done Done Sent false
3352 Acked Write 1 Done Done Sent false
3353 Acked Read 8 Done Done Sent false
3354 Acked Read 6 Done Done Sent false
3355 Acked Write 10 Done Done Sent false
3356 Acked Read 9 Done Done Sent false
3357 Acked Read 1 Done Done Sent false
3358 Acked Read 4 Done Done Sent false
3359 Acked Read 9 Done Done Sent false
3360 Acked Read 1 Done Done Sent false
3361 Acked Write 6 Done Done Sent false
3362 Acked Write 10 Done Done Sent false
3363 Acked Write 8 Done Done Sent false
3364 Acked Read 3 Done Done Sent false
3365 Acked Flush 0 Done Done Sent false
3366 Acked Write 6 Done Done Sent false
3367 Acked Write 10 Done Done Sent false
3368 Acked Write 3 Done Done Sent false
3369 Acked Write 7 Done Done Sent false
3370 Acked Flush 0 Done Done Sent false
3371 Acked Write 3 Done Done Sent false
3372 Acked Read 5 Done Done Sent false
3373 Acked Read 8 Done Done Sent false
3374 Acked Read 8 Done Done Sent false
3375 Acked Read 4 Done Done Sent false
3376 Acked Write 3 Done Done Sent false
3377 Acked Flush 0 Done Done Sent false
3378 Acked Write 5 Done Done Sent false
3379 Acked Read 5 Done Done Sent false
3380 Acked Flush 0 Done Done Sent false
3381 Acked Read 1 Done Done Sent false
3382 Acked Write 8 Done Done Sent false
3383 Acked Write 4 Done Done Sent false
3384 Acked Read 5 Done Done Sent false
3385 Acked Write 2 Done Done Sent false
3386 Acked Write 8 Done Done Sent false
3387 Acked Read 5 Done Done Sent false
3388 Acked Read 3 Done Done Sent false
3389 Acked Flush 0 Done Done Sent false
3390 Acked Read 6 Done Done Sent false
3391 Acked Read 3 Done Done Sent false
3392 Acked Read 10 Done Done Sent false
3393 Acked Read 5 Done Done Sent false
3394 Acked Read 2 Done Done Sent false
3395 Acked Write 9 Done Done Sent false
3396 Acked Write 3 Done Done Sent false
3397 Acked Write 1 Done Done Sent false
3398 Acked Read 5 Done Done Sent false
3399 Acked Read 8 Done Done Sent false
3400 Acked Read 3 Done Done Sent false
3401 Acked Write 3 Done Done Sent false
3402 Acked Write 2 Done Done Sent false
3403 Acked Write 8 Done Done Sent false
3404 Acked Write 5 Done Done Sent false
3405 Acked Read 8 Done Done Sent false
3406 Acked Write 2 Done Done Sent false
3407 Acked Write 1 Done Done Sent false
3408 Acked Read 8 Done Done Sent false
3409 Acked Read 5 Done Done Sent false
3410 Acked Write 9 Done Done Sent false
3411 Acked Read 10 Done Done Sent false
3412 Acked Read 9 Done Done Sent false
3413 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3413 3413 3111
Downstairs last five completed: 3111 3110 3109 3108 3107
Upstairs last five completed: 3413 3412 3411 3410 3409
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.652302686Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42755)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.652649066Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.653439744Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654054642Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654082969Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 3111","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654100687Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 3111","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654349068Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 3111","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654371915Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 3111","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:08.654648273Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3413 3412 3411 3410 3409
Upstairs last five completed: 3413 3412 3411 3410 3409
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
[8820] Got stop action so:false kr:false
Jan 27 19:38:10.887 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 156584cb-fbb9-44e3-aae0-b03361d9276c, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:10.887695366Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:10.887757129Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:10.887780326Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:38:10.887 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:10.887898722Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, method: GET, req_id: 2ff668b2-bf55-4138-9c13-47dba7ddf290, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:38:14.890 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/1, method: GET, req_id: 52509c1e-e90e-4c93-928d-4044d689f964, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 4321 len 24576 data: 2 2 2 2 3 3
002/300 Read block 442 len 20480
003/300 Write block 4294 len 8192 data: 2 2
004/300 Read block 3971 len 4096
005/300 Write block 2972 len 12288 data: 3 3 3
006/300 Read block 490 len 12288
007/300 Read block 3679 len 12288
008/300 Write block 1122 len 32768 data: 2 2 2 2 3 3 4 5
009/300 Read block 542 len 4096
010/300 Read block 2470 len 16384
011/300 Read block 378 len 16384
012/300 Write block 2156 len 40960 data: 4 4 4 4 4 5 5 5 5 5
013/300 Read block 781 len 20480
014/300 Read block 3459 len 12288
015/300 Read block 1915 len 12288
016/300 Write block 544 len 12288 data: 3 4 4
017/300 Read block 2709 len 28672
018/300 Write block 3215 len 40960 data: 3 3 3 4 4 4 4 4 4 4
019/300 Write block 3685 len 36864 data: 2 2 3 4 4 4 4 4 4
020/300 Flush
021/300 Flush
022/300 Write block 3765 len 28672 data: 4 4 4 4 3 3 3
023/300 Flush
024/300 Write block 3272 len 32768 data: 4 4 4 3 3 4 4 3
025/300 Read block 1385 len 8192
026/300 Write block 3539 len 16384 data: 2 2 2 2
027/300 Write block 3301 len 12288 data: 2 2 2
028/300 Write block 3732 len 20480 data: 2 2 2 4 4
029/300 Read block 2199 len 12288
030/300 Read block 4885 len 16384
031/300 Read block 1165 len 4096
032/300 Write block 4898 len 28672 data: 4 4 4 3 4 2 2
033/300 Write block 2875 len 16384 data: 2 2 2 2
034/300 Read block 2348 len 8192
035/300 Write block 3190 len 28672 data: 3 3 3 3 3 3 3
036/300 Read block 386 len 32768
037/300 Read block 3301 len 32768
038/300 Write block 689 len 24576 data: 3 2 2 2 2 3
039/300 Read block 4671 len 4096
040/300 Read block 4832 len 24576
041/300 Write block 3917 len 8192 data: 2 3
042/300 Read block 4128 len 8192
043/300 Read block 2823 len 32768
044/300 Read block 4104 len 20480
045/300 Read block 1606 len 24576
046/300 Write block 4236 len 28672 data: 3 3 4 3 3 3 4
047/300 Write block 1753 len 24576 data: 2 4 3 3 3 3
048/300 Write block 1958 len 8192 data: 4 4
049/300 Read block 4615 len 28672
050/300 Write block 409 len 36864 data: 3 3 2 2 2 2 2 2 2
051/300 Read block 395 len 32768
052/300 Write block 719 len 16384 data: 4 5 5 5
053/300 Read block 2395 len 36864
054/300 Read block 3873 len 28672
055/300 Read block 374 len 28672
056/300 Read block 4348 len 20480
057/300 Write block 4968 len 28672 data: 4 4 5 5 3 2 2
058/300 Read block 4368 len 40960
059/300 Read block 4405 len 28672
060/300 Write block 2182 len 24576 data: 4 3 4 3 4 4
061/300 Read block 2249 len 28672
062/300 Write block 2267 len 12288 data: 2 3 3
063/300 Flush
064/300 Write block 364 len 20480 data: 3 3 3 3 3
065/300 Write block 1524 len 16384 data: 3 3 2 3
066/300 Flush
067/300 Read block 2175 len 40960
068/300 Write block 2045 len 12288 data: 3 3 3
069/300 Write block 1337 len 12288 data: 3 3 3
070/300 Read block 4490 len 20480
071/300 Flush
072/300 Write block 3703 len 28672 data: 2 2 2 2 3 5 5
073/300 Read block 4866 len 32768
074/300 Write block 417 len 36864 data: 3 3 3 3 3 3 3 3 3
075/300 Read block 3791 len 24576
076/300 Read block 987 len 12288
077/300 Write block 1256 len 12288 data: 2 2 2
078/300 Read block 4367 len 32768
079/300 Write block 2728 len 40960 data: 2 2 2 2 2 2 2 2 3 3
080/300 Flush
081/300 Read block 4945 len 36864
082/300 Read block 533 len 36864
083/300 Read block 1942 len 20480
084/300 Read block 4063 len 12288
085/300 Read block 3489 len 28672
086/300 Read block 4382 len 8192
087/300 Write block 598 len 12288 data: 3 3 3
088/300 Read block 3629 len 40960
089/300 Read block 4451 len 12288
090/300 Flush
091/300 Read block 372 len 20480
092/300 Read block 1501 len 32768
093/300 Write block 1448 len 28672 data: 2 2 2 2 2 2 2
094/300 Write block 4746 len 20480 data: 2 2 2 2 2
095/300 Flush
096/300 Read block 2239 len 36864
097/300 Read block 934 len 28672
098/300 Write block 3963 len 12288 data: 3 2 3
099/300 Read block 2224 len 4096
100/300 Read block 2774 len 32768
101/300 Write block 1859 len 16384 data: 5 4 4 4
102/300 Write block 916 len 24576 data: 2 2 2 3 3 3
103/300 Read block 1769 len 4096
104/300 Write block 3267 len 28672 data: 2 2 2 3 4 5 5
105/300 Write block 4769 len 40960 data: 2 2 3 2 2 2 2 2 2 2
106/300 Write block 1771 len 12288 data: 2 3 3
107/300 Read block 2822 len 4096
108/300 Write block 4905 len 4096 data: 2
109/300 Write block 4043 len 16384 data: 4 3 3 3
110/300 Read block 2717 len 4096
111/300 Write block 1373 len 28672 data: 4 4 5 6 6 5 5
112/300 Read block 4914 len 12288
113/300 Write block 1407 len 12288 data: 3 2 2
114/300 Write block 1352 len 36864 data: 2 4 4 4 4 4 6 6 6
115/300 Read block 2895 len 20480
116/300 Write block 4302 len 12288 data: 2 2 3
117/300 Write block 1619 len 40960 data: 3 2 2 2 3 3 3 3 5 4
118/300 Read block 3961 len 40960
119/300 Read block 4685 len 36864
120/300 Write block 666 len 12288 data: 2 2 2
121/300 Read block 3381 len 40960
122/300 Write block 164 len 32768 data: 2 2 3 4 4 4 4 4
123/300 Write block 3424 len 12288 data: 2 2 3
124/300 Read block 934 len 36864
125/300 Read block 4741 len 8192
126/300 Read block 910 len 4096
127/300 Write block 3142 len 8192 data: 6 6
128/300 Write block 1294 len 24576 data: 3 3 3 4 4 4
129/300 Write block 3072 len 8192 data: 2 2
130/300 Read block 4094 len 12288
131/300 Write block 2342 len 12288 data: 3 3 3
132/300 Write block 4275 len 40960 data: 4 4 4 4 3 3 3 3 3 4
133/300 Read block 1561 len 36864
134/300 Write block 4505 len 40960 data: 3 3 3 2 3 3 3 4 4 5
135/300 Read block 2797 len 36864
136/300 Read block 1511 len 12288
137/300 Read block 4977 len 36864
138/300 Write block 4770 len 32768 data: 3 4 3 3 3 3 3 3
139/300 Read block 1366 len 28672
140/300 Write block 4050 len 4096 data: 2
141/300 Read block 3997 len 4096
142/300 Flush
143/300 Write block 2959 len 20480 data: 2 2 2 3 3
144/300 Read block 2844 len 8192
145/300 Flush
146/300 Read block 3728 len 40960
147/300 Write block 794 len 36864 data: 4 4 4 4 3 2 3 2 3
148/300 Read block 2074 len 20480
149/300 Write block 114 len 24576 data: 2 2 2 2 2 2
150/300 Read block 3217 len 16384
151/300 Flush
152/300 Read block 1701 len 40960
153/300 Write block 552 len 12288 data: 3 3 3
154/300 Read block 4541 len 24576
155/300 Read block 4100 len 4096
156/300 Read block 1970 len 24576
157/300 Write block 4941 len 16384 data: 3 4 5 5
158/300 Read block 840 len 16384
159/300 Write block 1728 len 12288 data: 3 3 2
160/300 Flush
161/300 Read block 2331 len 36864
162/300 Read block 1110 len 32768
163/300 Read block 3876 len 36864
164/300 Write block 1254 len 12288 data: 2 2 3
165/300 Flush
166/300 Flush
167/300 Read block 1724 len 4096
168/300 Read block 80 len 16384
169/300 Write block 3382 len 16384 data: 2 2 2 2
170/300 Read block 3102 len 24576
171/300 Read block 3414 len 12288
172/300 Flush
173/300 Read block 2731 len 4096
174/300 Write block 1530 len 12288 data: 2 3 3
175/300 Flush
176/300 Write block 1010 len 24576 data: 3 3 3 3 3 3
177/300 Read block 442 len 36864
178/300 Read block 1972 len 36864
179/300 Read block 2863 len 24576
180/300 Read block 1354 len 24576
181/300 Read block 4097 len 36864
182/300 Write block 1405 len 16384 data: 4 4 4 3
183/300 Write block 2236 len 16384 data: 3 3 3 4
184/300 Read block 2697 len 12288
185/300 Write block 2935 len 8192 data: 3 4
186/300 Flush
187/300 Write block 3758 len 20480 data: 4 5 5 5 5
188/300 Flush
189/300 Read block 254 len 8192
190/300 Read block 4536 len 40960
191/300 Read block 1698 len 40960
192/300 Write block 3050 len 8192 data: 4 4
193/300 Read block 1718 len 24576
194/300 Write block 2305 len 32768 data: 2 2 2 2 2 2 2 2
195/300 Write block 1338 len 4096 data: 4
196/300 Write block 2907 len 32768 data: 2 2 2 2 2 2 2 2
197/300 Write block 83 len 32768 data: 3 2 2 3 3 4 3 3
198/300 Read block 1527 len 28672
199/300 Read block 3153 len 36864
200/300 Read block 4319 len 32768
201/300 Write block 2788 len 32768 data: 2 2 3 3 3 3 4 4
202/300 Write block 60 len 36864 data: 3 2 3 3 4 4 4 3 4
203/300 Write block 1479 len 4096 data: 4
204/300 Flush
205/300 Write block 4829 len 4096 data: 3
206/300 Read block 228 len 40960
207/300 Write block 2425 len 4096 data: 3
208/300 Write block 1460 len 8192 data: 3 2
209/300 Flush
210/300 Read block 1412 len 32768
211/300 Flush
212/300 Read block 3148 len 36864
213/300 Read block 650 len 20480
214/300 Flush
215/300 Read block 1153 len 36864
216/300 Read block 2667 len 16384
217/300 Read block 1624 len 32768
218/300 Write block 3140 len 24576 data: 7 7 7 7 4 4
219/300 Flush
220/300 Write block 2719 len 8192 data: 5 4
221/300 Write block 3020 len 40960 data: 3 3 3 4 4 4 3 3 3 3
222/300 Write block 3893 len 24576 data: 3 3 3 4 4 4
223/300 Read block 1930 len 28672
224/300 Write block 4162 len 8192 data: 3 3
225/300 Write block 1853 len 16384 data: 4 5 6 5
226/300 Read block 2956 len 36864
227/300 Write block 4789 len 28672 data: 4 6 6 5 5 5 4
228/300 Write block 464 len 12288 data: 4 4 3
229/300 Write block 246 len 4096 data: 2
230/300 Write block 1734 len 12288 data: 2 2 2
231/300 Write block 196 len 20480 data: 4 4 4 5 5
232/300 Write block 658 len 32768 data: 3 3 2 2 2 2 2 2
233/300 Write block 4297 len 32768 data: 2 2 2 2 2 3 3 4
234/300 Flush
235/300 Read block 2486 len 32768
236/300 Read block 2896 len 20480
237/300 Read block 4350 len 4096
238/300 Read block 4141 len 20480
239/300 Flush
240/300 Write block 2934 len 12288 data: 3 4 5
241/300 Write block 4240 len 12288 data: 4 4 5
242/300 Flush
243/300 Read block 2117 len 28672
244/300 Flush
245/300 Read block 685 len 20480
246/300 Read block 3558 len 20480
247/300 Write block 3809 len 24576 data: 2 2 2 2 2 2
248/300 Flush
249/300 Write block 3984 len 16384 data: 2 2 2 2
250/300 Read block 1536 len 8192
251/300 Write block 773 len 24576 data: 2 2 2 2 2 2
252/300 Read block 269 len 12288
253/300 Write block 4218 len 20480 data: 2 2 2 2 2
254/300 Read block 370 len 28672
255/300 Read block 38 len 12288
256/300 Read block 3631 len 36864
257/300 Read block 247 len 28672
258/300 Read block 1195 len 40960
259/300 Write block 2440 len 8192 data: 5 6
260/300 Write block 1158 len 8192 data: 2 2
261/300 Flush
262/300 Flush
263/300 Flush
264/300 Read block 3489 len 40960
265/300 Write block 4615 len 24576 data: 4 4 4 4 4 3
266/300 Read block 3049 len 40960
267/300 Read block 2467 len 16384
268/300 Write block 1784 len 24576 data: 2 3 3 3 3 4
269/300 Write block 809 len 16384 data: 3 3 3 3
270/300 Read block 363 len 4096
271/300 Write block 3383 len 28672 data: 3 3 3 2 3 4 4
272/300 Write block 2299 len 28672 data: 2 2 2 3 3 3 3
273/300 Read block 2469 len 4096
274/300 Read block 4629 len 16384
275/300 Read block 4776 len 4096
276/300 Flush
277/300 Read block 1410 len 32768
278/300 Write block 2811 len 20480 data: 3 3 3 3 3
279/300 Write block 3020 len 40960 data: 4 4 4 5 5 5 4 4 4 4
280/300 Read block 2385 len 40960
281/300 Read block 4246 len 24576
282/300 Write block 3117 len 8192 data: 2 2
283/300 Read block 109 len 24576
284/300 Read block 4252 len 28672
285/300 Write block 545 len 40960 data: 5 5 4 4 4 3 2 4 4 4
286/300 Read block 1692 len 24576
287/300 Flush
288/300 Read block 1840 len 8192
289/300 Write block 1606 len 24576 data: 3 2 2 2 2 2
290/300 Write block 3169 len 36864 data: 2 2 2 3 3 2 2 4 4
291/300 Write block 3880 len 4096 data: 4
292/300 Read block 3589 len 28672
293/300 Write block 3422 len 40960 data: 2 2 3 3 4 3 3 3 3 3
294/300 Read block 1824 len 4096
295/300 Read block 2164 len 40960
296/300 Read block 979 len 12288
297/300 Read block 565 len 32768
298/300 Read block 3005 len 8192
299/300 Read block 2105 len 20480
300/300 Read block 1450 len 32768
301/300 Flush
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:38:15.125 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/1, method: POST, req_id: f4c56b86-01ec-48a3-b83e-7ca4b7633510, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3414 Acked Write 6 Done Sent Done false
3415 Acked Read 5 Done Sent Done false
3416 Acked Write 2 Done Sent Done false
3417 Acked Read 1 Done Sent Done false
3418 Acked Write 3 Done Sent Done false
3419 Acked Read 3 Done Sent Done false
3420 Acked Read 3 Done Sent Done false
3421 Acked Write 8 Done Sent Done false
3422 Acked Read 1 Done Sent Done false
3423 Acked Read 4 Done Sent Done false
3424 Acked Read 4 Done Sent Done false
3425 Acked Write 10 Done Sent Done false
3426 Acked Read 5 Done Sent Done false
3427 Acked Read 3 Done Sent Done false
3428 Acked Read 3 Done Sent Done false
3429 Acked Write 3 Done Sent Done false
3430 Acked Read 7 Done Sent Done false
3431 Acked Write 10 Done Sent Done false
3432 Acked Write 9 Done Sent Done false
3433 Acked Flush 0 Done Sent Done false
3434 Acked Flush 0 Done Sent Done false
3435 Acked Write 7 Done Sent Done false
3436 Acked Flush 0 Done Sent Done false
3437 Acked Write 8 Done Sent Done false
3438 Acked Read 2 Done Sent Done false
3439 Acked Write 4 Done Sent Done false
3440 Acked Write 3 Done Sent Done false
3441 Acked Write 5 Done Sent Done false
3442 Acked Read 3 Done Sent Done false
3443 Acked Read 4 Done Sent Done false
3444 Acked Read 1 Done Sent Done false
3445 Acked Write 7 Done Sent Done false
3446 Acked Write 4 Done Sent Done false
3447 Acked Read 2 Done Sent Done false
3448 Acked Write 7 Done Sent Done false
3449 Acked Read 8 Done Sent Done false
3450 Acked Read 8 Done Sent Done false
3451 Acked Write 6 Done Sent Done false
3452 Acked Read 1 Done Sent Done false
3453 Acked Read 6 Done Sent Done false
3454 Acked Write 2 Done Sent Done false
3455 Acked Read 2 Done Sent Done false
3456 Acked Read 8 Done Sent Done false
3457 Acked Read 5 Done Sent Done false
3458 Acked Read 6 Done Sent Done false
3459 Acked Write 7 Done Sent Done false
3460 Acked Write 6 Done Sent Done false
3461 Acked Write 2 Done Sent Done false
3462 Acked Read 7 Done Sent Done false
3463 Acked Write 9 Done Sent Done false
3464 Acked Read 8 Done Sent Done false
3465 Acked Write 4 Done Sent Done false
3466 Acked Read 9 Done Sent Done false
3467 Acked Read 7 Done Sent Done false
3468 Acked Read 7 Done Sent Done false
3469 Acked Read 5 Done Sent Done false
3470 Acked Write 7 Done Sent Done false
3471 Acked Read 10 Done Sent Done false
3472 Acked Read 7 Done Sent Done false
3473 Acked Write 6 Done Sent Done false
3474 Acked Read 7 Done Sent Done false
3475 Acked Write 3 Done Sent Done false
3476 Acked Flush 0 Done Sent Done false
3477 Acked Write 5 Done Sent Done false
3478 Acked Write 4 Done Sent Done false
3479 Acked Flush 0 Done Sent Done false
3480 Acked Read 10 Done Sent Done false
3481 Acked Write 3 Done Sent Done false
3482 Acked Write 3 Done Sent Done false
3483 Acked Read 5 Done Sent Done false
3484 Acked Flush 0 Done Sent Done false
3485 Acked Write 7 Done Sent Done false
3486 Acked Read 8 Done Sent Done false
3487 Acked Write 9 Done Sent Done false
3488 Acked Read 6 Done Sent Done false
3489 Acked Read 3 Done Sent Done false
3490 Acked Write 3 Done Sent Done false
3491 Acked Read 8 Done Sent Done false
3492 Acked Write 10 Done Sent Done false
3493 Acked Flush 0 Done Sent Done false
3494 Acked Read 9 Done Sent Done false
3495 Acked Read 9 Done Sent Done false
3496 Acked Read 5 Done Sent Done false
3497 Acked Read 3 Done Sent Done false
3498 Acked Read 7 Done Sent Done false
3499 Acked Read 2 Done Sent Done false
3500 Acked Write 3 Done Sent Done false
3501 Acked Read 10 Done Sent Done false
3502 Acked Read 3 Done Sent Done false
3503 Acked Flush 0 Done Sent Done false
3504 Acked Read 5 Done Sent Done false
3505 Acked Read 8 Done Sent Done false
3506 Acked Write 7 Done Sent Done false
3507 Acked Write 5 Done Sent Done false
3508 Acked Flush 0 Done Sent Done false
3509 Acked Read 9 Done Sent Done false
3510 Acked Read 7 Done Sent Done false
3511 Acked Write 3 Done Sent Done false
3512 Acked Read 1 Done Sent Done false
3513 Acked Read 8 Done Sent Done false
3514 Acked Write 4 Done Sent Done false
3515 Acked Write 6 Done Sent Done false
3516 Acked Read 1 Done Sent Done false
3517 Acked Write 7 Done Sent Done false
3518 Acked Write 10 Done Sent Done false
3519 Acked Write 3 Done Sent Done false
3520 Acked Read 1 Done Sent Done false
3521 Acked Write 1 Done Sent Done false
3522 Acked Write 4 Done Sent Done false
3523 Acked Read 1 Done Sent Done false
3524 Acked Write 7 Done Sent Done false
3525 Acked Read 3 Done Sent Done false
3526 Acked Write 3 Done Sent Done false
3527 Acked Write 9 Done Sent Done false
3528 Acked Read 5 Done Sent Done false
3529 Acked Write 3 Done Sent Done false
3530 Acked Write 10 Done Sent Done false
3531 Acked Read 10 Done Sent Done false
3532 Acked Read 9 Done Sent Done false
3533 Acked Write 3 Done Sent Done false
3534 Acked Read 10 Done Sent Done false
3535 Acked Write 8 Done Sent Done false
3536 Acked Write 3 Done Sent Done false
3537 Acked Read 9 Done Sent Done false
3538 Acked Read 2 Done Sent Done false
3539 Acked Read 1 Done Sent Done false
3540 Acked Write 2 Done Sent Done false
3541 Acked Write 6 Done Sent Done false
3542 Acked Write 2 Done Sent Done false
3543 Acked Read 3 Done Sent Done false
3544 Acked Write 3 Done Sent Done false
3545 Acked Write 10 Done Sent Done false
3546 Acked Read 9 Done Sent Done false
3547 Acked Write 10 Done Sent Done false
3548 Acked Read 9 Done Sent Done false
3549 Acked Read 3 Done Sent Done false
3550 Acked Read 9 Done Sent Done false
3551 Acked Write 8 Done Sent Done false
3552 Acked Read 7 Done Sent Done false
3553 Acked Write 1 Done Sent Done false
3554 Acked Read 1 Done Sent Done false
3555 Acked Flush 0 Done Sent Done false
3556 Acked Write 5 Done Sent Done false
3557 Acked Read 2 Done Sent Done false
3558 Acked Flush 0 Done Sent Done false
3559 Acked Read 10 Done Sent Done false
3560 Acked Write 9 Done Sent Done false
3561 Acked Read 5 Done Sent Done false
3562 Acked Write 6 Done Sent Done false
3563 Acked Read 4 Done Sent Done false
3564 Acked Flush 0 Done Sent Done false
3565 Acked Read 10 Done Sent Done false
3566 Acked Write 3 Done Sent Done false
3567 Acked Read 6 Done Sent Done false
3568 Acked Read 1 Done Sent Done false
3569 Acked Read 6 Done Sent Done false
3570 Acked Write 4 Done Sent Done false
3571 Acked Read 4 Done Sent Done false
3572 Acked Write 3 Done Sent Done false
3573 Acked Flush 0 Done Sent Done false
3574 Acked Read 9 Done Sent Done false
3575 Acked Read 8 Done Sent Done false
3576 Acked Read 9 Done Sent Done false
3577 Acked Write 3 Done Sent Done false
3578 Acked Flush 0 Done Sent Done false
3579 Acked Flush 0 Done Sent Done false
3580 Acked Read 1 Done Sent Done false
3581 Acked Read 4 Done Sent Done false
3582 Acked Write 4 Done Sent Done false
3583 Acked Read 6 Done Sent Done false
3584 Acked Read 3 Done Sent Done false
3585 Acked Flush 0 Done Sent Done false
3586 Acked Read 1 Done Sent Done false
3587 Acked Write 3 Done Sent Done false
3588 Acked Flush 0 Done Sent Done false
3589 Acked Write 6 Done Sent Done false
3590 Acked Read 9 Done Sent Done false
3591 Acked Read 9 Done Sent Done false
3592 Acked Read 6 Done Sent Done false
3593 Acked Read 6 Done Sent Done false
3594 Acked Read 9 Done Sent Done false
3595 Acked Write 4 Done Sent Done false
3596 Acked Write 4 Done Sent Done false
3597 Acked Read 3 Done Sent Done false
3598 Acked Write 2 Done Sent Done false
3599 Acked Flush 0 Done Sent Done false
3600 Acked Write 5 Done Sent Done false
3601 Acked Flush 0 Done Sent Done false
3602 Acked Read 2 Done Sent Done false
3603 Acked Read 10 Done Sent Done false
3604 Acked Read 10 Done Sent Done false
3605 Acked Write 2 Done Sent Done false
3606 Acked Read 6 Done Sent Done false
3607 Acked Write 8 Done Sent Done false
3608 Acked Write 1 Done Sent Done false
3609 Acked Write 8 Done Sent Done false
3610 Acked Write 8 Done Sent Done false
3611 Acked Read 7 Done Sent Done false
3612 Acked Read 9 Done Sent Done false
3613 Acked Read 8 Done Sent Done false
3614 Acked Write 8 Done Sent Done false
3615 Acked Write 9 Done Sent Done false
3616 Acked Write 1 Done Sent Done false
3617 Acked Flush 0 Done Sent Done false
3618 Acked Write 1 Done Sent Done false
3619 Acked Read 10 Done Sent Done false
3620 Acked Write 1 Done Sent Done false
3621 Acked Write 2 Done Sent Done false
3622 Acked Flush 0 Done Sent Done false
3623 Acked Read 8 Done Sent Done false
3624 Acked Flush 0 Done Sent Done false
3625 Acked Read 9 Done Sent Done false
3626 Acked Read 5 Done Sent Done false
3627 Acked Flush 0 Done Sent Done false
3628 Acked Read 9 Done Sent Done false
3629 Acked Read 4 Done Sent Done false
3630 Acked Read 8 Done Sent Done false
3631 Acked Write 6 Done Sent Done false
3632 Acked Flush 0 Done Sent Done false
3633 Acked Write 2 Done Sent Done false
3634 Acked Write 10 Done Sent Done false
3635 Acked Write 6 Done Sent Done false
3636 Acked Read 7 Done Sent Done false
3637 Acked Write 2 Done Sent Done false
3638 Acked Write 4 Done Sent Done false
3639 Acked Read 9 Done Sent Done false
3640 Acked Write 7 Done Sent Done false
3641 Acked Write 3 Done Sent Done false
3642 Acked Write 1 Done Sent Done false
3643 Acked Write 3 Done Sent Done false
3644 Acked Write 5 Done Sent Done false
3645 Acked Write 8 Done Sent Done false
3646 Acked Write 8 Done Sent Done false
3647 Acked Flush 0 Done Sent Done false
3648 Acked Read 8 Done Sent Done false
3649 Acked Read 5 Done Sent Done false
3650 Acked Read 1 Done Sent Done false
3651 Acked Read 5 Done Sent Done false
3652 Acked Flush 0 Done Sent Done false
3653 Acked Write 3 Done Sent Done false
3654 Acked Write 3 Done Sent Done false
3655 Acked Flush 0 Done Sent Done false
3656 Acked Read 7 Done Sent Done false
3657 Acked Flush 0 Done Sent Done false
3658 Acked Read 5 Done Sent Done false
3659 Acked Read 5 Done Sent Done false
3660 Acked Write 6 Done Sent Done false
3661 Acked Flush 0 Done Sent Done false
3662 Acked Write 4 Done Sent Done false
3663 Acked Read 2 Done Sent Done false
3664 Acked Write 6 Done Sent Done false
3665 Acked Read 3 Done Sent Done false
3666 Acked Write 5 Done Sent Done false
3667 Acked Read 7 Done Sent Done false
3668 Acked Read 3 Done Sent Done false
3669 Acked Read 9 Done Sent Done false
3670 Acked Read 7 Done Sent Done false
3671 Acked Read 10 Done Sent Done false
3672 Acked Write 2 Done Sent Done false
3673 Acked Write 2 Done Sent Done false
3674 Acked Flush 0 Done Sent Done false
3675 Acked Flush 0 Done Sent Done false
3676 Acked Flush 0 Done Sent Done false
3677 Acked Read 10 Done Sent Done false
3678 Acked Write 6 Done Sent Done false
3679 Acked Read 10 Done Sent Done false
3680 Acked Read 4 Done Sent Done false
3681 Acked Write 6 Done Sent Done false
3682 Acked Write 4 Done Sent Done false
3683 Acked Read 1 Done Sent Done false
3684 Acked Write 7 Done Sent Done false
3685 Acked Write 7 Done Sent Done false
3686 Acked Read 1 Done Sent Done false
3687 Acked Read 4 Done Sent Done false
3688 Acked Read 1 Done Sent Done false
3689 Acked Flush 0 Done Sent Done false
3690 Acked Read 8 Done Sent Done false
3691 Acked Write 5 Done Sent Done false
3692 Acked Write 10 Done Sent Done false
3693 Acked Read 10 Done Sent Done false
3694 Acked Read 6 Done Sent Done false
3695 Acked Write 2 Done Sent Done false
3696 Acked Read 6 Done Sent Done false
3697 Acked Read 7 Done Sent Done false
3698 Acked Write 10 Done Sent Done false
3699 Acked Read 6 Done Sent Done false
3700 Acked Flush 0 Done Sent Done false
3701 Acked Read 2 Done Sent Done false
3702 Acked Write 6 Done Sent Done false
3703 Acked Write 9 Done Sent Done false
3704 Acked Write 1 Done Sent Done false
3705 Acked Read 7 Done Sent Done false
3706 Acked Write 10 Done Sent Done false
3707 Acked Read 1 Done Sent Done false
3708 Acked Read 10 Done Sent Done false
3709 Acked Read 3 Done Sent Done false
3710 Acked Read 8 Done Sent Done false
3711 Acked Read 2 Done Sent Done false
3712 Acked Read 5 Done Sent Done false
3713 Acked Read 8 Done Sent Done false
3714 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3714 3413 3714
Downstairs last five completed: 3413 3412 3411 3410 3409
Upstairs last five completed: 3714 3713 3712 3711 3710
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1209)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3414 Acked Write 6 Done Sent Done false
3415 Acked Read 5 Done Sent Done false
3416 Acked Write 2 Done Sent Done false
3417 Acked Read 1 Done Sent Done false
3418 Acked Write 3 Done Sent Done false
3419 Acked Read 3 Done Sent Done false
3420 Acked Read 3 Done Sent Done false
3421 Acked Write 8 Done Sent Done false
3422 Acked Read 1 Done Sent Done false
3423 Acked Read 4 Done Sent Done false
3424 Acked Read 4 Done Sent Done false
3425 Acked Write 10 Done Sent Done false
3426 Acked Read 5 Done Sent Done false
3427 Acked Read 3 Done Sent Done false
3428 Acked Read 3 Done Sent Done false
3429 Acked Write 3 Done Sent Done false
3430 Acked Read 7 Done Sent Done false
3431 Acked Write 10 Done Sent Done false
3432 Acked Write 9 Done Sent Done false
3433 Acked Flush 0 Done Sent Done false
3434 Acked Flush 0 Done Sent Done false
3435 Acked Write 7 Done Sent Done false
3436 Acked Flush 0 Done Sent Done false
3437 Acked Write 8 Done Sent Done false
3438 Acked Read 2 Done Sent Done false
3439 Acked Write 4 Done Sent Done false
3440 Acked Write 3 Done Sent Done false
3441 Acked Write 5 Done Sent Done false
3442 Acked Read 3 Done Sent Done false
3443 Acked Read 4 Done Sent Done false
3444 Acked Read 1 Done Sent Done false
3445 Acked Write 7 Done Sent Done false
3446 Acked Write 4 Done Sent Done false
3447 Acked Read 2 Done Sent Done false
3448 Acked Write 7 Done Sent Done false
3449 Acked Read 8 Done Sent Done false
3450 Acked Read 8 Done Sent Done false
3451 Acked Write 6 Done Sent Done false
3452 Acked Read 1 Done Sent Done false
3453 Acked Read 6 Done Sent Done false
3454 Acked Write 2 Done Sent Done false
3455 Acked Read 2 Done Sent Done false
3456 Acked Read 8 Done Sent Done false
3457 Acked Read 5 Done Sent Done false
3458 Acked Read 6 Done Sent Done false
3459 Acked Write 7 Done Sent Done false
3460 Acked Write 6 Done Sent Done false
3461 Acked Write 2 Done Sent Done false
3462 Acked Read 7 Done Sent Done false
3463 Acked Write 9 Done Sent Done false
3464 Acked Read 8 Done Sent Done false
3465 Acked Write 4 Done Sent Done false
3466 Acked Read 9 Done Sent Done false
3467 Acked Read 7 Done Sent Done false
3468 Acked Read 7 Done Sent Done false
3469 Acked Read 5 Done Sent Done false
3470 Acked Write 7 Done Sent Done false
3471 Acked Read 10 Done Sent Done false
3472 Acked Read 7 Done Sent Done false
3473 Acked Write 6 Done Sent Done false
3474 Acked Read 7 Done Sent Done false
3475 Acked Write 3 Done Sent Done false
3476 Acked Flush 0 Done Sent Done false
3477 Acked Write 5 Done Sent Done false
3478 Acked Write 4 Done Sent Done false
3479 Acked Flush 0 Done Sent Done false
3480 Acked Read 10 Done Sent Done false
3481 Acked Write 3 Done Sent Done false
3482 Acked Write 3 Done Sent Done false
3483 Acked Read 5 Done Sent Done false
3484 Acked Flush 0 Done Sent Done false
3485 Acked Write 7 Done Sent Done false
3486 Acked Read 8 Done Sent Done false
3487 Acked Write 9 Done Sent Done false
3488 Acked Read 6 Done Sent Done false
3489 Acked Read 3 Done Sent Done false
3490 Acked Write 3 Done Sent Done false
3491 Acked Read 8 Done Sent Done false
3492 Acked Write 10 Done Sent Done false
3493 Acked Flush 0 Done Sent Done false
3494 Acked Read 9 Done Sent Done false
3495 Acked Read 9 Done Sent Done false
3496 Acked Read 5 Done Sent Done false
3497 Acked Read 3 Done Sent Done false
3498 Acked Read 7 Done Sent Done false
3499 Acked Read 2 Done Sent Done false
3500 Acked Write 3 Done Sent Done false
3501 Acked Read 10 Done Sent Done false
3502 Acked Read 3 Done Sent Done false
3503 Acked Flush 0 Done Sent Done false
3504 Acked Read 5 Done Sent Done false
3505 Acked Read 8 Done Sent Done false
3506 Acked Write 7 Done Sent Done false
3507 Acked Write 5 Done Sent Done false
3508 Acked Flush 0 Done Sent Done false
3509 Acked Read 9 Done Sent Done false
3510 Acked Read 7 Done Sent Done false
3511 Acked Write 3 Done Sent Done false
3512 Acked Read 1 Done Sent Done false
3513 Acked Read 8 Done Sent Done false
3514 Acked Write 4 Done Sent Done false
3515 Acked Write 6 Done Sent Done false
3516 Acked Read 1 Done Sent Done false
3517 Acked Write 7 Done Sent Done false
3518 Acked Write 10 Done Sent Done false
3519 Acked Write 3 Done Sent Done false
3520 Acked Read 1 Done Sent Done false
3521 Acked Write 1 Done Sent Done false
3522 Acked Write 4 Done Sent Done false
3523 Acked Read 1 Done Sent Done false
3524 Acked Write 7 Done Sent Done false
3525 Acked Read 3 Done Sent Done false
3526 Acked Write 3 Done Sent Done false
3527 Acked Write 9 Done Sent Done false
3528 Acked Read 5 Done Sent Done false
3529 Acked Write 3 Done Sent Done false
3530 Acked Write 10 Done Sent Done false
3531 Acked Read 10 Done Sent Done false
3532 Acked Read 9 Done Sent Done false
3533 Acked Write 3 Done Sent Done false
3534 Acked Read 10 Done Sent Done false
3535 Acked Write 8 Done Sent Done false
3536 Acked Write 3 Done Sent Done false
3537 Acked Read 9 Done Sent Done false
3538 Acked Read 2 Done Sent Done false
3539 Acked Read 1 Done Sent Done false
3540 Acked Write 2 Done Sent Done false
3541 Acked Write 6 Done Sent Done false
3542 Acked Write 2 Done Sent Done false
3543 Acked Read 3 Done Sent Done false
3544 Acked Write 3 Done Sent Done false
3545 Acked Write 10 Done Sent Done false
3546 Acked Read 9 Done Sent Done false
3547 Acked Write 10 Done Sent Done false
3548 Acked Read 9 Done Sent Done false
3549 Acked Read 3 Done Sent Done false
3550 Acked Read 9 Done Sent Done false
3551 Acked Write 8 Done Sent Done false
3552 Acked Read 7 Done Sent Done false
3553 Acked Write 1 Done Sent Done false
3554 Acked Read 1 Done Sent Done false
3555 Acked Flush 0 Done Sent Done false
3556 Acked Write 5 Done Sent Done false
3557 Acked Read 2 Done Sent Done false
3558 Acked Flush 0 Done Sent Done false
3559 Acked Read 10 Done Sent Done false
3560 Acked Write 9 Done Sent Done false
3561 Acked Read 5 Done Sent Done false
3562 Acked Write 6 Done Sent Done false
3563 Acked Read 4 Done Sent Done false
3564 Acked Flush 0 Done Sent Done false
3565 Acked Read 10 Done Sent Done false
3566 Acked Write 3 Done Sent Done false
3567 Acked Read 6 Done Sent Done false
3568 Acked Read 1 Done Sent Done false
3569 Acked Read 6 Done Sent Done false
3570 Acked Write 4 Done Sent Done false
3571 Acked Read 4 Done Sent Done false
3572 Acked Write 3 Done Sent Done false
3573 Acked Flush 0 Done Sent Done false
3574 Acked Read 9 Done Sent Done false
3575 Acked Read 8 Done Sent Done false
3576 Acked Read 9 Done Sent Done false
3577 Acked Write 3 Done Sent Done false
3578 Acked Flush 0 Done Sent Done false
3579 Acked Flush 0 Done Sent Done false
3580 Acked Read 1 Done Sent Done false
3581 Acked Read 4 Done Sent Done false
3582 Acked Write 4 Done Sent Done false
3583 Acked Read 6 Done Sent Done false
3584 Acked Read 3 Done Sent Done false
3585 Acked Flush 0 Done Sent Done false
3586 Acked Read 1 Done Sent Done false
3587 Acked Write 3 Done Sent Done false
3588 Acked Flush 0 Done Sent Done false
3589 Acked Write 6 Done Sent Done false
3590 Acked Read 9 Done Sent Done false
3591 Acked Read 9 Done Sent Done false
3592 Acked Read 6 Done Sent Done false
3593 Acked Read 6 Done Sent Done false
3594 Acked Read 9 Done Sent Done false
3595 Acked Write 4 Done Sent Done false
3596 Acked Write 4 Done Sent Done false
3597 Acked Read 3 Done Sent Done false
3598 Acked Write 2 Done Sent Done false
3599 Acked Flush 0 Done Sent Done false
3600 Acked Write 5 Done Sent Done false
3601 Acked Flush 0 Done Sent Done false
3602 Acked Read 2 Done Sent Done false
3603 Acked Read 10 Done Sent Done false
3604 Acked Read 10 Done Sent Done false
3605 Acked Write 2 Done Sent Done false
3606 Acked Read 6 Done Sent Done false
3607 Acked Write 8 Done Sent Done false
3608 Acked Write 1 Done Sent Done false
3609 Acked Write 8 Done Sent Done false
3610 Acked Write 8 Done Sent Done false
3611 Acked Read 7 Done Sent Done false
3612 Acked Read 9 Done Sent Done false
3613 Acked Read 8 Done Sent Done false
3614 Acked Write 8 Done Sent Done false
3615 Acked Write 9 Done Sent Done false
3616 Acked Write 1 Done Sent Done false
3617 Acked Flush 0 Done Sent Done false
3618 Acked Write 1 Done Sent Done false
3619 Acked Read 10 Done Sent Done false
3620 Acked Write 1 Done Sent Done false
3621 Acked Write 2 Done Sent Done false
3622 Acked Flush 0 Done Sent Done false
3623 Acked Read 8 Done Sent Done false
3624 Acked Flush 0 Done Sent Done false
3625 Acked Read 9 Done Sent Done false
3626 Acked Read 5 Done Sent Done false
3627 Acked Flush 0 Done Sent Done false
3628 Acked Read 9 Done Sent Done false
3629 Acked Read 4 Done Sent Done false
3630 Acked Read 8 Done Sent Done false
3631 Acked Write 6 Done Sent Done false
3632 Acked Flush 0 Done Sent Done false
3633 Acked Write 2 Done Sent Done false
3634 Acked Write 10 Done Sent Done false
3635 Acked Write 6 Done Sent Done false
3636 Acked Read 7 Done Sent Done false
3637 Acked Write 2 Done Sent Done false
3638 Acked Write 4 Done Sent Done false
3639 Acked Read 9 Done Sent Done false
3640 Acked Write 7 Done Sent Done false
3641 Acked Write 3 Done Sent Done false
3642 Acked Write 1 Done Sent Done false
3643 Acked Write 3 Done Sent Done false
3644 Acked Write 5 Done Sent Done false
3645 Acked Write 8 Done Sent Done false
3646 Acked Write 8 Done Sent Done false
3647 Acked Flush 0 Done Sent Done false
3648 Acked Read 8 Done Sent Done false
3649 Acked Read 5 Done Sent Done false
3650 Acked Read 1 Done Sent Done false
3651 Acked Read 5 Done Sent Done false
3652 Acked Flush 0 Done Sent Done false
3653 Acked Write 3 Done Sent Done false
3654 Acked Write 3 Done Sent Done false
3655 Acked Flush 0 Done Sent Done false
3656 Acked Read 7 Done Sent Done false
3657 Acked Flush 0 Done Sent Done false
3658 Acked Read 5 Done Sent Done false
3659 Acked Read 5 Done Sent Done false
3660 Acked Write 6 Done Sent Done false
3661 Acked Flush 0 Done Sent Done false
3662 Acked Write 4 Done Sent Done false
3663 Acked Read 2 Done Sent Done false
3664 Acked Write 6 Done Sent Done false
3665 Acked Read 3 Done Sent Done false
3666 Acked Write 5 Done Sent Done false
3667 Acked Read 7 Done Sent Done false
3668 Acked Read 3 Done Sent Done false
3669 Acked Read 9 Done Sent Done false
3670 Acked Read 7 Done Sent Done false
3671 Acked Read 10 Done Sent Done false
3672 Acked Write 2 Done Sent Done false
3673 Acked Write 2 Done Sent Done false
3674 Acked Flush 0 Done Sent Done false
3675 Acked Flush 0 Done Sent Done false
3676 Acked Flush 0 Done Sent Done false
3677 Acked Read 10 Done Sent Done false
3678 Acked Write 6 Done Sent Done false
3679 Acked Read 10 Done Sent Done false
3680 Acked Read 4 Done Sent Done false
3681 Acked Write 6 Done Sent Done false
3682 Acked Write 4 Done Sent Done false
3683 Acked Read 1 Done Sent Done false
3684 Acked Write 7 Done Sent Done false
3685 Acked Write 7 Done Sent Done false
3686 Acked Read 1 Done Sent Done false
3687 Acked Read 4 Done Sent Done false
3688 Acked Read 1 Done Sent Done false
3689 Acked Flush 0 Done Sent Done false
3690 Acked Read 8 Done Sent Done false
3691 Acked Write 5 Done Sent Done false
3692 Acked Write 10 Done Sent Done false
3693 Acked Read 10 Done Sent Done false
3694 Acked Read 6 Done Sent Done false
3695 Acked Write 2 Done Sent Done false
3696 Acked Read 6 Done Sent Done false
3697 Acked Read 7 Done Sent Done false
3698 Acked Write 10 Done Sent Done false
3699 Acked Read 6 Done Sent Done false
3700 Acked Flush 0 Done Sent Done false
3701 Acked Read 2 Done Sent Done false
3702 Acked Write 6 Done Sent Done false
3703 Acked Write 9 Done Sent Done false
3704 Acked Write 1 Done Sent Done false
3705 Acked Read 7 Done Sent Done false
3706 Acked Write 10 Done Sent Done false
3707 Acked Read 1 Done Sent Done false
3708 Acked Read 10 Done Sent Done false
3709 Acked Read 3 Done Sent Done false
3710 Acked Read 8 Done Sent Done false
3711 Acked Read 2 Done Sent Done false
3712 Acked Read 5 Done Sent Done false
3713 Acked Read 8 Done Sent Done false
3714 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3714 3413 3714
Downstairs last five completed: 3413 3412 3411 3410 3409
Upstairs last five completed: 3714 3713 3712 3711 3710
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.88857422Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42395)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.888802473Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.889831103Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.890452291Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.890480887Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 3413","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.890497035Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 3413","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.890753565Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 301 jobs since flush 3413","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.890772653Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 301 jobs since flush: 3413","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:20.891024604Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 3714 3713 3712 3711 3710
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Jan 27 19:38:23.130 INFO request completed, latency_us[8820] Got stop action so:false kr:false
: 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b5291f64-d6a5-4bfd-b344-4af562e6ab2c, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:38:23.130 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/1, method: GET, req_id: f1526735-a161-48ea-9133-fd70e5f89fad, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:23.130981131Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:23.131054703Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:23.131072551Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:23.131136653Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:38:27.132 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/1, method: GET, req_id: fbf6302c-421d-4026-a24f-bf8ea784a049, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2651 len 24576 data: 4 4 4 4 4 4
002/300 Write block 42 len 16384 data: 3 4 4 3
003/300 Flush
004/300 Read block 104 len 4096
005/300 Read block 1019 len 16384
006/300 Flush
007/300 Read block 85 len 36864
008/300 Read block 3102 len 28672
009/300 Write block 4863 len 40960 data: 3 3 2 2 3 3 4 3 3 3
010/300 Flush
011/300 Read block 913 len 4096
012/300 Flush
013/300 Write block 2847 len 40960 data: 3 4 4 4 5 5 5 4 4 4
014/300 Read block 3879 len 40960
015/300 Write block 2870 len 32768 data: 2 2 2 2 2 3 3 3
016/300 Write block 658 len 16384 data: 4 4 3 3
017/300 Write block 3261 len 20480 data: 2 2 2 2 2
018/300 Read block 3353 len 20480
019/300 Read block 4167 len 20480
020/300 Write block 3748 len 8192 data: 4 4
021/300 Flush
022/300 Write block 1298 len 20480 data: 5 5 4 5 4
023/300 Read block 1958 len 28672
024/300 Read block 763 len 12288
025/300 Read block 4867 len 4096
026/300 Read block 2795 len 8192
027/300 Write block 2034 len 20480 data: 3 3 2 2 2
028/300 Write block 1729 len 12288 data: 4 3 2
029/300 Flush
030/300 Read block 2343 len 36864
031/300 Flush
032/300 Flush
033/300 Write block 2724 len 36864 data: 3 3 3 2 3 3 3 3 3
034/300 Read block 4494 len 8192
035/300 Write block 1674 len 12288 data: 5 4 4
036/300 Write block 4556 len 4096 data: 2
037/300 Write block 1694 len 24576 data: 2 2 2 2 2 2
038/300 Read block 1405 len 20480
039/300 Read block 397 len 28672
040/300 Read block 4817 len 40960
041/300 Write block 3648 len 8192 data: 2 2
042/300 Write block 919 len 16384 data: 4 4 4 3
043/300 Read block 2762 len 28672
044/300 Read block 4465 len 12288
045/300 Read block 1102 len 40960
046/300 Flush
047/300 Write block 4774 len 12288 data: 4 4 4
048/300 Read block 2472 len 24576
049/300 Flush
050/300 Read block 2293 len 12288
051/300 Read block 3060 len 12288
052/300 Write block 1888 len 24576 data: 3 3 3 3 3 3
053/300 Flush
054/300 Read block 1044 len 12288
055/300 Flush
056/300 Read block 1898 len 4096
057/300 Write block 2798 len 36864 data: 3 3 3 4 4 2 2 2 2
058/300 Write block 2395 len 12288 data: 2 2 2
059/300 Write block 1751 len 28672 data: 2 2 3 5 4 4 4
060/300 Read block 4329 len 36864
061/300 Read block 925 len 16384
062/300 Write block 4475 len 32768 data: 3 3 3 3 2 3 3 3
063/300 Write block 1991 len 36864 data: 2 2 2 2 3 3 3 3 3
064/300 Read block 3971 len 12288
065/300 Flush
066/300 Read block 991 len 28672
067/300 Write block 1017 len 8192 data: 3 4
068/300 Read block 3690 len 40960
069/300 Write block 3452 len 28672 data: 3 4 4 5 5 5 3
070/300 Read block 4791 len 8192
071/300 Write block 717 len 12288 data: 3 3 5
072/300 Read block 2458 len 4096
073/300 Read block 3171 len 8192
074/300 Read block 4564 len 36864
075/300 Write block 3853 len 12288 data: 2 2 2
076/300 Write block 2672 len 40960 data: 2 3 3 3 3 4 4 5 5 5
077/300 Read block 4043 len 12288
078/300 Write block 4393 len 8192 data: 3 3
079/300 Read block 906 len 12288
080/300 Read block 1567 len 40960
081/300 Write block 249 len 24576 data: 2 2 2 2 2 2
082/300 Read block 265 len 20480
083/300 Read block 2347 len 24576
084/300 Read block 2238 len 40960
085/300 Read block 3568 len 16384
086/300 Read block 2768 len 32768
087/300 Read block 3105 len 40960
088/300 Read block 2357 len 4096
089/300 Write block 1215 len 20480 data: 3 2 2 2 3
090/300 Read block 3819 len 16384
091/300 Write block 2680 len 28672 data: 6 6 5 5 5 3 3
092/300 Write block 2154 len 24576 data: 4 4 5 5 5 5
093/300 Read block 1934 len 24576
094/300 Read block 3521 len 40960
095/300 Write block 2658 len 8192 data: 5 4
096/300 Read block 2585 len 8192
097/300 Flush
098/300 Read block 1567 len 16384
099/300 Read block 4093 len 32768
100/300 Write block 4989 len 16384 data: 4 3 2 2
101/300 Write block 4231 len 8192 data: 3 3
102/300 Write block 3649 len 16384 data: 3 2 2 2
103/300 Flush
104/300 Read block 903 len 32768
105/300 Read block 1448 len 36864
106/300 Write block 2137 len 40960 data: 2 2 2 2 3 3 3 4 3 3
107/300 Write block 2611 len 20480 data: 2 2 2 2 2
108/300 Write block 3054 len 12288 data: 3 2 3
109/300 Read block 2301 len 32768
110/300 Read block 731 len 24576
111/300 Read block 124 len 40960
112/300 Read block 3999 len 16384
113/300 Read block 1920 len 8192
114/300 Read block 3926 len 40960
115/300 Write block 4250 len 12288 data: 7 5 4
116/300 Flush
117/300 Write block 1199 len 4096 data: 2
118/300 Read block 2787 len 28672
119/300 Read block 3333 len 32768
120/300 Write block 2744 len 40960 data: 2 2 2 2 2 2 2 4 3 3
121/300 Read block 2595 len 24576
122/300 Write block 1068 len 20480 data: 2 2 3 3 3
123/300 Read block 71 len 32768
124/300 Read block 1920 len 8192
125/300 Write block 187 len 4096 data: 3
126/300 Read block 3858 len 16384
127/300 Read block 4773 len 24576
128/300 Flush
129/300 Flush
130/300 Write block 1104 len 12288 data: 3 4 3
131/300 Read block 4884 len 36864
132/300 Read block 3279 len 8192
133/300 Write block 1120 len 40960 data: 2 2 3 3 3 3 4 4 5 6
134/300 Read block 3266 len 40960
135/300 Flush
136/300 Read block 1227 len 24576
137/300 Read block 3138 len 12288
138/300 Flush
139/300 Read block 4548 len 40960
140/300 Flush
141/300 Flush
142/300 Write block 2197 len 28672 data: 4 4 4 3 4 3 4
143/300 Write block 3721 len 4096 data: 4
144/300 Write block 1784 len 8192 data: 3 4
145/300 Read block 1021 len 12288
146/300 Read block 3497 len 16384
147/300 Read block 3681 len 20480
148/300 Write block 4967 len 36864 data: 4 5 5 6 6 4 3 3 2
149/300 Write block 3878 len 16384 data: 4 3 5 4
150/300 Read block 4122 len 8192
151/300 Read block 1640 len 28672
152/300 Write block 1234 len 28672 data: 5 5 5 5 5 5 4
153/300 Write block 3517 len 24576 data: 3 4 6 6 6 6
154/300 Write block 3334 len 28672 data: 3 4 4 4 3 3 3
155/300 Write block 2638 len 32768 data: 5 5 5 5 4 3 3 2
156/300 Read block 1041 len 20480
157/300 Read block 4187 len 24576
158/300 Write block 2209 len 24576 data: 2 2 2 2 2 2
159/300 Read block 2352 len 4096
160/300 Write block 4275 len 4096 data: 5
161/300 Read block 3359 len 40960
162/300 Write block 4610 len 16384 data: 3 2 2 3
163/300 Flush
164/300 Read block 1813 len 24576
165/300 Write block 2345 len 32768 data: 3 3 3 3 2 3 3 3
166/300 Read block 4825 len 36864
167/300 Read block 211 len 20480
168/300 Write block 3442 len 4096 data: 2
169/300 Flush
170/300 Read block 307 len 40960
171/300 Read block 248 len 36864
172/300 Read block 4129 len 4096
173/300 Read block 1027 len 28672
174/300 Flush
175/300 Write block 2995 len 24576 data: 3 3 3 3 3 4
176/300 Read block 1283 len 16384
177/300 Read block 3962 len 8192
178/300 Write block 3491 len 12288 data: 3 4 4
179/300 Read block 2304 len 32768
180/300 Write block 3889 len 4096 data: 2
181/300 Write block 4416 len 12288 data: 3 3 3
182/300 Read block 1866 len 28672
183/300 Read block 4510 len 8192
184/300 Write block 2490 len 24576 data: 3 3 3 4 4 3
185/300 Write block 767 len 28672 data: 2 2 2 2 2 2 3
186/300 Read block 1556 len 8192
187/300 Read block 2247 len 8192
188/300 Flush
189/300 Write block 3653 len 12288 data: 2 2 2
190/300 Read block 2185 len 16384
191/300 Write block 648 len 12288 data: 4 4 3
192/300 Write block 3658 len 8192 data: 3 3
193/300 Write block 4855 len 32768 data: 3 2 3 3 3 3 3 2
194/300 Write block 4667 len 4096 data: 3
195/300 Write block 4127 len 32768 data: 2 2 2 2 2 2 2 3
196/300 Read block 641 len 20480
197/300 Write block 1449 len 40960 data: 3 3 3 3 3 3 2 4 4 5
198/300 Write block 3020 len 4096 data: 5
199/300 Read block 2079 len 20480
200/300 Write block 1229 len 4096 data: 2
201/300 Read block 4884 len 36864
202/300 Read block 954 len 40960
203/300 Write block 4056 len 16384 data: 3 3 3 3
204/300 Write block 2971 len 12288 data: 3 4 4
205/300 Read block 213 len 20480
206/300 Write block 972 len 28672 data: 3 3 4 4 4 3 3
207/300 Read block 1815 len 16384
208/300 Read block 4406 len 32768
209/300 Write block 4111 len 36864 data: 4 4 4 2 2 3 3 5 5
210/300 Write block 1941 len 32768 data: 2 2 3 3 3 3 3 3
211/300 Read block 1828 len 4096
212/300 Write block 4640 len 8192 data: 2 2
213/300 Read block 2900 len 16384
214/300 Write block 235 len 16384 data: 3 3 3 3
215/300 Read block 275 len 32768
216/300 Write block 4713 len 24576 data: 4 4 4 4 4 4
217/300 Flush
218/300 Write block 4549 len 40960 data: 2 2 2 2 2 2 2 3 2 2
219/300 Write block 2182 len 20480 data: 5 4 5 4 5
220/300 Read block 4679 len 4096
221/300 Write block 1838 len 16384 data: 2 2 2 2
222/300 Read block 3051 len 20480
223/300 Read block 3995 len 4096
224/300 Read block 1455 len 4096
225/300 Flush
226/300 Read block 2686 len 40960
227/300 Write block 3323 len 20480 data: 3 3 4 5 3
228/300 Write block 1339 len 40960 data: 4 3 2 3 3 3 3 3 3 3
229/300 Read block 223 len 24576
230/300 Read block 3053 len 40960
231/300 Write block 1966 len 20480 data: 2 2 2 2 2
232/300 Read block 4533 len 20480
233/300 Write block 4055 len 36864 data: 3 4 4 4 4 2 2 2 2
234/300 Read block 529 len 40960
235/300 Read block 2446 len 20480
236/300 Read block 3386 len 36864
237/300 Read block 1565 len 8192
238/300 Write block 709 len 40960 data: 3 3 4 4 9 8 7 6 4 4
239/300 Write block 3660 len 36864 data: 3 3 3 3 3 4 4 4 4
240/300 Write block 337 len 28672 data: 3 3 3 3 3 3 3
241/300 Read block 2409 len 28672
242/300 Write block 4927 len 24576 data: 4 4 3 2 2 2
243/300 Write block 3894 len 40960 data: 4 4 5 5 5 4 3 3 4 3
244/300 Read block 3000 len 28672
245/300 Write block 2111 len 8192 data: 2 2
246/300 Write block 2683 len 24576 data: 6 6 4 4 3 3
247/300 Read block 229 len 24576
248/300 Write block 3389 len 40960 data: 5 4 4 3 3 2 2 2 5 5
249/300 Read block 4576 len 4096
250/300 Read block 1278 len 32768
251/300 Write block 168 len 8192 data: 5 5
252/300 Read block 2579 len 24576
253/300 Write block 4023 len 28672 data: 3 2 2 2 3 2 2
254/300 Write block 3664 len 8192 data: 4 5
255/300 Read block 2132 len 40960
256/300 Write block 288 len 36864 data: 4 4 3 3 3 3 4 4 4
257/300 Write block 393 len 28672 data: 3 3 3 3 4 3 3
258/300 Read block 2342 len 32768
259/300 Flush
260/300 Write block 866 len 24576 data: 5 5 6 4 4 5
261/300 Read block 1028 len 36864
262/300 Read block 80 len 28672
263/300 Read block 926 len 16384
264/300 Read block 1733 len 4096
265/300 Read block 3461 len 20480
266/300 Read block 1884 len 24576
267/300 Read block 4677 len 28672
268/300 Read block 2017 len 28672
269/300 Read block 1544 len 28672
270/300 Read block 3208 len 16384
271/300 Write block 1444 len 4096 data: 2
272/300 Read block 2859 len 4096
273/300 Read block 4178 len 20480
274/300 Read block 1994 len 8192
275/300 Read block 4035 len 4096
276/300 Read block 3576 len 24576
277/300 Read block 4589 len 12288
278/300 Read block 205 len 28672
279/300 Write block 1554 len 36864 data: 3 3 3 4 4 4 3 2 2
280/300 Write block 1556 len 20480 data: 4 5 5 5 4
281/300 Write block 1434 len 12288 data: 2 2 2
282/300 Write block 860 len 40960 data: 3 4 4 3 3 3 6 6 7 5
283/300 Write block 830 len 40960 data: 4 4 5 4 4 4 3 2 2 2
284/300 Write block 4457 len 8192 data: 2 2
285/300 Write block 1529 len 24576 data: 2 3 4 4 3 3
286/300 Write block 1866 len 40960 data: 3 2 2 2 3 3 3 3 3 3
287/300 Read block 4713 len 8192
288/300 Write block 3287 len 8192 data: 3 2
289/300 Read block 3725 len 12288
290/300 Read block 3860 len 16384
291/300 Flush
292/300 Write block 1978 len 4096 data: 3
293/300 Flush
294/300 Read block 3797 len 16384
295/300 Read block 1401 len 32768
296/300 Write block 2773 len 4096 data: 4
297/300 Write block 4638 len 24576 data: 3 3 3 3 2 3
298/300 Write block 3090 len 32768 data: 2 4 4 4 4 4 3 3
299/300 Flush
300/300 Flush
301/300 Read block 3479 len 8192
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:38:27.364 INFO request completed, latency_us: 200, response_code: 204, uri: /start/cid/1, method: POST, req_id: fd56c5bd-ba34-4c12-9e20-b3a9856d9550, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3715 Acked Write 6 Done Sent Done false
3716 Acked Write 4 Done Sent Done false
3717 Acked Flush 0 Done Sent Done false
3718 Acked Read 1 Done Sent Done false
3719 Acked Read 4 Done Sent Done false
3720 Acked Flush 0 Done Sent Done false
3721 Acked Read 9 Done Sent Done false
3722 Acked Read 7 Done Sent Done false
3723 Acked Write 10 Done Sent Done false
3724 Acked Flush 0 Done Sent Done false
3725 Acked Read 1 Done Sent Done false
3726 Acked Flush 0 Done Sent Done false
3727 Acked Write 10 Done Sent Done false
3728 Acked Read 10 Done Sent Done false
3729 Acked Write 8 Done Sent Done false
3730 Acked Write 4 Done Sent Done false
3731 Acked Write 5 Done Sent Done false
3732 Acked Read 5 Done Sent Done false
3733 Acked Read 5 Done Sent Done false
3734 Acked Write 2 Done Sent Done false
3735 Acked Flush 0 Done Sent Done false
3736 Acked Write 5 Done Sent Done false
3737 Acked Read 7 Done Sent Done false
3738 Acked Read 3 Done Sent Done false
3739 Acked Read 1 Done Sent Done false
3740 Acked Read 2 Done Sent Done false
3741 Acked Write 5 Done Sent Done false
3742 Acked Write 3 Done Sent Done false
3743 Acked Flush 0 Done Sent Done false
3744 Acked Read 9 Done Sent Done false
3745 Acked Flush 0 Done Sent Done false
3746 Acked Flush 0 Done Sent Done false
3747 Acked Write 9 Done Sent Done false
3748 Acked Read 2 Done Sent Done false
3749 Acked Write 3 Done Sent Done false
3750 Acked Write 1 Done Sent Done false
3751 Acked Write 6 Done Sent Done false
3752 Acked Read 5 Done Sent Done false
3753 Acked Read 7 Done Sent Done false
3754 Acked Read 10 Done Sent Done false
3755 Acked Write 2 Done Sent Done false
3756 Acked Write 4 Done Sent Done false
3757 Acked Read 7 Done Sent Done false
3758 Acked Read 3 Done Sent Done false
3759 Acked Read 10 Done Sent Done false
3760 Acked Flush 0 Done Sent Done false
3761 Acked Write 3 Done Sent Done false
3762 Acked Read 6 Done Sent Done false
3763 Acked Flush 0 Done Sent Done false
3764 Acked Read 3 Done Sent Done false
3765 Acked Read 3 Done Sent Done false
3766 Acked Write 6 Done Sent Done false
3767 Acked Flush 0 Done Sent Done false
3768 Acked Read 3 Done Sent Done false
3769 Acked Flush 0 Done Sent Done false
3770 Acked Read 1 Done Sent Done false
3771 Acked Write 9 Done Sent Done false
3772 Acked Write 3 Done Sent Done false
3773 Acked Write 7 Done Sent Done false
3774 Acked Read 9 Done Sent Done false
3775 Acked Read 4 Done Sent Done false
3776 Acked Write 8 Done Sent Done false
3777 Acked Write 9 Done Sent Done false
3778 Acked Read 3 Done Sent Done false
3779 Acked Flush 0 Done Sent Done false
3780 Acked Read 7 Done Sent Done false
3781 Acked Write 2 Done Sent Done false
3782 Acked Read 10 Done Sent Done false
3783 Acked Write 7 Done Sent Done false
3784 Acked Read 2 Done Sent Done false
3785 Acked Write 3 Done Sent Done false
3786 Acked Read 1 Done Sent Done false
3787 Acked Read 2 Done Sent Done false
3788 Acked Read 9 Done Sent Done false
3789 Acked Write 3 Done Sent Done false
3790 Acked Write 10 Done Sent Done false
3791 Acked Read 3 Done Sent Done false
3792 Acked Write 2 Done Sent Done false
3793 Acked Read 3 Done Sent Done false
3794 Acked Read 10 Done Sent Done false
3795 Acked Write 6 Done Sent Done false
3796 Acked Read 5 Done Sent Done false
3797 Acked Read 6 Done Sent Done false
3798 Acked Read 10 Done Sent Done false
3799 Acked Read 4 Done Sent Done false
3800 Acked Read 8 Done Sent Done false
3801 Acked Read 10 Done Sent Done false
3802 Acked Read 1 Done Sent Done false
3803 Acked Write 5 Done Sent Done false
3804 Acked Read 4 Done Sent Done false
3805 Acked Write 7 Done Sent Done false
3806 Acked Write 6 Done Sent Done false
3807 Acked Read 6 Done Sent Done false
3808 Acked Read 10 Done Sent Done false
3809 Acked Write 2 Done Sent Done false
3810 Acked Read 2 Done Sent Done false
3811 Acked Flush 0 Done Sent Done false
3812 Acked Read 4 Done Sent Done false
3813 Acked Read 8 Done Sent Done false
3814 Acked Write 4 Done Sent Done false
3815 Acked Write 2 Done Sent Done false
3816 Acked Write 4 Done Sent Done false
3817 Acked Flush 0 Done Sent Done false
3818 Acked Read 8 Done Sent Done false
3819 Acked Read 9 Done Sent Done false
3820 Acked Write 10 Done Sent Done false
3821 Acked Write 5 Done Sent Done false
3822 Acked Write 3 Done Sent Done false
3823 Acked Read 8 Done Sent Done false
3824 Acked Read 6 Done Sent Done false
3825 Acked Read 10 Done Sent Done false
3826 Acked Read 4 Done Sent Done false
3827 Acked Read 2 Done Sent Done false
3828 Acked Read 10 Done Sent Done false
3829 Acked Write 3 Done Sent Done false
3830 Acked Flush 0 Done Sent Done false
3831 Acked Write 1 Done Sent Done false
3832 Acked Read 7 Done Sent Done false
3833 Acked Read 8 Done Sent Done false
3834 Acked Write 10 Done Sent Done false
3835 Acked Read 6 Done Sent Done false
3836 Acked Write 5 Done Sent Done false
3837 Acked Read 8 Done Sent Done false
3838 Acked Read 2 Done Sent Done false
3839 Acked Write 1 Done Sent Done false
3840 Acked Read 4 Done Sent Done false
3841 Acked Read 6 Done Sent Done false
3842 Acked Flush 0 Done Sent Done false
3843 Acked Flush 0 Done Sent Done false
3844 Acked Write 3 Done Sent Done false
3845 Acked Read 9 Done Sent Done false
3846 Acked Read 2 Done Sent Done false
3847 Acked Write 10 Done Sent Done false
3848 Acked Read 10 Done Sent Done false
3849 Acked Flush 0 Done Sent Done false
3850 Acked Read 6 Done Sent Done false
3851 Acked Read 3 Done Sent Done false
3852 Acked Flush 0 Done Sent Done false
3853 Acked Read 10 Done Sent Done false
3854 Acked Flush 0 Done Sent Done false
3855 Acked Flush 0 Done Sent Done false
3856 Acked Write 7 Done Sent Done false
3857 Acked Write 1 Done Sent Done false
3858 Acked Write 2 Done Sent Done false
3859 Acked Read 3 Done Sent Done false
3860 Acked Read 4 Done Sent Done false
3861 Acked Read 5 Done Sent Done false
3862 Acked Write 9 Done Sent Done false
3863 Acked Write 4 Done Sent Done false
3864 Acked Read 2 Done Sent Done false
3865 Acked Read 7 Done Sent Done false
3866 Acked Write 7 Done Sent Done false
3867 Acked Write 6 Done Sent Done false
3868 Acked Write 7 Done Sent Done false
3869 Acked Write 8 Done Sent Done false
3870 Acked Read 5 Done Sent Done false
3871 Acked Read 6 Done Sent Done false
3872 Acked Write 6 Done Sent Done false
3873 Acked Read 1 Done Sent Done false
3874 Acked Write 1 Done Sent Done false
3875 Acked Read 10 Done Sent Done false
3876 Acked Write 4 Done Sent Done false
3877 Acked Flush 0 Done Sent Done false
3878 Acked Read 6 Done Sent Done false
3879 Acked Write 8 Done Sent Done false
3880 Acked Read 9 Done Sent Done false
3881 Acked Read 5 Done Sent Done false
3882 Acked Write 1 Done Sent Done false
3883 Acked Flush 0 Done Sent Done false
3884 Acked Read 10 Done Sent Done false
3885 Acked Read 9 Done Sent Done false
3886 Acked Read 1 Done Sent Done false
3887 Acked Read 7 Done Sent Done false
3888 Acked Flush 0 Done Sent Done false
3889 Acked Write 6 Done Sent Done false
3890 Acked Read 4 Done Sent Done false
3891 Acked Read 2 Done Sent Done false
3892 Acked Write 3 Done Sent Done false
3893 Acked Read 8 Done Sent Done false
3894 Acked Write 1 Done Sent Done false
3895 Acked Write 3 Done Sent Done false
3896 Acked Read 7 Done Sent Done false
3897 Acked Read 2 Done Sent Done false
3898 Acked Write 6 Done Sent Done false
3899 Acked Write 7 Done Sent Done false
3900 Acked Read 2 Done Sent Done false
3901 Acked Read 2 Done Sent Done false
3902 Acked Flush 0 Done Sent Done false
3903 Acked Write 3 Done Sent Done false
3904 Acked Read 4 Done Sent Done false
3905 Acked Write 3 Done Sent Done false
3906 Acked Write 2 Done Sent Done false
3907 Acked Write 8 Done Sent Done false
3908 Acked Write 1 Done Sent Done false
3909 Acked Write 8 Done Sent Done false
3910 Acked Read 5 Done Sent Done false
3911 Acked Write 10 Done Sent Done false
3912 Acked Write 1 Done Sent Done false
3913 Acked Read 5 Done Sent Done false
3914 Acked Write 1 Done Sent Done false
3915 Acked Read 9 Done Sent Done false
3916 Acked Read 10 Done Sent Done false
3917 Acked Write 4 Done Sent Done false
3918 Acked Write 3 Done Sent Done false
3919 Acked Read 5 Done Sent Done false
3920 Acked Write 7 Done Sent Done false
3921 Acked Read 4 Done Sent Done false
3922 Acked Read 8 Done Sent Done false
3923 Acked Write 9 Done Sent Done false
3924 Acked Write 8 Done Sent Done false
3925 Acked Read 1 Done Sent Done false
3926 Acked Write 2 Done Sent Done false
3927 Acked Read 4 Done Sent Done false
3928 Acked Write 4 Done Sent Done false
3929 Acked Read 8 Done Sent Done false
3930 Acked Write 6 Done Sent Done false
3931 Acked Flush 0 Done Sent Done false
3932 Acked Write 10 Done Sent Done false
3933 Acked Write 5 Done Sent Done false
3934 Acked Read 1 Done Sent Done false
3935 Acked Write 4 Done Sent Done false
3936 Acked Read 5 Done Sent Done false
3937 Acked Read 1 Done Sent Done false
3938 Acked Read 1 Done Sent Done false
3939 Acked Flush 0 Done Sent Done false
3940 Acked Read 10 Done Sent Done false
3941 Acked Write 5 Done Sent Done false
3942 Acked Write 10 Done Sent Done false
3943 Acked Read 6 Done Sent Done false
3944 Acked Read 10 Done Sent Done false
3945 Acked Write 5 Done Sent Done false
3946 Acked Read 5 Done Sent Done false
3947 Acked Write 9 Done Sent Done false
3948 Acked Read 10 Done Sent Done false
3949 Acked Read 5 Done Sent Done false
3950 Acked Read 9 Done Sent Done false
3951 Acked Read 2 Done Sent Done false
3952 Acked Write 10 Done Sent Done false
3953 Acked Write 9 Done Sent Done false
3954 Acked Write 7 Done Sent Done false
3955 Acked Read 7 Done Sent Done false
3956 Acked Write 6 Done Sent Done false
3957 Acked Write 10 Done Sent Done false
3958 Acked Read 7 Done Sent Done false
3959 Acked Write 2 Done Sent Done false
3960 Acked Write 6 Done Sent Done false
3961 Acked Read 6 Done Sent Done false
3962 Acked Write 10 Done Sent Done false
3963 Acked Read 1 Done Sent Done false
3964 Acked Read 8 Done Sent Done false
3965 Acked Write 2 Done Sent Done false
3966 Acked Read 6 Done Sent Done false
3967 Acked Write 7 Done Sent Done false
3968 Acked Write 2 Done Sent Done false
3969 Acked Read 10 Done Sent Done false
3970 Acked Write 9 Done Sent Done false
3971 Acked Write 7 Done Sent Done false
3972 Acked Read 8 Done Sent Done false
3973 Acked Flush 0 Done Sent Done false
3974 Acked Write 6 Done Sent Done false
3975 Acked Read 9 Done Sent Done false
3976 Acked Read 7 Done Sent Done false
3977 Acked Read 4 Done Sent Done false
3978 Acked Read 1 Done Sent Done false
3979 Acked Read 5 Done Sent Done false
3980 Acked Read 6 Done Sent Done false
3981 Acked Read 7 Done Sent Done false
3982 Acked Read 7 Done Sent Done false
3983 Acked Read 7 Done Sent Done false
3984 Acked Read 4 Done Sent Done false
3985 Acked Write 1 Done Sent Done false
3986 Acked Read 1 Done Sent Done false
3987 Acked Read 5 Done Sent Done false
3988 Acked Read 2 Done Sent Done false
3989 Acked Read 1 Done Sent Done false
3990 Acked Read 6 Done Sent Done false
3991 Acked Read 3 Done Sent Done false
3992 Acked Read 7 Done Sent Done false
3993 Acked Write 9 Done Sent Done false
3994 Acked Write 5 Done Sent Done false
3995 Acked Write 3 Done Sent Done false
3996 Acked Write 10 Done Sent Done false
3997 Acked Write 10 Done Sent Done false
3998 Acked Write 2 Done Sent Done false
3999 Acked Write 6 Done Sent Done false
4000 Acked Write 10 Done Sent Done false
4001 Acked Read 2 Done Sent Done false
4002 Acked Write 2 Done Sent Done false
4003 Acked Read 3 Done Sent Done false
4004 Acked Read 4 Done Sent Done false
4005 Acked Flush 0 Done Sent Done false
4006 Acked Write 1 Done Sent Done false
4007 Acked Flush 0 Done Sent Done false
4008 Acked Read 4 Done Sent Done false
4009 Acked Read 8 Done Sent Done false
4010 Acked Write 1 Done Sent Done false
4011 Acked Write 6 Done Sent Done false
4012 Acked Write 8 Done Sent Done false
4013 Acked Flush 0 Done Sent Done false
4014 Acked Flush 0 Done Sent Done false
4015 Acked Read 2 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4014 3714 4014
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 4015 4014 4013 4012 4011
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1210)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3715 Acked Write 6 Done Sent Done false
3716 Acked Write 4 Done Sent Done false
3717 Acked Flush 0 Done Sent Done false
3718 Acked Read 1 Done Sent Done false
3719 Acked Read 4 Done Sent Done false
3720 Acked Flush 0 Done Sent Done false
3721 Acked Read 9 Done Sent Done false
3722 Acked Read 7 Done Sent Done false
3723 Acked Write 10 Done Sent Done false
3724 Acked Flush 0 Done Sent Done false
3725 Acked Read 1 Done Sent Done false
3726 Acked Flush 0 Done Sent Done false
3727 Acked Write 10 Done Sent Done false
3728 Acked Read 10 Done Sent Done false
3729 Acked Write 8 Done Sent Done false
3730 Acked Write 4 Done Sent Done false
3731 Acked Write 5 Done Sent Done false
3732 Acked Read 5 Done Sent Done false
3733 Acked Read 5 Done Sent Done false
3734 Acked Write 2 Done Sent Done false
3735 Acked Flush 0 Done Sent Done false
3736 Acked Write 5 Done Sent Done false
3737 Acked Read 7 Done Sent Done false
3738 Acked Read 3 Done Sent Done false
3739 Acked Read 1 Done Sent Done false
3740 Acked Read 2 Done Sent Done false
3741 Acked Write 5 Done Sent Done false
3742 Acked Write 3 Done Sent Done false
3743 Acked Flush 0 Done Sent Done false
3744 Acked Read 9 Done Sent Done false
3745 Acked Flush 0 Done Sent Done false
3746 Acked Flush 0 Done Sent Done false
3747 Acked Write 9 Done Sent Done false
3748 Acked Read 2 Done Sent Done false
3749 Acked Write 3 Done Sent Done false
3750 Acked Write 1 Done Sent Done false
3751 Acked Write 6 Done Sent Done false
3752 Acked Read 5 Done Sent Done false
3753 Acked Read 7 Done Sent Done false
3754 Acked Read 10 Done Sent Done false
3755 Acked Write 2 Done Sent Done false
3756 Acked Write 4 Done Sent Done false
3757 Acked Read 7 Done Sent Done false
3758 Acked Read 3 Done Sent Done false
3759 Acked Read 10 Done Sent Done false
3760 Acked Flush 0 Done Sent Done false
3761 Acked Write 3 Done Sent Done false
3762 Acked Read 6 Done Sent Done false
3763 Acked Flush 0 Done Sent Done false
3764 Acked Read 3 Done Sent Done false
3765 Acked Read 3 Done Sent Done false
3766 Acked Write 6 Done Sent Done false
3767 Acked Flush 0 Done Sent Done false
3768 Acked Read 3 Done Sent Done false
3769 Acked Flush 0 Done Sent Done false
3770 Acked Read 1 Done Sent Done false
3771 Acked Write 9 Done Sent Done false
3772 Acked Write 3 Done Sent Done false
3773 Acked Write 7 Done Sent Done false
3774 Acked Read 9 Done Sent Done false
3775 Acked Read 4 Done Sent Done false
3776 Acked Write 8 Done Sent Done false
3777 Acked Write 9 Done Sent Done false
3778 Acked Read 3 Done Sent Done false
3779 Acked Flush 0 Done Sent Done false
3780 Acked Read 7 Done Sent Done false
3781 Acked Write 2 Done Sent Done false
3782 Acked Read 10 Done Sent Done false
3783 Acked Write 7 Done Sent Done false
3784 Acked Read 2 Done Sent Done false
3785 Acked Write 3 Done Sent Done false
3786 Acked Read 1 Done Sent Done false
3787 Acked Read 2 Done Sent Done false
3788 Acked Read 9 Done Sent Done false
3789 Acked Write 3 Done Sent Done false
3790 Acked Write 10 Done Sent Done false
3791 Acked Read 3 Done Sent Done false
3792 Acked Write 2 Done Sent Done false
3793 Acked Read 3 Done Sent Done false
3794 Acked Read 10 Done Sent Done false
3795 Acked Write 6 Done Sent Done false
3796 Acked Read 5 Done Sent Done false
3797 Acked Read 6 Done Sent Done false
3798 Acked Read 10 Done Sent Done false
3799 Acked Read 4 Done Sent Done false
3800 Acked Read 8 Done Sent Done false
3801 Acked Read 10 Done Sent Done false
3802 Acked Read 1 Done Sent Done false
3803 Acked Write 5 Done Sent Done false
3804 Acked Read 4 Done Sent Done false
3805 Acked Write 7 Done Sent Done false
3806 Acked Write 6 Done Sent Done false
3807 Acked Read 6 Done Sent Done false
3808 Acked Read 10 Done Sent Done false
3809 Acked Write 2 Done Sent Done false
3810 Acked Read 2 Done Sent Done false
3811 Acked Flush 0 Done Sent Done false
3812 Acked Read 4 Done Sent Done false
3813 Acked Read 8 Done Sent Done false
3814 Acked Write 4 Done Sent Done false
3815 Acked Write 2 Done Sent Done false
3816 Acked Write 4 Done Sent Done false
3817 Acked Flush 0 Done Sent Done false
3818 Acked Read 8 Done Sent Done false
3819 Acked Read 9 Done Sent Done false
3820 Acked Write 10 Done Sent Done false
3821 Acked Write 5 Done Sent Done false
3822 Acked Write 3 Done Sent Done false
3823 Acked Read 8 Done Sent Done false
3824 Acked Read 6 Done Sent Done false
3825 Acked Read 10 Done Sent Done false
3826 Acked Read 4 Done Sent Done false
3827 Acked Read 2 Done Sent Done false
3828 Acked Read 10 Done Sent Done false
3829 Acked Write 3 Done Sent Done false
3830 Acked Flush 0 Done Sent Done false
3831 Acked Write 1 Done Sent Done false
3832 Acked Read 7 Done Sent Done false
3833 Acked Read 8 Done Sent Done false
3834 Acked Write 10 Done Sent Done false
3835 Acked Read 6 Done Sent Done false
3836 Acked Write 5 Done Sent Done false
3837 Acked Read 8 Done Sent Done false
3838 Acked Read 2 Done Sent Done false
3839 Acked Write 1 Done Sent Done false
3840 Acked Read 4 Done Sent Done false
3841 Acked Read 6 Done Sent Done false
3842 Acked Flush 0 Done Sent Done false
3843 Acked Flush 0 Done Sent Done false
3844 Acked Write 3 Done Sent Done false
3845 Acked Read 9 Done Sent Done false
3846 Acked Read 2 Done Sent Done false
3847 Acked Write 10 Done Sent Done false
3848 Acked Read 10 Done Sent Done false
3849 Acked Flush 0 Done Sent Done false
3850 Acked Read 6 Done Sent Done false
3851 Acked Read 3 Done Sent Done false
3852 Acked Flush 0 Done Sent Done false
3853 Acked Read 10 Done Sent Done false
3854 Acked Flush 0 Done Sent Done false
3855 Acked Flush 0 Done Sent Done false
3856 Acked Write 7 Done Sent Done false
3857 Acked Write 1 Done Sent Done false
3858 Acked Write 2 Done Sent Done false
3859 Acked Read 3 Done Sent Done false
3860 Acked Read 4 Done Sent Done false
3861 Acked Read 5 Done Sent Done false
3862 Acked Write 9 Done Sent Done false
3863 Acked Write 4 Done Sent Done false
3864 Acked Read 2 Done Sent Done false
3865 Acked Read 7 Done Sent Done false
3866 Acked Write 7 Done Sent Done false
3867 Acked Write 6 Done Sent Done false
3868 Acked Write 7 Done Sent Done false
3869 Acked Write 8 Done Sent Done false
3870 Acked Read 5 Done Sent Done false
3871 Acked Read 6 Done Sent Done false
3872 Acked Write 6 Done Sent Done false
3873 Acked Read 1 Done Sent Done false
3874 Acked Write 1 Done Sent Done false
3875 Acked Read 10 Done Sent Done false
3876 Acked Write 4 Done Sent Done false
3877 Acked Flush 0 Done Sent Done false
3878 Acked Read 6 Done Sent Done false
3879 Acked Write 8 Done Sent Done false
3880 Acked Read 9 Done Sent Done false
3881 Acked Read 5 Done Sent Done false
3882 Acked Write 1 Done Sent Done false
3883 Acked Flush 0 Done Sent Done false
3884 Acked Read 10 Done Sent Done false
3885 Acked Read 9 Done Sent Done false
3886 Acked Read 1 Done Sent Done false
3887 Acked Read 7 Done Sent Done false
3888 Acked Flush 0 Done Sent Done false
3889 Acked Write 6 Done Sent Done false
3890 Acked Read 4 Done Sent Done false
3891 Acked Read 2 Done Sent Done false
3892 Acked Write 3 Done Sent Done false
3893 Acked Read 8 Done Sent Done false
3894 Acked Write 1 Done Sent Done false
3895 Acked Write 3 Done Sent Done false
3896 Acked Read 7 Done Sent Done false
3897 Acked Read 2 Done Sent Done false
3898 Acked Write 6 Done Sent Done false
3899 Acked Write 7 Done Sent Done false
3900 Acked Read 2 Done Sent Done false
3901 Acked Read 2 Done Sent Done false
3902 Acked Flush 0 Done Sent Done false
3903 Acked Write 3 Done Sent Done false
3904 Acked Read 4 Done Sent Done false
3905 Acked Write 3 Done Sent Done false
3906 Acked Write 2 Done Sent Done false
3907 Acked Write 8 Done Sent Done false
3908 Acked Write 1 Done Sent Done false
3909 Acked Write 8 Done Sent Done false
3910 Acked Read 5 Done Sent Done false
3911 Acked Write 10 Done Sent Done false
3912 Acked Write 1 Done Sent Done false
3913 Acked Read 5 Done Sent Done false
3914 Acked Write 1 Done Sent Done false
3915 Acked Read 9 Done Sent Done false
3916 Acked Read 10 Done Sent Done false
3917 Acked Write 4 Done Sent Done false
3918 Acked Write 3 Done Sent Done false
3919 Acked Read 5 Done Sent Done false
3920 Acked Write 7 Done Sent Done false
3921 Acked Read 4 Done Sent Done false
3922 Acked Read 8 Done Sent Done false
3923 Acked Write 9 Done Sent Done false
3924 Acked Write 8 Done Sent Done false
3925 Acked Read 1 Done Sent Done false
3926 Acked Write 2 Done Sent Done false
3927 Acked Read 4 Done Sent Done false
3928 Acked Write 4 Done Sent Done false
3929 Acked Read 8 Done Sent Done false
3930 Acked Write 6 Done Sent Done false
3931 Acked Flush 0 Done Sent Done false
3932 Acked Write 10 Done Sent Done false
3933 Acked Write 5 Done Sent Done false
3934 Acked Read 1 Done Sent Done false
3935 Acked Write 4 Done Sent Done false
3936 Acked Read 5 Done Sent Done false
3937 Acked Read 1 Done Sent Done false
3938 Acked Read 1 Done Sent Done false
3939 Acked Flush 0 Done Sent Done false
3940 Acked Read 10 Done Sent Done false
3941 Acked Write 5 Done Sent Done false
3942 Acked Write 10 Done Sent Done false
3943 Acked Read 6 Done Sent Done false
3944 Acked Read 10 Done Sent Done false
3945 Acked Write 5 Done Sent Done false
3946 Acked Read 5 Done Sent Done false
3947 Acked Write 9 Done Sent Done false
3948 Acked Read 10 Done Sent Done false
3949 Acked Read 5 Done Sent Done false
3950 Acked Read 9 Done Sent Done false
3951 Acked Read 2 Done Sent Done false
3952 Acked Write 10 Done Sent Done false
3953 Acked Write 9 Done Sent Done false
3954 Acked Write 7 Done Sent Done false
3955 Acked Read 7 Done Sent Done false
3956 Acked Write 6 Done Sent Done false
3957 Acked Write 10 Done Sent Done false
3958 Acked Read 7 Done Sent Done false
3959 Acked Write 2 Done Sent Done false
3960 Acked Write 6 Done Sent Done false
3961 Acked Read 6 Done Sent Done false
3962 Acked Write 10 Done Sent Done false
3963 Acked Read 1 Done Sent Done false
3964 Acked Read 8 Done Sent Done false
3965 Acked Write 2 Done Sent Done false
3966 Acked Read 6 Done Sent Done false
3967 Acked Write 7 Done Sent Done false
3968 Acked Write 2 Done Sent Done false
3969 Acked Read 10 Done Sent Done false
3970 Acked Write 9 Done Sent Done false
3971 Acked Write 7 Done Sent Done false
3972 Acked Read 8 Done Sent Done false
3973 Acked Flush 0 Done Sent Done false
3974 Acked Write 6 Done Sent Done false
3975 Acked Read 9 Done Sent Done false
3976 Acked Read 7 Done Sent Done false
3977 Acked Read 4 Done Sent Done false
3978 Acked Read 1 Done Sent Done false
3979 Acked Read 5 Done Sent Done false
3980 Acked Read 6 Done Sent Done false
3981 Acked Read 7 Done Sent Done false
3982 Acked Read 7 Done Sent Done false
3983 Acked Read 7 Done Sent Done false
3984 Acked Read 4 Done Sent Done false
3985 Acked Write 1 Done Sent Done false
3986 Acked Read 1 Done Sent Done false
3987 Acked Read 5 Done Sent Done false
3988 Acked Read 2 Done Sent Done false
3989 Acked Read 1 Done Sent Done false
3990 Acked Read 6 Done Sent Done false
3991 Acked Read 3 Done Sent Done false
3992 Acked Read 7 Done Sent Done false
3993 Acked Write 9 Done Sent Done false
3994 Acked Write 5 Done Sent Done false
3995 Acked Write 3 Done Sent Done false
3996 Acked Write 10 Done Sent Done false
3997 Acked Write 10 Done Sent Done false
3998 Acked Write 2 Done Sent Done false
3999 Acked Write 6 Done Sent Done false
4000 Acked Write 10 Done Sent Done false
4001 Acked Read 2 Done Sent Done false
4002 Acked Write 2 Done Sent Done false
4003 Acked Read 3 Done Sent Done false
4004 Acked Read 4 Done Sent Done false
4005 Acked Flush 0 Done Sent Done false
4006 Acked Write 1 Done Sent Done false
4007 Acked Flush 0 Done Sent Done false
4008 Acked Read 4 Done Sent Done false
4009 Acked Read 8 Done Sent Done false
4010 Acked Write 1 Done Sent Done false
4011 Acked Write 6 Done Sent Done false
4012 Acked Write 8 Done Sent Done false
4013 Acked Flush 0 Done Sent Done false
4014 Acked Flush 0 Done Sent Done false
4015 Acked Read 2 Done Sent Done false
4016 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4016 3714 4016
Downstairs last five completed: 3714 3713 3712 3711 3710
Upstairs last five completed: 4016 4015 4014 4013 4012
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.132861327Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35527)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.132984522Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.133901975Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.134488847Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.134512564Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 3714","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.134526902Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 3714","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.134788692Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 3714","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.134821568Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 3714","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:33.135040233Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4016 4015 4014 4013 4012
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:38:35.369 [8830] Got stop action so:false kr:false
INFO request completed, latency_us: 428, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f40ac462-28d4-4bca-a934-e9b5c95475ba, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:38:35.370 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7eb26a11-abda-4c83-9f62-ce86232cb9f5, remote_addr: 127.0.0.1:55047, local_addr: 127{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:35.370435952Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
.0.0.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:35.370504154Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:35.370534411Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:35.370583065Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:38:39.372 INFO request completed, latency_us: 310, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6714c97f-9527-4f5e-9bbc-f600b6b72a76, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2041 len 12288 data: 2 3 3
002/300 Write block 2299 len 28672 data: 3 3 3 4 4 4 4
003/300 Read block 2457 len 16384
004/300 Write block 3959 len 40960 data: 4 3 3 3 4 3 4 3 3 3
005/300 Read block 1493 len 8192
006/300 Write block 3103 len 28672 data: 3 3 3 3 3 3 2
007/300 Read block 1019 len 36864
008/300 Read block 416 len 28672
009/300 Read block 4022 len 32768
010/300 Write block 1949 len 20480 data: 2 3 3 4 4
011/300 Flush
012/300 Read block 3237 len 20480
013/300 Write block 1225 len 16384 data: 4 3 2 2
014/300 Read block 1717 len 8192
015/300 Flush
016/300 Read block 136 len 4096
017/300 Write block 1873 len 16384 data: 4 4 4 3
018/300 Read block 148 len 40960
019/300 Read block 3739 len 16384
020/300 Read block 3812 len 12288
021/300 Write block 1920 len 24576 data: 2 2 2 3 4 4
022/300 Write block 1815 len 4096 data: 4
023/300 Write block 281 len 32768 data: 9 7 5 4 4 4 3 5
024/300 Read block 4209 len 20480
025/300 Read block 1045 len 12288
026/300 Write block 842 len 4096 data: 2
027/300 Read block 2091 len 36864
028/300 Write block 3696 len 32768 data: 3 3 2 2 2 2 2 3
029/300 Read block 1254 len 40960
030/300 Read block 2690 len 32768
031/300 Read block 4134 len 20480
032/300 Read block 4150 len 12288
033/300 Read block 331 len 4096
034/300 Write block 1735 len 28672 data: 3 3 2 2 2 2 2
035/300 Flush
036/300 Read block 1119 len 4096
037/300 Write block 1996 len 32768 data: 4 4 4 4 2 2 2 2
038/300 Write block 843 len 16384 data: 2 2 2 3
039/300 Read block 3491 len 4096
040/300 Read block 3553 len 24576
041/300 Write block 54 len 28672 data: 3 2 2 2 3 3 4
042/300 Read block 2601 len 16384
043/300 Read block 2162 len 16384
044/300 Read block 3036 len 16384
045/300 Write block 3474 len 40960 data: 2 2 2 2 2 3 3 3 3 2
046/300 Write block 2234 len 4096 data: 3
047/300 Write block 4347 len 36864 data: 3 3 3 3 3 3 3 2 3
048/300 Read block 2867 len 40960
049/300 Read block 1148 len 40960
050/300 Read block 3476 len 4096
051/300 Read block 4845 len 40960
052/300 Write block 1203 len 28672 data: 3 3 3 3 4 3 3
053/300 Read block 3738 len 40960
054/300 Read block 4115 len 36864
055/300 Read block 2893 len 12288
056/300 Read block 1123 len 36864
057/300 Read block 2774 len 12288
058/300 Read block 3859 len 32768
059/300 Write block 3869 len 24576 data: 3 3 2 3 4 4
060/300 Read block 3330 len 12288
061/300 Write block 2288 len 20480 data: 7 6 6 4 4
062/300 Read block 2956 len 40960
063/300 Write block 3711 len 16384 data: 5 4 4 4
064/300 Write block 1033 len 28672 data: 6 6 5 5 4 2 2
065/300 Read block 3176 len 32768
066/300 Read block 2526 len 24576
067/300 Write block 2087 len 4096 data: 2
068/300 Write block 179 len 32768 data: 3 3 3 3 3 4 4 4
069/300 Write block 2308 len 12288 data: 3 3 3
070/300 Read block 2820 len 28672
071/300 Read block 558 len 20480
072/300 Read block 952 len 28672
073/300 Read block 698 len 20480
074/300 Write block 2659 len 36864 data: 5 3 3 2 4 4 4 3 2
075/300 Write block 2633 len 4096 data: 3
076/300 Read block 724 len 32768
077/300 Read block 2563 len 24576
078/300 Flush
079/300 Read block 1226 len 20480
080/300 Flush
081/300 Read block 497 len 16384
082/300 Read block 1501 len 8192
083/300 Read block 1292 len 8192
084/300 Flush
085/300 Write block 259 len 20480 data: 3 3 3 3 3
086/300 Read block 1978 len 16384
087/300 Write block 4223 len 12288 data: 2 3 3
088/300 Write block 749 len 16384 data: 3 2 2 2
089/300 Read block 3709 len 16384
090/300 Read block 3432 len 40960
091/300 Read block 2024 len 4096
092/300 Write block 3292 len 32768 data: 4 5 5 5 5 3 3 2
093/300 Write block 1451 len 36864 data: 4 4 4 4 3 5 5 6 4
094/300 Read block 2056 len 4096
095/300 Read block 4607 len 8192
096/300 Write block 2307 len 36864 data: 3 4 4 4 3 3 2 2 2
097/300 Read block 3846 len 20480
098/300 Write block 1049 len 8192 data: 3 3
099/300 Write block 4700 len 16384 data: 3 3 2 2
100/300 Read block 3013 len 40960
101/300 Write block 1849 len 4096 data: 4
102/300 Write block 3405 len 28672 data: 4 3 3 4 4 3 2
103/300 Flush
104/300 Flush
105/300 Read block 902 len 24576
106/300 Write block 4497 len 32768 data: 2 3 3 3 2 3 3 3
107/300 Read block 4803 len 36864
108/300 Read block 779 len 36864
109/300 Write block 2538 len 8192 data: 3 3
110/300 Flush
111/300 Read block 2776 len 24576
112/300 Read block 1568 len 8192
113/300 Read block 1355 len 16384
114/300 Write block 3328 len 4096 data: 3
115/300 Read block 2086 len 16384
116/300 Flush
117/300 Write block 3965 len 28672 data: 5 4 4 4 3 3 3
118/300 Read block 353 len 4096
119/300 Write block 2171 len 32768 data: 4 3 4 4 5 5 5 5
120/300 Write block 819 len 4096 data: 3
121/300 Read block 851 len 40960
122/300 Read block 968 len 8192
123/300 Flush
124/300 Write block 1965 len 36864 data: 2 3 3 3 3 3 2 2 2
125/300 Write block 1553 len 28672 data: 3 4 4 5 6 6 6
126/300 Read block 4964 len 36864
127/300 Write block 320 len 40960 data: 5 5 3 3 3 3 3 3 3 3
128/300 Write block 2298 len 12288 data: 2 4 4
129/300 Write block 3330 len 12288 data: 4 4 5
130/300 Write block 1890 len 4096 data: 4
131/300 Flush
132/300 Write block 3479 len 32768 data: 4 4 4 4 3 2 2 2
133/300 Read block 3874 len 20480
134/300 Write block 124 len 4096 data: 5
135/300 Flush
136/300 Read block 4441 len 32768
137/300 Write block 3768 len 36864 data: 5 4 4 4 4 6 6 6 6
138/300 Read block 3736 len 24576
139/300 Read block 2633 len 28672
140/300 Read block 664 len 32768
141/300 Write block 2498 len 20480 data: 3 3 2 2 2
142/300 Read block 642 len 40960
143/300 Read block 65 len 40960
144/300 Read block 841 len 8192
145/300 Write block 494 len 24576 data: 3 3 3 4 5 5
146/300 Write block 2353 len 4096 data: 5
147/300 Read block 3097 len 28672
148/300 Flush
149/300 Write block 1019 len 16384 data: 5 5 4 2
150/300 Read block 3588 len 16384
151/300 Write block 2604 len 16384 data: 3 3 3 3
152/300 Write block 3817 len 12288 data: 2 2 2
153/300 Write block 1515 len 12288 data: 2 2 2
154/300 Read block 3051 len 36864
155/300 Flush
156/300 Write block 1197 len 40960 data: 3 3 3 2 3 3 4 4 4 4
157/300 Write block 48 len 20480 data: 4 4 4 3 3
158/300 Read block 1060 len 8192
159/300 Read block 2481 len 40960
160/300 Read block 3636 len 24576
161/300 Flush
162/300 Write block 1216 len 20480 data: 3 3 3 4 3
163/300 Flush
164/300 Write block 4680 len 12288 data: 3 4 3
165/300 Read block 3802 len 24576
166/300 Read block 4928 len 24576
167/300 Read block 3168 len 24576
168/300 Read block 3035 len 32768
169/300 Read block 3232 len 40960
170/300 Read block 100 len 12288
171/300 Read block 2772 len 16384
172/300 Write block 1068 len 4096 data: 3
173/300 Write block 2918 len 12288 data: 2 2 2
174/300 Read block 4057 len 20480
175/300 Read block 1329 len 36864
176/300 Read block 1473 len 12288
177/300 Read block 1489 len 32768
178/300 Write block 3222 len 36864 data: 5 5 5 4 2 2 2 2 3
179/300 Write block 4101 len 28672 data: 3 3 4 3 3 3 4
180/300 Write block 3829 len 20480 data: 4 4 3 3 3
181/300 Read block 657 len 28672
182/300 Read block 560 len 40960
183/300 Write block 1454 len 32768 data: 5 4 6 6 7 5 4 3
184/300 Read block 3439 len 36864
185/300 Write block 2388 len 20480 data: 4 4 5 6 4
186/300 Write block 602 len 8192 data: 4 5
187/300 Read block 3605 len 4096
188/300 Read block 729 len 36864
189/300 Read block 3420 len 28672
190/300 Read block 4960 len 40960
191/300 Read block 4973 len 32768
192/300 Write block 4080 len 16384 data: 2 2 2 2
193/300 Read block 3707 len 36864
194/300 Read block 1546 len 8192
195/300 Write block 4730 len 32768 data: 4 4 4 5 4 5 5 5
196/300 Write block 2586 len 32768 data: 2 2 2 2 2 2 2 2
197/300 Flush
198/300 Write block 4979 len 4096 data: 2
199/300 Flush
200/300 Read block 1208 len 24576
201/300 Flush
202/300 Flush
203/300 Write block 4829 len 40960 data: 4 4 3 3 3 2 2 2 2 3
204/300 Write block 2793 len 8192 data: 4 5
205/300 Read block 4295 len 12288
206/300 Write block 578 len 24576 data: 4 4 4 4 4 5
207/300 Read block 4556 len 40960
208/300 Read block 1952 len 32768
209/300 Read block 2018 len 36864
210/300 Read block 2880 len 20480
211/300 Read block 3240 len 36864
212/300 Write block 228 len 16384 data: 3 3 2 2
213/300 Read block 4840 len 12288
214/300 Read block 2021 len 24576
215/300 Write block 1614 len 20480 data: 2 2 3 3 3
216/300 Read block 2641 len 12288
217/300 Read block 4979 len 32768
218/300 Write block 901 len 16384 data: 2 2 2 3
219/300 Write block 4380 len 32768 data: 3 3 3 4 3 3 3 3
220/300 Read block 3886 len 20480
221/300 Flush
222/300 Write block 860 len 20480 data: 4 5 5 4 4
223/300 Read block 2932 len 36864
224/300 Read block 2991 len 40960
225/300 Read block 3604 len 12288
226/300 Write block 170 len 40960 data: 5 5 3 3 3 3 4 4 3 4
227/300 Read block 2232 len 8192
228/300 Read block 4174 len 40960
229/300 Read block 2957 len 16384
230/300 Write block 2642 len 40960 data: 5 4 4 3 2 2 2 3 3 5
231/300 Write block 1404 len 24576 data: 3 5 5 5 4 3
232/300 Read block 2944 len 8192
233/300 Write block 1193 len 32768 data: 3 3 3 5 4 4 4 3
234/300 Read block 344 len 20480
235/300 Read block 861 len 36864
236/300 Read block 4946 len 16384
237/300 Read block 2624 len 4096
238/300 Write block 1962 len 32768 data: 2 2 2 3 4 4 4 4
239/300 Write block 776 len 28672 data: 3 3 3 2 2 3 3
240/300 Read block 1920 len 16384
241/300 Write block 2846 len 4096 data: 5
242/300 Flush
243/300 Read block 851 len 36864
244/300 Write block 1903 len 20480 data: 2 3 3 3 3
245/300 Read block 1846 len 24576
246/300 Read block 959 len 36864
247/300 Write block 2669 len 28672 data: 2 2 2 3 4 4 4
248/300 Write block 4718 len 4096 data: 5
249/300 Read block 3058 len 32768
250/300 Read block 453 len 24576
251/300 Flush
252/300 Read block 3421 len 8192
253/300 Read block 2462 len 36864
254/300 Write block 3140 len 4096 data: 8
255/300 Write block 683 len 16384 data: 3 3 2 2
256/300 Write block 3616 len 36864 data: 3 3 3 3 3 4 4 4 4
257/300 Write block 1312 len 40960 data: 3 2 4 4 4 4 4 4 4 4
258/300 Read block 4246 len 24576
259/300 Write block 4072 len 8192 data: 3 3
260/300 Read block 3955 len 8192
261/300 Write block 1548 len 16384 data: 3 3 3 4
262/300 Write block 626 len 28672 data: 2 2 2 2 3 2 2
263/300 Write block 3915 len 36864 data: 2 2 3 4 4 3 4 4 3
264/300 Write block 870 len 32768 data: 5 6 5 6 4 3 3 3
265/300 Read block 3583 len 36864
266/300 Write block 4475 len 16384 data: 4 4 4 4
267/300 Write block 3425 len 12288 data: 4 5 4
268/300 Write block 2862 len 36864 data: 4 4 4 4 4 3 2 2 3
269/300 Read block 666 len 8192
270/300 Read block 4002 len 16384
271/300 Read block 696 len 4096
272/300 Write block 2346 len 28672 data: 4 4 4 3 4 4 4
273/300 Read block 1198 len 24576
274/300 Write block 1106 len 28672 data: 4 3 2 2 3 4 3
275/300 Read block 3564 len 40960
276/300 Write block 183 len 4096 data: 4
277/300 Read block 3075 len 8192
278/300 Read block 170 len 36864
279/300 Read block 3410 len 40960
280/300 Read block 3250 len 32768
281/300 Read block 1619 len 28672
282/300 Write block 4923 len 12288 data: 5 4 4
283/300 Write block 3426 len 36864 data: 6 5 4 4 4 4 3 3 3
284/300 Flush
285/300 Flush
286/300 Write block 2011 len 12288 data: 4 3 4
287/300 Write block 1627 len 4096 data: 6
288/300 Read block 760 len 20480
289/300 Write block 704 len 20480 data: 2 3 3 3 4
290/300 Write block 2050 len 28672 data: 3 3 2 2 2 2 2
291/300 Flush
292/300 Write block 3171 len 36864 data: 3 4 4 3 3 5 5 5 5
293/300 Flush
294/300 Read block 2133 len 16384
295/300 Read block 4974 len 8192
296/300 Write block 1364 len 32768 data: 5 5 5 5 3 3 4 3
297/300 Read block 3596 len 20480
298/300 Read block 3302 len 24576
299/300 Flush
300/300 Write block 2751 len 24576 data: 5 4 4 3 3 3
301/300 Write block 2281 len 8192 data: 2 2
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:38:39.615 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/2, method: POST, req_id: 00eee15a-2609-4943-be28-6e70ad9a4454, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Write 3 Done Done Sent false
4018 Acked Write 7 Done Done Sent false
4019 Acked Read 4 Done Done Sent false
4020 Acked Write 10 Done Done Sent false
4021 Acked Read 2 Done Done Sent false
4022 Acked Write 7 Done Done Sent false
4023 Acked Read 9 Done Done Sent false
4024 Acked Read 7 Done Done Sent false
4025 Acked Read 8 Done Done Sent false
4026 Acked Write 5 Done Done Sent false
4027 Acked Flush 0 Done Done Sent false
4028 Acked Read 5 Done Done Sent false
4029 Acked Write 4 Done Done Sent false
4030 Acked Read 2 Done Done Sent false
4031 Acked Flush 0 Done Done Sent false
4032 Acked Read 1 Done Done Sent false
4033 Acked Write 4 Done Done Sent false
4034 Acked Read 10 Done Done Sent false
4035 Acked Read 4 Done Done Sent false
4036 Acked Read 3 Done Done Sent false
4037 Acked Write 6 Done Done Sent false
4038 Acked Write 1 Done Done Sent false
4039 Acked Write 8 Done Done Sent false
4040 Acked Read 5 Done Done Sent false
4041 Acked Read 3 Done Done Sent false
4042 Acked Write 1 Done Done Sent false
4043 Acked Read 9 Done Done Sent false
4044 Acked Write 8 Done Done Sent false
4045 Acked Read 10 Done Done Sent false
4046 Acked Read 8 Done Done Sent false
4047 Acked Read 5 Done Done Sent false
4048 Acked Read 3 Done Done Sent false
4049 Acked Read 1 Done Done Sent false
4050 Acked Write 7 Done Done Sent false
4051 Acked Flush 0 Done Done Sent false
4052 Acked Read 1 Done Done Sent false
4053 Acked Write 8 Done Done Sent false
4054 Acked Write 4 Done Done Sent false
4055 Acked Read 1 Done Done Sent false
4056 Acked Read 6 Done Done Sent false
4057 Acked Write 7 Done Done Sent false
4058 Acked Read 4 Done Done Sent false
4059 Acked Read 4 Done Done Sent false
4060 Acked Read 4 Done Done Sent false
4061 Acked Write 10 Done Done Sent false
4062 Acked Write 1 Done Done Sent false
4063 Acked Write 9 Done Done Sent false
4064 Acked Read 10 Done Done Sent false
4065 Acked Read 10 Done Done Sent false
4066 Acked Read 1 Done Done Sent false
4067 Acked Read 10 Done Done Sent false
4068 Acked Write 7 Done Done Sent false
4069 Acked Read 10 Done Done Sent false
4070 Acked Read 9 Done Done Sent false
4071 Acked Read 3 Done Done Sent false
4072 Acked Read 9 Done Done Sent false
4073 Acked Read 3 Done Done Sent false
4074 Acked Read 8 Done Done Sent false
4075 Acked Write 6 Done Done Sent false
4076 Acked Read 3 Done Done Sent false
4077 Acked Write 5 Done Done Sent false
4078 Acked Read 10 Done Done Sent false
4079 Acked Write 4 Done Done Sent false
4080 Acked Write 7 Done Done Sent false
4081 Acked Read 8 Done Done Sent false
4082 Acked Read 6 Done Done Sent false
4083 Acked Write 1 Done Done Sent false
4084 Acked Write 8 Done Done Sent false
4085 Acked Write 3 Done Done Sent false
4086 Acked Read 7 Done Done Sent false
4087 Acked Read 5 Done Done Sent false
4088 Acked Read 7 Done Done Sent false
4089 Acked Read 5 Done Done Sent false
4090 Acked Write 9 Done Done Sent false
4091 Acked Write 1 Done Done Sent false
4092 Acked Read 8 Done Done Sent false
4093 Acked Read 6 Done Done Sent false
4094 Acked Flush 0 Done Done Sent false
4095 Acked Read 5 Done Done Sent false
4096 Acked Flush 0 Done Done Sent false
4097 Acked Read 4 Done Done Sent false
4098 Acked Read 2 Done Done Sent false
4099 Acked Read 2 Done Done Sent false
4100 Acked Flush 0 Done Done Sent false
4101 Acked Write 5 Done Done Sent false
4102 Acked Read 4 Done Done Sent false
4103 Acked Write 3 Done Done Sent false
4104 Acked Write 4 Done Done Sent false
4105 Acked Read 4 Done Done Sent false
4106 Acked Read 10 Done Done Sent false
4107 Acked Read 1 Done Done Sent false
4108 Acked Write 8 Done Done Sent false
4109 Acked Write 9 Done Done Sent false
4110 Acked Read 1 Done Done Sent false
4111 Acked Read 2 Done Done Sent false
4112 Acked Write 9 Done Done Sent false
4113 Acked Read 5 Done Done Sent false
4114 Acked Write 2 Done Done Sent false
4115 Acked Write 4 Done Done Sent false
4116 Acked Read 10 Done Done Sent false
4117 Acked Write 1 Done Done Sent false
4118 Acked Write 7 Done Done Sent false
4119 Acked Flush 0 Done Done Sent false
4120 Acked Flush 0 Done Done Sent false
4121 Acked Read 6 Done Done Sent false
4122 Acked Write 8 Done Done Sent false
4123 Acked Read 9 Done Done Sent false
4124 Acked Read 9 Done Done Sent false
4125 Acked Write 2 Done Done Sent false
4126 Acked Flush 0 Done Done Sent false
4127 Acked Read 6 Done Done Sent false
4128 Acked Read 2 Done Done Sent false
4129 Acked Read 4 Done Done Sent false
4130 Acked Write 1 Done Done Sent false
4131 Acked Read 4 Done Done Sent false
4132 Acked Flush 0 Done Done Sent false
4133 Acked Write 7 Done Done Sent false
4134 Acked Read 1 Done Done Sent false
4135 Acked Write 8 Done Done Sent false
4136 Acked Write 1 Done Done Sent false
4137 Acked Read 10 Done Done Sent false
4138 Acked Read 2 Done Done Sent false
4139 Acked Flush 0 Done Done Sent false
4140 Acked Write 9 Done Done Sent false
4141 Acked Write 7 Done Done Sent false
4142 Acked Read 9 Done Done Sent false
4143 Acked Write 10 Done Done Sent false
4144 Acked Write 3 Done Done Sent false
4145 Acked Write 3 Done Done Sent false
4146 Acked Write 1 Done Done Sent false
4147 Acked Flush 0 Done Done Sent false
4148 Acked Write 8 Done Done Sent false
4149 Acked Read 5 Done Done Sent false
4150 Acked Write 1 Done Done Sent false
4151 Acked Flush 0 Done Done Sent false
4152 Acked Read 8 Done Done Sent false
4153 Acked Write 9 Done Done Sent false
4154 Acked Read 6 Done Done Sent false
4155 Acked Read 7 Done Done Sent false
4156 Acked Read 8 Done Done Sent false
4157 Acked Write 5 Done Done Sent false
4158 Acked Read 10 Done Done Sent false
4159 Acked Read 10 Done Done Sent false
4160 Acked Read 2 Done Done Sent false
4161 Acked Write 6 Done Done Sent false
4162 Acked Write 1 Done Done Sent false
4163 Acked Read 7 Done Done Sent false
4164 Acked Flush 0 Done Done Sent false
4165 Acked Write 4 Done Done Sent false
4166 Acked Read 4 Done Done Sent false
4167 Acked Write 4 Done Done Sent false
4168 Acked Write 3 Done Done Sent false
4169 Acked Write 3 Done Done Sent false
4170 Acked Read 9 Done Done Sent false
4171 Acked Flush 0 Done Done Sent false
4172 Acked Write 10 Done Done Sent false
4173 Acked Write 5 Done Done Sent false
4174 Acked Read 2 Done Done Sent false
4175 Acked Read 10 Done Done Sent false
4176 Acked Read 6 Done Done Sent false
4177 Acked Flush 0 Done Done Sent false
4178 Acked Write 5 Done Done Sent false
4179 Acked Flush 0 Done Done Sent false
4180 Acked Write 3 Done Done Sent false
4181 Acked Read 6 Done Done Sent false
4182 Acked Read 6 Done Done Sent false
4183 Acked Read 6 Done Done Sent false
4184 Acked Read 8 Done Done Sent false
4185 Acked Read 10 Done Done Sent false
4186 Acked Read 3 Done Done Sent false
4187 Acked Read 4 Done Done Sent false
4188 Acked Write 1 Done Done Sent false
4189 Acked Write 3 Done Done Sent false
4190 Acked Read 5 Done Done Sent false
4191 Acked Read 9 Done Done Sent false
4192 Acked Read 3 Done Done Sent false
4193 Acked Read 8 Done Done Sent false
4194 Acked Write 9 Done Done Sent false
4195 Acked Write 7 Done Done Sent false
4196 Acked Write 5 Done Done Sent false
4197 Acked Read 7 Done Done Sent false
4198 Acked Read 10 Done Done Sent false
4199 Acked Write 8 Done Done Sent false
4200 Acked Read 9 Done Done Sent false
4201 Acked Write 5 Done Done Sent false
4202 Acked Write 2 Done Done Sent false
4203 Acked Read 1 Done Done Sent false
4204 Acked Read 9 Done Done Sent false
4205 Acked Read 7 Done Done Sent false
4206 Acked Read 10 Done Done Sent false
4207 Acked Read 8 Done Done Sent false
4208 Acked Write 4 Done Done Sent false
4209 Acked Read 9 Done Done Sent false
4210 Acked Read 2 Done Done Sent false
4211 Acked Write 8 Done Done Sent false
4212 Acked Write 8 Done Done Sent false
4213 Acked Flush 0 Done Done Sent false
4214 Acked Write 1 Done Done Sent false
4215 Acked Flush 0 Done Done Sent false
4216 Acked Read 6 Done Done Sent false
4217 Acked Flush 0 Done Done Sent false
4218 Acked Flush 0 Done Done Sent false
4219 Acked Write 10 Done Done Sent false
4220 Acked Write 2 Done Done Sent false
4221 Acked Read 3 Done Done Sent false
4222 Acked Write 6 Done Done Sent false
4223 Acked Read 10 Done Done Sent false
4224 Acked Read 8 Done Done Sent false
4225 Acked Read 9 Done Done Sent false
4226 Acked Read 5 Done Done Sent false
4227 Acked Read 9 Done Done Sent false
4228 Acked Write 4 Done Done Sent false
4229 Acked Read 3 Done Done Sent false
4230 Acked Read 6 Done Done Sent false
4231 Acked Write 5 Done Done Sent false
4232 Acked Read 3 Done Done Sent false
4233 Acked Read 8 Done Done Sent false
4234 Acked Write 4 Done Done Sent false
4235 Acked Write 8 Done Done Sent false
4236 Acked Read 5 Done Done Sent false
4237 Acked Flush 0 Done Done Sent false
4238 Acked Write 5 Done Done Sent false
4239 Acked Read 9 Done Done Sent false
4240 Acked Read 10 Done Done Sent false
4241 Acked Read 3 Done Done Sent false
4242 Acked Write 10 Done Done Sent false
4243 Acked Read 2 Done Done Sent false
4244 Acked Read 10 Done Done Sent false
4245 Acked Read 4 Done Done Sent false
4246 Acked Write 10 Done Done Sent false
4247 Acked Write 6 Done Done Sent false
4248 Acked Read 2 Done Done Sent false
4249 Acked Write 8 Done Done Sent false
4250 Acked Read 5 Done Done Sent false
4251 Acked Read 9 Done Done Sent false
4252 Acked Read 4 Done Done Sent false
4253 Acked Read 1 Done Done Sent false
4254 Acked Write 8 Done Done Sent false
4255 Acked Write 7 Done Done Sent false
4256 Acked Read 4 Done Done Sent false
4257 Acked Write 1 Done Done Sent false
4258 Acked Flush 0 Done Done Sent false
4259 Acked Read 9 Done Done Sent false
4260 Acked Write 5 Done Done Sent false
4261 Acked Read 6 Done Done Sent false
4262 Acked Read 9 Done Done Sent false
4263 Acked Write 7 Done Done Sent false
4264 Acked Write 1 Done Done Sent false
4265 Acked Read 8 Done Done Sent false
4266 Acked Read 6 Done Done Sent false
4267 Acked Flush 0 Done Done Sent false
4268 Acked Read 2 Done Done Sent false
4269 Acked Read 9 Done Done Sent false
4270 Acked Write 1 Done Done Sent false
4271 Acked Write 4 Done Done Sent false
4272 Acked Write 9 Done Done Sent false
4273 Acked Write 10 Done Done Sent false
4274 Acked Read 6 Done Done Sent false
4275 Acked Write 2 Done Done Sent false
4276 Acked Read 2 Done Done Sent false
4277 Acked Write 4 Done Done Sent false
4278 Acked Write 7 Done Done Sent false
4279 Acked Write 9 Done Done Sent false
4280 Acked Write 8 Done Done Sent false
4281 Acked Read 9 Done Done Sent false
4282 Acked Write 4 Done Done Sent false
4283 Acked Write 3 Done Done Sent false
4284 Acked Write 9 Done Done Sent false
4285 Acked Read 2 Done Done Sent false
4286 Acked Read 4 Done Done Sent false
4287 Acked Read 1 Done Done Sent false
4288 Acked Write 7 Done Done Sent false
4289 Acked Read 6 Done Done Sent false
4290 Acked Write 7 Done Done Sent false
4291 Acked Read 10 Done Done Sent false
4292 Acked Write 1 Done Done Sent false
4293 Acked Read 2 Done Done Sent false
4294 Acked Read 9 Done Done Sent false
4295 Acked Read 10 Done Done Sent false
4296 Acked Read 8 Done Done Sent false
4297 Acked Read 7 Done Done Sent false
4298 Acked Write 3 Done Done Sent false
4299 Acked Write 9 Done Done Sent false
4300 Acked Flush 0 Done Done Sent false
4301 Acked Flush 0 Done Done Sent false
4302 Acked Write 3 Done Done Sent false
4303 Acked Write 1 Done Done Sent false
4304 Acked Read 5 Done Done Sent false
4305 Acked Write 5 Done Done Sent false
4306 Acked Write 7 Done Done Sent false
4307 Acked Flush 0 Done Done Sent false
4308 Acked Write 9 Done Done Sent false
4309 Acked Flush 0 Done Done Sent false
4310 Acked Read 4 Done Done Sent false
4311 Acked Read 2 Done Done Sent false
4312 Acked Write 8 Done Done Sent false
4313 Acked Read 5 Done Done Sent false
4314 Acked Read 6 Done Done Sent false
4315 Acked Flush 0 Done Done Sent false
4316 Acked Write 6 Done Done Sent false
4317 Acked Write 2 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4315 4315 4016
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4317 4316 4315 4314 4313
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Write 3 Done Done Sent false
4018 Acked Write 7 Done Done Sent false
4019 Acked Read 4 Done Done Sent false
4020 Acked Write 10 Done Done Sent false
4021 Acked Read 2 Done Done Sent false
4022 Acked Write 7 Done Done Sent false
4023 Acked Read 9 Done Done Sent false
4024 Acked Read 7 Done Done Sent false
4025 Acked Read 8 Done Done Sent false
4026 Acked Write 5 Done Done Sent false
4027 Acked Flush 0 Done Done Sent false
4028 Acked Read 5 Done Done Sent false
4029 Acked Write 4 Done Done Sent false
4030 Acked Read 2 Done Done Sent false
4031 Acked Flush 0 Done Done Sent false
4032 Acked Read 1 Done Done Sent false
4033 Acked Write 4 Done Done Sent false
4034 Acked Read 10 Done Done Sent false
4035 Acked Read 4 Done Done Sent false
4036 Acked Read 3 Done Done Sent false
4037 Acked Write 6 Done Done Sent false
4038 Acked Write 1 Done Done Sent false
4039 Acked Write 8 Done Done Sent false
4040 Acked Read 5 Done Done Sent false
4041 Acked Read 3 Done Done Sent false
4042 Acked Write 1 Done Done Sent false
4043 Acked Read 9 Done Done Sent false
4044 Acked Write 8 Done Done Sent false
4045 Acked Read 10 Done Done Sent false
4046 Acked Read 8 Done Done Sent false
4047 Acked Read 5 Done Done Sent false
4048 Acked Read 3 Done Done Sent false
4049 Acked Read 1 Done Done Sent false
4050 Acked Write 7 Done Done Sent false
4051 Acked Flush 0 Done Done Sent false
4052 Acked Read 1 Done Done Sent false
4053 Acked Write 8 Done Done Sent false
4054 Acked Write 4 Done Done Sent false
4055 Acked Read 1 Done Done Sent false
4056 Acked Read 6 Done Done Sent false
4057 Acked Write 7 Done Done Sent false
4058 Acked Read 4 Done Done Sent false
4059 Acked Read 4 Done Done Sent false
4060 Acked Read 4 Done Done Sent false
4061 Acked Write 10 Done Done Sent false
4062 Acked Write 1 Done Done Sent false
4063 Acked Write 9 Done Done Sent false
4064 Acked Read 10 Done Done Sent false
4065 Acked Read 10 Done Done Sent false
4066 Acked Read 1 Done Done Sent false
4067 Acked Read 10 Done Done Sent false
4068 Acked Write 7 Done Done Sent false
4069 Acked Read 10 Done Done Sent false
4070 Acked Read 9 Done Done Sent false
4071 Acked Read 3 Done Done Sent false
4072 Acked Read 9 Done Done Sent false
4073 Acked Read 3 Done Done Sent false
4074 Acked Read 8 Done Done Sent false
4075 Acked Write 6 Done Done Sent false
4076 Acked Read 3 Done Done Sent false
4077 Acked Write 5 Done Done Sent false
4078 Acked Read 10 Done Done Sent false
4079 Acked Write 4 Done Done Sent false
4080 Acked Write 7 Done Done Sent false
4081 Acked Read 8 Done Done Sent false
4082 Acked Read 6 Done Done Sent false
4083 Acked Write 1 Done Done Sent false
4084 Acked Write 8 Done Done Sent false
4085 Acked Write 3 Done Done Sent false
4086 Acked Read 7 Done Done Sent false
4087 Acked Read 5 Done Done Sent false
4088 Acked Read 7 Done Done Sent false
4089 Acked Read 5 Done Done Sent false
4090 Acked Write 9 Done Done Sent false
4091 Acked Write 1 Done Done Sent false
4092 Acked Read 8 Done Done Sent false
4093 Acked Read 6 Done Done Sent false
4094 Acked Flush 0 Done Done Sent false
4095 Acked Read 5 Done Done Sent false
4096 Acked Flush 0 Done Done Sent false
4097 Acked Read 4 Done Done Sent false
4098 Acked Read 2 Done Done Sent false
4099 Acked Read 2 Done Done Sent false
4100 Acked Flush 0 Done Done Sent false
4101 Acked Write 5 Done Done Sent false
4102 Acked Read 4 Done Done Sent false
4103 Acked Write 3 Done Done Sent false
4104 Acked Write 4 Done Done Sent false
4105 Acked Read 4 Done Done Sent false
4106 Acked Read 10 Done Done Sent false
4107 Acked Read 1 Done Done Sent false
4108 Acked Write 8 Done Done Sent false
4109 Acked Write 9 Done Done Sent false
4110 Acked Read 1 Done Done Sent false
4111 Acked Read 2 Done Done Sent false
4112 Acked Write 9 Done Done Sent false
4113 Acked Read 5 Done Done Sent false
4114 Acked Write 2 Done Done Sent false
4115 Acked Write 4 Done Done Sent false
4116 Acked Read 10 Done Done Sent false
4117 Acked Write 1 Done Done Sent false
4118 Acked Write 7 Done Done Sent false
4119 Acked Flush 0 Done Done Sent false
4120 Acked Flush 0 Done Done Sent false
4121 Acked Read 6 Done Done Sent false
4122 Acked Write 8 Done Done Sent false
4123 Acked Read 9 Done Done Sent false
4124 Acked Read 9 Done Done Sent false
4125 Acked Write 2 Done Done Sent false
4126 Acked Flush 0 Done Done Sent false
4127 Acked Read 6 Done Done Sent false
4128 Acked Read 2 Done Done Sent false
4129 Acked Read 4 Done Done Sent false
4130 Acked Write 1 Done Done Sent false
4131 Acked Read 4 Done Done Sent false
4132 Acked Flush 0 Done Done Sent false
4133 Acked Write 7 Done Done Sent false
4134 Acked Read 1 Done Done Sent false
4135 Acked Write 8 Done Done Sent false
4136 Acked Write 1 Done Done Sent false
4137 Acked Read 10 Done Done Sent false
4138 Acked Read 2 Done Done Sent false
4139 Acked Flush 0 Done Done Sent false
4140 Acked Write 9 Done Done Sent false
4141 Acked Write 7 Done Done Sent false
4142 Acked Read 9 Done Done Sent false
4143 Acked Write 10 Done Done Sent false
4144 Acked Write 3 Done Done Sent false
4145 Acked Write 3 Done Done Sent false
4146 Acked Write 1 Done Done Sent false
4147 Acked Flush 0 Done Done Sent false
4148 Acked Write 8 Done Done Sent false
4149 Acked Read 5 Done Done Sent false
4150 Acked Write 1 Done Done Sent false
4151 Acked Flush 0 Done Done Sent false
4152 Acked Read 8 Done Done Sent false
4153 Acked Write 9 Done Done Sent false
4154 Acked Read 6 Done Done Sent false
4155 Acked Read 7 Done Done Sent false
4156 Acked Read 8 Done Done Sent false
4157 Acked Write 5 Done Done Sent false
4158 Acked Read 10 Done Done Sent false
4159 Acked Read 10 Done Done Sent false
4160 Acked Read 2 Done Done Sent false
4161 Acked Write 6 Done Done Sent false
4162 Acked Write 1 Done Done Sent false
4163 Acked Read 7 Done Done Sent false
4164 Acked Flush 0 Done Done Sent false
4165 Acked Write 4 Done Done Sent false
4166 Acked Read 4 Done Done Sent false
4167 Acked Write 4 Done Done Sent false
4168 Acked Write 3 Done Done Sent false
4169 Acked Write 3 Done Done Sent false
4170 Acked Read 9 Done Done Sent false
4171 Acked Flush 0 Done Done Sent false
4172 Acked Write 10 Done Done Sent false
4173 Acked Write 5 Done Done Sent false
4174 Acked Read 2 Done Done Sent false
4175 Acked Read 10 Done Done Sent false
4176 Acked Read 6 Done Done Sent false
4177 Acked Flush 0 Done Done Sent false
4178 Acked Write 5 Done Done Sent false
4179 Acked Flush 0 Done Done Sent false
4180 Acked Write 3 Done Done Sent false
4181 Acked Read 6 Done Done Sent false
4182 Acked Read 6 Done Done Sent false
4183 Acked Read 6 Done Done Sent false
4184 Acked Read 8 Done Done Sent false
4185 Acked Read 10 Done Done Sent false
4186 Acked Read 3 Done Done Sent false
4187 Acked Read 4 Done Done Sent false
4188 Acked Write 1 Done Done Sent false
4189 Acked Write 3 Done Done Sent false
4190 Acked Read 5 Done Done Sent false
4191 Acked Read 9 Done Done Sent false
4192 Acked Read 3 Done Done Sent false
4193 Acked Read 8 Done Done Sent false
4194 Acked Write 9 Done Done Sent false
4195 Acked Write 7 Done Done Sent false
4196 Acked Write 5 Done Done Sent false
4197 Acked Read 7 Done Done Sent false
4198 Acked Read 10 Done Done Sent false
4199 Acked Write 8 Done Done Sent false
4200 Acked Read 9 Done Done Sent false
4201 Acked Write 5 Done Done Sent false
4202 Acked Write 2 Done Done Sent false
4203 Acked Read 1 Done Done Sent false
4204 Acked Read 9 Done Done Sent false
4205 Acked Read 7 Done Done Sent false
4206 Acked Read 10 Done Done Sent false
4207 Acked Read 8 Done Done Sent false
4208 Acked Write 4 Done Done Sent false
4209 Acked Read 9 Done Done Sent false
4210 Acked Read 2 Done Done Sent false
4211 Acked Write 8 Done Done Sent false
4212 Acked Write 8 Done Done Sent false
4213 Acked Flush 0 Done Done Sent false
4214 Acked Write 1 Done Done Sent false
4215 Acked Flush 0 Done Done Sent false
4216 Acked Read 6 Done Done Sent false
4217 Acked Flush 0 Done Done Sent false
4218 Acked Flush 0 Done Done Sent false
4219 Acked Write 10 Done Done Sent false
4220 Acked Write 2 Done Done Sent false
4221 Acked Read 3 Done Done Sent false
4222 Acked Write 6 Done Done Sent false
4223 Acked Read 10 Done Done Sent false
4224 Acked Read 8 Done Done Sent false
4225 Acked Read 9 Done Done Sent false
4226 Acked Read 5 Done Done Sent false
4227 Acked Read 9 Done Done Sent false
4228 Acked Write 4 Done Done Sent false
4229 Acked Read 3 Done Done Sent false
4230 Acked Read 6 Done Done Sent false
4231 Acked Write 5 Done Done Sent false
4232 Acked Read 3 Done Done Sent false
4233 Acked Read 8 Done Done Sent false
4234 Acked Write 4 Done Done Sent false
4235 Acked Write 8 Done Done Sent false
4236 Acked Read 5 Done Done Sent false
4237 Acked Flush 0 Done Done Sent false
4238 Acked Write 5 Done Done Sent false
4239 Acked Read 9 Done Done Sent false
4240 Acked Read 10 Done Done Sent false
4241 Acked Read 3 Done Done Sent false
4242 Acked Write 10 Done Done Sent false
4243 Acked Read 2 Done Done Sent false
4244 Acked Read 10 Done Done Sent false
4245 Acked Read 4 Done Done Sent false
4246 Acked Write 10 Done Done Sent false
4247 Acked Write 6 Done Done Sent false
4248 Acked Read 2 Done Done Sent false
4249 Acked Write 8 Done Done Sent false
4250 Acked Read 5 Done Done Sent false
4251 Acked Read 9 Done Done Sent false
4252 Acked Read 4 Done Done Sent false
4253 Acked Read 1 Done Done Sent false
4254 Acked Write 8 Done Done Sent false
4255 Acked Write 7 Done Done Sent false
4256 Acked Read 4 Done Done Sent false
4257 Acked Write 1 Done Done Sent false
4258 Acked Flush 0 Done Done Sent false
4259 Acked Read 9 Done Done Sent false
4260 Acked Write 5 Done Done Sent false
4261 Acked Read 6 Done Done Sent false
4262 Acked Read 9 Done Done Sent false
4263 Acked Write 7 Done Done Sent false
4264 Acked Write 1 Done Done Sent false
4265 Acked Read 8 Done Done Sent false
4266 Acked Read 6 Done Done Sent false
4267 Acked Flush 0 Done Done Sent false
4268 Acked Read 2 Done Done Sent false
4269 Acked Read 9 Done Done Sent false
4270 Acked Write 1 Done Done Sent false
4271 Acked Write 4 Done Done Sent false
4272 Acked Write 9 Done Done Sent false
4273 Acked Write 10 Done Done Sent false
4274 Acked Read 6 Done Done Sent false
4275 Acked Write 2 Done Done Sent false
4276 Acked Read 2 Done Done Sent false
4277 Acked Write 4 Done Done Sent false
4278 Acked Write 7 Done Done Sent false
4279 Acked Write 9 Done Done Sent false
4280 Acked Write 8 Done Done Sent false
4281 Acked Read 9 Done Done Sent false
4282 Acked Write 4 Done Done Sent false
4283 Acked Write 3 Done Done Sent false
4284 Acked Write 9 Done Done Sent false
4285 Acked Read 2 Done Done Sent false
4286 Acked Read 4 Done Done Sent false
4287 Acked Read 1 Done Done Sent false
4288 Acked Write 7 Done Done Sent false
4289 Acked Read 6 Done Done Sent false
4290 Acked Write 7 Done Done Sent false
4291 Acked Read 10 Done Done Sent false
4292 Acked Write 1 Done Done Sent false
4293 Acked Read 2 Done Done Sent false
4294 Acked Read 9 Done Done Sent false
4295 Acked Read 10 Done Done Sent false
4296 Acked Read 8 Done Done Sent false
4297 Acked Read 7 Done Done Sent false
4298 Acked Write 3 Done Done Sent false
4299 Acked Write 9 Done Done Sent false
4300 Acked Flush 0 Done Done Sent false
4301 Acked Flush 0 Done Done Sent false
4302 Acked Write 3 Done Done Sent false
4303 Acked Write 1 Done Done Sent false
4304 Acked Read 5 Done Done Sent false
4305 Acked Write 5 Done Done Sent false
4306 Acked Write 7 Done Done Sent false
4307 Acked Flush 0 Done Done Sent false
4308 Acked Write 9 Done Done Sent false
4309 Acked Flush 0 Done Done Sent false
4310 Acked Read 4 Done Done Sent false
4311 Acked Read 2 Done Done Sent false
4312 Acked Write 8 Done Done Sent false
4313 Acked Read 5 Done Done Sent false
4314 Acked Read 6 Done Done Sent false
4315 Acked Flush 0 Done Done Sent false
4316 Acked Write 6 Done Done Sent false
4317 Acked Write 2 Done Done Sent false
4318 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4318 4318 4016
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4318 4317 4316 4315 4314
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:45.371819827Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:45.371916006Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:45.371957391Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:45.371996047Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:45.372091176Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Write 3 Done Done Sent false
4018 Acked Write 7 Done Done Sent false
4019 Acked Read 4 Done Done Sent false
4020 Acked Write 10 Done Done Sent false
4021 Acked Read 2 Done Done Sent false
4022 Acked Write 7 Done Done Sent false
4023 Acked Read 9 Done Done Sent false
4024 Acked Read 7 Done Done Sent false
4025 Acked Read 8 Done Done Sent false
4026 Acked Write 5 Done Done Sent false
4027 Acked Flush 0 Done Done Sent false
4028 Acked Read 5 Done Done Sent false
4029 Acked Write 4 Done Done Sent false
4030 Acked Read 2 Done Done Sent false
4031 Acked Flush 0 Done Done Sent false
4032 Acked Read 1 Done Done Sent false
4033 Acked Write 4 Done Done Sent false
4034 Acked Read 10 Done Done Sent false
4035 Acked Read 4 Done Done Sent false
4036 Acked Read 3 Done Done Sent false
4037 Acked Write 6 Done Done Sent false
4038 Acked Write 1 Done Done Sent false
4039 Acked Write 8 Done Done Sent false
4040 Acked Read 5 Done Done Sent false
4041 Acked Read 3 Done Done Sent false
4042 Acked Write 1 Done Done Sent false
4043 Acked Read 9 Done Done Sent false
4044 Acked Write 8 Done Done Sent false
4045 Acked Read 10 Done Done Sent false
4046 Acked Read 8 Done Done Sent false
4047 Acked Read 5 Done Done Sent false
4048 Acked Read 3 Done Done Sent false
4049 Acked Read 1 Done Done Sent false
4050 Acked Write 7 Done Done Sent false
4051 Acked Flush 0 Done Done Sent false
4052 Acked Read 1 Done Done Sent false
4053 Acked Write 8 Done Done Sent false
4054 Acked Write 4 Done Done Sent false
4055 Acked Read 1 Done Done Sent false
4056 Acked Read 6 Done Done Sent false
4057 Acked Write 7 Done Done Sent false
4058 Acked Read 4 Done Done Sent false
4059 Acked Read 4 Done Done Sent false
4060 Acked Read 4 Done Done Sent false
4061 Acked Write 10 Done Done Sent false
4062 Acked Write 1 Done Done Sent false
4063 Acked Write 9 Done Done Sent false
4064 Acked Read 10 Done Done Sent false
4065 Acked Read 10 Done Done Sent false
4066 Acked Read 1 Done Done Sent false
4067 Acked Read 10 Done Done Sent false
4068 Acked Write 7 Done Done Sent false
4069 Acked Read 10 Done Done Sent false
4070 Acked Read 9 Done Done Sent false
4071 Acked Read 3 Done Done Sent false
4072 Acked Read 9 Done Done Sent false
4073 Acked Read 3 Done Done Sent false
4074 Acked Read 8 Done Done Sent false
4075 Acked Write 6 Done Done Sent false
4076 Acked Read 3 Done Done Sent false
4077 Acked Write 5 Done Done Sent false
4078 Acked Read 10 Done Done Sent false
4079 Acked Write 4 Done Done Sent false
4080 Acked Write 7 Done Done Sent false
4081 Acked Read 8 Done Done Sent false
4082 Acked Read 6 Done Done Sent false
4083 Acked Write 1 Done Done Sent false
4084 Acked Write 8 Done Done Sent false
4085 Acked Write 3 Done Done Sent false
4086 Acked Read 7 Done Done Sent false
4087 Acked Read 5 Done Done Sent false
4088 Acked Read 7 Done Done Sent false
4089 Acked Read 5 Done Done Sent false
4090 Acked Write 9 Done Done Sent false
4091 Acked Write 1 Done Done Sent false
4092 Acked Read 8 Done Done Sent false
4093 Acked Read 6 Done Done Sent false
4094 Acked Flush 0 Done Done Sent false
4095 Acked Read 5 Done Done Sent false
4096 Acked Flush 0 Done Done Sent false
4097 Acked Read 4 Done Done Sent false
4098 Acked Read 2 Done Done Sent false
4099 Acked Read 2 Done Done Sent false
4100 Acked Flush 0 Done Done Sent false
4101 Acked Write 5 Done Done Sent false
4102 Acked Read 4 Done Done Sent false
4103 Acked Write 3 Done Done Sent false
4104 Acked Write 4 Done Done Sent false
4105 Acked Read 4 Done Done Sent false
4106 Acked Read 10 Done Done Sent false
4107 Acked Read 1 Done Done Sent false
4108 Acked Write 8 Done Done Sent false
4109 Acked Write 9 Done Done Sent false
4110 Acked Read 1 Done Done Sent false
4111 Acked Read 2 Done Done Sent false
4112 Acked Write 9 Done Done Sent false
4113 Acked Read 5 Done Done Sent false
4114 Acked Write 2 Done Done Sent false
4115 Acked Write 4 Done Done Sent false
4116 Acked Read 10 Done Done Sent false
4117 Acked Write 1 Done Done Sent false
4118 Acked Write 7 Done Done Sent false
4119 Acked Flush 0 Done Done Sent false
4120 Acked Flush 0 Done Done Sent false
4121 Acked Read 6 Done Done Sent false
4122 Acked Write 8 Done Done Sent false
4123 Acked Read 9 Done Done Sent false
4124 Acked Read 9 Done Done Sent false
4125 Acked Write 2 Done Done Sent false
4126 Acked Flush 0 Done Done Sent false
4127 Acked Read 6 Done Done Sent false
4128 Acked Read 2 Done Done Sent false
4129 Acked Read 4 Done Done Sent false
4130 Acked Write 1 Done Done Sent false
4131 Acked Read 4 Done Done Sent false
4132 Acked Flush 0 Done Done Sent false
4133 Acked Write 7 Done Done Sent false
4134 Acked Read 1 Done Done Sent false
4135 Acked Write 8 Done Done Sent false
4136 Acked Write 1 Done Done Sent false
4137 Acked Read 10 Done Done Sent false
4138 Acked Read 2 Done Done Sent false
4139 Acked Flush 0 Done Done Sent false
4140 Acked Write 9 Done Done Sent false
4141 Acked Write 7 Done Done Sent false
4142 Acked Read 9 Done Done Sent false
4143 Acked Write 10 Done Done Sent false
4144 Acked Write 3 Done Done Sent false
4145 Acked Write 3 Done Done Sent false
4146 Acked Write 1 Done Done Sent false
4147 Acked Flush 0 Done Done Sent false
4148 Acked Write 8 Done Done Sent false
4149 Acked Read 5 Done Done Sent false
4150 Acked Write 1 Done Done Sent false
4151 Acked Flush 0 Done Done Sent false
4152 Acked Read 8 Done Done Sent false
4153 Acked Write 9 Done Done Sent false
4154 Acked Read 6 Done Done Sent false
4155 Acked Read 7 Done Done Sent false
4156 Acked Read 8 Done Done Sent false
4157 Acked Write 5 Done Done Sent false
4158 Acked Read 10 Done Done Sent false
4159 Acked Read 10 Done Done Sent false
4160 Acked Read 2 Done Done Sent false
4161 Acked Write 6 Done Done Sent false
4162 Acked Write 1 Done Done Sent false
4163 Acked Read 7 Done Done Sent false
4164 Acked Flush 0 Done Done Sent false
4165 Acked Write 4 Done Done Sent false
4166 Acked Read 4 Done Done Sent false
4167 Acked Write 4 Done Done Sent false
4168 Acked Write 3 Done Done Sent false
4169 Acked Write 3 Done Done Sent false
4170 Acked Read 9 Done Done Sent false
4171 Acked Flush 0 Done Done Sent false
4172 Acked Write 10 Done Done Sent false
4173 Acked Write 5 Done Done Sent false
4174 Acked Read 2 Done Done Sent false
4175 Acked Read 10 Done Done Sent false
4176 Acked Read 6 Done Done Sent false
4177 Acked Flush 0 Done Done Sent false
4178 Acked Write 5 Done Done Sent false
4179 Acked Flush 0 Done Done Sent false
4180 Acked Write 3 Done Done Sent false
4181 Acked Read 6 Done Done Sent false
4182 Acked Read 6 Done Done Sent false
4183 Acked Read 6 Done Done Sent false
4184 Acked Read 8 Done Done Sent false
4185 Acked Read 10 Done Done Sent false
4186 Acked Read 3 Done Done Sent false
4187 Acked Read 4 Done Done Sent false
4188 Acked Write 1 Done Done Sent false
4189 Acked Write 3 Done Done Sent false
4190 Acked Read 5 Done Done Sent false
4191 Acked Read 9 Done Done Sent false
4192 Acked Read 3 Done Done Sent false
4193 Acked Read 8 Done Done Sent false
4194 Acked Write 9 Done Done Sent false
4195 Acked Write 7 Done Done Sent false
4196 Acked Write 5 Done Done Sent false
4197 Acked Read 7 Done Done Sent false
4198 Acked Read 10 Done Done Sent false
4199 Acked Write 8 Done Done Sent false
4200 Acked Read 9 Done Done Sent false
4201 Acked Write 5 Done Done Sent false
4202 Acked Write 2 Done Done Sent false
4203 Acked Read 1 Done Done Sent false
4204 Acked Read 9 Done Done Sent false
4205 Acked Read 7 Done Done Sent false
4206 Acked Read 10 Done Done Sent false
4207 Acked Read 8 Done Done Sent false
4208 Acked Write 4 Done Done Sent false
4209 Acked Read 9 Done Done Sent false
4210 Acked Read 2 Done Done Sent false
4211 Acked Write 8 Done Done Sent false
4212 Acked Write 8 Done Done Sent false
4213 Acked Flush 0 Done Done Sent false
4214 Acked Write 1 Done Done Sent false
4215 Acked Flush 0 Done Done Sent false
4216 Acked Read 6 Done Done Sent false
4217 Acked Flush 0 Done Done Sent false
4218 Acked Flush 0 Done Done Sent false
4219 Acked Write 10 Done Done Sent false
4220 Acked Write 2 Done Done Sent false
4221 Acked Read 3 Done Done Sent false
4222 Acked Write 6 Done Done Sent false
4223 Acked Read 10 Done Done Sent false
4224 Acked Read 8 Done Done Sent false
4225 Acked Read 9 Done Done Sent false
4226 Acked Read 5 Done Done Sent false
4227 Acked Read 9 Done Done Sent false
4228 Acked Write 4 Done Done Sent false
4229 Acked Read 3 Done Done Sent false
4230 Acked Read 6 Done Done Sent false
4231 Acked Write 5 Done Done Sent false
4232 Acked Read 3 Done Done Sent false
4233 Acked Read 8 Done Done Sent false
4234 Acked Write 4 Done Done Sent false
4235 Acked Write 8 Done Done Sent false
4236 Acked Read 5 Done Done Sent false
4237 Acked Flush 0 Done Done Sent false
4238 Acked Write 5 Done Done Sent false
4239 Acked Read 9 Done Done Sent false
4240 Acked Read 10 Done Done Sent false
4241 Acked Read 3 Done Done Sent false
4242 Acked Write 10 Done Done Sent false
4243 Acked Read 2 Done Done Sent false
4244 Acked Read 10 Done Done Sent false
4245 Acked Read 4 Done Done Sent false
4246 Acked Write 10 Done Done Sent false
4247 Acked Write 6 Done Done Sent false
4248 Acked Read 2 Done Done Sent false
4249 Acked Write 8 Done Done Sent false
4250 Acked Read 5 Done Done Sent false
4251 Acked Read 9 Done Done Sent false
4252 Acked Read 4 Done Done Sent false
4253 Acked Read 1 Done Done Sent false
4254 Acked Write 8 Done Done Sent false
4255 Acked Write 7 Done Done Sent false
4256 Acked Read 4 Done Done Sent false
4257 Acked Write 1 Done Done Sent false
4258 Acked Flush 0 Done Done Sent false
4259 Acked Read 9 Done Done Sent false
4260 Acked Write 5 Done Done Sent false
4261 Acked Read 6 Done Done Sent false
4262 Acked Read 9 Done Done Sent false
4263 Acked Write 7 Done Done Sent false
4264 Acked Write 1 Done Done Sent false
4265 Acked Read 8 Done Done Sent false
4266 Acked Read 6 Done Done Sent false
4267 Acked Flush 0 Done Done Sent false
4268 Acked Read 2 Done Done Sent false
4269 Acked Read 9 Done Done Sent false
4270 Acked Write 1 Done Done Sent false
4271 Acked Write 4 Done Done Sent false
4272 Acked Write 9 Done Done Sent false
4273 Acked Write 10 Done Done Sent false
4274 Acked Read 6 Done Done Sent false
4275 Acked Write 2 Done Done Sent false
4276 Acked Read 2 Done Done Sent false
4277 Acked Write 4 Done Done Sent false
4278 Acked Write 7 Done Done Sent false
4279 Acked Write 9 Done Done Sent false
4280 Acked Write 8 Done Done Sent false
4281 Acked Read 9 Done Done Sent false
4282 Acked Write 4 Done Done Sent false
4283 Acked Write 3 Done Done Sent false
4284 Acked Write 9 Done Done Sent false
4285 Acked Read 2 Done Done Sent false
4286 Acked Read 4 Done Done Sent false
4287 Acked Read 1 Done Done Sent false
4288 Acked Write 7 Done Done Sent false
4289 Acked Read 6 Done Done Sent false
4290 Acked Write 7 Done Done Sent false
4291 Acked Read 10 Done Done Sent false
4292 Acked Write 1 Done Done Sent false
4293 Acked Read 2 Done Done Sent false
4294 Acked Read 9 Done Done Sent false
4295 Acked Read 10 Done Done Sent false
4296 Acked Read 8 Done Done Sent false
4297 Acked Read 7 Done Done Sent false
4298 Acked Write 3 Done Done Sent false
4299 Acked Write 9 Done Done Sent false
4300 Acked Flush 0 Done Done Sent false
4301 Acked Flush 0 Done Done Sent false
4302 Acked Write 3 Done Done Sent false
4303 Acked Write 1 Done Done Sent false
4304 Acked Read 5 Done Done Sent false
4305 Acked Write 5 Done Done Sent false
4306 Acked Write 7 Done Done Sent false
4307 Acked Flush 0 Done Done Sent false
4308 Acked Write 9 Done Done Sent false
4309 Acked Flush 0 Done Done Sent false
4310 Acked Read 4 Done Done Sent false
4311 Acked Read 2 Done Done Sent false
4312 Acked Write 8 Done Done Sent false
4313 Acked Read 5 Done Done Sent false
4314 Acked Read 6 Done Done Sent false
4315 Acked Flush 0 Done Done Sent false
4316 Acked Write 6 Done Done Sent false
4317 Acked Write 2 Done Done Sent false
4318 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4318 4318 4016
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4318 4317 4316 4315 4314
CLIENT: Up:0 ds:302 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4017 Acked Write 3 Done Done Sent false
4018 Acked Write 7 Done Done Sent false
4019 Acked Read 4 Done Done Sent false
4020 Acked Write 10 Done Done Sent false
4021 Acked Read 2 Done Done Sent false
4022 Acked Write 7 Done Done Sent false
4023 Acked Read 9 Done Done Sent false
4024 Acked Read 7 Done Done Sent false
4025 Acked Read 8 Done Done Sent false
4026 Acked Write 5 Done Done Sent false
4027 Acked Flush 0 Done Done Sent false
4028 Acked Read 5 Done Done Sent false
4029 Acked Write 4 Done Done Sent false
4030 Acked Read 2 Done Done Sent false
4031 Acked Flush 0 Done Done Sent false
4032 Acked Read 1 Done Done Sent false
4033 Acked Write 4 Done Done Sent false
4034 Acked Read 10 Done Done Sent false
4035 Acked Read 4 Done Done Sent false
4036 Acked Read 3 Done Done Sent false
4037 Acked Write 6 Done Done Sent false
4038 Acked Write 1 Done Done Sent false
4039 Acked Write 8 Done Done Sent false
4040 Acked Read 5 Done Done Sent false
4041 Acked Read 3 Done Done Sent false
4042 Acked Write 1 Done Done Sent false
4043 Acked Read 9 Done Done Sent false
4044 Acked Write 8 Done Done Sent false
4045 Acked Read 10 Done Done Sent false
4046 Acked Read 8 Done Done Sent false
4047 Acked Read 5 Done Done Sent false
4048 Acked Read 3 Done Done Sent false
4049 Acked Read 1 Done Done Sent false
4050 Acked Write 7 Done Done Sent false
4051 Acked Flush 0 Done Done Sent false
4052 Acked Read 1 Done Done Sent false
4053 Acked Write 8 Done Done Sent false
4054 Acked Write 4 Done Done Sent false
4055 Acked Read 1 Done Done Sent false
4056 Acked Read 6 Done Done Sent false
4057 Acked Write 7 Done Done Sent false
4058 Acked Read 4 Done Done Sent false
4059 Acked Read 4 Done Done Sent false
4060 Acked Read 4 Done Done Sent false
4061 Acked Write 10 Done Done Sent false
4062 Acked Write 1 Done Done Sent false
4063 Acked Write 9 Done Done Sent false
4064 Acked Read 10 Done Done Sent false
4065 Acked Read 10 Done Done Sent false
4066 Acked Read 1 Done Done Sent false
4067 Acked Read 10 Done Done Sent false
4068 Acked Write 7 Done Done Sent false
4069 Acked Read 10 Done Done Sent false
4070 Acked Read 9 Done Done Sent false
4071 Acked Read 3 Done Done Sent false
4072 Acked Read 9 Done Done Sent false
4073 Acked Read 3 Done Done Sent false
4074 Acked Read 8 Done Done Sent false
4075 Acked Write 6 Done Done Sent false
4076 Acked Read 3 Done Done Sent false
4077 Acked Write 5 Done Done Sent false
4078 Acked Read 10 Done Done Sent false
4079 Acked Write 4 Done Done Sent false
4080 Acked Write 7 Done Done Sent false
4081 Acked Read 8 Done Done Sent false
4082 Acked Read 6 Done Done Sent false
4083 Acked Write 1 Done Done Sent false
4084 Acked Write 8 Done Done Sent false
4085 Acked Write 3 Done Done Sent false
4086 Acked Read 7 Done Done Sent false
4087 Acked Read 5 Done Done Sent false
4088 Acked Read 7 Done Done Sent false
4089 Acked Read 5 Done Done Sent false
4090 Acked Write 9 Done Done Sent false
4091 Acked Write 1 Done Done Sent false
4092 Acked Read 8 Done Done Sent false
4093 Acked Read 6 Done Done Sent false
4094 Acked Flush 0 Done Done Sent false
4095 Acked Read 5 Done Done Sent false
4096 Acked Flush 0 Done Done Sent false
4097 Acked Read 4 Done Done Sent false
4098 Acked Read 2 Done Done Sent false
4099 Acked Read 2 Done Done Sent false
4100 Acked Flush 0 Done Done Sent false
4101 Acked Write 5 Done Done Sent false
4102 Acked Read 4 Done Done Sent false
4103 Acked Write 3 Done Done Sent false
4104 Acked Write 4 Done Done Sent false
4105 Acked Read 4 Done Done Sent false
4106 Acked Read 10 Done Done Sent false
4107 Acked Read 1 Done Done Sent false
4108 Acked Write 8 Done Done Sent false
4109 Acked Write 9 Done Done Sent false
4110 Acked Read 1 Done Done Sent false
4111 Acked Read 2 Done Done Sent false
4112 Acked Write 9 Done Done Sent false
4113 Acked Read 5 Done Done Sent false
4114 Acked Write 2 Done Done Sent false
4115 Acked Write 4 Done Done Sent false
4116 Acked Read 10 Done Done Sent false
4117 Acked Write 1 Done Done Sent false
4118 Acked Write 7 Done Done Sent false
4119 Acked Flush 0 Done Done Sent false
4120 Acked Flush 0 Done Done Sent false
4121 Acked Read 6 Done Done Sent false
4122 Acked Write 8 Done Done Sent false
4123 Acked Read 9 Done Done Sent false
4124 Acked Read 9 Done Done Sent false
4125 Acked Write 2 Done Done Sent false
4126 Acked Flush 0 Done Done Sent false
4127 Acked Read 6 Done Done Sent false
4128 Acked Read 2 Done Done Sent false
4129 Acked Read 4 Done Done Sent false
4130 Acked Write 1 Done Done Sent false
4131 Acked Read 4 Done Done Sent false
4132 Acked Flush 0 Done Done Sent false
4133 Acked Write 7 Done Done Sent false
4134 Acked Read 1 Done Done Sent false
4135 Acked Write 8 Done Done Sent false
4136 Acked Write 1 Done Done Sent false
4137 Acked Read 10 Done Done Sent false
4138 Acked Read 2 Done Done Sent false
4139 Acked Flush 0 Done Done Sent false
4140 Acked Write 9 Done Done Sent false
4141 Acked Write 7 Done Done Sent false
4142 Acked Read 9 Done Done Sent false
4143 Acked Write 10 Done Done Sent false
4144 Acked Write 3 Done Done Sent false
4145 Acked Write 3 Done Done Sent false
4146 Acked Write 1 Done Done Sent false
4147 Acked Flush 0 Done Done Sent false
4148 Acked Write 8 Done Done Sent false
4149 Acked Read 5 Done Done Sent false
4150 Acked Write 1 Done Done Sent false
4151 Acked Flush 0 Done Done Sent false
4152 Acked Read 8 Done Done Sent false
4153 Acked Write 9 Done Done Sent false
4154 Acked Read 6 Done Done Sent false
4155 Acked Read 7 Done Done Sent false
4156 Acked Read 8 Done Done Sent false
4157 Acked Write 5 Done Done Sent false
4158 Acked Read 10 Done Done Sent false
4159 Acked Read 10 Done Done Sent false
4160 Acked Read 2 Done Done Sent false
4161 Acked Write 6 Done Done Sent false
4162 Acked Write 1 Done Done Sent false
4163 Acked Read 7 Done Done Sent false
4164 Acked Flush 0 Done Done Sent false
4165 Acked Write 4 Done Done Sent false
4166 Acked Read 4 Done Done Sent false
4167 Acked Write 4 Done Done Sent false
4168 Acked Write 3 Done Done Sent false
4169 Acked Write 3 Done Done Sent false
4170 Acked Read 9 Done Done Sent false
4171 Acked Flush 0 Done Done Sent false
4172 Acked Write 10 Done Done Sent false
4173 Acked Write 5 Done Done Sent false
4174 Acked Read 2 Done Done Sent false
4175 Acked Read 10 Done Done Sent false
4176 Acked Read 6 Done Done Sent false
4177 Acked Flush 0 Done Done Sent false
4178 Acked Write 5 Done Done Sent false
4179 Acked Flush 0 Done Done Sent false
4180 Acked Write 3 Done Done Sent false
4181 Acked Read 6 Done Done Sent false
4182 Acked Read 6 Done Done Sent false
4183 Acked Read 6 Done Done Sent false
4184 Acked Read 8 Done Done Sent false
4185 Acked Read 10 Done Done Sent false
4186 Acked Read 3 Done Done Sent false
4187 Acked Read 4 Done Done Sent false
4188 Acked Write 1 Done Done Sent false
4189 Acked Write 3 Done Done Sent false
4190 Acked Read 5 Done Done Sent false
4191 Acked Read 9 Done Done Sent false
4192 Acked Read 3 Done Done Sent false
4193 Acked Read 8 Done Done Sent false
4194 Acked Write 9 Done Done Sent false
4195 Acked Write 7 Done Done Sent false
4196 Acked Write 5 Done Done Sent false
4197 Acked Read 7 Done Done Sent false
4198 Acked Read 10 Done Done Sent false
4199 Acked Write 8 Done Done Sent false
4200 Acked Read 9 Done Done Sent false
4201 Acked Write 5 Done Done Sent false
4202 Acked Write 2 Done Done Sent false
4203 Acked Read 1 Done Done Sent false
4204 Acked Read 9 Done Done Sent false
4205 Acked Read 7 Done Done Sent false
4206 Acked Read 10 Done Done Sent false
4207 Acked Read 8 Done Done Sent false
4208 Acked Write 4 Done Done Sent false
4209 Acked Read 9 Done Done Sent false
4210 Acked Read 2 Done Done Sent false
4211 Acked Write 8 Done Done Sent false
4212 Acked Write 8 Done Done Sent false
4213 Acked Flush 0 Done Done Sent false
4214 Acked Write 1 Done Done Sent false
4215 Acked Flush 0 Done Done Sent false
4216 Acked Read 6 Done Done Sent false
4217 Acked Flush 0 Done Done Sent false
4218 Acked Flush 0 Done Done Sent false
4219 Acked Write 10 Done Done Sent false
4220 Acked Write 2 Done Done Sent false
4221 Acked Read 3 Done Done Sent false
4222 Acked Write 6 Done Done Sent false
4223 Acked Read 10 Done Done Sent false
4224 Acked Read 8 Done Done Sent false
4225 Acked Read 9 Done Done Sent false
4226 Acked Read 5 Done Done Sent false
4227 Acked Read 9 Done Done Sent false
4228 Acked Write 4 Done Done Sent false
4229 Acked Read 3 Done Done Sent false
4230 Acked Read 6 Done Done Sent false
4231 Acked Write 5 Done Done Sent false
4232 Acked Read 3 Done Done Sent false
4233 Acked Read 8 Done Done Sent false
4234 Acked Write 4 Done Done Sent false
4235 Acked Write 8 Done Done Sent false
4236 Acked Read 5 Done Done Sent false
4237 Acked Flush 0 Done Done Sent false
4238 Acked Write 5 Done Done Sent false
4239 Acked Read 9 Done Done Sent false
4240 Acked Read 10 Done Done Sent false
4241 Acked Read 3 Done Done Sent false
4242 Acked Write 10 Done Done Sent false
4243 Acked Read 2 Done Done Sent false
4244 Acked Read 10 Done Done Sent false
4245 Acked Read 4 Done Done Sent false
4246 Acked Write 10 Done Done Sent false
4247 Acked Write 6 Done Done Sent false
4248 Acked Read 2 Done Done Sent false
4249 Acked Write 8 Done Done Sent false
4250 Acked Read 5 Done Done Sent false
4251 Acked Read 9 Done Done Sent false
4252 Acked Read 4 Done Done Sent false
4253 Acked Read 1 Done Done Sent false
4254 Acked Write 8 Done Done Sent false
4255 Acked Write 7 Done Done Sent false
4256 Acked Read 4 Done Done Sent false
4257 Acked Write 1 Done Done Sent false
4258 Acked Flush 0 Done Done Sent false
4259 Acked Read 9 Done Done Sent false
4260 Acked Write 5 Done Done Sent false
4261 Acked Read 6 Done Done Sent false
4262 Acked Read 9 Done Done Sent false
4263 Acked Write 7 Done Done Sent false
4264 Acked Write 1 Done Done Sent false
4265 Acked Read 8 Done Done Sent false
4266 Acked Read 6 Done Done Sent false
4267 Acked Flush 0 Done Done Sent false
4268 Acked Read 2 Done Done Sent false
4269 Acked Read 9 Done Done Sent false
4270 Acked Write 1 Done Done Sent false
4271 Acked Write 4 Done Done Sent false
4272 Acked Write 9 Done Done Sent false
4273 Acked Write 10 Done Done Sent false
4274 Acked Read 6 Done Done Sent false
4275 Acked Write 2 Done Done Sent false
4276 Acked Read 2 Done Done Sent false
4277 Acked Write 4 Done Done Sent false
4278 Acked Write 7 Done Done Sent false
4279 Acked Write 9 Done Done Sent false
4280 Acked Write 8 Done Done Sent false
4281 Acked Read 9 Done Done Sent false
4282 Acked Write 4 Done Done Sent false
4283 Acked Write 3 Done Done Sent false
4284 Acked Write 9 Done Done Sent false
4285 Acked Read 2 Done Done Sent false
4286 Acked Read 4 Done Done Sent false
4287 Acked Read 1 Done Done Sent false
4288 Acked Write 7 Done Done Sent false
4289 Acked Read 6 Done Done Sent false
4290 Acked Write 7 Done Done Sent false
4291 Acked Read 10 Done Done Sent false
4292 Acked Write 1 Done Done Sent false
4293 Acked Read 2 Done Done Sent false
4294 Acked Read 9 Done Done Sent false
4295 Acked Read 10 Done Done Sent false
4296 Acked Read 8 Done Done Sent false
4297 Acked Read 7 Done Done Sent false
4298 Acked Write 3 Done Done Sent false
4299 Acked Write 9 Done Done Sent false
4300 Acked Flush 0 Done Done Sent false
4301 Acked Flush 0 Done Done Sent false
4302 Acked Write 3 Done Done Sent false
4303 Acked Write 1 Done Done Sent false
4304 Acked Read 5 Done Done Sent false
4305 Acked Write 5 Done Done Sent false
4306 Acked Write 7 Done Done Sent false
4307 Acked Flush 0 Done Done Sent false
4308 Acked Write 9 Done Done Sent false
4309 Acked Flush 0 Done Done Sent false
4310 Acked Read 4 Done Done Sent false
4311 Acked Read 2 Done Done Sent false
4312 Acked Write 8 Done Done Sent false
4313 Acked Read 5 Done Done Sent false
4314 Acked Read 6 Done Done Sent false
4315 Acked Flush 0 Done Done Sent false
4316 Acked Write 6 Done Done Sent false
4317 Acked Write 2 Done Done Sent false
4318 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4318 4318 4016
Downstairs last five completed: 4016 4015 4014 4013 4012
Upstairs last five completed: 4318 4317 4316 4315 4314
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.374031176Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57441)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.374098608Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.375149185Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.375760884Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.375787151Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 4016","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.375804389Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 4016","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.376073517Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 4016","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.376101924Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 4016","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.376327468Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4318 4317 4316 4315 4314
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Jan 27 19:38:55.626 INFO request completed, latency_us: 362, response_code:[8820] Got stop action so:false kr:false
204, uri: /stop/cid/1, method: POST, req_id: 41c367a2-4aae-4c76-8413-378f226f9180, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:55.626706242Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:55.626747627Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:55.626765135Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:38:55.626 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/1, method: GET, req_id: 81bf71fe-f088-4a0c-9922-f1031751c5f3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:55.626856165Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:38:59.629 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0e1005a3-0a20-49c2-b295-984e50526c6d, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2715 len 32768 data: 2 2 2 4 6 5 4 4
002/300 Read block 4247 len 40960
003/300 Flush
004/300 Write block 552 len 24576 data: 5 5 5 3 2 3
005/300 Read block 405 len 36864
006/300 Read block 1158 len 40960
007/300 Read block 3800 len 24576
008/300 Read block 2726 len 20480
009/300 Write block 4893 len 8192 data: 2 2
010/300 Write block 2773 len 28672 data: 5 4 3 2 2 2 3
011/300 Write block 828 len 32768 data: 2 3 5 5 6 5 5 5
012/300 Read block 3703 len 20480
013/300 Read block 4092 len 20480
014/300 Write block 4480 len 28672 data: 4 4 4 3 2 2 3
015/300 Read block 1777 len 16384
016/300 Read block 4231 len 12288
017/300 Write block 3465 len 20480 data: 4 4 4 3 3
018/300 Read block 1402 len 24576
019/300 Write block 1342 len 36864 data: 4 4 4 4 4 4 4 3 3
020/300 Read block 2565 len 4096
021/300 Read block 649 len 40960
022/300 Read block 4590 len 24576
023/300 Write block 2004 len 12288 data: 2 2 2
024/300 Write block 2949 len 24576 data: 3 3 4 4 4 4
025/300 Read block 2814 len 16384
026/300 Read block 389 len 12288
027/300 Read block 3827 len 16384
028/300 Write block 1399 len 20480 data: 5 5 5 4 4
029/300 Read block 2606 len 40960
030/300 Read block 4670 len 8192
031/300 Read block 475 len 8192
032/300 Read block 2121 len 12288
033/300 Read block 1618 len 36864
034/300 Read block 1124 len 36864
035/300 Write block 3350 len 12288 data: 3 2 2
036/300 Write block 2209 len 16384 data: 3 3 3 3
037/300 Write block 3528 len 12288 data: 3 2 2
038/300 Flush
039/300 Write block 1510 len 4096 data: 2
040/300 Write block 1249 len 24576 data: 3 2 3 3 3 3
041/300 Write block 2610 len 16384 data: 2 3 3 3
042/300 Read block 149 len 28672
043/300 Write block 3367 len 24576 data: 4 4 4 4 4 5
044/300 Read block 4543 len 12288
045/300 Read block 2773 len 40960
046/300 Read block 1255 len 8192
047/300 Read block 997 len 20480
048/300 Read block 3390 len 8192
049/300 Read block 3943 len 8192
050/300 Read block 189 len 16384
051/300 Write block 4759 len 32768 data: 2 2 2 2 2 3 3 2
052/300 Flush
053/300 Read block 334 len 40960
054/300 Flush
055/300 Read block 3164 len 40960
056/300 Write block 768 len 24576 data: 3 3 3 3 3 4
057/300 Write block 478 len 24576 data: 5 4 4 4 4 3
058/300 Read block 4167 len 20480
059/300 Write block 4256 len 36864 data: 3 3 2 2 3 3 3 3 3
060/300 Read block 2853 len 24576
061/300 Read block 1502 len 12288
062/300 Read block 4134 len 8192
063/300 Read block 2912 len 12288
064/300 Read block 2743 len 4096
065/300 Write block 2287 len 36864 data: 6 8 7 7 5 5 2 2 2
066/300 Write block 2285 len 8192 data: 4 5
067/300 Read block 1492 len 20480
068/300 Write block 2256 len 20480 data: 5 4 4 3 3
069/300 Write block 3031 len 16384 data: 3 2 2 2
070/300 Read block 3226 len 40960
071/300 Read block 2388 len 16384
072/300 Write block 4335 len 12288 data: 5 6 6
073/300 Read block 347 len 20480
074/300 Read block 4319 len 8192
075/300 Write block 2259 len 24576 data: 4 4 3 3 3 2
076/300 Read block 4150 len 28672
077/300 Flush
078/300 Read block 1800 len 24576
079/300 Read block 4562 len 40960
080/300 Write block 1697 len 40960 data: 3 3 3 2 2 2 2 2 2 2
081/300 Write block 2441 len 4096 data: 7
082/300 Write block 529 len 28672 data: 2 2 3 3 3 3 3
083/300 Write block 1998 len 32768 data: 5 5 3 3 3 3 3 3
084/300 Flush
085/300 Read block 1422 len 20480
086/300 Read block 1560 len 8192
087/300 Write block 3975 len 36864 data: 2 2 2 2 2 2 2 2 2
088/300 Write block 4762 len 40960 data: 3 3 4 4 3 2 2 3 4 5
089/300 Read block 4120 len 36864
090/300 Read block 2406 len 32768
091/300 Write block 4821 len 28672 data: 2 2 2 3 3 2 2
092/300 Write block 2409 len 8192 data: 3 4
093/300 Read block 4365 len 4096
094/300 Read block 3056 len 16384
095/300 Read block 4950 len 12288
096/300 Write block 3001 len 8192 data: 4 4
097/300 Write block 3143 len 40960 data: 8 5 5 4 3 3 3 4 5 4
098/300 Flush
099/300 Read block 2305 len 40960
100/300 Read block 2436 len 28672
101/300 Write block 1751 len 4096 data: 3
102/300 Write block 1409 len 8192 data: 4 3
103/300 Write block 2392 len 4096 data: 5
104/300 Write block 2872 len 28672 data: 3 3 3 4 4 4 3
105/300 Write block 2264 len 32768 data: 3 2 2 3 4 4 2 2
106/300 Write block 3074 len 32768 data: 2 2 2 2 2 2 2 2
107/300 Read block 1819 len 8192
108/300 Read block 3265 len 40960
109/300 Flush
110/300 Write block 1264 len 36864 data: 3 3 3 2 2 2 2 2 2
111/300 Read block 1803 len 4096
112/300 Write block 4056 len 36864 data: 5 5 5 5 3 3 3 3 2
113/300 Read block 2316 len 20480
114/300 Write block 2508 len 4096 data: 3
115/300 Write block 2172 len 20480 data: 4 5 5 6 6
116/300 Read block 866 len 24576
117/300 Read block 4496 len 40960
118/300 Flush
119/300 Read block 3621 len 36864
120/300 Read block 1391 len 40960
121/300 Read block 1612 len 24576
122/300 Write block 2927 len 12288 data: 2 3 3
123/300 Read block 509 len 12288
124/300 Read block 4173 len 4096
125/300 Write block 222 len 4096 data: 3
126/300 Read block 1304 len 12288
127/300 Write block 4805 len 4096 data: 5
128/300 Read block 3873 len 20480
129/300 Read block 780 len 40960
130/300 Write block 1981 len 32768 data: 2 2 3 3 4 4 4 4
131/300 Flush
132/300 Read block 705 len 12288
133/300 Read block 4362 len 28672
134/300 Read block 1368 len 40960
135/300 Write block 3346 len 4096 data: 3
136/300 Flush
137/300 Write block 3606 len 36864 data: 4 4 4 3 3 3 2 2 3
138/300 Write block 1615 len 24576 data: 3 4 4 4 4 3
139/300 Read block 2533 len 16384
140/300 Read block 4933 len 40960
141/300 Read block 1218 len 28672
142/300 Write block 658 len 32768 data: 5 5 4 4 3 3 3 3
143/300 Flush
144/300 Write block 676 len 12288 data: 3 3 3
145/300 Read block 69 len 32768
146/300 Read block 2946 len 28672
147/300 Flush
148/300 Read block 3463 len 12288
149/300 Read block 756 len 12288
150/300 Write block 4736 len 8192 data: 6 6
151/300 Write block 1814 len 36864 data: 4 5 4 4 3 2 3 3 3
152/300 Write block 3509 len 16384 data: 2 2 2 2
153/300 Read block 3773 len 32768
154/300 Write block 2375 len 16384 data: 4 4 4 5
155/300 Write block 2716 len 16384 data: 3 3 5 7
156/300 Read block 3878 len 32768
157/300 Write block 2029 len 8192 data: 4 4
158/300 Read block 2221 len 8192
159/300 Read block 4293 len 16384
160/300 Write block 409 len 20480 data: 4 4 3 3 3
161/300 Write block 3153 len 20480 data: 3 3 3 3 3
162/300 Write block 1626 len 40960 data: 4 7 5 4 3 3 3 3 3 3
163/300 Read block 4676 len 16384
164/300 Read block 3468 len 32768
165/300 Read block 1801 len 16384
166/300 Write block 1891 len 8192 data: 4 4
167/300 Write block 3567 len 32768 data: 3 4 4 4 4 3 3 3
168/300 Write block 3931 len 28672 data: 4 4 4 3 3 4 5
169/300 Read block 1249 len 8192
170/300 Read block 3710 len 16384
171/300 Read block 1675 len 36864
172/300 Write block 3338 len 16384 data: 4 4 4 2
173/300 Read block 2757 len 32768
174/300 Read block 2278 len 32768
175/300 Write block 2611 len 16384 data: 4 4 4 3
176/300 Write block 2201 len 12288 data: 5 4 5
177/300 Read block 1425 len 12288
178/300 Read block 3625 len 36864
179/300 Write block 3606 len 16384 data: 5 5 5 4
180/300 Write block 2974 len 28672 data: 4 3 3 3 4 6 6
181/300 Read block 3711 len 24576
182/300 Write block 3850 len 32768 data: 3 3 2 3 3 3 2 2
183/300 Write block 4336 len 16384 data: 7 7 6 5
184/300 Write block 1127 len 36864 data: 5 6 7 5 5 5 4 4 3
185/300 Write block 3062 len 28672 data: 3 3 2 2 2 2 2
186/300 Read block 291 len 40960
187/300 Read block 1903 len 40960
188/300 Write block 4740 len 28672 data: 4 4 3 2 2 2 3
189/300 Read block 2581 len 32768
190/300 Read block 305 len 8192
191/300 Read block 2034 len 28672
192/300 Write block 4861 len 12288 data: 4 3 4
193/300 Write block 517 len 28672 data: 2 2 2 2 2 2 2
194/300 Read block 4269 len 24576
195/300 Write block 2076 len 36864 data: 3 3 3 3 3 3 3 3 3
196/300 Write block 1388 len 16384 data: 3 3 3 4
197/300 Read block 4407 len 24576
198/300 Flush
199/300 Write block 3537 len 16384 data: 2 2 3 3
200/300 Read block 1114 len 28672
201/300 Read block 3685 len 32768
202/300 Read block 834 len 36864
203/300 Flush
204/300 Write block 3115 len 20480 data: 2 2 3 3 2
205/300 Write block 3462 len 4096 data: 3
206/300 Read block 3955 len 36864
207/300 Read block 3728 len 8192
208/300 Read block 3057 len 28672
209/300 Read block 3374 len 16384
210/300 Write block 934 len 20480 data: 3 2 2 2 2
211/300 Read block 4425 len 36864
212/300 Read block 142 len 40960
213/300 Read block 397 len 28672
214/300 Write block 3051 len 24576 data: 5 4 3 4 3 4
215/300 Read block 4865 len 20480
216/300 Read block 428 len 28672
217/300 Write block 3767 len 4096 data: 5
218/300 Read block 1927 len 20480
219/300 Read block 912 len 8192
220/300 Write block 2679 len 36864 data: 6 7 7 6 7 7 5 5 4
221/300 Read block 3690 len 32768
222/300 Read block 2440 len 4096
223/300 Write block 4211 len 12288 data: 2 2 2
224/300 Read block 2304 len 20480
225/300 Read block 980 len 40960
226/300 Read block 287 len 28672
227/300 Write block 272 len 28672 data: 3 3 3 4 4 4 5
228/300 Read block 797 len 8192
229/300 Read block 4470 len 36864
230/300 Read block 1933 len 40960
231/300 Write block 115 len 40960 data: 3 3 3 3 3 4 4 5 5 6
232/300 Write block 497 len 24576 data: 5 6 6 4 4 4
233/300 Read block 2741 len 24576
234/300 Flush
235/300 Read block 523 len 16384
236/300 Write block 4224 len 4096 data: 4
237/300 Read block 1936 len 16384
238/300 Write block 3241 len 16384 data: 4 5 5 3
239/300 Write block 3754 len 36864 data: 4 4 4 4 5 6 6 6 6
240/300 Read block 2507 len 40960
241/300 Read block 2179 len 36864
242/300 Read block 4213 len 28672
243/300 Flush
244/300 Read block 3316 len 36864
245/300 Read block 3627 len 12288
246/300 Write block 1715 len 36864 data: 3 3 3 2 3 3 3 3 3
247/300 Flush
248/300 Read block 2721 len 4096
249/300 Read block 4116 len 32768
250/300 Read block 2001 len 12288
251/300 Read block 1370 len 8192
252/300 Write block 1147 len 16384 data: 3 4 4 2
253/300 Read block 826 len 12288
254/300 Read block 2873 len 32768
255/300 Read block 2546 len 20480
256/300 Read block 4405 len 40960
257/300 Read block 4473 len 4096
258/300 Read block 3782 len 28672
259/300 Read block 4384 len 32768
260/300 Read block 4471 len 28672
261/300 Write block 2624 len 40960 data: 5 3 2 2 2 3 2 2 2 4
262/300 Read block 1920 len 24576
263/300 Read block 2423 len 20480
264/300 Write block 2157 len 24576 data: 6 6 6 5 6 6
265/300 Flush
266/300 Read block 2974 len 36864
267/300 Read block 4788 len 36864
268/300 Flush
269/300 Write block 114 len 12288 data: 3 4 4
270/300 Read block 3719 len 24576
271/300 Write block 2386 len 20480 data: 4 4 5 5 6
272/300 Flush
273/300 Flush
274/300 Read block 3166 len 4096
275/300 Write block 4797 len 20480 data: 4 3 4 4 4
276/300 Write block 4864 len 28672 data: 4 3 3 4 4 5 4
277/300 Read block 1754 len 20480
278/300 Read block 4323 len 28672
279/300 Read block 1725 len 20480
280/300 Write block 3455 len 20480 data: 6 6 6 4 3
281/300 Write block 2820 len 40960 data: 2 2 2 2 2 2 2 2 2 2
282/300 Read block 1312 len 16384
283/300 Read block 2617 len 20480
284/300 Write block 4363 len 16384 data: 3 3 3 3
285/300 Read block 3716 len 8192
286/300 Read block 414 len 28672
287/300 Read block 4608 len 12288
288/300 Read block 859 len 20480
289/300 Read block 949 len 32768
290/300 Write block 4102 len 40960 data: 4 5 4 4 4 5 3 4 4 5
291/300 Read block 1606 len 8192
292/300 Read block 4336 len 12288
293/300 Write block 2266 len 16384 data: 3 4 5 5
294/300 Write block 4225 len 24576 data: 4 3 2 2 3 3
295/300 Write block 1889 len 4096 data: 4
296/300 Write block 2216 len 16384 data: 2 2 2 2
297/300 Write block 4041 len 28672 data: 3 4 5 4 4 4 3
298/300 Read block 707 len 36864
299/300 Write block 1572 len 36864 data: 4 4 3 4 4 3 3 2 3
300/300 Read block 3643 len 36864
301/300 Read block 1542 len 12288
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:38:59.879 INFO request completed, latency_us: 161, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2f0548c5-4a27-4305-945c-01644facebf5, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4319 Acked Write 8 Done Sent Done false
4320 Acked Read 10 Done Sent Done false
4321 Acked Flush 0 Done Sent Done false
4322 Acked Write 6 Done Sent Done false
4323 Acked Read 9 Done Sent Done false
4324 Acked Read 10 Done Sent Done false
4325 Acked Read 6 Done Sent Done false
4326 Acked Read 5 Done Sent Done false
4327 Acked Write 2 Done Sent Done false
4328 Acked Write 7 Done Sent Done false
4329 Acked Write 8 Done Sent Done false
4330 Acked Read 5 Done Sent Done false
4331 Acked Read 5 Done Sent Done false
4332 Acked Write 7 Done Sent Done false
4333 Acked Read 4 Done Sent Done false
4334 Acked Read 3 Done Sent Done false
4335 Acked Write 5 Done Sent Done false
4336 Acked Read 6 Done Sent Done false
4337 Acked Write 9 Done Sent Done false
4338 Acked Read 1 Done Sent Done false
4339 Acked Read 10 Done Sent Done false
4340 Acked Read 6 Done Sent Done false
4341 Acked Write 3 Done Sent Done false
4342 Acked Write 6 Done Sent Done false
4343 Acked Read 4 Done Sent Done false
4344 Acked Read 3 Done Sent Done false
4345 Acked Read 4 Done Sent Done false
4346 Acked Write 5 Done Sent Done false
4347 Acked Read 10 Done Sent Done false
4348 Acked Read 2 Done Sent Done false
4349 Acked Read 2 Done Sent Done false
4350 Acked Read 3 Done Sent Done false
4351 Acked Read 9 Done Sent Done false
4352 Acked Read 9 Done Sent Done false
4353 Acked Write 3 Done Sent Done false
4354 Acked Write 4 Done Sent Done false
4355 Acked Write 3 Done Sent Done false
4356 Acked Flush 0 Done Sent Done false
4357 Acked Write 1 Done Sent Done false
4358 Acked Write 6 Done Sent Done false
4359 Acked Write 4 Done Sent Done false
4360 Acked Read 7 Done Sent Done false
4361 Acked Write 6 Done Sent Done false
4362 Acked Read 3 Done Sent Done false
4363 Acked Read 10 Done Sent Done false
4364 Acked Read 2 Done Sent Done false
4365 Acked Read 5 Done Sent Done false
4366 Acked Read 2 Done Sent Done false
4367 Acked Read 2 Done Sent Done false
4368 Acked Read 4 Done Sent Done false
4369 Acked Write 8 Done Sent Done false
4370 Acked Flush 0 Done Sent Done false
4371 Acked Read 10 Done Sent Done false
4372 Acked Flush 0 Done Sent Done false
4373 Acked Read 10 Done Sent Done false
4374 Acked Write 6 Done Sent Done false
4375 Acked Write 6 Done Sent Done false
4376 Acked Read 5 Done Sent Done false
4377 Acked Write 9 Done Sent Done false
4378 Acked Read 6 Done Sent Done false
4379 Acked Read 3 Done Sent Done false
4380 Acked Read 2 Done Sent Done false
4381 Acked Read 3 Done Sent Done false
4382 Acked Read 1 Done Sent Done false
4383 Acked Write 9 Done Sent Done false
4384 Acked Write 2 Done Sent Done false
4385 Acked Read 5 Done Sent Done false
4386 Acked Write 5 Done Sent Done false
4387 Acked Write 4 Done Sent Done false
4388 Acked Read 10 Done Sent Done false
4389 Acked Read 4 Done Sent Done false
4390 Acked Write 3 Done Sent Done false
4391 Acked Read 5 Done Sent Done false
4392 Acked Read 2 Done Sent Done false
4393 Acked Write 6 Done Sent Done false
4394 Acked Read 7 Done Sent Done false
4395 Acked Flush 0 Done Sent Done false
4396 Acked Read 6 Done Sent Done false
4397 Acked Read 10 Done Sent Done false
4398 Acked Write 10 Done Sent Done false
4399 Acked Write 1 Done Sent Done false
4400 Acked Write 7 Done Sent Done false
4401 Acked Write 8 Done Sent Done false
4402 Acked Flush 0 Done Sent Done false
4403 Acked Read 5 Done Sent Done false
4404 Acked Read 2 Done Sent Done false
4405 Acked Write 9 Done Sent Done false
4406 Acked Write 10 Done Sent Done false
4407 Acked Read 9 Done Sent Done false
4408 Acked Read 8 Done Sent Done false
4409 Acked Write 7 Done Sent Done false
4410 Acked Write 2 Done Sent Done false
4411 Acked Read 1 Done Sent Done false
4412 Acked Read 4 Done Sent Done false
4413 Acked Read 3 Done Sent Done false
4414 Acked Write 2 Done Sent Done false
4415 Acked Write 10 Done Sent Done false
4416 Acked Flush 0 Done Sent Done false
4417 Acked Read 10 Done Sent Done false
4418 Acked Read 7 Done Sent Done false
4419 Acked Write 1 Done Sent Done false
4420 Acked Write 2 Done Sent Done false
4421 Acked Write 1 Done Sent Done false
4422 Acked Write 7 Done Sent Done false
4423 Acked Write 8 Done Sent Done false
4424 Acked Write 8 Done Sent Done false
4425 Acked Read 2 Done Sent Done false
4426 Acked Read 10 Done Sent Done false
4427 Acked Flush 0 Done Sent Done false
4428 Acked Write 9 Done Sent Done false
4429 Acked Read 1 Done Sent Done false
4430 Acked Write 9 Done Sent Done false
4431 Acked Read 5 Done Sent Done false
4432 Acked Write 1 Done Sent Done false
4433 Acked Write 5 Done Sent Done false
4434 Acked Read 6 Done Sent Done false
4435 Acked Read 10 Done Sent Done false
4436 Acked Flush 0 Done Sent Done false
4437 Acked Read 9 Done Sent Done false
4438 Acked Read 10 Done Sent Done false
4439 Acked Read 6 Done Sent Done false
4440 Acked Write 3 Done Sent Done false
4441 Acked Read 3 Done Sent Done false
4442 Acked Read 1 Done Sent Done false
4443 Acked Write 1 Done Sent Done false
4444 Acked Read 3 Done Sent Done false
4445 Acked Write 1 Done Sent Done false
4446 Acked Read 5 Done Sent Done false
4447 Acked Read 10 Done Sent Done false
4448 Acked Write 8 Done Sent Done false
4449 Acked Flush 0 Done Sent Done false
4450 Acked Read 3 Done Sent Done false
4451 Acked Read 7 Done Sent Done false
4452 Acked Read 10 Done Sent Done false
4453 Acked Write 1 Done Sent Done false
4454 Acked Flush 0 Done Sent Done false
4455 Acked Write 9 Done Sent Done false
4456 Acked Write 6 Done Sent Done false
4457 Acked Read 4 Done Sent Done false
4458 Acked Read 10 Done Sent Done false
4459 Acked Read 7 Done Sent Done false
4460 Acked Write 8 Done Sent Done false
4461 Acked Flush 0 Done Sent Done false
4462 Acked Write 3 Done Sent Done false
4463 Acked Read 8 Done Sent Done false
4464 Acked Read 7 Done Sent Done false
4465 Acked Flush 0 Done Sent Done false
4466 Acked Read 3 Done Sent Done false
4467 Acked Read 3 Done Sent Done false
4468 Acked Write 2 Done Sent Done false
4469 Acked Write 9 Done Sent Done false
4470 Acked Write 4 Done Sent Done false
4471 Acked Read 8 Done Sent Done false
4472 Acked Write 4 Done Sent Done false
4473 Acked Write 4 Done Sent Done false
4474 Acked Read 8 Done Sent Done false
4475 Acked Write 2 Done Sent Done false
4476 Acked Read 2 Done Sent Done false
4477 Acked Read 4 Done Sent Done false
4478 Acked Write 5 Done Sent Done false
4479 Acked Write 5 Done Sent Done false
4480 Acked Write 10 Done Sent Done false
4481 Acked Read 4 Done Sent Done false
4482 Acked Read 8 Done Sent Done false
4483 Acked Read 4 Done Sent Done false
4484 Acked Write 2 Done Sent Done false
4485 Acked Write 8 Done Sent Done false
4486 Acked Write 7 Done Sent Done false
4487 Acked Read 2 Done Sent Done false
4488 Acked Read 4 Done Sent Done false
4489 Acked Read 9 Done Sent Done false
4490 Acked Write 4 Done Sent Done false
4491 Acked Read 8 Done Sent Done false
4492 Acked Read 8 Done Sent Done false
4493 Acked Write 4 Done Sent Done false
4494 Acked Write 3 Done Sent Done false
4495 Acked Read 3 Done Sent Done false
4496 Acked Read 9 Done Sent Done false
4497 Acked Write 4 Done Sent Done false
4498 Acked Write 7 Done Sent Done false
4499 Acked Read 6 Done Sent Done false
4500 Acked Write 8 Done Sent Done false
4501 Acked Write 4 Done Sent Done false
4502 Acked Write 9 Done Sent Done false
4503 Acked Write 7 Done Sent Done false
4504 Acked Read 10 Done Sent Done false
4505 Acked Read 10 Done Sent Done false
4506 Acked Write 7 Done Sent Done false
4507 Acked Read 8 Done Sent Done false
4508 Acked Read 2 Done Sent Done false
4509 Acked Read 7 Done Sent Done false
4510 Acked Write 3 Done Sent Done false
4511 Acked Write 7 Done Sent Done false
4512 Acked Read 6 Done Sent Done false
4513 Acked Write 9 Done Sent Done false
4514 Acked Write 4 Done Sent Done false
4515 Acked Read 6 Done Sent Done false
4516 Acked Flush 0 Done Sent Done false
4517 Acked Write 4 Done Sent Done false
4518 Acked Read 7 Done Sent Done false
4519 Acked Read 8 Done Sent Done false
4520 Acked Read 9 Done Sent Done false
4521 Acked Flush 0 Done Sent Done false
4522 Acked Write 5 Done Sent Done false
4523 Acked Write 1 Done Sent Done false
4524 Acked Read 9 Done Sent Done false
4525 Acked Read 2 Done Sent Done false
4526 Acked Read 7 Done Sent Done false
4527 Acked Read 4 Done Sent Done false
4528 Acked Write 5 Done Sent Done false
4529 Acked Read 9 Done Sent Done false
4530 Acked Read 10 Done Sent Done false
4531 Acked Read 7 Done Sent Done false
4532 Acked Write 6 Done Sent Done false
4533 Acked Read 5 Done Sent Done false
4534 Acked Read 7 Done Sent Done false
4535 Acked Write 1 Done Sent Done false
4536 Acked Read 5 Done Sent Done false
4537 Acked Read 2 Done Sent Done false
4538 Acked Write 9 Done Sent Done false
4539 Acked Read 8 Done Sent Done false
4540 Acked Read 1 Done Sent Done false
4541 Acked Write 3 Done Sent Done false
4542 Acked Read 5 Done Sent Done false
4543 Acked Read 10 Done Sent Done false
4544 Acked Read 7 Done Sent Done false
4545 Acked Write 7 Done Sent Done false
4546 Acked Read 2 Done Sent Done false
4547 Acked Read 9 Done Sent Done false
4548 Acked Read 10 Done Sent Done false
4549 Acked Write 10 Done Sent Done false
4550 Acked Write 6 Done Sent Done false
4551 Acked Read 6 Done Sent Done false
4552 Acked Flush 0 Done Sent Done false
4553 Acked Read 4 Done Sent Done false
4554 Acked Write 1 Done Sent Done false
4555 Acked Read 4 Done Sent Done false
4556 Acked Write 4 Done Sent Done false
4557 Acked Write 9 Done Sent Done false
4558 Acked Read 10 Done Sent Done false
4559 Acked Read 9 Done Sent Done false
4560 Acked Read 7 Done Sent Done false
4561 Acked Flush 0 Done Sent Done false
4562 Acked Read 9 Done Sent Done false
4563 Acked Read 3 Done Sent Done false
4564 Acked Write 9 Done Sent Done false
4565 Acked Flush 0 Done Sent Done false
4566 Acked Read 1 Done Sent Done false
4567 Acked Read 8 Done Sent Done false
4568 Acked Read 3 Done Sent Done false
4569 Acked Read 2 Done Sent Done false
4570 Acked Write 4 Done Sent Done false
4571 Acked Read 3 Done Sent Done false
4572 Acked Read 8 Done Sent Done false
4573 Acked Read 5 Done Sent Done false
4574 Acked Read 10 Done Sent Done false
4575 Acked Read 1 Done Sent Done false
4576 Acked Read 7 Done Sent Done false
4577 Acked Read 8 Done Sent Done false
4578 Acked Read 7 Done Sent Done false
4579 Acked Write 10 Done Sent Done false
4580 Acked Read 6 Done Sent Done false
4581 Acked Read 5 Done Sent Done false
4582 Acked Write 6 Done Sent Done false
4583 Acked Flush 0 Done Sent Done false
4584 Acked Read 9 Done Sent Done false
4585 Acked Read 9 Done Sent Done false
4586 Acked Flush 0 Done Sent Done false
4587 Acked Write 3 Done Sent Done false
4588 Acked Read 6 Done Sent Done false
4589 Acked Write 5 Done Sent Done false
4590 Acked Flush 0 Done Sent Done false
4591 Acked Flush 0 Done Sent Done false
4592 Acked Read 1 Done Sent Done false
4593 Acked Write 5 Done Sent Done false
4594 Acked Write 7 Done Sent Done false
4595 Acked Read 5 Done Sent Done false
4596 Acked Read 7 Done Sent Done false
4597 Acked Read 5 Done Sent Done false
4598 Acked Write 5 Done Sent Done false
4599 Acked Write 10 Done Sent Done false
4600 Acked Read 4 Done Sent Done false
4601 Acked Read 5 Done Sent Done false
4602 Acked Write 4 Done Sent Done false
4603 Acked Read 2 Done Sent Done false
4604 Acked Read 7 Done Sent Done false
4605 Acked Read 3 Done Sent Done false
4606 Acked Read 5 Done Sent Done false
4607 Acked Read 8 Done Sent Done false
4608 Acked Write 10 Done Sent Done false
4609 Acked Read 2 Done Sent Done false
4610 Acked Read 3 Done Sent Done false
4611 Acked Write 4 Done Sent Done false
4612 Acked Write 6 Done Sent Done false
4613 Acked Write 1 Done Sent Done false
4614 Acked Write 4 Done Sent Done false
4615 Acked Write 7 Done Sent Done false
4616 Acked Read 9 Done Sent Done false
4617 Acked Write 9 Done Sent Done false
4618 Acked Read 9 Done Sent Done false
4619 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4591 4318 4591
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4619 4618 4617 4616 4615
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4319 Acked Write 8 Done Sent Done false
4320 Acked Read 10 Done Sent Done false
4321 Acked Flush 0 Done Sent Done false
4322 Acked Write 6 Done Sent Done false
4323 Acked Read 9 Done Sent Done false
4324 Acked Read 10 Done Sent Done false
4325 Acked Read 6 Done Sent Done false
4326 Acked Read 5 Done Sent Done false
4327 Acked Write 2 Done Sent Done false
4328 Acked Write 7 Done Sent Done false
4329 Acked Write 8 Done Sent Done false
4330 Acked Read 5 Done Sent Done false
4331 Acked Read 5 Done Sent Done false
4332 Acked Write 7 Done Sent Done false
4333 Acked Read 4 Done Sent Done false
4334 Acked Read 3 Done Sent Done false
4335 Acked Write 5 Done Sent Done false
4336 Acked Read 6 Done Sent Done false
4337 Acked Write 9 Done Sent Done false
4338 Acked Read 1 Done Sent Done false
4339 Acked Read 10 Done Sent Done false
4340 Acked Read 6 Done Sent Done false
4341 Acked Write 3 Done Sent Done false
4342 Acked Write 6 Done Sent Done false
4343 Acked Read 4 Done Sent Done false
4344 Acked Read 3 Done Sent Done false
4345 Acked Read 4 Done Sent Done false
4346 Acked Write 5 Done Sent Done false
4347 Acked Read 10 Done Sent Done false
4348 Acked Read 2 Done Sent Done false
4349 Acked Read 2 Done Sent Done false
4350 Acked Read 3 Done Sent Done false
4351 Acked Read 9 Done Sent Done false
4352 Acked Read 9 Done Sent Done false
4353 Acked Write 3 Done Sent Done false
4354 Acked Write 4 Done Sent Done false
4355 Acked Write 3 Done Sent Done false
4356 Acked Flush 0 Done Sent Done false
4357 Acked Write 1 Done Sent Done false
4358 Acked Write 6 Done Sent Done false
4359 Acked Write 4 Done Sent Done false
4360 Acked Read 7 Done Sent Done false
4361 Acked Write 6 Done Sent Done false
4362 Acked Read 3 Done Sent Done false
4363 Acked Read 10 Done Sent Done false
4364 Acked Read 2 Done Sent Done false
4365 Acked Read 5 Done Sent Done false
4366 Acked Read 2 Done Sent Done false
4367 Acked Read 2 Done Sent Done false
4368 Acked Read 4 Done Sent Done false
4369 Acked Write 8 Done Sent Done false
4370 Acked Flush 0 Done Sent Done false
4371 Acked Read 10 Done Sent Done false
4372 Acked Flush 0 Done Sent Done false
4373 Acked Read 10 Done Sent Done false
4374 Acked Write 6 Done Sent Done false
4375 Acked Write 6 Done Sent Done false
4376 Acked Read 5 Done Sent Done false
4377 Acked Write 9 Done Sent Done false
4378 Acked Read 6 Done Sent Done false
4379 Acked Read 3 Done Sent Done false
4380 Acked Read 2 Done Sent Done false
4381 Acked Read 3 Done Sent Done false
4382 Acked Read 1 Done Sent Done false
4383 Acked Write 9 Done Sent Done false
4384 Acked Write 2 Done Sent Done false
4385 Acked Read 5 Done Sent Done false
4386 Acked Write 5 Done Sent Done false
4387 Acked Write 4 Done Sent Done false
4388 Acked Read 10 Done Sent Done false
4389 Acked Read 4 Done Sent Done false
4390 Acked Write 3 Done Sent Done false
4391 Acked Read 5 Done Sent Done false
4392 Acked Read 2 Done Sent Done false
4393 Acked Write 6 Done Sent Done false
4394 Acked Read 7 Done Sent Done false
4395 Acked Flush 0 Done Sent Done false
4396 Acked Read 6 Done Sent Done false
4397 Acked Read 10 Done Sent Done false
4398 Acked Write 10 Done Sent Done false
4399 Acked Write 1 Done Sent Done false
4400 Acked Write 7 Done Sent Done false
4401 Acked Write 8 Done Sent Done false
4402 Acked Flush 0 Done Sent Done false
4403 Acked Read 5 Done Sent Done false
4404 Acked Read 2 Done Sent Done false
4405 Acked Write 9 Done Sent Done false
4406 Acked Write 10 Done Sent Done false
4407 Acked Read 9 Done Sent Done false
4408 Acked Read 8 Done Sent Done false
4409 Acked Write 7 Done Sent Done false
4410 Acked Write 2 Done Sent Done false
4411 Acked Read 1 Done Sent Done false
4412 Acked Read 4 Done Sent Done false
4413 Acked Read 3 Done Sent Done false
4414 Acked Write 2 Done Sent Done false
4415 Acked Write 10 Done Sent Done false
4416 Acked Flush 0 Done Sent Done false
4417 Acked Read 10 Done Sent Done false
4418 Acked Read 7 Done Sent Done false
4419 Acked Write 1 Done Sent Done false
4420 Acked Write 2 Done Sent Done false
4421 Acked Write 1 Done Sent Done false
4422 Acked Write 7 Done Sent Done false
4423 Acked Write 8 Done Sent Done false
4424 Acked Write 8 Done Sent Done false
4425 Acked Read 2 Done Sent Done false
4426 Acked Read 10 Done Sent Done false
4427 Acked Flush 0 Done Sent Done false
4428 Acked Write 9 Done Sent Done false
4429 Acked Read 1 Done Sent Done false
4430 Acked Write 9 Done Sent Done false
4431 Acked Read 5 Done Sent Done false
4432 Acked Write 1 Done Sent Done false
4433 Acked Write 5 Done Sent Done false
4434 Acked Read 6 Done Sent Done false
4435 Acked Read 10 Done Sent Done false
4436 Acked Flush 0 Done Sent Done false
4437 Acked Read 9 Done Sent Done false
4438 Acked Read 10 Done Sent Done false
4439 Acked Read 6 Done Sent Done false
4440 Acked Write 3 Done Sent Done false
4441 Acked Read 3 Done Sent Done false
4442 Acked Read 1 Done Sent Done false
4443 Acked Write 1 Done Sent Done false
4444 Acked Read 3 Done Sent Done false
4445 Acked Write 1 Done Sent Done false
4446 Acked Read 5 Done Sent Done false
4447 Acked Read 10 Done Sent Done false
4448 Acked Write 8 Done Sent Done false
4449 Acked Flush 0 Done Sent Done false
4450 Acked Read 3 Done Sent Done false
4451 Acked Read 7 Done Sent Done false
4452 Acked Read 10 Done Sent Done false
4453 Acked Write 1 Done Sent Done false
4454 Acked Flush 0 Done Sent Done false
4455 Acked Write 9 Done Sent Done false
4456 Acked Write 6 Done Sent Done false
4457 Acked Read 4 Done Sent Done false
4458 Acked Read 10 Done Sent Done false
4459 Acked Read 7 Done Sent Done false
4460 Acked Write 8 Done Sent Done false
4461 Acked Flush 0 Done Sent Done false
4462 Acked Write 3 Done Sent Done false
4463 Acked Read 8 Done Sent Done false
4464 Acked Read 7 Done Sent Done false
4465 Acked Flush 0 Done Sent Done false
4466 Acked Read 3 Done Sent Done false
4467 Acked Read 3 Done Sent Done false
4468 Acked Write 2 Done Sent Done false
4469 Acked Write 9 Done Sent Done false
4470 Acked Write 4 Done Sent Done false
4471 Acked Read 8 Done Sent Done false
4472 Acked Write 4 Done Sent Done false
4473 Acked Write 4 Done Sent Done false
4474 Acked Read 8 Done Sent Done false
4475 Acked Write 2 Done Sent Done false
4476 Acked Read 2 Done Sent Done false
4477 Acked Read 4 Done Sent Done false
4478 Acked Write 5 Done Sent Done false
4479 Acked Write 5 Done Sent Done false
4480 Acked Write 10 Done Sent Done false
4481 Acked Read 4 Done Sent Done false
4482 Acked Read 8 Done Sent Done false
4483 Acked Read 4 Done Sent Done false
4484 Acked Write 2 Done Sent Done false
4485 Acked Write 8 Done Sent Done false
4486 Acked Write 7 Done Sent Done false
4487 Acked Read 2 Done Sent Done false
4488 Acked Read 4 Done Sent Done false
4489 Acked Read 9 Done Sent Done false
4490 Acked Write 4 Done Sent Done false
4491 Acked Read 8 Done Sent Done false
4492 Acked Read 8 Done Sent Done false
4493 Acked Write 4 Done Sent Done false
4494 Acked Write 3 Done Sent Done false
4495 Acked Read 3 Done Sent Done false
4496 Acked Read 9 Done Sent Done false
4497 Acked Write 4 Done Sent Done false
4498 Acked Write 7 Done Sent Done false
4499 Acked Read 6 Done Sent Done false
4500 Acked Write 8 Done Sent Done false
4501 Acked Write 4 Done Sent Done false
4502 Acked Write 9 Done Sent Done false
4503 Acked Write 7 Done Sent Done false
4504 Acked Read 10 Done Sent Done false
4505 Acked Read 10 Done Sent Done false
4506 Acked Write 7 Done Sent Done false
4507 Acked Read 8 Done Sent Done false
4508 Acked Read 2 Done Sent Done false
4509 Acked Read 7 Done Sent Done false
4510 Acked Write 3 Done Sent Done false
4511 Acked Write 7 Done Sent Done false
4512 Acked Read 6 Done Sent Done false
4513 Acked Write 9 Done Sent Done false
4514 Acked Write 4 Done Sent Done false
4515 Acked Read 6 Done Sent Done false
4516 Acked Flush 0 Done Sent Done false
4517 Acked Write 4 Done Sent Done false
4518 Acked Read 7 Done Sent Done false
4519 Acked Read 8 Done Sent Done false
4520 Acked Read 9 Done Sent Done false
4521 Acked Flush 0 Done Sent Done false
4522 Acked Write 5 Done Sent Done false
4523 Acked Write 1 Done Sent Done false
4524 Acked Read 9 Done Sent Done false
4525 Acked Read 2 Done Sent Done false
4526 Acked Read 7 Done Sent Done false
4527 Acked Read 4 Done Sent Done false
4528 Acked Write 5 Done Sent Done false
4529 Acked Read 9 Done Sent Done false
4530 Acked Read 10 Done Sent Done false
4531 Acked Read 7 Done Sent Done false
4532 Acked Write 6 Done Sent Done false
4533 Acked Read 5 Done Sent Done false
4534 Acked Read 7 Done Sent Done false
4535 Acked Write 1 Done Sent Done false
4536 Acked Read 5 Done Sent Done false
4537 Acked Read 2 Done Sent Done false
4538 Acked Write 9 Done Sent Done false
4539 Acked Read 8 Done Sent Done false
4540 Acked Read 1 Done Sent Done false
4541 Acked Write 3 Done Sent Done false
4542 Acked Read 5 Done Sent Done false
4543 Acked Read 10 Done Sent Done false
4544 Acked Read 7 Done Sent Done false
4545 Acked Write 7 Done Sent Done false
4546 Acked Read 2 Done Sent Done false
4547 Acked Read 9 Done Sent Done false
4548 Acked Read 10 Done Sent Done false
4549 Acked Write 10 Done Sent Done false
4550 Acked Write 6 Done Sent Done false
4551 Acked Read 6 Done Sent Done false
4552 Acked Flush 0 Done Sent Done false
4553 Acked Read 4 Done Sent Done false
4554 Acked Write 1 Done Sent Done false
4555 Acked Read 4 Done Sent Done false
4556 Acked Write 4 Done Sent Done false
4557 Acked Write 9 Done Sent Done false
4558 Acked Read 10 Done Sent Done false
4559 Acked Read 9 Done Sent Done false
4560 Acked Read 7 Done Sent Done false
4561 Acked Flush 0 Done Sent Done false
4562 Acked Read 9 Done Sent Done false
4563 Acked Read 3 Done Sent Done false
4564 Acked Write 9 Done Sent Done false
4565 Acked Flush 0 Done Sent Done false
4566 Acked Read 1 Done Sent Done false
4567 Acked Read 8 Done Sent Done false
4568 Acked Read 3 Done Sent Done false
4569 Acked Read 2 Done Sent Done false
4570 Acked Write 4 Done Sent Done false
4571 Acked Read 3 Done Sent Done false
4572 Acked Read 8 Done Sent Done false
4573 Acked Read 5 Done Sent Done false
4574 Acked Read 10 Done Sent Done false
4575 Acked Read 1 Done Sent Done false
4576 Acked Read 7 Done Sent Done false
4577 Acked Read 8 Done Sent Done false
4578 Acked Read 7 Done Sent Done false
4579 Acked Write 10 Done Sent Done false
4580 Acked Read 6 Done Sent Done false
4581 Acked Read 5 Done Sent Done false
4582 Acked Write 6 Done Sent Done false
4583 Acked Flush 0 Done Sent Done false
4584 Acked Read 9 Done Sent Done false
4585 Acked Read 9 Done Sent Done false
4586 Acked Flush 0 Done Sent Done false
4587 Acked Write 3 Done Sent Done false
4588 Acked Read 6 Done Sent Done false
4589 Acked Write 5 Done Sent Done false
4590 Acked Flush 0 Done Sent Done false
4591 Acked Flush 0 Done Sent Done false
4592 Acked Read 1 Done Sent Done false
4593 Acked Write 5 Done Sent Done false
4594 Acked Write 7 Done Sent Done false
4595 Acked Read 5 Done Sent Done false
4596 Acked Read 7 Done Sent Done false
4597 Acked Read 5 Done Sent Done false
4598 Acked Write 5 Done Sent Done false
4599 Acked Write 10 Done Sent Done false
4600 Acked Read 4 Done Sent Done false
4601 Acked Read 5 Done Sent Done false
4602 Acked Write 4 Done Sent Done false
4603 Acked Read 2 Done Sent Done false
4604 Acked Read 7 Done Sent Done false
4605 Acked Read 3 Done Sent Done false
4606 Acked Read 5 Done Sent Done false
4607 Acked Read 8 Done Sent Done false
4608 Acked Write 10 Done Sent Done false
4609 Acked Read 2 Done Sent Done false
4610 Acked Read 3 Done Sent Done false
4611 Acked Write 4 Done Sent Done false
4612 Acked Write 6 Done Sent Done false
4613 Acked Write 1 Done Sent Done false
4614 Acked Write 4 Done Sent Done false
4615 Acked Write 7 Done Sent Done false
4616 Acked Read 9 Done Sent Done false
4617 Acked Write 9 Done Sent Done false
4618 Acked Read 9 Done Sent Done false
4619 Acked Read 3 Done Sent Done false
4620 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4620 4318 4620
Downstairs last five completed: 4318 4317 4316 4315 4314
Upstairs last five completed: 4620 4619 4618 4617 4616
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.627495566Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53696)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.62771761Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.628740501Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629377776Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629411393Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 4318","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629428101Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 4318","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629663003Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 4318","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629682691Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 4318","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:05.629963758Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4620 4619 4618 4617 4616
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Jan 27 19:39:07.885 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 84401d90-f99c-402c-9426-c7b1c60dd0f3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:07.886185877Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:39:07.886 INFO request completed, latency_us{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:07.886260058Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
: 144, response_code: 200, uri: /state/cid/0, method: GET, {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:07.886295454Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
req_id: 46f3d705-0d89-4e92-ac73-7b425e36205a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.886395762Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:39:11.888 INFO request completed, latency_us: 306, response_code: 200, uri: /state/cid/0, method: GET, req_id: 108bfa34-f9ab-48eb-921d-f07cf5a77813, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2558 len 24576 data: 3 3 4 4 4 5
002/300 Read block 1394 len 12288
003/300 Read block 182 len 4096
004/300 Write block 4348 len 40960 data: 4 4 4 4 4 4 3 4 3 3
005/300 Write block 1311 len 12288 data: 3 4 3
006/300 Read block 2736 len 8192
007/300 Read block 684 len 24576
008/300 Write block 279 len 24576 data: 7 9 10 8 6 5
009/300 Read block 4962 len 32768
010/300 Read block 2213 len 36864
011/300 Read block 3854 len 40960
012/300 Read block 1499 len 28672
013/300 Read block 3655 len 36864
014/300 Write block 511 len 24576 data: 3 2 2 2 2 2
015/300 Write block 558 len 24576 data: 3 4 4 4 4 4
016/300 Read block 201 len 12288
017/300 Read block 4500 len 8192
018/300 Read block 4114 len 28672
019/300 Write block 1739 len 40960 data: 3 3 3 2 2 2 2 2 2 2
020/300 Write block 3638 len 36864 data: 4 3 3 2 2 2 2 2 2
021/300 Write block 3275 len 32768 data: 4 4 5 5 4 4 4 4
022/300 Write block 3329 len 36864 data: 4 5 5 6 3 4 5 5 5
023/300 Read block 2348 len 16384
024/300 Write block 2247 len 4096 data: 4
025/300 Write block 3471 len 40960 data: 2 2 2 3 3 3 3 3 5 5
026/300 Read block 4718 len 4096
027/300 Write block 1818 len 40960 data: 4 3 4 4 4 2 2 2 2 2
028/300 Read block 4473 len 24576
029/300 Write block 3924 len 20480 data: 3 2 3 3 3
030/300 Read block 3881 len 36864
031/300 Flush
032/300 Write block 2140 len 32768 data: 3 4 4 4 5 4 4 3
033/300 Write block 4254 len 12288 data: 3 3 4
034/300 Read block 81 len 8192
035/300 Read block 3712 len 36864
036/300 Write block 1303 len 32768 data: 5 5 4 5 5 4 4 4
037/300 Write block 2952 len 20480 data: 5 5 5 4 4
038/300 Read block 487 len 32768
039/300 Write block 890 len 12288 data: 2 2 2
040/300 Read block 4583 len 24576
041/300 Write block 3678 len 4096 data: 2
042/300 Flush
043/300 Write block 1854 len 28672 data: 6 7 6 5 5 6 5
044/300 Read block 4176 len 40960
045/300 Write block 3777 len 20480 data: 6 5 4 3 3
046/300 Read block 263 len 8192
047/300 Read block 4070 len 20480
048/300 Read block 1171 len 40960
049/300 Read block 1201 len 12288
050/300 Read block 2420 len 36864
051/300 Read block 530 len 20480
052/300 Read block 2305 len 16384
053/300 Read block 45 len 16384
054/300 Write block 24 len 28672 data: 3 3 2 2 3 3 3
055/300 Read block 875 len 28672
056/300 Write block 501 len 8192 data: 5 5
057/300 Read block 2101 len 36864
058/300 Write block 1445 len 36864 data: 2 2 2 3 4 4 5 5 5
059/300 Read block 2288 len 40960
060/300 Read block 3570 len 16384
061/300 Read block 1138 len 16384
062/300 Read block 4311 len 24576
063/300 Read block 2179 len 16384
064/300 Flush
065/300 Write block 765 len 32768 data: 2 2 3 4 4 4 4 4
066/300 Read block 3577 len 16384
067/300 Flush
068/300 Flush
069/300 Read block 1031 len 40960
070/300 Flush
071/300 Write block 3104 len 28672 data: 4 4 4 4 4 3 2
072/300 Write block 4086 len 28672 data: 2 2 2 2 2 2 2
073/300 Flush
074/300 Read block 971 len 32768
075/300 Read block 4351 len 8192
076/300 Read block 534 len 28672
077/300 Read block 821 len 16384
078/300 Write block 3174 len 8192 data: 4 4
079/300 Write block 137 len 16384 data: 2 4 4 3
080/300 Read block 228 len 4096
081/300 Write block 1463 len 36864 data: 3 4 4 4 4 3 3 5 5
082/300 Read block 1506 len 8192
083/300 Write block 3774 len 8192 data: 7 7
084/300 Flush
085/300 Read block 4921 len 12288
086/300 Write block 1807 len 32768 data: 4 4 4 6 7 6 4 5
087/300 Write block 2828 len 24576 data: 3 3 4 4 5 5
088/300 Read block 1322 len 8192
089/300 Write block 35 len 28672 data: 2 2 3 3 3 4 3
090/300 Read block 2982 len 32768
091/300 Read block 1030 len 24576
092/300 Write block 2604 len 4096 data: 4
093/300 Read block 717 len 16384
094/300 Read block 2326 len 36864
095/300 Write block 1819 len 16384 data: 4 5 5 5
096/300 Read block 3206 len 40960
097/300 Write block 3436 len 16384 data: 3 3 3 3
098/300 Read block 238 len 28672
099/300 Read block 817 len 4096
100/300 Flush
101/300 Read block 1008 len 16384
102/300 Read block 2813 len 8192
103/300 Read block 3274 len 24576
104/300 Flush
105/300 Read block 2718 len 16384
106/300 Write block 2389 len 4096 data: 6
107/300 Read block 663 len 16384
108/300 Write block 983 len 20480 data: 3 3 3 3 3
109/300 Write block 2766 len 8192 data: 3 4
110/300 Read block 1140 len 36864
111/300 Read block 2779 len 40960
112/300 Write block 638 len 16384 data: 3 3 3 3
113/300 Read block 2749 len 28672
114/300 Write block 528 len 12288 data: 2 3 3
115/300 Write block 4427 len 8192 data: 4 4
116/300 Read block 3116 len 40960
117/300 Read block 4461 len 32768
118/300 Flush
119/300 Flush
120/300 Write block 4925 len 36864 data: 5 4 5 5 4 3 3 3 2
121/300 Write block 2718 len 4096 data: 6
122/300 Write block 426 len 16384 data: 3 2 2 2
123/300 Read block 2357 len 20480
124/300 Read block 2297 len 36864
125/300 Write block 2605 len 24576 data: 4 4 4 4 3 3
126/300 Read block 1160 len 12288
127/300 Write block 1395 len 12288 data: 4 3 3
128/300 Read block 1635 len 40960
129/300 Write block 4161 len 28672 data: 4 4 4 4 5 4 4
130/300 Read block 4898 len 16384
131/300 Read block 1464 len 20480
132/300 Write block 2669 len 40960 data: 3 3 3 4 5 5 5 4 5 5
133/300 Write block 4035 len 20480 data: 4 5 5 3 3
134/300 Read block 2645 len 28672
135/300 Read block 4487 len 16384
136/300 Flush
137/300 Flush
138/300 Write block 2841 len 36864 data: 4 4 4 5 5 6 4 5 5
139/300 Write block 2353 len 8192 data: 6 5
140/300 Write block 4008 len 16384 data: 4 5 6 4
141/300 Write block 1376 len 36864 data: 7 7 6 6 5 4 4 4 4
142/300 Write block 1690 len 28672 data: 2 2 2 2 3 3 3
143/300 Write block 1190 len 16384 data: 3 3 3 4
144/300 Read block 3868 len 4096
145/300 Read block 3357 len 40960
146/300 Read block 4408 len 28672
147/300 Read block 2653 len 32768
148/300 Write block 1122 len 24576 data: 4 4 4 4 5 6
149/300 Flush
150/300 Write block 3502 len 36864 data: 2 2 3 3 2 2 2 3 3
151/300 Write block 2194 len 12288 data: 4 4 4
152/300 Read block 971 len 4096
153/300 Read block 1824 len 12288
154/300 Write block 3215 len 32768 data: 4 4 4 5 5 5 5 6
155/300 Read block 1905 len 32768
156/300 Read block 1759 len 24576
157/300 Read block 138 len 16384
158/300 Read block 2372 len 36864
159/300 Read block 710 len 4096
160/300 Write block 2204 len 12288 data: 3 4 3
161/300 Write block 2016 len 36864 data: 3 3 4 3 3 3 3 3 2
162/300 Read block 3815 len 28672
163/300 Write block 3988 len 28672 data: 2 2 2 2 2 2 2
164/300 Read block 4048 len 36864
165/300 Read block 4669 len 4096
166/300 Write block 3426 len 40960 data: 7 6 5 5 5 5 4 4 4 3
167/300 Read block 3044 len 28672
168/300 Flush
169/300 Read block 286 len 32768
170/300 Read block 1959 len 12288
171/300 Write block 2242 len 36864 data: 3 3 3 4 4 5 4 3 3
172/300 Write block 3710 len 36864 data: 5 6 5 5 5 4 2 2 2
173/300 Read block 4742 len 4096
174/300 Write block 1635 len 20480 data: 4 3 3 3 3
175/300 Read block 4969 len 20480
176/300 Write block 2361 len 4096 data: 2
177/300 Flush
178/300 Read block 3110 len 4096
179/300 Write block 4112 len 8192 data: 5 5
180/300 Read block 3079 len 24576
181/300 Read block 1058 len 40960
182/300 Write block 4487 len 36864 data: 3 3 2 2 2 4 4 4 4
183/300 Read block 3745 len 40960
184/300 Write block 2378 len 4096 data: 6
185/300 Read block 3174 len 8192
186/300 Read block 4549 len 12288
187/300 Read block 4584 len 4096
188/300 Read block 3309 len 32768
189/300 Read block 784 len 40960
190/300 Read block 395 len 16384
191/300 Read block 2560 len 24576
192/300 Write block 2459 len 8192 data: 2 2
193/300 Read block 3759 len 16384
194/300 Write block 3191 len 32768 data: 4 4 4 4 4 4 3 3
195/300 Read block 1583 len 8192
196/300 Read block 969 len 28672
197/300 Read block 4424 len 16384
198/300 Write block 4133 len 20480 data: 3 4 3 3 3
199/300 Write block 1072 len 36864 data: 4 3 3 3 3 3 3 4 4
200/300 Write block 4503 len 4096 data: 4
201/300 Write block 1240 len 8192 data: 5 3
202/300 Flush
203/300 Read block 799 len 32768
204/300 Flush
205/300 Read block 3063 len 20480
206/300 Write block 3854 len 16384 data: 4 4 3 3
207/300 Write block 3838 len 28672 data: 4 4 4 5 6 5 4
208/300 Read block 1612 len 32768
209/300 Read block 2191 len 40960
210/300 Flush
211/300 Read block 1361 len 4096
212/300 Read block 2726 len 28672
213/300 Read block 552 len 20480
214/300 Read block 93 len 28672
215/300 Read block 978 len 16384
216/300 Write block 2793 len 12288 data: 5 6 5
217/300 Write block 1123 len 8192 data: 5 5
218/300 Read block 883 len 8192
219/300 Write block 2592 len 40960 data: 3 3 2 2 2 2 3 3 4 3
220/300 Write block 3562 len 8192 data: 5 5
221/300 Read block 477 len 28672
222/300 Write block 2807 len 4096 data: 2
223/300 Flush
224/300 Read block 4343 len 32768
225/300 Write block 2590 len 28672 data: 3 3 4 4 3 3 3
226/300 Write block 300 len 36864 data: 6 4 4 3 2 2 2 2 2
227/300 Write block 4495 len 40960 data: 5 3 3 4 4 4 3 4 5 4
228/300 Read block 4913 len 40960
229/300 Read block 4792 len 24576
230/300 Read block 1392 len 24576
231/300 Read block 1892 len 28672
232/300 Read block 2914 len 4096
233/300 Write block 2372 len 40960 data: 3 3 3 5 5 5 7 4 4 3
234/300 Flush
235/300 Read block 1251 len 36864
236/300 Read block 1489 len 8192
237/300 Write block 2550 len 24576 data: 2 2 2 2 2 2
238/300 Write block 3684 len 32768 data: 2 3 3 4 5 5 5 5
239/300 Read block 3357 len 36864
240/300 Write block 217 len 16384 data: 2 2 2 2
241/300 Write block 2675 len 12288 data: 6 5 6
242/300 Read block 2905 len 32768
243/300 Write block 3048 len 32768 data: 4 4 5 6 5 4 5 4
244/300 Write block 439 len 24576 data: 4 4 3 3 4 2
245/300 Write block 1665 len 12288 data: 2 3 2
246/300 Write block 4911 len 28672 data: 3 3 3 3 3 3 2
247/300 Write block 3051 len 20480 data: 7 6 5 6 5
248/300 Write block 1003 len 28672 data: 2 3 3 3 3 3 4
249/300 Read block 3330 len 36864
250/300 Read block 4194 len 8192
251/300 Write block 356 len 28672 data: 5 4 4 2 2 2 2
252/300 Write block 2928 len 24576 data: 4 4 3 2 2 3
253/300 Read block 1951 len 28672
254/300 Write block 2341 len 20480 data: 5 4 4 4 4
255/300 Write block 1009 len 8192 data: 5 4
256/300 Write block 4949 len 16384 data: 3 3 3 2
257/300 Read block 1844 len 16384
258/300 Read block 247 len 20480
259/300 Write block 4089 len 20480 data: 3 3 3 3 3
260/300 Read block 2874 len 8192
261/300 Write block 2024 len 20480 data: 3 2 4 4 4
262/300 Flush
263/300 Read block 695 len 16384
264/300 Write block 2682 len 8192 data: 7 8
265/300 Read block 2721 len 24576
266/300 Read block 3894 len 4096
267/300 Read block 3848 len 12288
268/300 Read block 4147 len 24576
269/300 Write block 1137 len 20480 data: 3 3 3 3 3
270/300 Read block 4397 len 28672
271/300 Write block 997 len 32768 data: 2 2 2 2 2 2 3 4
272/300 Read block 4755 len 20480
273/300 Write block 3888 len 8192 data: 3 3
274/300 Read block 2291 len 12288
275/300 Flush
276/300 Flush
277/300 Read block 994 len 32768
278/300 Flush
279/300 Flush
280/300 Write block 4305 len 4096 data: 3
281/300 Write block 4630 len 16384 data: 3 3 3 4
282/300 Read block 422 len 28672
283/300 Read block 2373 len 20480
284/300 Read block 2590 len 40960
285/300 Flush
286/300 Read block 3643 len 12288
287/300 Write block 982 len 8192 data: 3 4
288/300 Write block 4993 len 16384 data: 3 3 4 4
289/300 Write block 4858 len 40960 data: 4 4 4 5 4 5 5 4 4 5
290/300 Write block 3805 len 28672 data: 2 2 2 2 3 3 3
291/300 Write block 4329 len 40960 data: 3 4 5 4 4 4 6 8 8 7
292/300 Write block 1072 len 24576 data: 5 4 4 4 4 4
293/300 Read block 4654 len 8192
294/300 Read block 4064 len 24576
295/300 Write block 429 len 12288 data: 3 2 2
296/300 Read block 3291 len 4096
297/300 Read block 4972 len 4096
298/300 Write block 3715 len 20480 data: 5 3 3 3 2
299/300 Write block 4181 len 8192 data: 2 2
300/300 Write block 1895 len 4096 data: 3
301/300 Write block 3060 len 40960 data: 4 4 4 4 3 3 3 3 3 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:39:12.124 INFO request completed, latency_us: 195, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6275d77b-902d-4c57-864b-588bdc66c430, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4621 Acked Write 6 Sent Done Done false
4622 Acked Read 3 Sent Done Done false
4623 Acked Read 1 Sent Done Done false
4624 Acked Write 10 Sent Done Done false
4625 Acked Write 3 Sent Done Done false
4626 Acked Read 2 Sent Done Done false
4627 Acked Read 6 Sent Done Done false
4628 Acked Write 6 Sent Done Done false
4629 Acked Read 8 Sent Done Done false
4630 Acked Read 9 Sent Done Done false
4631 Acked Read 10 Sent Done Done false
4632 Acked Read 7 Sent Done Done false
4633 Acked Read 9 Sent Done Done false
4634 Acked Write 6 Sent Done Done false
4635 Acked Write 6 Sent Done Done false
4636 Acked Read 3 Sent Done Done false
4637 Acked Read 2 Sent Done Done false
4638 Acked Read 7 Sent Done Done false
4639 Acked Write 10 Sent Done Done false
4640 Acked Write 9 Sent Done Done false
4641 Acked Write 8 Sent Done Done false
4642 Acked Write 9 Sent Done Done false
4643 Acked Read 4 Sent Done Done false
4644 Acked Write 1 Sent Done Done false
4645 Acked Write 10 Sent Done Done false
4646 Acked Read 1 Sent Done Done false
4647 Acked Write 10 Sent Done Done false
4648 Acked Read 6 Sent Done Done false
4649 Acked Write 5 Sent Done Done false
4650 Acked Read 9 Sent Done Done false
4651 Acked Flush 0 Sent Done Done false
4652 Acked Write 8 Sent Done Done false
4653 Acked Write 3 Sent Done Done false
4654 Acked Read 2 Sent Done Done false
4655 Acked Read 9 Sent Done Done false
4656 Acked Write 8 Sent Done Done false
4657 Acked Write 5 Sent Done Done false
4658 Acked Read 8 Sent Done Done false
4659 Acked Write 3 Sent Done Done false
4660 Acked Read 6 Sent Done Done false
4661 Acked Write 1 Sent Done Done false
4662 Acked Flush 0 Sent Done Done false
4663 Acked Write 7 Sent Done Done false
4664 Acked Read 10 Sent Done Done false
4665 Acked Write 5 Sent Done Done false
4666 Acked Read 2 Sent Done Done false
4667 Acked Read 5 Sent Done Done false
4668 Acked Read 10 Sent Done Done false
4669 Acked Read 3 Sent Done Done false
4670 Acked Read 9 Sent Done Done false
4671 Acked Read 5 Sent Done Done false
4672 Acked Read 4 Sent Done Done false
4673 Acked Read 4 Sent Done Done false
4674 Acked Write 7 Sent Done Done false
4675 Acked Read 7 Sent Done Done false
4676 Acked Write 2 Sent Done Done false
4677 Acked Read 9 Sent Done Done false
4678 Acked Write 9 Sent Done Done false
4679 Acked Read 10 Sent Done Done false
4680 Acked Read 4 Sent Done Done false
4681 Acked Read 4 Sent Done Done false
4682 Acked Read 6 Sent Done Done false
4683 Acked Read 4 Sent Done Done false
4684 Acked Flush 0 Sent Done Done false
4685 Acked Write 8 Sent Done Done false
4686 Acked Read 4 Sent Done Done false
4687 Acked Flush 0 Sent Done Done false
4688 Acked Flush 0 Sent Done Done false
4689 Acked Read 10 Sent Done Done false
4690 Acked Flush 0 Sent Done Done false
4691 Acked Write 7 Sent Done Done false
4692 Acked Write 7 Sent Done Done false
4693 Acked Flush 0 Sent Done Done false
4694 Acked Read 8 Sent Done Done false
4695 Acked Read 2 Sent Done Done false
4696 Acked Read 7 Sent Done Done false
4697 Acked Read 4 Sent Done Done false
4698 Acked Write 2 Sent Done Done false
4699 Acked Write 4 Sent Done Done false
4700 Acked Read 1 Sent Done Done false
4701 Acked Write 9 Sent Done Done false
4702 Acked Read 2 Sent Done Done false
4703 Acked Write 2 Sent Done Done false
4704 Acked Flush 0 Sent Done Done false
4705 Acked Read 3 Sent Done Done false
4706 Acked Write 8 Sent Done Done false
4707 Acked Write 6 Sent Done Done false
4708 Acked Read 2 Sent Done Done false
4709 Acked Write 7 Sent Done Done false
4710 Acked Read 8 Sent Done Done false
4711 Acked Read 6 Sent Done Done false
4712 Acked Write 1 Sent Done Done false
4713 Acked Read 4 Sent Done Done false
4714 Acked Read 9 Sent Done Done false
4715 Acked Write 4 Sent Done Done false
4716 Acked Read 10 Sent Done Done false
4717 Acked Write 4 Sent Done Done false
4718 Acked Read 7 Sent Done Done false
4719 Acked Read 1 Sent Done Done false
4720 Acked Flush 0 Sent Done Done false
4721 Acked Read 4 Sent Done Done false
4722 Acked Read 2 Sent Done Done false
4723 Acked Read 6 Sent Done Done false
4724 Acked Flush 0 Sent Done Done false
4725 Acked Read 4 Sent Done Done false
4726 Acked Write 1 Sent Done Done false
4727 Acked Read 4 Sent Done Done false
4728 Acked Write 5 Sent Done Done false
4729 Acked Write 2 Sent Done Done false
4730 Acked Read 9 Sent Done Done false
4731 Acked Read 10 Sent Done Done false
4732 Acked Write 4 Sent Done Done false
4733 Acked Read 7 Sent Done Done false
4734 Acked Write 3 Sent Done Done false
4735 Acked Write 2 Sent Done Done false
4736 Acked Read 10 Sent Done Done false
4737 Acked Read 8 Sent Done Done false
4738 Acked Flush 0 Sent Done Done false
4739 Acked Flush 0 Sent Done Done false
4740 Acked Write 9 Sent Done Done false
4741 Acked Write 1 Sent Done Done false
4742 Acked Write 4 Sent Done Done false
4743 Acked Read 5 Sent Done Done false
4744 Acked Read 9 Sent Done Done false
4745 Acked Write 6 Sent Done Done false
4746 Acked Read 3 Sent Done Done false
4747 Acked Write 3 Sent Done Done false
4748 Acked Read 10 Sent Done Done false
4749 Acked Write 7 Sent Done Done false
4750 Acked Read 4 Sent Done Done false
4751 Acked Read 5 Sent Done Done false
4752 Acked Write 10 Sent Done Done false
4753 Acked Write 5 Sent Done Done false
4754 Acked Read 7 Sent Done Done false
4755 Acked Read 4 Sent Done Done false
4756 Acked Flush 0 Sent Done Done false
4757 Acked Flush 0 Sent Done Done false
4758 Acked Write 9 Sent Done Done false
4759 Acked Write 2 Sent Done Done false
4760 Acked Write 4 Sent Done Done false
4761 Acked Write 9 Sent Done Done false
4762 Acked Write 7 Sent Done Done false
4763 Acked Write 4 Sent Done Done false
4764 Acked Read 1 Sent Done Done false
4765 Acked Read 10 Sent Done Done false
4766 Acked Read 7 Sent Done Done false
4767 Acked Read 8 Sent Done Done false
4768 Acked Write 6 Sent Done Done false
4769 Acked Flush 0 Sent Done Done false
4770 Acked Write 9 Sent Done Done false
4771 Acked Write 3 Sent Done Done false
4772 Acked Read 1 Sent Done Done false
4773 Acked Read 3 Sent Done Done false
4774 Acked Write 8 Sent Done Done false
4775 Acked Read 8 Sent Done Done false
4776 Acked Read 6 Sent Done Done false
4777 Acked Read 4 Sent Done Done false
4778 Acked Read 9 Sent Done Done false
4779 Acked Read 1 Sent Done Done false
4780 Acked Write 3 Sent Done Done false
4781 Acked Write 9 Sent Done Done false
4782 Acked Read 7 Sent Done Done false
4783 Acked Write 7 Sent Done Done false
4784 Acked Read 9 Sent Done Done false
4785 Acked Read 1 Sent Done Done false
4786 Acked Write 10 Sent Done Done false
4787 Acked Read 7 Sent Done Done false
4788 Acked Flush 0 Sent Done Done false
4789 Acked Read 8 Sent Done Done false
4790 Acked Read 3 Sent Done Done false
4791 Acked Write 9 Sent Done Done false
4792 Acked Write 9 Sent Done Done false
4793 Acked Read 1 Sent Done Done false
4794 Acked Write 5 Sent Done Done false
4795 Acked Read 5 Sent Done Done false
4796 Acked Write 1 Sent Done Done false
4797 Acked Flush 0 Sent Done Done false
4798 Acked Read 1 Sent Done Done false
4799 Acked Write 2 Sent Done Done false
4800 Acked Read 6 Sent Done Done false
4801 Acked Read 10 Sent Done Done false
4802 Acked Write 9 Sent Done Done false
4803 Acked Read 10 Sent Done Done false
4804 Acked Write 1 Sent Done Done false
4805 Acked Read 2 Sent Done Done false
4806 Acked Read 3 Sent Done Done false
4807 Acked Read 1 Sent Done Done false
4808 Acked Read 8 Sent Done Done false
4809 Acked Read 10 Sent Done Done false
4810 Acked Read 4 Sent Done Done false
4811 Acked Read 6 Sent Done Done false
4812 Acked Write 2 Sent Done Done false
4813 Acked Read 4 Sent Done Done false
4814 Acked Write 8 Sent Done Done false
4815 Acked Read 2 Sent Done Done false
4816 Acked Read 7 Sent Done Done false
4817 Acked Read 4 Sent Done Done false
4818 Acked Write 5 Sent Done Done false
4819 Acked Write 9 Sent Done Done false
4820 Acked Write 1 Sent Done Done false
4821 Acked Write 2 Sent Done Done false
4822 Acked Flush 0 Sent Done Done false
4823 Acked Read 8 Sent Done Done false
4824 Acked Flush 0 Sent Done Done false
4825 Acked Read 5 Sent Done Done false
4826 Acked Write 4 Sent Done Done false
4827 Acked Write 7 Sent Done Done false
4828 Acked Read 8 Sent Done Done false
4829 Acked Read 10 Sent Done Done false
4830 Acked Flush 0 Sent Done Done false
4831 Acked Read 1 Sent Done Done false
4832 Acked Read 7 Sent Done Done false
4833 Acked Read 5 Sent Done Done false
4834 Acked Read 7 Sent Done Done false
4835 Acked Read 4 Sent Done Done false
4836 Acked Write 3 Sent Done Done false
4837 Acked Write 2 Sent Done Done false
4838 Acked Read 2 Sent Done Done false
4839 Acked Write 10 Sent Done Done false
4840 Acked Write 2 Sent Done Done false
4841 Acked Read 7 Sent Done Done false
4842 Acked Write 1 Sent Done Done false
4843 Acked Flush 0 Sent Done Done false
4844 Acked Read 8 Sent Done Done false
4845 Acked Write 7 Sent Done Done false
4846 Acked Write 9 Sent Done Done false
4847 Acked Write 10 Sent Done Done false
4848 Acked Read 10 Sent Done Done false
4849 Acked Read 6 Sent Done Done false
4850 Acked Read 6 Sent Done Done false
4851 Acked Read 7 Sent Done Done false
4852 Acked Read 1 Sent Done Done false
4853 Acked Write 10 Sent Done Done false
4854 Acked Flush 0 Sent Done Done false
4855 Acked Read 9 Sent Done Done false
4856 Acked Read 2 Sent Done Done false
4857 Acked Write 6 Sent Done Done false
4858 Acked Write 8 Sent Done Done false
4859 Acked Read 9 Sent Done Done false
4860 Acked Write 4 Sent Done Done false
4861 Acked Write 3 Sent Done Done false
4862 Acked Read 8 Sent Done Done false
4863 Acked Write 8 Sent Done Done false
4864 Acked Write 6 Sent Done Done false
4865 Acked Write 3 Sent Done Done false
4866 Acked Write 7 Sent Done Done false
4867 Acked Write 5 Sent Done Done false
4868 Acked Write 7 Sent Done Done false
4869 Acked Read 9 Sent Done Done false
4870 Acked Read 2 Sent Done Done false
4871 Acked Write 7 Sent Done Done false
4872 Acked Write 6 Sent Done Done false
4873 Acked Read 7 Sent Done Done false
4874 Acked Write 5 Sent Done Done false
4875 Acked Write 2 Sent Done Done false
4876 Acked Write 4 Sent Done Done false
4877 Acked Read 4 Sent Done Done false
4878 Acked Read 5 Sent Done Done false
4879 Acked Write 5 Sent Done Done false
4880 Acked Read 2 Sent Done Done false
4881 Acked Write 5 Sent Done Done false
4882 Acked Flush 0 Sent Done Done false
4883 Acked Read 4 Sent Done Done false
4884 Acked Write 2 Sent Done Done false
4885 Acked Read 6 Sent Done Done false
4886 Acked Read 1 Sent Done Done false
4887 Acked Read 3 Sent Done Done false
4888 Acked Read 6 Sent Done Done false
4889 Acked Write 5 Sent Done Done false
4890 Acked Read 7 Sent Done Done false
4891 Acked Write 8 Sent Done Done false
4892 Acked Read 5 Sent Done Done false
4893 Acked Write 2 Sent Done Done false
4894 Acked Read 3 Sent Done Done false
4895 Acked Flush 0 Sent Done Done false
4896 Acked Flush 0 Sent Done Done false
4897 Acked Read 8 Sent Done Done false
4898 Acked Flush 0 Sent Done Done false
4899 Acked Flush 0 Sent Done Done false
4900 Acked Write 1 Sent Done Done false
4901 Acked Write 4 Sent Done Done false
4902 Acked Read 7 Sent Done Done false
4903 Acked Read 5 Sent Done Done false
4904 Acked Read 10 Sent Done Done false
4905 Acked Flush 0 Sent Done Done false
4906 Acked Read 3 Sent Done Done false
4907 Acked Write 2 Sent Done Done false
4908 Acked Write 4 Sent Done Done false
4909 Acked Write 10 Sent Done Done false
4910 Acked Write 7 Sent Done Done false
4911 Acked Write 10 Sent Done Done false
4912 Acked Write 6 Sent Done Done false
4913 Acked Read 2 Sent Done Done false
4914 Acked Read 6 Sent Done Done false
4915 Acked Write 3 Sent Done Done false
4916 Acked Read 1 Sent Done Done false
4917 Acked Read 1 Sent Done Done false
4918 Acked Write 5 Sent Done Done false
4919 Acked Write 2 Sent Done Done false
4920 Acked Write 1 Sent Done Done false
4921 Acked Write 10 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4620 4905 4905
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4921 4920 4919 4918 4917
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4621 Acked Write 6 Sent Done Done false
4622 Acked Read 3 Sent Done Done false
4623 Acked Read 1 Sent Done Done false
4624 Acked Write 10 Sent Done Done false
4625 Acked Write 3 Sent Done Done false
4626 Acked Read 2 Sent Done Done false
4627 Acked Read 6 Sent Done Done false
4628 Acked Write 6 Sent Done Done false
4629 Acked Read 8 Sent Done Done false
4630 Acked Read 9 Sent Done Done false
4631 Acked Read 10 Sent Done Done false
4632 Acked Read 7 Sent Done Done false
4633 Acked Read 9 Sent Done Done false
4634 Acked Write 6 Sent Done Done false
4635 Acked Write 6 Sent Done Done false
4636 Acked Read 3 Sent Done Done false
4637 Acked Read 2 Sent Done Done false
4638 Acked Read 7 Sent Done Done false
4639 Acked Write 10 Sent Done Done false
4640 Acked Write 9 Sent Done Done false
4641 Acked Write 8 Sent Done Done false
4642 Acked Write 9 Sent Done Done false
4643 Acked Read 4 Sent Done Done false
4644 Acked Write 1 Sent Done Done false
4645 Acked Write 10 Sent Done Done false
4646 Acked Read 1 Sent Done Done false
4647 Acked Write 10 Sent Done Done false
4648 Acked Read 6 Sent Done Done false
4649 Acked Write 5 Sent Done Done false
4650 Acked Read 9 Sent Done Done false
4651 Acked Flush 0 Sent Done Done false
4652 Acked Write 8 Sent Done Done false
4653 Acked Write 3 Sent Done Done false
4654 Acked Read 2 Sent Done Done false
4655 Acked Read 9 Sent Done Done false
4656 Acked Write 8 Sent Done Done false
4657 Acked Write 5 Sent Done Done false
4658 Acked Read 8 Sent Done Done false
4659 Acked Write 3 Sent Done Done false
4660 Acked Read 6 Sent Done Done false
4661 Acked Write 1 Sent Done Done false
4662 Acked Flush 0 Sent Done Done false
4663 Acked Write 7 Sent Done Done false
4664 Acked Read 10 Sent Done Done false
4665 Acked Write 5 Sent Done Done false
4666 Acked Read 2 Sent Done Done false
4667 Acked Read 5 Sent Done Done false
4668 Acked Read 10 Sent Done Done false
4669 Acked Read 3 Sent Done Done false
4670 Acked Read 9 Sent Done Done false
4671 Acked Read 5 Sent Done Done false
4672 Acked Read 4 Sent Done Done false
4673 Acked Read 4 Sent Done Done false
4674 Acked Write 7 Sent Done Done false
4675 Acked Read 7 Sent Done Done false
4676 Acked Write 2 Sent Done Done false
4677 Acked Read 9 Sent Done Done false
4678 Acked Write 9 Sent Done Done false
4679 Acked Read 10 Sent Done Done false
4680 Acked Read 4 Sent Done Done false
4681 Acked Read 4 Sent Done Done false
4682 Acked Read 6 Sent Done Done false
4683 Acked Read 4 Sent Done Done false
4684 Acked Flush 0 Sent Done Done false
4685 Acked Write 8 Sent Done Done false
4686 Acked Read 4 Sent Done Done false
4687 Acked Flush 0 Sent Done Done false
4688 Acked Flush 0 Sent Done Done false
4689 Acked Read 10 Sent Done Done false
4690 Acked Flush 0 Sent Done Done false
4691 Acked Write 7 Sent Done Done false
4692 Acked Write 7 Sent Done Done false
4693 Acked Flush 0 Sent Done Done false
4694 Acked Read 8 Sent Done Done false
4695 Acked Read 2 Sent Done Done false
4696 Acked Read 7 Sent Done Done false
4697 Acked Read 4 Sent Done Done false
4698 Acked Write 2 Sent Done Done false
4699 Acked Write 4 Sent Done Done false
4700 Acked Read 1 Sent Done Done false
4701 Acked Write 9 Sent Done Done false
4702 Acked Read 2 Sent Done Done false
4703 Acked Write 2 Sent Done Done false
4704 Acked Flush 0 Sent Done Done false
4705 Acked Read 3 Sent Done Done false
4706 Acked Write 8 Sent Done Done false
4707 Acked Write 6 Sent Done Done false
4708 Acked Read 2 Sent Done Done false
4709 Acked Write 7 Sent Done Done false
4710 Acked Read 8 Sent Done Done false
4711 Acked Read 6 Sent Done Done false
4712 Acked Write 1 Sent Done Done false
4713 Acked Read 4 Sent Done Done false
4714 Acked Read 9 Sent Done Done false
4715 Acked Write 4 Sent Done Done false
4716 Acked Read 10 Sent Done Done false
4717 Acked Write 4 Sent Done Done false
4718 Acked Read 7 Sent Done Done false
4719 Acked Read 1 Sent Done Done false
4720 Acked Flush 0 Sent Done Done false
4721 Acked Read 4 Sent Done Done false
4722 Acked Read 2 Sent Done Done false
4723 Acked Read 6 Sent Done Done false
4724 Acked Flush 0 Sent Done Done false
4725 Acked Read 4 Sent Done Done false
4726 Acked Write 1 Sent Done Done false
4727 Acked Read 4 Sent Done Done false
4728 Acked Write 5 Sent Done Done false
4729 Acked Write 2 Sent Done Done false
4730 Acked Read 9 Sent Done Done false
4731 Acked Read 10 Sent Done Done false
4732 Acked Write 4 Sent Done Done false
4733 Acked Read 7 Sent Done Done false
4734 Acked Write 3 Sent Done Done false
4735 Acked Write 2 Sent Done Done false
4736 Acked Read 10 Sent Done Done false
4737 Acked Read 8 Sent Done Done false
4738 Acked Flush 0 Sent Done Done false
4739 Acked Flush 0 Sent Done Done false
4740 Acked Write 9 Sent Done Done false
4741 Acked Write 1 Sent Done Done false
4742 Acked Write 4 Sent Done Done false
4743 Acked Read 5 Sent Done Done false
4744 Acked Read 9 Sent Done Done false
4745 Acked Write 6 Sent Done Done false
4746 Acked Read 3 Sent Done Done false
4747 Acked Write 3 Sent Done Done false
4748 Acked Read 10 Sent Done Done false
4749 Acked Write 7 Sent Done Done false
4750 Acked Read 4 Sent Done Done false
4751 Acked Read 5 Sent Done Done false
4752 Acked Write 10 Sent Done Done false
4753 Acked Write 5 Sent Done Done false
4754 Acked Read 7 Sent Done Done false
4755 Acked Read 4 Sent Done Done false
4756 Acked Flush 0 Sent Done Done false
4757 Acked Flush 0 Sent Done Done false
4758 Acked Write 9 Sent Done Done false
4759 Acked Write 2 Sent Done Done false
4760 Acked Write 4 Sent Done Done false
4761 Acked Write 9 Sent Done Done false
4762 Acked Write 7 Sent Done Done false
4763 Acked Write 4 Sent Done Done false
4764 Acked Read 1 Sent Done Done false
4765 Acked Read 10 Sent Done Done false
4766 Acked Read 7 Sent Done Done false
4767 Acked Read 8 Sent Done Done false
4768 Acked Write 6 Sent Done Done false
4769 Acked Flush 0 Sent Done Done false
4770 Acked Write 9 Sent Done Done false
4771 Acked Write 3 Sent Done Done false
4772 Acked Read 1 Sent Done Done false
4773 Acked Read 3 Sent Done Done false
4774 Acked Write 8 Sent Done Done false
4775 Acked Read 8 Sent Done Done false
4776 Acked Read 6 Sent Done Done false
4777 Acked Read 4 Sent Done Done false
4778 Acked Read 9 Sent Done Done false
4779 Acked Read 1 Sent Done Done false
4780 Acked Write 3 Sent Done Done false
4781 Acked Write 9 Sent Done Done false
4782 Acked Read 7 Sent Done Done false
4783 Acked Write 7 Sent Done Done false
4784 Acked Read 9 Sent Done Done false
4785 Acked Read 1 Sent Done Done false
4786 Acked Write 10 Sent Done Done false
4787 Acked Read 7 Sent Done Done false
4788 Acked Flush 0 Sent Done Done false
4789 Acked Read 8 Sent Done Done false
4790 Acked Read 3 Sent Done Done false
4791 Acked Write 9 Sent Done Done false
4792 Acked Write 9 Sent Done Done false
4793 Acked Read 1 Sent Done Done false
4794 Acked Write 5 Sent Done Done false
4795 Acked Read 5 Sent Done Done false
4796 Acked Write 1 Sent Done Done false
4797 Acked Flush 0 Sent Done Done false
4798 Acked Read 1 Sent Done Done false
4799 Acked Write 2 Sent Done Done false
4800 Acked Read 6 Sent Done Done false
4801 Acked Read 10 Sent Done Done false
4802 Acked Write 9 Sent Done Done false
4803 Acked Read 10 Sent Done Done false
4804 Acked Write 1 Sent Done Done false
4805 Acked Read 2 Sent Done Done false
4806 Acked Read 3 Sent Done Done false
4807 Acked Read 1 Sent Done Done false
4808 Acked Read 8 Sent Done Done false
4809 Acked Read 10 Sent Done Done false
4810 Acked Read 4 Sent Done Done false
4811 Acked Read 6 Sent Done Done false
4812 Acked Write 2 Sent Done Done false
4813 Acked Read 4 Sent Done Done false
4814 Acked Write 8 Sent Done Done false
4815 Acked Read 2 Sent Done Done false
4816 Acked Read 7 Sent Done Done false
4817 Acked Read 4 Sent Done Done false
4818 Acked Write 5 Sent Done Done false
4819 Acked Write 9 Sent Done Done false
4820 Acked Write 1 Sent Done Done false
4821 Acked Write 2 Sent Done Done false
4822 Acked Flush 0 Sent Done Done false
4823 Acked Read 8 Sent Done Done false
4824 Acked Flush 0 Sent Done Done false
4825 Acked Read 5 Sent Done Done false
4826 Acked Write 4 Sent Done Done false
4827 Acked Write 7 Sent Done Done false
4828 Acked Read 8 Sent Done Done false
4829 Acked Read 10 Sent Done Done false
4830 Acked Flush 0 Sent Done Done false
4831 Acked Read 1 Sent Done Done false
4832 Acked Read 7 Sent Done Done false
4833 Acked Read 5 Sent Done Done false
4834 Acked Read 7 Sent Done Done false
4835 Acked Read 4 Sent Done Done false
4836 Acked Write 3 Sent Done Done false
4837 Acked Write 2 Sent Done Done false
4838 Acked Read 2 Sent Done Done false
4839 Acked Write 10 Sent Done Done false
4840 Acked Write 2 Sent Done Done false
4841 Acked Read 7 Sent Done Done false
4842 Acked Write 1 Sent Done Done false
4843 Acked Flush 0 Sent Done Done false
4844 Acked Read 8 Sent Done Done false
4845 Acked Write 7 Sent Done Done false
4846 Acked Write 9 Sent Done Done false
4847 Acked Write 10 Sent Done Done false
4848 Acked Read 10 Sent Done Done false
4849 Acked Read 6 Sent Done Done false
4850 Acked Read 6 Sent Done Done false
4851 Acked Read 7 Sent Done Done false
4852 Acked Read 1 Sent Done Done false
4853 Acked Write 10 Sent Done Done false
4854 Acked Flush 0 Sent Done Done false
4855 Acked Read 9 Sent Done Done false
4856 Acked Read 2 Sent Done Done false
4857 Acked Write 6 Sent Done Done false
4858 Acked Write 8 Sent Done Done false
4859 Acked Read 9 Sent Done Done false
4860 Acked Write 4 Sent Done Done false
4861 Acked Write 3 Sent Done Done false
4862 Acked Read 8 Sent Done Done false
4863 Acked Write 8 Sent Done Done false
4864 Acked Write 6 Sent Done Done false
4865 Acked Write 3 Sent Done Done false
4866 Acked Write 7 Sent Done Done false
4867 Acked Write 5 Sent Done Done false
4868 Acked Write 7 Sent Done Done false
4869 Acked Read 9 Sent Done Done false
4870 Acked Read 2 Sent Done Done false
4871 Acked Write 7 Sent Done Done false
4872 Acked Write 6 Sent Done Done false
4873 Acked Read 7 Sent Done Done false
4874 Acked Write 5 Sent Done Done false
4875 Acked Write 2 Sent Done Done false
4876 Acked Write 4 Sent Done Done false
4877 Acked Read 4 Sent Done Done false
4878 Acked Read 5 Sent Done Done false
4879 Acked Write 5 Sent Done Done false
4880 Acked Read 2 Sent Done Done false
4881 Acked Write 5 Sent Done Done false
4882 Acked Flush 0 Sent Done Done false
4883 Acked Read 4 Sent Done Done false
4884 Acked Write 2 Sent Done Done false
4885 Acked Read 6 Sent Done Done false
4886 Acked Read 1 Sent Done Done false
4887 Acked Read 3 Sent Done Done false
4888 Acked Read 6 Sent Done Done false
4889 Acked Write 5 Sent Done Done false
4890 Acked Read 7 Sent Done Done false
4891 Acked Write 8 Sent Done Done false
4892 Acked Read 5 Sent Done Done false
4893 Acked Write 2 Sent Done Done false
4894 Acked Read 3 Sent Done Done false
4895 Acked Flush 0 Sent Done Done false
4896 Acked Flush 0 Sent Done Done false
4897 Acked Read 8 Sent Done Done false
4898 Acked Flush 0 Sent Done Done false
4899 Acked Flush 0 Sent Done Done false
4900 Acked Write 1 Sent Done Done false
4901 Acked Write 4 Sent Done Done false
4902 Acked Read 7 Sent Done Done false
4903 Acked Read 5 Sent Done Done false
4904 Acked Read 10 Sent Done Done false
4905 Acked Flush 0 Sent Done Done false
4906 Acked Read 3 Sent Done Done false
4907 Acked Write 2 Sent Done Done false
4908 Acked Write 4 Sent Done Done false
4909 Acked Write 10 Sent Done Done false
4910 Acked Write 7 Sent Done Done false
4911 Acked Write 10 Sent Done Done false
4912 Acked Write 6 Sent Done Done false
4913 Acked Read 2 Sent Done Done false
4914 Acked Read 6 Sent Done Done false
4915 Acked Write 3 Sent Done Done false
4916 Acked Read 1 Sent Done Done false
4917 Acked Read 1 Sent Done Done false
4918 Acked Write 5 Sent Done Done false
4919 Acked Write 2 Sent Done Done false
4920 Acked Write 1 Sent Done Done false
4921 Acked Write 10 Sent Done Done false
4922 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4620 4922 4922
Downstairs last five completed: 4620 4619 4618 4617 4616
Upstairs last five completed: 4922 4921 4920 4919 4918
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.887507479Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58328)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.887775278Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.888856662Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.889498817Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.889521594Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 4620","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.889535133Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 4620","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.889787743Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 4620","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.88981023Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 4620","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:17.890055422Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 4922 4921 4920 4919 4918
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:39:20.129 INFO request completed[8810] Got stop action so:false kr:false
, latency_us: 317, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 48abf409-74ce-4465-9a05-7aaacf83661d, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.13009212Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.130157792Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.130181819Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:39:20.130 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/0, method: GET, req_id: 754bed8b-5c38-49d3-b6f9-06f619e9878e, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.130301335Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:39:24.132 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: bc870c15-9322-441b-9e92-e80023ea7b78, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 397 len 12288 data: 5 4 4
002/300 Write block 1763 len 32768 data: 2 3 3 2 2 3 3 3
003/300 Write block 2702 len 8192 data: 2 2
004/300 Read block 941 len 32768
005/300 Read block 2027 len 24576
006/300 Read block 1015 len 32768
007/300 Write block 196 len 40960 data: 5 5 5 6 6 4 3 3 3 4
008/300 Read block 1555 len 20480
009/300 Read block 3680 len 28672
010/300 Write block 3380 len 8192 data: 3 2
011/300 Write block 114 len 24576 data: 4 5 5 4 4 4
012/300 Write block 1244 len 20480 data: 2 2 2 2 2
013/300 Read block 2780 len 16384
014/300 Read block 495 len 8192
015/300 Read block 2507 len 32768
016/300 Write block 2120 len 4096 data: 2
017/300 Read block 4053 len 8192
018/300 Write block 375 len 24576 data: 4 4 4 3 3 3
019/300 Write block 3365 len 36864 data: 3 3 5 5 5 5 5 6 4
020/300 Read block 4537 len 4096
021/300 Write block 4275 len 24576 data: 6 5 5 5 4 4
022/300 Read block 4409 len 4096
023/300 Read block 865 len 16384
024/300 Write block 3553 len 40960 data: 2 2 2 3 3 3 5 5 5 6
025/300 Write block 3262 len 12288 data: 3 3 3
026/300 Write block 4161 len 24576 data: 5 5 5 5 6 5
027/300 Read block 2869 len 24576
028/300 Read block 4055 len 24576
029/300 Read block 1999 len 28672
030/300 Read block 389 len 28672
031/300 Write block 2007 len 20480 data: 2 2 2 3 5
032/300 Write block 157 len 40960 data: 3 3 3 3 2 2 2 3 3 4
033/300 Read block 2950 len 36864
034/300 Write block 2508 len 36864 data: 4 3 2 2 3 3 3 3 3
035/300 Write block 1883 len 32768 data: 3 3 3 4 3 4 5 5
036/300 Read block 455 len 16384
037/300 Read block 837 len 12288
038/300 Write block 1034 len 40960 data: 7 6 6 5 3 3 2 2 3 3
039/300 Write block 2549 len 36864 data: 2 3 3 3 3 3 3 3 4
040/300 Read block 1272 len 36864
041/300 Read block 2642 len 12288
042/300 Write block 2072 len 32768 data: 2 2 2 2 4 4 4 4
043/300 Read block 4079 len 24576
044/300 Write block 2002 len 4096 data: 4
045/300 Read block 2460 len 28672
046/300 Read block 1003 len 4096
047/300 Read block 3422 len 4096
048/300 Write block 3644 len 4096 data: 3
049/300 Read block 2105 len 40960
050/300 Read block 2215 len 16384
051/300 Write block 3256 len 36864 data: 2 2 2 2 2 3 4 4 4
052/300 Read block 3035 len 24576
053/300 Write block 1065 len 40960 data: 2 2 2 4 3 4 4 6 5 5
054/300 Read block 1254 len 12288
055/300 Read block 2256 len 8192
056/300 Flush
057/300 Write block 1355 len 20480 data: 5 5 5 7 7
058/300 Read block 4667 len 40960
059/300 Read block 1923 len 16384
060/300 Write block 2806 len 36864 data: 3 3 3 3 3 4 4 4 4
061/300 Read block 3925 len 36864
062/300 Read block 1404 len 12288
063/300 Write block 3705 len 32768 data: 3 3 4 6 6 6 7 6
064/300 Flush
065/300 Write block 4207 len 20480 data: 3 3 3 3 3
066/300 Write block 278 len 4096 data: 6
067/300 Write block 460 len 8192 data: 3 3
068/300 Write block 627 len 32768 data: 3 3 3 4 3 3 2 2
069/300 Write block 2049 len 12288 data: 3 4 4
070/300 Read block 4110 len 20480
071/300 Read block 2900 len 8192
072/300 Read block 330 len 12288
073/300 Read block 195 len 8192
074/300 Read block 2090 len 20480
075/300 Read block 4228 len 4096
076/300 Read block 3218 len 8192
077/300 Read block 564 len 24576
078/300 Write block 4349 len 28672 data: 5 5 5 5 5 4 5
079/300 Read block 3588 len 36864
080/300 Read block 4648 len 4096
081/300 Write block 2125 len 40960 data: 3 2 2 2 2 2 2 2 2 2
082/300 Flush
083/300 Read block 2539 len 28672
084/300 Read block 1127 len 32768
085/300 Read block 3611 len 20480
086/300 Read block 4918 len 16384
087/300 Write block 2885 len 20480 data: 3 3 3 3 3
088/300 Write block 3618 len 28672 data: 4 4 4 5 5 5 5
089/300 Read block 1751 len 8192
090/300 Read block 2000 len 36864
091/300 Read block 1560 len 28672
092/300 Flush
093/300 Read block 3574 len 24576
094/300 Read block 1523 len 36864
095/300 Write block 3962 len 32768 data: 4 5 4 6 5 5 5 4
096/300 Read block 2326 len 8192
097/300 Write block 776 len 28672 data: 4 4 4 3 3 4 4
098/300 Flush
099/300 Read block 4636 len 24576
100/300 Read block 22 len 20480
101/300 Write block 2088 len 12288 data: 2 2 2
102/300 Write block 130 len 40960 data: 4 5 4 3 2 2 2 3 5 5
103/300 Write block 2685 len 16384 data: 6 6 5 4
104/300 Write block 4345 len 32768 data: 3 3 4 5 6 6 6 6
105/300 Read block 2491 len 20480
106/300 Read block 1577 len 24576
107/300 Read block 2834 len 16384
108/300 Write block 94 len 36864 data: 2 2 3 3 3 3 3 3 4
109/300 Write block 4119 len 36864 data: 6 5 5 5 4 2 2 2 3
110/300 Write block 564 len 4096 data: 5
111/300 Read block 580 len 16384
112/300 Read block 3984 len 36864
113/300 Read block 1519 len 36864
114/300 Read block 4178 len 12288
115/300 Write block 4886 len 28672 data: 3 4 4 3 3 2 2
116/300 Read block 3879 len 16384
117/300 Write block 1383 len 40960 data: 5 5 3 3 3 4 4 4 5 4
118/300 Write block 1865 len 4096 data: 3
119/300 Write block 309 len 32768 data: 2 2 2 3 3 4 4 4
120/300 Read block 2143 len 40960
121/300 Read block 3541 len 8192
122/300 Read block 4598 len 28672
123/300 Write block 3724 len 40960 data: 4 5 5 6 5 5 4 3 3 3
124/300 Write block 1673 len 36864 data: 4 6 5 5 3 3 3 2 2
125/300 Flush
126/300 Read block 3417 len 12288
127/300 Read block 1044 len 32768
128/300 Read block 3891 len 4096
129/300 Write block 891 len 20480 data: 3 3 2 2 2
130/300 Write block 3176 len 36864 data: 6 6 6 6 4 4 4 3 3
131/300 Read block 1448 len 4096
132/300 Read block 2551 len 16384
133/300 Write block 743 len 32768 data: 5 4 4 4 4 4 4 3
134/300 Write block 4513 len 24576 data: 5 6 4 4 4 4
135/300 Write block 78 len 12288 data: 4 4 3
136/300 Read block 3696 len 4096
137/300 Read block 92 len 4096
138/300 Write block 946 len 32768 data: 3 3 2 2 2 2 3 3
139/300 Write block 3860 len 8192 data: 2 3
140/300 Write block 519 len 8192 data: 3 3
141/300 Write block 4613 len 8192 data: 4 3
142/300 Read block 4301 len 20480
143/300 Write block 3469 len 4096 data: 4
144/300 Write block 495 len 28672 data: 4 4 6 7 7 5 6
145/300 Read block 2835 len 4096
146/300 Write block 1335 len 16384 data: 4 4 4 5
147/300 Read block 3330 len 40960
148/300 Read block 2837 len 36864
149/300 Read block 2169 len 8192
150/300 Flush
151/300 Write block 1185 len 28672 data: 3 2 2 2 3 4 4
152/300 Write block 4712 len 12288 data: 4 5 5
153/300 Write block 79 len 20480 data: 5 4 3 3 4
154/300 Read block 1418 len 36864
155/300 Read block 3701 len 20480
156/300 Write block 4040 len 36864 data: 3 4 5 6 5 5 5 4 3
157/300 Flush
158/300 Read block 4103 len 12288
159/300 Write block 2133 len 8192 data: 3 3
160/300 Read block 1402 len 36864
161/300 Read block 4251 len 28672
162/300 Read block 363 len 28672
163/300 Write block 3606 len 12288 data: 6 6 6
164/300 Read block 665 len 4096
165/300 Read block 4310 len 20480
166/300 Flush
167/300 Read block 3532 len 4096
168/300 Read block 1967 len 20480
169/300 Read block 2260 len 12288
170/300 Read block 3433 len 28672
171/300 Read block 1908 len 32768
172/300 Read block 187 len 16384
173/300 Read block 559 len 32768
174/300 Write block 3461 len 32768 data: 3 4 3 4 5 5 5 4
175/300 Write block 4916 len 36864 data: 4 3 2 2 2 2 5 6 5
176/300 Write block 1751 len 4096 data: 4
177/300 Read block 1816 len 8192
178/300 Read block 3488 len 4096
179/300 Read block 1448 len 32768
180/300 Flush
181/300 Read block 2948 len 12288
182/300 Write block 192 len 8192 data: 4 4
183/300 Write block 1234 len 12288 data: 6 6 6
184/300 Read block 683 len 12288
185/300 Read block 2305 len 32768
186/300 Write block 4750 len 28672 data: 3 2 2 2 2 3 2
187/300 Read block 1948 len 8192
188/300 Write block 1738 len 28672 data: 3 4 4 4 3 3 3
189/300 Read block 1711 len 36864
190/300 Flush
191/300 Write block 3004 len 36864 data: 3 3 3 3 3 3 2 2 2
192/300 Write block 3623 len 16384 data: 6 6 3 2
193/300 Read block 1240 len 12288
194/300 Flush
195/300 Read block 1733 len 28672
196/300 Write block 1276 len 36864 data: 2 2 2 2 2 2 2 2 2
197/300 Read block 1372 len 28672
198/300 Read block 2388 len 20480
199/300 Read block 3316 len 4096
200/300 Flush
201/300 Read block 4745 len 20480
202/300 Read block 4060 len 4096
203/300 Read block 3405 len 4096
204/300 Write block 420 len 28672 data: 4 4 4 4 4 4 4
205/300 Read block 3524 len 4096
206/300 Read block 2458 len 16384
207/300 Read block 1597 len 36864
208/300 Write block 2014 len 16384 data: 4 4 4 4
209/300 Read block 4609 len 20480
210/300 Read block 3042 len 36864
211/300 Read block 4045 len 40960
212/300 Flush
213/300 Read block 2537 len 12288
214/300 Flush
215/300 Read block 4255 len 32768
216/300 Read block 4878 len 28672
217/300 Read block 2915 len 24576
218/300 Read block 2708 len 12288
219/300 Read block 2111 len 28672
220/300 Read block 4096 len 12288
221/300 Write block 4383 len 24576 data: 5 4 4 4 4 3
222/300 Read block 3028 len 28672
223/300 Write block 2269 len 20480 data: 6 3 3 2 2
224/300 Read block 1401 len 40960
225/300 Write block 1117 len 28672 data: 3 3 2 3 3 5 6
226/300 Read block 4496 len 32768
227/300 Write block 3451 len 20480 data: 4 4 5 5 7
228/300 Write block 815 len 28672 data: 3 3 3 3 4 4 3
229/300 Write block 2320 len 12288 data: 2 2 2
230/300 Write block 1368 len 32768 data: 4 4 5 4 3 5 5 6
231/300 Write block 1815 len 8192 data: 6 5
232/300 Read block 4094 len 36864
233/300 Read block 1670 len 28672
234/300 Write block 4835 len 32768 data: 3 3 3 4 3 3 3 4
235/300 Read block 2857 len 20480
236/300 Read block 2218 len 40960
237/300 Read block 2157 len 20480
238/300 Read block 2772 len 20480
239/300 Read block 2239 len 32768
240/300 Write block 4173 len 32768 data: 5 4 4 4 4 4 2 2
241/300 Read block 2566 len 24576
242/300 Write block 2032 len 12288 data: 4 3 4
243/300 Write block 453 len 8192 data: 3 3
244/300 Read block 2561 len 20480
245/300 Write block 168 len 20480 data: 6 6 6 6 4
246/300 Write block 4348 len 8192 data: 6 7
247/300 Write block 505 len 12288 data: 4 4 3
248/300 Read block 1485 len 16384
249/300 Read block 4509 len 12288
250/300 Read block 1402 len 4096
251/300 Read block 3015 len 32768
252/300 Write block 4834 len 24576 data: 3 4 4 4 5 4
253/300 Write block 3428 len 16384 data: 6 6 6 6
254/300 Write block 1596 len 4096 data: 3
255/300 Write block 4870 len 12288 data: 5 4 4
256/300 Write block 707 len 20480 data: 4 5 4 4 5
257/300 Read block 392 len 28672
258/300 Read block 2250 len 40960
259/300 Read block 361 len 12288
260/300 Read block 2603 len 8192
261/300 Read block 4743 len 4096
262/300 Read block 755 len 28672
263/300 Write block 4480 len 4096 data: 5
264/300 Write block 2201 len 36864 data: 6 5 6 4 5 4 4 4 4
265/300 Read block 1549 len 8192
266/300 Write block 2106 len 20480 data: 2 2 2 2 2
267/300 Write block 1815 len 12288 data: 7 6 5
268/300 Read block 1138 len 40960
269/300 Read block 365 len 28672
270/300 Write block 4935 len 24576 data: 2 2 2 3 3 3
271/300 Read block 3163 len 4096
272/300 Write block 1204 len 28672 data: 5 5 5 5 4 4 3
273/300 Read block 1266 len 28672
274/300 Write block 4489 len 24576 data: 3 3 3 5 5 5
275/300 Read block 2636 len 4096
276/300 Write block 451 len 8192 data: 3 3
277/300 Read block 1270 len 4096
278/300 Read block 1534 len 40960
279/300 Write block 2036 len 36864 data: 3 3 3 2 2 3 4 4 3
280/300 Read block 4921 len 32768
281/300 Read block 1687 len 32768
282/300 Write block 46 len 36864 data: 4 4 5 5 5 4 4 3 4
283/300 Write block 3628 len 36864 data: 2 2 2 2 2 3 3 5 5
284/300 Read block 4148 len 28672
285/300 Write block 4690 len 36864 data: 2 2 2 3 3 3 3 3 3
286/300 Write block 1344 len 24576 data: 5 5 5 5 5 4
287/300 Read block 2972 len 20480
288/300 Read block 3249 len 16384
289/300 Read block 1545 len 12288
290/300 Read block 2504 len 36864
291/300 Flush
292/300 Read block 1062 len 36864
293/300 Read block 2952 len 28672
294/300 Write block 4805 len 16384 data: 6 3 4 4
295/300 Flush
296/300 Read block 1608 len 24576
297/300 Read block 3804 len 4096
298/300 Read block 117 len 12288
299/300 Write block 2925 len 40960 data: 2 2 3 5 5 4 3 3 4 4
300/300 Write block 2936 len 32768 data: 6 4 3 3 3 3 3 2
301/300 Read block 1686 len 28672
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:39:24.372 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/0, method: POST, req_id: ad283adf-1532-4024-bdd0-9bdf6e832c02, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Write 3 Sent Done Done false
4924 Acked Write 8 Sent Done Done false
4925 Acked Write 2 Sent Done Done false
4926 Acked Read 8 Sent Done Done false
4927 Acked Read 6 Sent Done Done false
4928 Acked Read 8 Sent Done Done false
4929 Acked Write 10 Sent Done Done false
4930 Acked Read 5 Sent Done Done false
4931 Acked Read 7 Sent Done Done false
4932 Acked Write 2 Sent Done Done false
4933 Acked Write 6 Sent Done Done false
4934 Acked Write 5 Sent Done Done false
4935 Acked Read 4 Sent Done Done false
4936 Acked Read 2 Sent Done Done false
4937 Acked Read 8 Sent Done Done false
4938 Acked Write 1 Sent Done Done false
4939 Acked Read 2 Sent Done Done false
4940 Acked Write 6 Sent Done Done false
4941 Acked Write 9 Sent Done Done false
4942 Acked Read 1 Sent Done Done false
4943 Acked Write 6 Sent Done Done false
4944 Acked Read 1 Sent Done Done false
4945 Acked Read 4 Sent Done Done false
4946 Acked Write 10 Sent Done Done false
4947 Acked Write 3 Sent Done Done false
4948 Acked Write 6 Sent Done Done false
4949 Acked Read 6 Sent Done Done false
4950 Acked Read 6 Sent Done Done false
4951 Acked Read 7 Sent Done Done false
4952 Acked Read 7 Sent Done Done false
4953 Acked Write 5 Sent Done Done false
4954 Acked Write 10 Sent Done Done false
4955 Acked Read 9 Sent Done Done false
4956 Acked Write 9 Sent Done Done false
4957 Acked Write 8 Sent Done Done false
4958 Acked Read 4 Sent Done Done false
4959 Acked Read 3 Sent Done Done false
4960 Acked Write 10 Sent Done Done false
4961 Acked Write 9 Sent Done Done false
4962 Acked Read 9 Sent Done Done false
4963 Acked Read 3 Sent Done Done false
4964 Acked Write 8 Sent Done Done false
4965 Acked Read 6 Sent Done Done false
4966 Acked Write 1 Sent Done Done false
4967 Acked Read 7 Sent Done Done false
4968 Acked Read 1 Sent Done Done false
4969 Acked Read 1 Sent Done Done false
4970 Acked Write 1 Sent Done Done false
4971 Acked Read 10 Sent Done Done false
4972 Acked Read 4 Sent Done Done false
4973 Acked Write 9 Sent Done Done false
4974 Acked Read 6 Sent Done Done false
4975 Acked Write 10 Sent Done Done false
4976 Acked Read 3 Sent Done Done false
4977 Acked Read 2 Sent Done Done false
4978 Acked Flush 0 Sent Done Done false
4979 Acked Write 5 Sent Done Done false
4980 Acked Read 10 Sent Done Done false
4981 Acked Read 4 Sent Done Done false
4982 Acked Write 9 Sent Done Done false
4983 Acked Read 9 Sent Done Done false
4984 Acked Read 3 Sent Done Done false
4985 Acked Write 8 Sent Done Done false
4986 Acked Flush 0 Sent Done Done false
4987 Acked Write 5 Sent Done Done false
4988 Acked Write 1 Sent Done Done false
4989 Acked Write 2 Sent Done Done false
4990 Acked Write 8 Sent Done Done false
4991 Acked Write 3 Sent Done Done false
4992 Acked Read 5 Sent Done Done false
4993 Acked Read 2 Sent Done Done false
4994 Acked Read 3 Sent Done Done false
4995 Acked Read 2 Sent Done Done false
4996 Acked Read 5 Sent Done Done false
4997 Acked Read 1 Sent Done Done false
4998 Acked Read 2 Sent Done Done false
4999 Acked Read 6 Sent Done Done false
5000 Acked Write 7 Sent Done Done false
5001 Acked Read 9 Sent Done Done false
5002 Acked Read 1 Sent Done Done false
5003 Acked Write 10 Sent Done Done false
5004 Acked Flush 0 Sent Done Done false
5005 Acked Read 7 Sent Done Done false
5006 Acked Read 8 Sent Done Done false
5007 Acked Read 5 Sent Done Done false
5008 Acked Read 4 Sent Done Done false
5009 Acked Write 5 Sent Done Done false
5010 Acked Write 7 Sent Done Done false
5011 Acked Read 2 Sent Done Done false
5012 Acked Read 9 Sent Done Done false
5013 Acked Read 7 Sent Done Done false
5014 Acked Flush 0 Sent Done Done false
5015 Acked Read 6 Sent Done Done false
5016 Acked Read 9 Sent Done Done false
5017 Acked Write 8 Sent Done Done false
5018 Acked Read 2 Sent Done Done false
5019 Acked Write 7 Sent Done Done false
5020 Acked Flush 0 Sent Done Done false
5021 Acked Read 6 Sent Done Done false
5022 Acked Read 5 Sent Done Done false
5023 Acked Write 3 Sent Done Done false
5024 Acked Write 10 Sent Done Done false
5025 Acked Write 4 Sent Done Done false
5026 Acked Write 8 Sent Done Done false
5027 Acked Read 5 Sent Done Done false
5028 Acked Read 6 Sent Done Done false
5029 Acked Read 4 Sent Done Done false
5030 Acked Write 9 Sent Done Done false
5031 Acked Write 9 Sent Done Done false
5032 Acked Write 1 Sent Done Done false
5033 Acked Read 4 Sent Done Done false
5034 Acked Read 9 Sent Done Done false
5035 Acked Read 9 Sent Done Done false
5036 Acked Read 3 Sent Done Done false
5037 Acked Write 7 Sent Done Done false
5038 Acked Read 4 Sent Done Done false
5039 Acked Write 10 Sent Done Done false
5040 Acked Write 1 Sent Done Done false
5041 Acked Write 8 Sent Done Done false
5042 Acked Read 10 Sent Done Done false
5043 Acked Read 2 Sent Done Done false
5044 Acked Read 7 Sent Done Done false
5045 Acked Write 10 Sent Done Done false
5046 Acked Write 9 Sent Done Done false
5047 Acked Flush 0 Sent Done Done false
5048 Acked Read 3 Sent Done Done false
5049 Acked Read 8 Sent Done Done false
5050 Acked Read 1 Sent Done Done false
5051 Acked Write 5 Sent Done Done false
5052 Acked Write 9 Sent Done Done false
5053 Acked Read 1 Sent Done Done false
5054 Acked Read 4 Sent Done Done false
5055 Acked Write 8 Sent Done Done false
5056 Acked Write 6 Sent Done Done false
5057 Acked Write 3 Sent Done Done false
5058 Acked Read 1 Sent Done Done false
5059 Acked Read 1 Sent Done Done false
5060 Acked Write 8 Sent Done Done false
5061 Acked Write 2 Sent Done Done false
5062 Acked Write 2 Sent Done Done false
5063 Acked Write 2 Sent Done Done false
5064 Acked Read 5 Sent Done Done false
5065 Acked Write 1 Sent Done Done false
5066 Acked Write 7 Sent Done Done false
5067 Acked Read 1 Sent Done Done false
5068 Acked Write 4 Sent Done Done false
5069 Acked Read 10 Sent Done Done false
5070 Acked Read 9 Sent Done Done false
5071 Acked Read 2 Sent Done Done false
5072 Acked Flush 0 Sent Done Done false
5073 Acked Write 7 Sent Done Done false
5074 Acked Write 3 Sent Done Done false
5075 Acked Write 5 Sent Done Done false
5076 Acked Read 9 Sent Done Done false
5077 Acked Read 5 Sent Done Done false
5078 Acked Write 9 Sent Done Done false
5079 Acked Flush 0 Sent Done Done false
5080 Acked Read 3 Sent Done Done false
5081 Acked Write 2 Sent Done Done false
5082 Acked Read 9 Sent Done Done false
5083 Acked Read 7 Sent Done Done false
5084 Acked Read 7 Sent Done Done false
5085 Acked Write 3 Sent Done Done false
5086 Acked Read 1 Sent Done Done false
5087 Acked Read 5 Sent Done Done false
5088 Acked Flush 0 Sent Done Done false
5089 Acked Read 1 Sent Done Done false
5090 Acked Read 5 Sent Done Done false
5091 Acked Read 3 Sent Done Done false
5092 Acked Read 7 Sent Done Done false
5093 Acked Read 8 Sent Done Done false
5094 Acked Read 4 Sent Done Done false
5095 Acked Read 8 Sent Done Done false
5096 Acked Write 8 Sent Done Done false
5097 Acked Write 9 Sent Done Done false
5098 Acked Write 1 Sent Done Done false
5099 Acked Read 2 Sent Done Done false
5100 Acked Read 1 Sent Done Done false
5101 Acked Read 8 Sent Done Done false
5102 Acked Flush 0 Sent Done Done false
5103 Acked Read 3 Sent Done Done false
5104 Acked Write 2 Sent Done Done false
5105 Acked Write 3 Sent Done Done false
5106 Acked Read 3 Sent Done Done false
5107 Acked Read 8 Sent Done Done false
5108 Acked Write 7 Sent Done Done false
5109 Acked Read 2 Sent Done Done false
5110 Acked Write 7 Sent Done Done false
5111 Acked Read 9 Sent Done Done false
5112 Acked Flush 0 Sent Done Done false
5113 Acked Write 9 Sent Done Done false
5114 Acked Write 4 Sent Done Done false
5115 Acked Read 3 Sent Done Done false
5116 Acked Flush 0 Sent Done Done false
5117 Acked Read 7 Sent Done Done false
5118 Acked Write 9 Sent Done Done false
5119 Acked Read 7 Sent Done Done false
5120 Acked Read 5 Sent Done Done false
5121 Acked Read 1 Sent Done Done false
5122 Acked Flush 0 Sent Done Done false
5123 Acked Read 5 Sent Done Done false
5124 Acked Read 1 Sent Done Done false
5125 Acked Read 1 Sent Done Done false
5126 Acked Write 7 Sent Done Done false
5127 Acked Read 1 Sent Done Done false
5128 Acked Read 4 Sent Done Done false
5129 Acked Read 9 Sent Done Done false
5130 Acked Write 4 Sent Done Done false
5131 Acked Read 5 Sent Done Done false
5132 Acked Read 9 Sent Done Done false
5133 Acked Read 10 Sent Done Done false
5134 Acked Flush 0 Sent Done Done false
5135 Acked Read 3 Sent Done Done false
5136 Acked Flush 0 Sent Done Done false
5137 Acked Read 8 Sent Done Done false
5138 Acked Read 7 Sent Done Done false
5139 Acked Read 6 Sent Done Done false
5140 Acked Read 3 Sent Done Done false
5141 Acked Read 7 Sent Done Done false
5142 Acked Read 3 Sent Done Done false
5143 Acked Write 6 Sent Done Done false
5144 Acked Read 7 Sent Done Done false
5145 Acked Write 5 Sent Done Done false
5146 Acked Read 10 Sent Done Done false
5147 Acked Write 7 Sent Done Done false
5148 Acked Read 8 Sent Done Done false
5149 Acked Write 5 Sent Done Done false
5150 Acked Write 7 Sent Done Done false
5151 Acked Write 3 Sent Done Done false
5152 Acked Write 8 Sent Done Done false
5153 Acked Write 2 Sent Done Done false
5154 Acked Read 9 Sent Done Done false
5155 Acked Read 7 Sent Done Done false
5156 Acked Write 8 Sent Done Done false
5157 Acked Read 5 Sent Done Done false
5158 Acked Read 10 Sent Done Done false
5159 Acked Read 5 Sent Done Done false
5160 Acked Read 5 Sent Done Done false
5161 Acked Read 8 Sent Done Done false
5162 Acked Write 8 Sent Done Done false
5163 Acked Read 6 Sent Done Done false
5164 Acked Write 3 Sent Done Done false
5165 Acked Write 2 Sent Done Done false
5166 Acked Read 5 Sent Done Done false
5167 Acked Write 5 Sent Done Done false
5168 Acked Write 2 Sent Done Done false
5169 Acked Write 3 Sent Done Done false
5170 Acked Read 4 Sent Done Done false
5171 Acked Read 3 Sent Done Done false
5172 Acked Read 1 Sent Done Done false
5173 Acked Read 8 Sent Done Done false
5174 Acked Write 6 Sent Done Done false
5175 Acked Write 4 Sent Done Done false
5176 Acked Write 1 Sent Done Done false
5177 Acked Write 3 Sent Done Done false
5178 Acked Write 5 Sent Done Done false
5179 Acked Read 7 Sent Done Done false
5180 Acked Read 10 Sent Done Done false
5181 Acked Read 3 Sent Done Done false
5182 Acked Read 2 Sent Done Done false
5183 Acked Read 1 Sent Done Done false
5184 Acked Read 7 Sent Done Done false
5185 Acked Write 1 Sent Done Done false
5186 Acked Write 9 Sent Done Done false
5187 Acked Read 2 Sent Done Done false
5188 Acked Write 5 Sent Done Done false
5189 Acked Write 3 Sent Done Done false
5190 Acked Read 10 Sent Done Done false
5191 Acked Read 7 Sent Done Done false
5192 Acked Write 6 Sent Done Done false
5193 Acked Read 1 Sent Done Done false
5194 Acked Write 7 Sent Done Done false
5195 Acked Read 7 Sent Done Done false
5196 Acked Write 6 Sent Done Done false
5197 Acked Read 1 Sent Done Done false
5198 Acked Write 2 Sent Done Done false
5199 Acked Read 1 Sent Done Done false
5200 Acked Read 10 Sent Done Done false
5201 Acked Write 9 Sent Done Done false
5202 Acked Read 8 Sent Done Done false
5203 Acked Read 8 Sent Done Done false
5204 Acked Write 9 Sent Done Done false
5205 Acked Write 9 Sent Done Done false
5206 Acked Read 7 Sent Done Done false
5207 Acked Write 9 Sent Done Done false
5208 Acked Write 6 Sent Done Done false
5209 Acked Read 5 Sent Done Done false
5210 Acked Read 4 Sent Done Done false
5211 Acked Read 3 Sent Done Done false
5212 Acked Read 9 Sent Done Done false
5213 Acked Flush 0 Sent Done Done false
5214 Acked Read 9 Sent Done Done false
5215 Acked Read 7 Sent Done Done false
5216 Acked Write 4 Sent Done Done false
5217 Acked Flush 0 Sent Done Done false
5218 Acked Read 6 Sent Done Done false
5219 Acked Read 1 Sent Done Done false
5220 Acked Read 3 Sent Done Done false
5221 Acked Write 10 Sent Done Done false
5222 Acked Write 8 Sent Done Done false
5223 Acked Read 7 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4922 5217 5217
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5223 5222 5221 5220 5219
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Write 3 Sent Done Done false
4924 Acked Write 8 Sent Done Done false
4925 Acked Write 2 Sent Done Done false
4926 Acked Read 8 Sent Done Done false
4927 Acked Read 6 Sent Done Done false
4928 Acked Read 8 Sent Done Done false
4929 Acked Write 10 Sent Done Done false
4930 Acked Read 5 Sent Done Done false
4931 Acked Read 7 Sent Done Done false
4932 Acked Write 2 Sent Done Done false
4933 Acked Write 6 Sent Done Done false
4934 Acked Write 5 Sent Done Done false
4935 Acked Read 4 Sent Done Done false
4936 Acked Read 2 Sent Done Done false
4937 Acked Read 8 Sent Done Done false
4938 Acked Write 1 Sent Done Done false
4939 Acked Read 2 Sent Done Done false
4940 Acked Write 6 Sent Done Done false
4941 Acked Write 9 Sent Done Done false
4942 Acked Read 1 Sent Done Done false
4943 Acked Write 6 Sent Done Done false
4944 Acked Read 1 Sent Done Done false
4945 Acked Read 4 Sent Done Done false
4946 Acked Write 10 Sent Done Done false
4947 Acked Write 3 Sent Done Done false
4948 Acked Write 6 Sent Done Done false
4949 Acked Read 6 Sent Done Done false
4950 Acked Read 6 Sent Done Done false
4951 Acked Read 7 Sent Done Done false
4952 Acked Read 7 Sent Done Done false
4953 Acked Write 5 Sent Done Done false
4954 Acked Write 10 Sent Done Done false
4955 Acked Read 9 Sent Done Done false
4956 Acked Write 9 Sent Done Done false
4957 Acked Write 8 Sent Done Done false
4958 Acked Read 4 Sent Done Done false
4959 Acked Read 3 Sent Done Done false
4960 Acked Write 10 Sent Done Done false
4961 Acked Write 9 Sent Done Done false
4962 Acked Read 9 Sent Done Done false
4963 Acked Read 3 Sent Done Done false
4964 Acked Write 8 Sent Done Done false
4965 Acked Read 6 Sent Done Done false
4966 Acked Write 1 Sent Done Done false
4967 Acked Read 7 Sent Done Done false
4968 Acked Read 1 Sent Done Done false
4969 Acked Read 1 Sent Done Done false
4970 Acked Write 1 Sent Done Done false
4971 Acked Read 10 Sent Done Done false
4972 Acked Read 4 Sent Done Done false
4973 Acked Write 9 Sent Done Done false
4974 Acked Read 6 Sent Done Done false
4975 Acked Write 10 Sent Done Done false
4976 Acked Read 3 Sent Done Done false
4977 Acked Read 2 Sent Done Done false
4978 Acked Flush 0 Sent Done Done false
4979 Acked Write 5 Sent Done Done false
4980 Acked Read 10 Sent Done Done false
4981 Acked Read 4 Sent Done Done false
4982 Acked Write 9 Sent Done Done false
4983 Acked Read 9 Sent Done Done false
4984 Acked Read 3 Sent Done Done false
4985 Acked Write 8 Sent Done Done false
4986 Acked Flush 0 Sent Done Done false
4987 Acked Write 5 Sent Done Done false
4988 Acked Write 1 Sent Done Done false
4989 Acked Write 2 Sent Done Done false
4990 Acked Write 8 Sent Done Done false
4991 Acked Write 3 Sent Done Done false
4992 Acked Read 5 Sent Done Done false
4993 Acked Read 2 Sent Done Done false
4994 Acked Read 3 Sent Done Done false
4995 Acked Read 2 Sent Done Done false
4996 Acked Read 5 Sent Done Done false
4997 Acked Read 1 Sent Done Done false
4998 Acked Read 2 Sent Done Done false
4999 Acked Read 6 Sent Done Done false
5000 Acked Write 7 Sent Done Done false
5001 Acked Read 9 Sent Done Done false
5002 Acked Read 1 Sent Done Done false
5003 Acked Write 10 Sent Done Done false
5004 Acked Flush 0 Sent Done Done false
5005 Acked Read 7 Sent Done Done false
5006 Acked Read 8 Sent Done Done false
5007 Acked Read 5 Sent Done Done false
5008 Acked Read 4 Sent Done Done false
5009 Acked Write 5 Sent Done Done false
5010 Acked Write 7 Sent Done Done false
5011 Acked Read 2 Sent Done Done false
5012 Acked Read 9 Sent Done Done false
5013 Acked Read 7 Sent Done Done false
5014 Acked Flush 0 Sent Done Done false
5015 Acked Read 6 Sent Done Done false
5016 Acked Read 9 Sent Done Done false
5017 Acked Write 8 Sent Done Done false
5018 Acked Read 2 Sent Done Done false
5019 Acked Write 7 Sent Done Done false
5020 Acked Flush 0 Sent Done Done false
5021 Acked Read 6 Sent Done Done false
5022 Acked Read 5 Sent Done Done false
5023 Acked Write 3 Sent Done Done false
5024 Acked Write 10 Sent Done Done false
5025 Acked Write 4 Sent Done Done false
5026 Acked Write 8 Sent Done Done false
5027 Acked Read 5 Sent Done Done false
5028 Acked Read 6 Sent Done Done false
5029 Acked Read 4 Sent Done Done false
5030 Acked Write 9 Sent Done Done false
5031 Acked Write 9 Sent Done Done false
5032 Acked Write 1 Sent Done Done false
5033 Acked Read 4 Sent Done Done false
5034 Acked Read 9 Sent Done Done false
5035 Acked Read 9 Sent Done Done false
5036 Acked Read 3 Sent Done Done false
5037 Acked Write 7 Sent Done Done false
5038 Acked Read 4 Sent Done Done false
5039 Acked Write 10 Sent Done Done false
5040 Acked Write 1 Sent Done Done false
5041 Acked Write 8 Sent Done Done false
5042 Acked Read 10 Sent Done Done false
5043 Acked Read 2 Sent Done Done false
5044 Acked Read 7 Sent Done Done false
5045 Acked Write 10 Sent Done Done false
5046 Acked Write 9 Sent Done Done false
5047 Acked Flush 0 Sent Done Done false
5048 Acked Read 3 Sent Done Done false
5049 Acked Read 8 Sent Done Done false
5050 Acked Read 1 Sent Done Done false
5051 Acked Write 5 Sent Done Done false
5052 Acked Write 9 Sent Done Done false
5053 Acked Read 1 Sent Done Done false
5054 Acked Read 4 Sent Done Done false
5055 Acked Write 8 Sent Done Done false
5056 Acked Write 6 Sent Done Done false
5057 Acked Write 3 Sent Done Done false
5058 Acked Read 1 Sent Done Done false
5059 Acked Read 1 Sent Done Done false
5060 Acked Write 8 Sent Done Done false
5061 Acked Write 2 Sent Done Done false
5062 Acked Write 2 Sent Done Done false
5063 Acked Write 2 Sent Done Done false
5064 Acked Read 5 Sent Done Done false
5065 Acked Write 1 Sent Done Done false
5066 Acked Write 7 Sent Done Done false
5067 Acked Read 1 Sent Done Done false
5068 Acked Write 4 Sent Done Done false
5069 Acked Read 10 Sent Done Done false
5070 Acked Read 9 Sent Done Done false
5071 Acked Read 2 Sent Done Done false
5072 Acked Flush 0 Sent Done Done false
5073 Acked Write 7 Sent Done Done false
5074 Acked Write 3 Sent Done Done false
5075 Acked Write 5 Sent Done Done false
5076 Acked Read 9 Sent Done Done false
5077 Acked Read 5 Sent Done Done false
5078 Acked Write 9 Sent Done Done false
5079 Acked Flush 0 Sent Done Done false
5080 Acked Read 3 Sent Done Done false
5081 Acked Write 2 Sent Done Done false
5082 Acked Read 9 Sent Done Done false
5083 Acked Read 7 Sent Done Done false
5084 Acked Read 7 Sent Done Done false
5085 Acked Write 3 Sent Done Done false
5086 Acked Read 1 Sent Done Done false
5087 Acked Read 5 Sent Done Done false
5088 Acked Flush 0 Sent Done Done false
5089 Acked Read 1 Sent Done Done false
5090 Acked Read 5 Sent Done Done false
5091 Acked Read 3 Sent Done Done false
5092 Acked Read 7 Sent Done Done false
5093 Acked Read 8 Sent Done Done false
5094 Acked Read 4 Sent Done Done false
5095 Acked Read 8 Sent Done Done false
5096 Acked Write 8 Sent Done Done false
5097 Acked Write 9 Sent Done Done false
5098 Acked Write 1 Sent Done Done false
5099 Acked Read 2 Sent Done Done false
5100 Acked Read 1 Sent Done Done false
5101 Acked Read 8 Sent Done Done false
5102 Acked Flush 0 Sent Done Done false
5103 Acked Read 3 Sent Done Done false
5104 Acked Write 2 Sent Done Done false
5105 Acked Write 3 Sent Done Done false
5106 Acked Read 3 Sent Done Done false
5107 Acked Read 8 Sent Done Done false
5108 Acked Write 7 Sent Done Done false
5109 Acked Read 2 Sent Done Done false
5110 Acked Write 7 Sent Done Done false
5111 Acked Read 9 Sent Done Done false
5112 Acked Flush 0 Sent Done Done false
5113 Acked Write 9 Sent Done Done false
5114 Acked Write 4 Sent Done Done false
5115 Acked Read 3 Sent Done Done false
5116 Acked Flush 0 Sent Done Done false
5117 Acked Read 7 Sent Done Done false
5118 Acked Write 9 Sent Done Done false
5119 Acked Read 7 Sent Done Done false
5120 Acked Read 5 Sent Done Done false
5121 Acked Read 1 Sent Done Done false
5122 Acked Flush 0 Sent Done Done false
5123 Acked Read 5 Sent Done Done false
5124 Acked Read 1 Sent Done Done false
5125 Acked Read 1 Sent Done Done false
5126 Acked Write 7 Sent Done Done false
5127 Acked Read 1 Sent Done Done false
5128 Acked Read 4 Sent Done Done false
5129 Acked Read 9 Sent Done Done false
5130 Acked Write 4 Sent Done Done false
5131 Acked Read 5 Sent Done Done false
5132 Acked Read 9 Sent Done Done false
5133 Acked Read 10 Sent Done Done false
5134 Acked Flush 0 Sent Done Done false
5135 Acked Read 3 Sent Done Done false
5136 Acked Flush 0 Sent Done Done false
5137 Acked Read 8 Sent Done Done false
5138 Acked Read 7 Sent Done Done false
5139 Acked Read 6 Sent Done Done false
5140 Acked Read 3 Sent Done Done false
5141 Acked Read 7 Sent Done Done false
5142 Acked Read 3 Sent Done Done false
5143 Acked Write 6 Sent Done Done false
5144 Acked Read 7 Sent Done Done false
5145 Acked Write 5 Sent Done Done false
5146 Acked Read 10 Sent Done Done false
5147 Acked Write 7 Sent Done Done false
5148 Acked Read 8 Sent Done Done false
5149 Acked Write 5 Sent Done Done false
5150 Acked Write 7 Sent Done Done false
5151 Acked Write 3 Sent Done Done false
5152 Acked Write 8 Sent Done Done false
5153 Acked Write 2 Sent Done Done false
5154 Acked Read 9 Sent Done Done false
5155 Acked Read 7 Sent Done Done false
5156 Acked Write 8 Sent Done Done false
5157 Acked Read 5 Sent Done Done false
5158 Acked Read 10 Sent Done Done false
5159 Acked Read 5 Sent Done Done false
5160 Acked Read 5 Sent Done Done false
5161 Acked Read 8 Sent Done Done false
5162 Acked Write 8 Sent Done Done false
5163 Acked Read 6 Sent Done Done false
5164 Acked Write 3 Sent Done Done false
5165 Acked Write 2 Sent Done Done false
5166 Acked Read 5 Sent Done Done false
5167 Acked Write 5 Sent Done Done false
5168 Acked Write 2 Sent Done Done false
5169 Acked Write 3 Sent Done Done false
5170 Acked Read 4 Sent Done Done false
5171 Acked Read 3 Sent Done Done false
5172 Acked Read 1 Sent Done Done false
5173 Acked Read 8 Sent Done Done false
5174 Acked Write 6 Sent Done Done false
5175 Acked Write 4 Sent Done Done false
5176 Acked Write 1 Sent Done Done false
5177 Acked Write 3 Sent Done Done false
5178 Acked Write 5 Sent Done Done false
5179 Acked Read 7 Sent Done Done false
5180 Acked Read 10 Sent Done Done false
5181 Acked Read 3 Sent Done Done false
5182 Acked Read 2 Sent Done Done false
5183 Acked Read 1 Sent Done Done false
5184 Acked Read 7 Sent Done Done false
5185 Acked Write 1 Sent Done Done false
5186 Acked Write 9 Sent Done Done false
5187 Acked Read 2 Sent Done Done false
5188 Acked Write 5 Sent Done Done false
5189 Acked Write 3 Sent Done Done false
5190 Acked Read 10 Sent Done Done false
5191 Acked Read 7 Sent Done Done false
5192 Acked Write 6 Sent Done Done false
5193 Acked Read 1 Sent Done Done false
5194 Acked Write 7 Sent Done Done false
5195 Acked Read 7 Sent Done Done false
5196 Acked Write 6 Sent Done Done false
5197 Acked Read 1 Sent Done Done false
5198 Acked Write 2 Sent Done Done false
5199 Acked Read 1 Sent Done Done false
5200 Acked Read 10 Sent Done Done false
5201 Acked Write 9 Sent Done Done false
5202 Acked Read 8 Sent Done Done false
5203 Acked Read 8 Sent Done Done false
5204 Acked Write 9 Sent Done Done false
5205 Acked Write 9 Sent Done Done false
5206 Acked Read 7 Sent Done Done false
5207 Acked Write 9 Sent Done Done false
5208 Acked Write 6 Sent Done Done false
5209 Acked Read 5 Sent Done Done false
5210 Acked Read 4 Sent Done Done false
5211 Acked Read 3 Sent Done Done false
5212 Acked Read 9 Sent Done Done false
5213 Acked Flush 0 Sent Done Done false
5214 Acked Read 9 Sent Done Done false
5215 Acked Read 7 Sent Done Done false
5216 Acked Write 4 Sent Done Done false
5217 Acked Flush 0 Sent Done Done false
5218 Acked Read 6 Sent Done Done false
5219 Acked Read 1 Sent Done Done false
5220 Acked Read 3 Sent Done Done false
5221 Acked Write 10 Sent Done Done false
5222 Acked Write 8 Sent Done Done false
5223 Acked Read 7 Sent Done Done false
5224 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4922 5224 5224
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5224 5223 5222 5221 5220
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:30.131515431Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:30.131618409Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:30.131660224Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:30.13169719Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:30.131765262Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1214)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Write 3 Sent Done Done false
4924 Acked Write 8 Sent Done Done false
4925 Acked Write 2 Sent Done Done false
4926 Acked Read 8 Sent Done Done false
4927 Acked Read 6 Sent Done Done false
4928 Acked Read 8 Sent Done Done false
4929 Acked Write 10 Sent Done Done false
4930 Acked Read 5 Sent Done Done false
4931 Acked Read 7 Sent Done Done false
4932 Acked Write 2 Sent Done Done false
4933 Acked Write 6 Sent Done Done false
4934 Acked Write 5 Sent Done Done false
4935 Acked Read 4 Sent Done Done false
4936 Acked Read 2 Sent Done Done false
4937 Acked Read 8 Sent Done Done false
4938 Acked Write 1 Sent Done Done false
4939 Acked Read 2 Sent Done Done false
4940 Acked Write 6 Sent Done Done false
4941 Acked Write 9 Sent Done Done false
4942 Acked Read 1 Sent Done Done false
4943 Acked Write 6 Sent Done Done false
4944 Acked Read 1 Sent Done Done false
4945 Acked Read 4 Sent Done Done false
4946 Acked Write 10 Sent Done Done false
4947 Acked Write 3 Sent Done Done false
4948 Acked Write 6 Sent Done Done false
4949 Acked Read 6 Sent Done Done false
4950 Acked Read 6 Sent Done Done false
4951 Acked Read 7 Sent Done Done false
4952 Acked Read 7 Sent Done Done false
4953 Acked Write 5 Sent Done Done false
4954 Acked Write 10 Sent Done Done false
4955 Acked Read 9 Sent Done Done false
4956 Acked Write 9 Sent Done Done false
4957 Acked Write 8 Sent Done Done false
4958 Acked Read 4 Sent Done Done false
4959 Acked Read 3 Sent Done Done false
4960 Acked Write 10 Sent Done Done false
4961 Acked Write 9 Sent Done Done false
4962 Acked Read 9 Sent Done Done false
4963 Acked Read 3 Sent Done Done false
4964 Acked Write 8 Sent Done Done false
4965 Acked Read 6 Sent Done Done false
4966 Acked Write 1 Sent Done Done false
4967 Acked Read 7 Sent Done Done false
4968 Acked Read 1 Sent Done Done false
4969 Acked Read 1 Sent Done Done false
4970 Acked Write 1 Sent Done Done false
4971 Acked Read 10 Sent Done Done false
4972 Acked Read 4 Sent Done Done false
4973 Acked Write 9 Sent Done Done false
4974 Acked Read 6 Sent Done Done false
4975 Acked Write 10 Sent Done Done false
4976 Acked Read 3 Sent Done Done false
4977 Acked Read 2 Sent Done Done false
4978 Acked Flush 0 Sent Done Done false
4979 Acked Write 5 Sent Done Done false
4980 Acked Read 10 Sent Done Done false
4981 Acked Read 4 Sent Done Done false
4982 Acked Write 9 Sent Done Done false
4983 Acked Read 9 Sent Done Done false
4984 Acked Read 3 Sent Done Done false
4985 Acked Write 8 Sent Done Done false
4986 Acked Flush 0 Sent Done Done false
4987 Acked Write 5 Sent Done Done false
4988 Acked Write 1 Sent Done Done false
4989 Acked Write 2 Sent Done Done false
4990 Acked Write 8 Sent Done Done false
4991 Acked Write 3 Sent Done Done false
4992 Acked Read 5 Sent Done Done false
4993 Acked Read 2 Sent Done Done false
4994 Acked Read 3 Sent Done Done false
4995 Acked Read 2 Sent Done Done false
4996 Acked Read 5 Sent Done Done false
4997 Acked Read 1 Sent Done Done false
4998 Acked Read 2 Sent Done Done false
4999 Acked Read 6 Sent Done Done false
5000 Acked Write 7 Sent Done Done false
5001 Acked Read 9 Sent Done Done false
5002 Acked Read 1 Sent Done Done false
5003 Acked Write 10 Sent Done Done false
5004 Acked Flush 0 Sent Done Done false
5005 Acked Read 7 Sent Done Done false
5006 Acked Read 8 Sent Done Done false
5007 Acked Read 5 Sent Done Done false
5008 Acked Read 4 Sent Done Done false
5009 Acked Write 5 Sent Done Done false
5010 Acked Write 7 Sent Done Done false
5011 Acked Read 2 Sent Done Done false
5012 Acked Read 9 Sent Done Done false
5013 Acked Read 7 Sent Done Done false
5014 Acked Flush 0 Sent Done Done false
5015 Acked Read 6 Sent Done Done false
5016 Acked Read 9 Sent Done Done false
5017 Acked Write 8 Sent Done Done false
5018 Acked Read 2 Sent Done Done false
5019 Acked Write 7 Sent Done Done false
5020 Acked Flush 0 Sent Done Done false
5021 Acked Read 6 Sent Done Done false
5022 Acked Read 5 Sent Done Done false
5023 Acked Write 3 Sent Done Done false
5024 Acked Write 10 Sent Done Done false
5025 Acked Write 4 Sent Done Done false
5026 Acked Write 8 Sent Done Done false
5027 Acked Read 5 Sent Done Done false
5028 Acked Read 6 Sent Done Done false
5029 Acked Read 4 Sent Done Done false
5030 Acked Write 9 Sent Done Done false
5031 Acked Write 9 Sent Done Done false
5032 Acked Write 1 Sent Done Done false
5033 Acked Read 4 Sent Done Done false
5034 Acked Read 9 Sent Done Done false
5035 Acked Read 9 Sent Done Done false
5036 Acked Read 3 Sent Done Done false
5037 Acked Write 7 Sent Done Done false
5038 Acked Read 4 Sent Done Done false
5039 Acked Write 10 Sent Done Done false
5040 Acked Write 1 Sent Done Done false
5041 Acked Write 8 Sent Done Done false
5042 Acked Read 10 Sent Done Done false
5043 Acked Read 2 Sent Done Done false
5044 Acked Read 7 Sent Done Done false
5045 Acked Write 10 Sent Done Done false
5046 Acked Write 9 Sent Done Done false
5047 Acked Flush 0 Sent Done Done false
5048 Acked Read 3 Sent Done Done false
5049 Acked Read 8 Sent Done Done false
5050 Acked Read 1 Sent Done Done false
5051 Acked Write 5 Sent Done Done false
5052 Acked Write 9 Sent Done Done false
5053 Acked Read 1 Sent Done Done false
5054 Acked Read 4 Sent Done Done false
5055 Acked Write 8 Sent Done Done false
5056 Acked Write 6 Sent Done Done false
5057 Acked Write 3 Sent Done Done false
5058 Acked Read 1 Sent Done Done false
5059 Acked Read 1 Sent Done Done false
5060 Acked Write 8 Sent Done Done false
5061 Acked Write 2 Sent Done Done false
5062 Acked Write 2 Sent Done Done false
5063 Acked Write 2 Sent Done Done false
5064 Acked Read 5 Sent Done Done false
5065 Acked Write 1 Sent Done Done false
5066 Acked Write 7 Sent Done Done false
5067 Acked Read 1 Sent Done Done false
5068 Acked Write 4 Sent Done Done false
5069 Acked Read 10 Sent Done Done false
5070 Acked Read 9 Sent Done Done false
5071 Acked Read 2 Sent Done Done false
5072 Acked Flush 0 Sent Done Done false
5073 Acked Write 7 Sent Done Done false
5074 Acked Write 3 Sent Done Done false
5075 Acked Write 5 Sent Done Done false
5076 Acked Read 9 Sent Done Done false
5077 Acked Read 5 Sent Done Done false
5078 Acked Write 9 Sent Done Done false
5079 Acked Flush 0 Sent Done Done false
5080 Acked Read 3 Sent Done Done false
5081 Acked Write 2 Sent Done Done false
5082 Acked Read 9 Sent Done Done false
5083 Acked Read 7 Sent Done Done false
5084 Acked Read 7 Sent Done Done false
5085 Acked Write 3 Sent Done Done false
5086 Acked Read 1 Sent Done Done false
5087 Acked Read 5 Sent Done Done false
5088 Acked Flush 0 Sent Done Done false
5089 Acked Read 1 Sent Done Done false
5090 Acked Read 5 Sent Done Done false
5091 Acked Read 3 Sent Done Done false
5092 Acked Read 7 Sent Done Done false
5093 Acked Read 8 Sent Done Done false
5094 Acked Read 4 Sent Done Done false
5095 Acked Read 8 Sent Done Done false
5096 Acked Write 8 Sent Done Done false
5097 Acked Write 9 Sent Done Done false
5098 Acked Write 1 Sent Done Done false
5099 Acked Read 2 Sent Done Done false
5100 Acked Read 1 Sent Done Done false
5101 Acked Read 8 Sent Done Done false
5102 Acked Flush 0 Sent Done Done false
5103 Acked Read 3 Sent Done Done false
5104 Acked Write 2 Sent Done Done false
5105 Acked Write 3 Sent Done Done false
5106 Acked Read 3 Sent Done Done false
5107 Acked Read 8 Sent Done Done false
5108 Acked Write 7 Sent Done Done false
5109 Acked Read 2 Sent Done Done false
5110 Acked Write 7 Sent Done Done false
5111 Acked Read 9 Sent Done Done false
5112 Acked Flush 0 Sent Done Done false
5113 Acked Write 9 Sent Done Done false
5114 Acked Write 4 Sent Done Done false
5115 Acked Read 3 Sent Done Done false
5116 Acked Flush 0 Sent Done Done false
5117 Acked Read 7 Sent Done Done false
5118 Acked Write 9 Sent Done Done false
5119 Acked Read 7 Sent Done Done false
5120 Acked Read 5 Sent Done Done false
5121 Acked Read 1 Sent Done Done false
5122 Acked Flush 0 Sent Done Done false
5123 Acked Read 5 Sent Done Done false
5124 Acked Read 1 Sent Done Done false
5125 Acked Read 1 Sent Done Done false
5126 Acked Write 7 Sent Done Done false
5127 Acked Read 1 Sent Done Done false
5128 Acked Read 4 Sent Done Done false
5129 Acked Read 9 Sent Done Done false
5130 Acked Write 4 Sent Done Done false
5131 Acked Read 5 Sent Done Done false
5132 Acked Read 9 Sent Done Done false
5133 Acked Read 10 Sent Done Done false
5134 Acked Flush 0 Sent Done Done false
5135 Acked Read 3 Sent Done Done false
5136 Acked Flush 0 Sent Done Done false
5137 Acked Read 8 Sent Done Done false
5138 Acked Read 7 Sent Done Done false
5139 Acked Read 6 Sent Done Done false
5140 Acked Read 3 Sent Done Done false
5141 Acked Read 7 Sent Done Done false
5142 Acked Read 3 Sent Done Done false
5143 Acked Write 6 Sent Done Done false
5144 Acked Read 7 Sent Done Done false
5145 Acked Write 5 Sent Done Done false
5146 Acked Read 10 Sent Done Done false
5147 Acked Write 7 Sent Done Done false
5148 Acked Read 8 Sent Done Done false
5149 Acked Write 5 Sent Done Done false
5150 Acked Write 7 Sent Done Done false
5151 Acked Write 3 Sent Done Done false
5152 Acked Write 8 Sent Done Done false
5153 Acked Write 2 Sent Done Done false
5154 Acked Read 9 Sent Done Done false
5155 Acked Read 7 Sent Done Done false
5156 Acked Write 8 Sent Done Done false
5157 Acked Read 5 Sent Done Done false
5158 Acked Read 10 Sent Done Done false
5159 Acked Read 5 Sent Done Done false
5160 Acked Read 5 Sent Done Done false
5161 Acked Read 8 Sent Done Done false
5162 Acked Write 8 Sent Done Done false
5163 Acked Read 6 Sent Done Done false
5164 Acked Write 3 Sent Done Done false
5165 Acked Write 2 Sent Done Done false
5166 Acked Read 5 Sent Done Done false
5167 Acked Write 5 Sent Done Done false
5168 Acked Write 2 Sent Done Done false
5169 Acked Write 3 Sent Done Done false
5170 Acked Read 4 Sent Done Done false
5171 Acked Read 3 Sent Done Done false
5172 Acked Read 1 Sent Done Done false
5173 Acked Read 8 Sent Done Done false
5174 Acked Write 6 Sent Done Done false
5175 Acked Write 4 Sent Done Done false
5176 Acked Write 1 Sent Done Done false
5177 Acked Write 3 Sent Done Done false
5178 Acked Write 5 Sent Done Done false
5179 Acked Read 7 Sent Done Done false
5180 Acked Read 10 Sent Done Done false
5181 Acked Read 3 Sent Done Done false
5182 Acked Read 2 Sent Done Done false
5183 Acked Read 1 Sent Done Done false
5184 Acked Read 7 Sent Done Done false
5185 Acked Write 1 Sent Done Done false
5186 Acked Write 9 Sent Done Done false
5187 Acked Read 2 Sent Done Done false
5188 Acked Write 5 Sent Done Done false
5189 Acked Write 3 Sent Done Done false
5190 Acked Read 10 Sent Done Done false
5191 Acked Read 7 Sent Done Done false
5192 Acked Write 6 Sent Done Done false
5193 Acked Read 1 Sent Done Done false
5194 Acked Write 7 Sent Done Done false
5195 Acked Read 7 Sent Done Done false
5196 Acked Write 6 Sent Done Done false
5197 Acked Read 1 Sent Done Done false
5198 Acked Write 2 Sent Done Done false
5199 Acked Read 1 Sent Done Done false
5200 Acked Read 10 Sent Done Done false
5201 Acked Write 9 Sent Done Done false
5202 Acked Read 8 Sent Done Done false
5203 Acked Read 8 Sent Done Done false
5204 Acked Write 9 Sent Done Done false
5205 Acked Write 9 Sent Done Done false
5206 Acked Read 7 Sent Done Done false
5207 Acked Write 9 Sent Done Done false
5208 Acked Write 6 Sent Done Done false
5209 Acked Read 5 Sent Done Done false
5210 Acked Read 4 Sent Done Done false
5211 Acked Read 3 Sent Done Done false
5212 Acked Read 9 Sent Done Done false
5213 Acked Flush 0 Sent Done Done false
5214 Acked Read 9 Sent Done Done false
5215 Acked Read 7 Sent Done Done false
5216 Acked Write 4 Sent Done Done false
5217 Acked Flush 0 Sent Done Done false
5218 Acked Read 6 Sent Done Done false
5219 Acked Read 1 Sent Done Done false
5220 Acked Read 3 Sent Done Done false
5221 Acked Write 10 Sent Done Done false
5222 Acked Write 8 Sent Done Done false
5223 Acked Read 7 Sent Done Done false
5224 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4922 5224 5224
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5224 5223 5222 5221 5220
CLIENT: Up:0 ds:302 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4923 Acked Write 3 Sent Done Done false
4924 Acked Write 8 Sent Done Done false
4925 Acked Write 2 Sent Done Done false
4926 Acked Read 8 Sent Done Done false
4927 Acked Read 6 Sent Done Done false
4928 Acked Read 8 Sent Done Done false
4929 Acked Write 10 Sent Done Done false
4930 Acked Read 5 Sent Done Done false
4931 Acked Read 7 Sent Done Done false
4932 Acked Write 2 Sent Done Done false
4933 Acked Write 6 Sent Done Done false
4934 Acked Write 5 Sent Done Done false
4935 Acked Read 4 Sent Done Done false
4936 Acked Read 2 Sent Done Done false
4937 Acked Read 8 Sent Done Done false
4938 Acked Write 1 Sent Done Done false
4939 Acked Read 2 Sent Done Done false
4940 Acked Write 6 Sent Done Done false
4941 Acked Write 9 Sent Done Done false
4942 Acked Read 1 Sent Done Done false
4943 Acked Write 6 Sent Done Done false
4944 Acked Read 1 Sent Done Done false
4945 Acked Read 4 Sent Done Done false
4946 Acked Write 10 Sent Done Done false
4947 Acked Write 3 Sent Done Done false
4948 Acked Write 6 Sent Done Done false
4949 Acked Read 6 Sent Done Done false
4950 Acked Read 6 Sent Done Done false
4951 Acked Read 7 Sent Done Done false
4952 Acked Read 7 Sent Done Done false
4953 Acked Write 5 Sent Done Done false
4954 Acked Write 10 Sent Done Done false
4955 Acked Read 9 Sent Done Done false
4956 Acked Write 9 Sent Done Done false
4957 Acked Write 8 Sent Done Done false
4958 Acked Read 4 Sent Done Done false
4959 Acked Read 3 Sent Done Done false
4960 Acked Write 10 Sent Done Done false
4961 Acked Write 9 Sent Done Done false
4962 Acked Read 9 Sent Done Done false
4963 Acked Read 3 Sent Done Done false
4964 Acked Write 8 Sent Done Done false
4965 Acked Read 6 Sent Done Done false
4966 Acked Write 1 Sent Done Done false
4967 Acked Read 7 Sent Done Done false
4968 Acked Read 1 Sent Done Done false
4969 Acked Read 1 Sent Done Done false
4970 Acked Write 1 Sent Done Done false
4971 Acked Read 10 Sent Done Done false
4972 Acked Read 4 Sent Done Done false
4973 Acked Write 9 Sent Done Done false
4974 Acked Read 6 Sent Done Done false
4975 Acked Write 10 Sent Done Done false
4976 Acked Read 3 Sent Done Done false
4977 Acked Read 2 Sent Done Done false
4978 Acked Flush 0 Sent Done Done false
4979 Acked Write 5 Sent Done Done false
4980 Acked Read 10 Sent Done Done false
4981 Acked Read 4 Sent Done Done false
4982 Acked Write 9 Sent Done Done false
4983 Acked Read 9 Sent Done Done false
4984 Acked Read 3 Sent Done Done false
4985 Acked Write 8 Sent Done Done false
4986 Acked Flush 0 Sent Done Done false
4987 Acked Write 5 Sent Done Done false
4988 Acked Write 1 Sent Done Done false
4989 Acked Write 2 Sent Done Done false
4990 Acked Write 8 Sent Done Done false
4991 Acked Write 3 Sent Done Done false
4992 Acked Read 5 Sent Done Done false
4993 Acked Read 2 Sent Done Done false
4994 Acked Read 3 Sent Done Done false
4995 Acked Read 2 Sent Done Done false
4996 Acked Read 5 Sent Done Done false
4997 Acked Read 1 Sent Done Done false
4998 Acked Read 2 Sent Done Done false
4999 Acked Read 6 Sent Done Done false
5000 Acked Write 7 Sent Done Done false
5001 Acked Read 9 Sent Done Done false
5002 Acked Read 1 Sent Done Done false
5003 Acked Write 10 Sent Done Done false
5004 Acked Flush 0 Sent Done Done false
5005 Acked Read 7 Sent Done Done false
5006 Acked Read 8 Sent Done Done false
5007 Acked Read 5 Sent Done Done false
5008 Acked Read 4 Sent Done Done false
5009 Acked Write 5 Sent Done Done false
5010 Acked Write 7 Sent Done Done false
5011 Acked Read 2 Sent Done Done false
5012 Acked Read 9 Sent Done Done false
5013 Acked Read 7 Sent Done Done false
5014 Acked Flush 0 Sent Done Done false
5015 Acked Read 6 Sent Done Done false
5016 Acked Read 9 Sent Done Done false
5017 Acked Write 8 Sent Done Done false
5018 Acked Read 2 Sent Done Done false
5019 Acked Write 7 Sent Done Done false
5020 Acked Flush 0 Sent Done Done false
5021 Acked Read 6 Sent Done Done false
5022 Acked Read 5 Sent Done Done false
5023 Acked Write 3 Sent Done Done false
5024 Acked Write 10 Sent Done Done false
5025 Acked Write 4 Sent Done Done false
5026 Acked Write 8 Sent Done Done false
5027 Acked Read 5 Sent Done Done false
5028 Acked Read 6 Sent Done Done false
5029 Acked Read 4 Sent Done Done false
5030 Acked Write 9 Sent Done Done false
5031 Acked Write 9 Sent Done Done false
5032 Acked Write 1 Sent Done Done false
5033 Acked Read 4 Sent Done Done false
5034 Acked Read 9 Sent Done Done false
5035 Acked Read 9 Sent Done Done false
5036 Acked Read 3 Sent Done Done false
5037 Acked Write 7 Sent Done Done false
5038 Acked Read 4 Sent Done Done false
5039 Acked Write 10 Sent Done Done false
5040 Acked Write 1 Sent Done Done false
5041 Acked Write 8 Sent Done Done false
5042 Acked Read 10 Sent Done Done false
5043 Acked Read 2 Sent Done Done false
5044 Acked Read 7 Sent Done Done false
5045 Acked Write 10 Sent Done Done false
5046 Acked Write 9 Sent Done Done false
5047 Acked Flush 0 Sent Done Done false
5048 Acked Read 3 Sent Done Done false
5049 Acked Read 8 Sent Done Done false
5050 Acked Read 1 Sent Done Done false
5051 Acked Write 5 Sent Done Done false
5052 Acked Write 9 Sent Done Done false
5053 Acked Read 1 Sent Done Done false
5054 Acked Read 4 Sent Done Done false
5055 Acked Write 8 Sent Done Done false
5056 Acked Write 6 Sent Done Done false
5057 Acked Write 3 Sent Done Done false
5058 Acked Read 1 Sent Done Done false
5059 Acked Read 1 Sent Done Done false
5060 Acked Write 8 Sent Done Done false
5061 Acked Write 2 Sent Done Done false
5062 Acked Write 2 Sent Done Done false
5063 Acked Write 2 Sent Done Done false
5064 Acked Read 5 Sent Done Done false
5065 Acked Write 1 Sent Done Done false
5066 Acked Write 7 Sent Done Done false
5067 Acked Read 1 Sent Done Done false
5068 Acked Write 4 Sent Done Done false
5069 Acked Read 10 Sent Done Done false
5070 Acked Read 9 Sent Done Done false
5071 Acked Read 2 Sent Done Done false
5072 Acked Flush 0 Sent Done Done false
5073 Acked Write 7 Sent Done Done false
5074 Acked Write 3 Sent Done Done false
5075 Acked Write 5 Sent Done Done false
5076 Acked Read 9 Sent Done Done false
5077 Acked Read 5 Sent Done Done false
5078 Acked Write 9 Sent Done Done false
5079 Acked Flush 0 Sent Done Done false
5080 Acked Read 3 Sent Done Done false
5081 Acked Write 2 Sent Done Done false
5082 Acked Read 9 Sent Done Done false
5083 Acked Read 7 Sent Done Done false
5084 Acked Read 7 Sent Done Done false
5085 Acked Write 3 Sent Done Done false
5086 Acked Read 1 Sent Done Done false
5087 Acked Read 5 Sent Done Done false
5088 Acked Flush 0 Sent Done Done false
5089 Acked Read 1 Sent Done Done false
5090 Acked Read 5 Sent Done Done false
5091 Acked Read 3 Sent Done Done false
5092 Acked Read 7 Sent Done Done false
5093 Acked Read 8 Sent Done Done false
5094 Acked Read 4 Sent Done Done false
5095 Acked Read 8 Sent Done Done false
5096 Acked Write 8 Sent Done Done false
5097 Acked Write 9 Sent Done Done false
5098 Acked Write 1 Sent Done Done false
5099 Acked Read 2 Sent Done Done false
5100 Acked Read 1 Sent Done Done false
5101 Acked Read 8 Sent Done Done false
5102 Acked Flush 0 Sent Done Done false
5103 Acked Read 3 Sent Done Done false
5104 Acked Write 2 Sent Done Done false
5105 Acked Write 3 Sent Done Done false
5106 Acked Read 3 Sent Done Done false
5107 Acked Read 8 Sent Done Done false
5108 Acked Write 7 Sent Done Done false
5109 Acked Read 2 Sent Done Done false
5110 Acked Write 7 Sent Done Done false
5111 Acked Read 9 Sent Done Done false
5112 Acked Flush 0 Sent Done Done false
5113 Acked Write 9 Sent Done Done false
5114 Acked Write 4 Sent Done Done false
5115 Acked Read 3 Sent Done Done false
5116 Acked Flush 0 Sent Done Done false
5117 Acked Read 7 Sent Done Done false
5118 Acked Write 9 Sent Done Done false
5119 Acked Read 7 Sent Done Done false
5120 Acked Read 5 Sent Done Done false
5121 Acked Read 1 Sent Done Done false
5122 Acked Flush 0 Sent Done Done false
5123 Acked Read 5 Sent Done Done false
5124 Acked Read 1 Sent Done Done false
5125 Acked Read 1 Sent Done Done false
5126 Acked Write 7 Sent Done Done false
5127 Acked Read 1 Sent Done Done false
5128 Acked Read 4 Sent Done Done false
5129 Acked Read 9 Sent Done Done false
5130 Acked Write 4 Sent Done Done false
5131 Acked Read 5 Sent Done Done false
5132 Acked Read 9 Sent Done Done false
5133 Acked Read 10 Sent Done Done false
5134 Acked Flush 0 Sent Done Done false
5135 Acked Read 3 Sent Done Done false
5136 Acked Flush 0 Sent Done Done false
5137 Acked Read 8 Sent Done Done false
5138 Acked Read 7 Sent Done Done false
5139 Acked Read 6 Sent Done Done false
5140 Acked Read 3 Sent Done Done false
5141 Acked Read 7 Sent Done Done false
5142 Acked Read 3 Sent Done Done false
5143 Acked Write 6 Sent Done Done false
5144 Acked Read 7 Sent Done Done false
5145 Acked Write 5 Sent Done Done false
5146 Acked Read 10 Sent Done Done false
5147 Acked Write 7 Sent Done Done false
5148 Acked Read 8 Sent Done Done false
5149 Acked Write 5 Sent Done Done false
5150 Acked Write 7 Sent Done Done false
5151 Acked Write 3 Sent Done Done false
5152 Acked Write 8 Sent Done Done false
5153 Acked Write 2 Sent Done Done false
5154 Acked Read 9 Sent Done Done false
5155 Acked Read 7 Sent Done Done false
5156 Acked Write 8 Sent Done Done false
5157 Acked Read 5 Sent Done Done false
5158 Acked Read 10 Sent Done Done false
5159 Acked Read 5 Sent Done Done false
5160 Acked Read 5 Sent Done Done false
5161 Acked Read 8 Sent Done Done false
5162 Acked Write 8 Sent Done Done false
5163 Acked Read 6 Sent Done Done false
5164 Acked Write 3 Sent Done Done false
5165 Acked Write 2 Sent Done Done false
5166 Acked Read 5 Sent Done Done false
5167 Acked Write 5 Sent Done Done false
5168 Acked Write 2 Sent Done Done false
5169 Acked Write 3 Sent Done Done false
5170 Acked Read 4 Sent Done Done false
5171 Acked Read 3 Sent Done Done false
5172 Acked Read 1 Sent Done Done false
5173 Acked Read 8 Sent Done Done false
5174 Acked Write 6 Sent Done Done false
5175 Acked Write 4 Sent Done Done false
5176 Acked Write 1 Sent Done Done false
5177 Acked Write 3 Sent Done Done false
5178 Acked Write 5 Sent Done Done false
5179 Acked Read 7 Sent Done Done false
5180 Acked Read 10 Sent Done Done false
5181 Acked Read 3 Sent Done Done false
5182 Acked Read 2 Sent Done Done false
5183 Acked Read 1 Sent Done Done false
5184 Acked Read 7 Sent Done Done false
5185 Acked Write 1 Sent Done Done false
5186 Acked Write 9 Sent Done Done false
5187 Acked Read 2 Sent Done Done false
5188 Acked Write 5 Sent Done Done false
5189 Acked Write 3 Sent Done Done false
5190 Acked Read 10 Sent Done Done false
5191 Acked Read 7 Sent Done Done false
5192 Acked Write 6 Sent Done Done false
5193 Acked Read 1 Sent Done Done false
5194 Acked Write 7 Sent Done Done false
5195 Acked Read 7 Sent Done Done false
5196 Acked Write 6 Sent Done Done false
5197 Acked Read 1 Sent Done Done false
5198 Acked Write 2 Sent Done Done false
5199 Acked Read 1 Sent Done Done false
5200 Acked Read 10 Sent Done Done false
5201 Acked Write 9 Sent Done Done false
5202 Acked Read 8 Sent Done Done false
5203 Acked Read 8 Sent Done Done false
5204 Acked Write 9 Sent Done Done false
5205 Acked Write 9 Sent Done Done false
5206 Acked Read 7 Sent Done Done false
5207 Acked Write 9 Sent Done Done false
5208 Acked Write 6 Sent Done Done false
5209 Acked Read 5 Sent Done Done false
5210 Acked Read 4 Sent Done Done false
5211 Acked Read 3 Sent Done Done false
5212 Acked Read 9 Sent Done Done false
5213 Acked Flush 0 Sent Done Done false
5214 Acked Read 9 Sent Done Done false
5215 Acked Read 7 Sent Done Done false
5216 Acked Write 4 Sent Done Done false
5217 Acked Flush 0 Sent Done Done false
5218 Acked Read 6 Sent Done Done false
5219 Acked Read 1 Sent Done Done false
5220 Acked Read 3 Sent Done Done false
5221 Acked Write 10 Sent Done Done false
5222 Acked Write 8 Sent Done Done false
5223 Acked Read 7 Sent Done Done false
5224 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4922 5224 5224
Downstairs last five completed: 4922 4921 4920 4919 4918
Upstairs last five completed: 5224 5223 5222 5221 5220
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.132573183Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:62129)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.132749673Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.134068209Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.134620084Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.134641872Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 4922","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.13465562Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 4922","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.13491743Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 4922","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.134937298Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 4922","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.135207316Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5224 5223 5222 5221 5220
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:39:40.382 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8097fdf7-71e9-4f55-809f-4db79a52658a, remote_addr: 127.0[8830] Got stop action so:false kr:false
.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:39:40.383 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/2, method: GET, req_id: fdf06a36-0477-4bfa-b783-b55311cb23b1, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs client error Connection reset by peer (os error 131)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:40.384498657Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"client task is sending Done(ReadFailed(Connection reset by peer (os error 131)\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/backtrace.rs:27:14\n 1: <core::task::poll::Poll<core::option::Option<core::result::Result<T,F>>> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/task/poll.rs:290:44))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:40.552639138Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to ReadFailed(Connection reset by peer (os error 131)\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.92/src/backtrace.rs:27:14\n 1: <core::task::poll::Poll<core::option::Option<core::result::Result<T,F>>> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/task/poll.rs:290:44)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:40.552716598Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:40.552805118Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:39:44.385 INFO request completed, latency_us: 264, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8a9b823c-e628-4133-8eed-077729e8be94, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 2347 len 28672 data: 5 5 4 5 5 5 7
002/300 Write block 3561 len 24576 data: 6 7 6 5 5 3
003/300 Read block 2922 len 20480
004/300 Read block 4560 len 12288
005/300 Read block 3744 len 28672
006/300 Write block 2736 len 20480 data: 4 4 3 2 2
007/300 Write block 4442 len 8192 data: 5 3
008/300 Read block 622 len 28672
009/300 Write block 2686 len 8192 data: 7 6
010/300 Write block 470 len 16384 data: 3 3 2 2
011/300 Read block 1762 len 12288
012/300 Flush
013/300 Flush
014/300 Read block 1664 len 24576
015/300 Flush
016/300 Read block 2169 len 16384
017/300 Write block 546 len 36864 data: 6 5 5 5 4 3 6 6 6
018/300 Read block 1824 len 8192
019/300 Read block 4243 len 12288
020/300 Write block 1552 len 16384 data: 3 4 5 5
021/300 Read block 2446 len 4096
022/300 Read block 1435 len 36864
023/300 Write block 1117 len 32768 data: 4 4 3 4 4 6 7 6
024/300 Read block 2343 len 36864
025/300 Read block 3487 len 40960
026/300 Write block 3674 len 24576 data: 3 2 2 2 3 2
027/300 Write block 2256 len 28672 data: 6 5 5 5 5 4 4
028/300 Read block 4183 len 4096
029/300 Read block 2234 len 8192
030/300 Write block 3568 len 28672 data: 5 5 5 5 4 4 4
031/300 Read block 4519 len 4096
032/300 Write block 795 len 40960 data: 5 5 5 4 3 4 3 4 3 2
033/300 Read block 4423 len 40960
034/300 Write block 4279 len 12288 data: 5 5 4
035/300 Read block 3739 len 32768
036/300 Read block 4190 len 20480
037/300 Flush
038/300 Read block 677 len 12288
039/300 Read block 4463 len 32768
040/300 Read block 3189 len 36864
041/300 Write block 16 len 24576 data: 5 5 3 3 3 3
042/300 Read block 1322 len 36864
043/300 Flush
044/300 Read block 3679 len 20480
045/300 Read block 576 len 20480
046/300 Flush
047/300 Read block 4498 len 4096
048/300 Write block 4115 len 24576 data: 3 4 4 6 7 6
049/300 Read block 672 len 28672
050/300 Read block 1819 len 20480
051/300 Write block 619 len 24576 data: 2 2 2 2 3 3
052/300 Read block 96 len 12288
053/300 Write block 4200 len 24576 data: 3 3 3 2 2 2
054/300 Write block 2425 len 28672 data: 4 3 3 4 4 3 2
055/300 Read block 3819 len 24576
056/300 Read block 1299 len 8192
057/300 Read block 3735 len 36864
058/300 Read block 4995 len 8192
059/300 Read block 3168 len 12288
060/300 Flush
061/300 Write block 4773 len 28672 data: 4 5 5 5 4 3 2
062/300 Read block 4139 len 32768
063/300 Write block 4749 len 32768 data: 3 4 3 3 3 3 4 3
064/300 Read block 2839 len 40960
065/300 Write block 4680 len 32768 data: 4 5 4 4 3 3 3 3
066/300 Write block 1301 len 20480 data: 6 5 6 6 5
067/300 Write block 3088 len 4096 data: 2
068/300 Write block 178 len 40960 data: 4 5 4 4 4 5 5 5 5 4
069/300 Write block 3253 len 8192 data: 3 3
070/300 Write block 3725 len 8192 data: 6 6
071/300 Write block 4114 len 36864 data: 3 4 5 5 7 8 7 6 6
072/300 Read block 1243 len 40960
073/300 Read block 1910 len 8192
074/300 Write block 1934 len 8192 data: 2 2
075/300 Read block 4622 len 8192
076/300 Write block 1624 len 40960 data: 4 4 5 8 6 5 4 4 4 4
077/300 Write block 129 len 28672 data: 4 5 6 5 4 3 3
078/300 Read block 4157 len 4096
079/300 Read block 3094 len 12288
080/300 Read block 2387 len 12288
081/300 Read block 243 len 40960
082/300 Read block 2743 len 16384
083/300 Read block 3348 len 12288
084/300 Write block 1114 len 20480 data: 3 3 3 5 5
085/300 Read block 2376 len 40960
086/300 Read block 500 len 24576
087/300 Read block 2127 len 4096
088/300 Write block 3103 len 28672 data: 4 5 5 5 5 5 4
089/300 Read block 423 len 16384
090/300 Write block 583 len 12288 data: 6 3 3
091/300 Read block 776 len 16384
092/300 Read block 2316 len 8192
093/300 Write block 2488 len 8192 data: 3 3
094/300 Write block 4105 len 16384 data: 5 5 6 4
095/300 Write block 758 len 40960 data: 2 2 2 2 2 2 2 3 3 4
096/300 Read block 1862 len 4096
097/300 Write block 3396 len 24576 data: 3 6 6 4 4 4
098/300 Write block 440 len 4096 data: 5
099/300 Write block 2511 len 32768 data: 3 4 4 4 4 4 3 2
100/300 Flush
101/300 Write block 1382 len 24576 data: 5 6 6 4 4 4
102/300 Write block 3426 len 20480 data: 8 7 7 7 7
103/300 Flush
104/300 Write block 115 len 16384 data: 6 6 5 5
105/300 Read block 3159 len 4096
106/300 Read block 405 len 24576
107/300 Read block 2605 len 32768
108/300 Write block 2686 len 24576 data: 8 7 5 2 2 2
109/300 Read block 2876 len 12288
110/300 Read block 510 len 12288
111/300 Write block 1346 len 32768 data: 6 6 6 5 4 3 3 5
112/300 Flush
113/300 Write block 3162 len 12288 data: 3 3 2
114/300 Read block 3409 len 12288
115/300 Read block 45 len 8192
116/300 Write block 4981 len 12288 data: 2 3 3
117/300 Read block 4784 len 24576
118/300 Read block 2584 len 4096
119/300 Read block 3449 len 28672
120/300 Read block 1026 len 8192
121/300 Write block 1405 len 24576 data: 6 6 6 5 5 4
122/300 Read block 3300 len 24576
123/300 Read block 3958 len 8192
124/300 Write block 1251 len 8192 data: 4 4
125/300 Read block 172 len 40960
126/300 Write block 4481 len 20480 data: 5 5 4 3 3
127/300 Read block 3103 len 20480
128/300 Read block 4949 len 36864
129/300 Write block 1233 len 40960 data: 4 7 7 7 6 6 6 6 4 2
130/300 Write block 510 len 24576 data: 3 4 3 3 3 3
131/300 Read block 4575 len 16384
132/300 Read block 3064 len 28672
133/300 Read block 1836 len 28672
134/300 Write block 2450 len 4096 data: 4
135/300 Write block 3002 len 8192 data: 5 4
136/300 Flush
137/300 Write block 302 len 32768 data: 5 4 3 3 3 3 3 3
138/300 Read block 4268 len 24576
139/300 Flush
140/300 Read block 683 len 36864
141/300 Write block 4288 len 32768 data: 3 2 2 2 2 2 3 3
142/300 Read block 1966 len 20480
143/300 Read block 4852 len 20480
144/300 Write block 2283 len 32768 data: 3 3 5 6 7 9 8 8
145/300 Write block 4923 len 36864 data: 7 6 6 5 6 6 5 4 4
146/300 Write block 4094 len 28672 data: 3 3 3 2 2 2 4
147/300 Read block 877 len 24576
148/300 Read block 14 len 32768
149/300 Write block 3432 len 28672 data: 5 5 5 4 4 4 4
150/300 Write block 1605 len 36864 data: 3 4 3 3 3 3 3 2 2
151/300 Write block 4949 len 32768 data: 4 4 4 3 2 3 3 2
152/300 Read block 2511 len 36864
153/300 Read block 1551 len 28672
154/300 Write block 3108 len 16384 data: 6 5 3 2
155/300 Write block 4551 len 40960 data: 3 3 3 3 3 4 3 3 2 3
156/300 Write block 1422 len 24576 data: 4 3 3 4 4 4
157/300 Read block 489 len 20480
158/300 Write block 3098 len 28672 data: 3 3 3 3 3 5 6
159/300 Read block 1895 len 24576
160/300 Read block 14 len 12288
161/300 Write block 4689 len 28672 data: 2 3 3 3 4 4 4
162/300 Read block 104 len 12288
163/300 Write block 1275 len 12288 data: 2 3 3
164/300 Read block 2363 len 32768
165/300 Read block 1796 len 16384
166/300 Write block 2639 len 4096 data: 6
167/300 Write block 1479 len 20480 data: 5 5 5 5 3
168/300 Read block 899 len 4096
169/300 Read block 3799 len 28672
170/300 Read block 1864 len 16384
171/300 Write block 4782 len 8192 data: 2 2
172/300 Read block 2309 len 12288
173/300 Read block 1699 len 40960
174/300 Read block 4137 len 12288
175/300 Write block 2042 len 20480 data: 5 5 4 4 4
176/300 Flush
177/300 Read block 2817 len 8192
178/300 Read block 4347 len 4096
179/300 Flush
180/300 Read block 3576 len 16384
181/300 Read block 185 len 16384
182/300 Read block 2733 len 8192
183/300 Read block 1841 len 32768
184/300 Read block 1118 len 16384
185/300 Read block 2150 len 36864
186/300 Read block 3306 len 28672
187/300 Write block 2164 len 4096 data: 6
188/300 Write block 2574 len 4096 data: 3
189/300 Read block 2875 len 36864
190/300 Read block 2122 len 16384
191/300 Write block 2449 len 24576 data: 4 5 4 3 3 3
192/300 Flush
193/300 Write block 709 len 36864 data: 5 5 6 5 10 9 8 7 5
194/300 Read block 2815 len 24576
195/300 Read block 465 len 28672
196/300 Write block 1525 len 16384 data: 4 3 4 2
197/300 Write block 1558 len 16384 data: 7 7 5 3
198/300 Read block 4765 len 4096
199/300 Read block 4696 len 28672
200/300 Write block 819 len 40960 data: 5 5 4 3 3 3 3 2 2 3
201/300 Write block 621 len 12288 data: 3 3 4
202/300 Write block 502 len 4096 data: 6
203/300 Read block 2243 len 16384
204/300 Write block 3618 len 16384 data: 5 5 5 6
205/300 Write block 2331 len 12288 data: 3 3 3
206/300 Read block 1478 len 8192
207/300 Write block 2176 len 36864 data: 7 6 6 5 4 4 6 5 6
208/300 Read block 3218 len 12288
209/300 Flush
210/300 Write block 4745 len 8192 data: 3 4
211/300 Write block 3958 len 32768 data: 4 5 4 4 5 6 5 7
212/300 Write block 600 len 20480 data: 4 4 5 6 6
213/300 Read block 154 len 8192
214/300 Write block 4263 len 40960 data: 4 4 4 4 3 3 3 3 5 4
215/300 Write block 3032 len 32768 data: 3 3 3 2 3 3 4 4
216/300 Write block 1000 len 8192 data: 3 3
217/300 Read block 3655 len 4096
218/300 Read block 2046 len 16384
219/300 Read block 2288 len 32768
220/300 Write block 2800 len 20480 data: 4 5 5 3 3
221/300 Write block 858 len 32768 data: 3 3 5 6 6 5 5 4
222/300 Read block 649 len 36864
223/300 Write block 4933 len 24576 data: 3 2 3 3 3 4
224/300 Read block 4051 len 24576
225/300 Write block 828 len 32768 data: 4 4 6 6 7 6 6 6
226/300 Write block 321 len 20480 data: 6 4 4 4 4
227/300 Write block 3314 len 4096 data: 4
228/300 Write block 1394 len 4096 data: 5
229/300 Write block 3880 len 12288 data: 6 5 4
230/300 Read block 4944 len 12288
231/300 Write block 2782 len 20480 data: 3 3 3 3 2
232/300 Read block 276 len 36864
233/300 Write block 515 len 36864 data: 4 3 3 3 4 4 3 3 3
234/300 Read block 3551 len 24576
235/300 Read block 730 len 24576
236/300 Read block 874 len 20480
237/300 Read block 2390 len 12288
238/300 Write block 1612 len 12288 data: 3 3 3
239/300 Read block 112 len 24576
240/300 Read block 1386 len 32768
241/300 Read block 1832 len 36864
242/300 Read block 2464 len 40960
243/300 Flush
244/300 Write block 492 len 40960 data: 3 3 4 5 5 7 8 8 6 7
245/300 Read block 4931 len 24576
246/300 Write block 1866 len 8192 data: 4 3
247/300 Write block 4692 len 28672 data: 4 5 5 5 4 4 4
248/300 Read block 4577 len 4096
249/300 Write block 3301 len 36864 data: 3 3 3 3 3 3 2 2 3
250/300 Read block 3705 len 32768
251/300 Read block 2557 len 16384
252/300 Flush
253/300 Write block 2376 len 12288 data: 6 6 8
254/300 Read block 2420 len 8192
255/300 Write block 1397 len 12288 data: 4 5 6
256/300 Read block 3564 len 12288
257/300 Write block 2834 len 36864 data: 4 2 3 4 4 4 4 5 5
258/300 Write block 396 len 20480 data: 4 6 5 5 4
259/300 Write block 87 len 36864 data: 4 5 4 4 3 3 2 3 3
260/300 Write block 1658 len 28672 data: 3 3 3 3 4 4 3
261/300 Read block 3358 len 40960
262/300 Read block 927 len 28672
263/300 Write block 4158 len 28672 data: 3 3 4 6 6 6 6
264/300 Read block 1839 len 40960
265/300 Read block 3532 len 32768
266/300 Read block 3776 len 24576
267/300 Write block 2411 len 40960 data: 3 4 4 5 4 4 3 3 3 3
268/300 Read block 1463 len 12288
269/300 Write block 2962 len 20480 data: 4 4 3 3 3
270/300 Read block 1164 len 8192
271/300 Write block 3597 len 32768 data: 2 2 3 3 2 2 3 4
272/300 Read block 1517 len 40960
273/300 Read block 4053 len 24576
274/300 Flush
275/300 Read block 3506 len 28672
276/300 Read block 744 len 24576
277/300 Write block 3840 len 28672 data: 5 6 7 6 5 5 5
278/300 Write block 1771 len 24576 data: 3 4 4 3 3 3
279/300 Read block 3507 len 20480
280/300 Read block 4878 len 24576
281/300 Write block 3607 len 16384 data: 7 7 5 4
282/300 Read block 2805 len 40960
283/300 Read block 376 len 4096
284/300 Read block 3454 len 16384
285/300 Read block 570 len 24576
286/300 Flush
287/300 Write block 4989 len 20480 data: 5 4 3 3 4
288/300 Write block 3211 len 24576 data: 3 3 3 2 5 5
289/300 Write block 293 len 8192 data: 4 5
290/300 Write block 2713 len 12288 data: 2 2 3
291/300 Write block 1089 len 12288 data: 2 3 3
292/300 Write block 1079 len 24576 data: 5 5 3 3 2 2
293/300 Write block 2316 len 12288 data: 2 2 2
294/300 Write block 4407 len 20480 data: 4 4 3 3 4
295/300 Read block 1957 len 8192
296/300 Write block 1121 len 40960 data: 5 7 8 7 5 6 7 7 8 6
297/300 Write block 2124 len 36864 data: 3 4 3 3 3 3 3 3 3
298/300 Flush
299/300 Write block 1022 len 8192 data: 3 2
300/300 Read block 4205 len 20480
301/300 Read block 2841 len 28672
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:39:44.617 INFO request completed, latency_us: 172, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5bbab708-7902-448b-a1a4-29aca33376ef, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5225 Acked Write 7 Done Done Sent false
5226 Acked Write 6 Done Done Sent false
5227 Acked Read 5 Done Done Sent false
5228 Acked Read 3 Done Done Sent false
5229 Acked Read 7 Done Done Sent false
5230 Acked Write 5 Done Done Sent false
5231 Acked Write 2 Done Done Sent false
5232 Acked Read 7 Done Done Sent false
5233 Acked Write 2 Done Done Sent false
5234 Acked Write 4 Done Done Sent false
5235 Acked Read 3 Done Done Sent false
5236 Acked Flush 0 Done Done Sent false
5237 Acked Flush 0 Done Done Sent false
5238 Acked Read 6 Done Done Sent false
5239 Acked Flush 0 Done Done Sent false
5240 Acked Read 4 Done Done Sent false
5241 Acked Write 9 Done Done Sent false
5242 Acked Read 2 Done Done Sent false
5243 Acked Read 3 Done Done Sent false
5244 Acked Write 4 Done Done Sent false
5245 Acked Read 1 Done Done Sent false
5246 Acked Read 9 Done Done Sent false
5247 Acked Write 8 Done Done Sent false
5248 Acked Read 9 Done Done Sent false
5249 Acked Read 10 Done Done Sent false
5250 Acked Write 6 Done Done Sent false
5251 Acked Write 7 Done Done Sent false
5252 Acked Read 1 Done Done Sent false
5253 Acked Read 2 Done Done Sent false
5254 Acked Write 7 Done Done Sent false
5255 Acked Read 1 Done Done Sent false
5256 Acked Write 10 Done Done Sent false
5257 Acked Read 10 Done Done Sent false
5258 Acked Write 3 Done Done Sent false
5259 Acked Read 8 Done Done Sent false
5260 Acked Read 5 Done Done Sent false
5261 Acked Flush 0 Done Done Sent false
5262 Acked Read 3 Done Done Sent false
5263 Acked Read 8 Done Done Sent false
5264 Acked Read 9 Done Done Sent false
5265 Acked Write 6 Done Done Sent false
5266 Acked Read 9 Done Done Sent false
5267 Acked Flush 0 Done Done Sent false
5268 Acked Read 5 Done Done Sent false
5269 Acked Read 5 Done Done Sent false
5270 Acked Flush 0 Done Done Sent false
5271 Acked Read 1 Done Done Sent false
5272 Acked Write 6 Done Done Sent false
5273 Acked Read 7 Done Done Sent false
5274 Acked Read 5 Done Done Sent false
5275 Acked Write 6 Done Done Sent false
5276 Acked Read 3 Done Done Sent false
5277 Acked Write 6 Done Done Sent false
5278 Acked Write 7 Done Done Sent false
5279 Acked Read 6 Done Done Sent false
5280 Acked Read 2 Done Done Sent false
5281 Acked Read 9 Done Done Sent false
5282 Acked Read 2 Done Done Sent false
5283 Acked Read 3 Done Done Sent false
5284 Acked Flush 0 Done Done Sent false
5285 Acked Write 7 Done Done Sent false
5286 Acked Read 8 Done Done Sent false
5287 Acked Write 8 Done Done Sent false
5288 Acked Read 10 Done Done Sent false
5289 Acked Write 8 Done Done Sent false
5290 Acked Write 5 Done Done Sent false
5291 Acked Write 1 Done Done Sent false
5292 Acked Write 10 Done Done Sent false
5293 Acked Write 2 Done Done Sent false
5294 Acked Write 2 Done Done Sent false
5295 Acked Write 9 Done Done Sent false
5296 Acked Read 10 Done Done Sent false
5297 Acked Read 2 Done Done Sent false
5298 Acked Write 2 Done Done Sent false
5299 Acked Read 2 Done Done Sent false
5300 Acked Write 10 Done Done Sent false
5301 Acked Write 7 Done Done Sent false
5302 Acked Read 1 Done Done Sent false
5303 Acked Read 3 Done Done Sent false
5304 Acked Read 3 Done Done Sent false
5305 Acked Read 10 Done Done Sent false
5306 Acked Read 4 Done Done Sent false
5307 Acked Read 3 Done Done Sent false
5308 Acked Write 5 Done Done Sent false
5309 Acked Read 10 Done Done Sent false
5310 Acked Read 6 Done Done Sent false
5311 Acked Read 1 Done Done Sent false
5312 Acked Write 7 Done Done Sent false
5313 Acked Read 4 Done Done Sent false
5314 Acked Write 3 Done Done Sent false
5315 Acked Read 4 Done Done Sent false
5316 Acked Read 2 Done Done Sent false
5317 Acked Write 2 Done Done Sent false
5318 Acked Write 4 Done Done Sent false
5319 Acked Write 10 Done Done Sent false
5320 Acked Read 1 Done Done Sent false
5321 Acked Write 6 Done Done Sent false
5322 Acked Write 1 Done Done Sent false
5323 Acked Write 8 Done Done Sent false
5324 Acked Flush 0 Done Done Sent false
5325 Acked Write 6 Done Done Sent false
5326 Acked Write 5 Done Done Sent false
5327 Acked Flush 0 Done Done Sent false
5328 Acked Write 4 Done Done Sent false
5329 Acked Read 1 Done Done Sent false
5330 Acked Read 6 Done Done Sent false
5331 Acked Read 8 Done Done Sent false
5332 Acked Write 6 Done Done Sent false
5333 Acked Read 3 Done Done Sent false
5334 Acked Read 3 Done Done Sent false
5335 Acked Write 8 Done Done Sent false
5336 Acked Flush 0 Done Done Sent false
5337 Acked Write 3 Done Done Sent false
5338 Acked Read 3 Done Done Sent false
5339 Acked Read 2 Done Done Sent false
5340 Acked Write 3 Done Done Sent false
5341 Acked Read 6 Done Done Sent false
5342 Acked Read 1 Done Done Sent false
5343 Acked Read 7 Done Done Sent false
5344 Acked Read 2 Done Done Sent false
5345 Acked Write 6 Done Done Sent false
5346 Acked Read 6 Done Done Sent false
5347 Acked Read 2 Done Done Sent false
5348 Acked Write 2 Done Done Sent false
5349 Acked Read 10 Done Done Sent false
5350 Acked Write 5 Done Done Sent false
5351 Acked Read 5 Done Done Sent false
5352 Acked Read 9 Done Done Sent false
5353 Acked Write 10 Done Done Sent false
5354 Acked Write 6 Done Done Sent false
5355 Acked Read 4 Done Done Sent false
5356 Acked Read 7 Done Done Sent false
5357 Acked Read 7 Done Done Sent false
5358 Acked Write 1 Done Done Sent false
5359 Acked Write 2 Done Done Sent false
5360 Acked Flush 0 Done Done Sent false
5361 Acked Write 8 Done Done Sent false
5362 Acked Read 6 Done Done Sent false
5363 Acked Flush 0 Done Done Sent false
5364 Acked Read 9 Done Done Sent false
5365 Acked Write 8 Done Done Sent false
5366 Acked Read 5 Done Done Sent false
5367 Acked Read 5 Done Done Sent false
5368 Acked Write 8 Done Done Sent false
5369 Acked Write 9 Done Done Sent false
5370 Acked Write 7 Done Done Sent false
5371 Acked Read 6 Done Done Sent false
5372 Acked Read 8 Done Done Sent false
5373 Acked Write 7 Done Done Sent false
5374 Acked Write 9 Done Done Sent false
5375 Acked Write 8 Done Done Sent false
5376 Acked Read 9 Done Done Sent false
5377 Acked Read 7 Done Done Sent false
5378 Acked Write 4 Done Done Sent false
5379 Acked Write 10 Done Done Sent false
5380 Acked Write 6 Done Done Sent false
5381 Acked Read 5 Done Done Sent false
5382 Acked Write 7 Done Done Sent false
5383 Acked Read 6 Done Done Sent false
5384 Acked Read 3 Done Done Sent false
5385 Acked Write 7 Done Done Sent false
5386 Acked Read 3 Done Done Sent false
5387 Acked Write 3 Done Done Sent false
5388 Acked Read 8 Done Done Sent false
5389 Acked Read 4 Done Done Sent false
5390 Acked Write 1 Done Done Sent false
5391 Acked Write 5 Done Done Sent false
5392 Acked Read 1 Done Done Sent false
5393 Acked Read 7 Done Done Sent false
5394 Acked Read 4 Done Done Sent false
5395 Acked Write 2 Done Done Sent false
5396 Acked Read 3 Done Done Sent false
5397 Acked Read 10 Done Done Sent false
5398 Acked Read 3 Done Done Sent false
5399 Acked Write 5 Done Done Sent false
5400 Acked Flush 0 Done Done Sent false
5401 Acked Read 2 Done Done Sent false
5402 Acked Read 1 Done Done Sent false
5403 Acked Flush 0 Done Done Sent false
5404 Acked Read 4 Done Done Sent false
5405 Acked Read 4 Done Done Sent false
5406 Acked Read 2 Done Done Sent false
5407 Acked Read 8 Done Done Sent false
5408 Acked Read 4 Done Done Sent false
5409 Acked Read 9 Done Done Sent false
5410 Acked Read 7 Done Done Sent false
5411 Acked Write 1 Done Done Sent false
5412 Acked Write 1 Done Done Sent false
5413 Acked Read 9 Done Done Sent false
5414 Acked Read 4 Done Done Sent false
5415 Acked Write 6 Done Done Sent false
5416 Acked Flush 0 Done Done Sent false
5417 Acked Write 9 Done Done Sent false
5418 Acked Read 6 Done Done Sent false
5419 Acked Read 7 Done Done Sent false
5420 Acked Write 4 Done Done Sent false
5421 Acked Write 4 Done Done Sent false
5422 Acked Read 1 Done Done Sent false
5423 Acked Read 7 Done Done Sent false
5424 Acked Write 10 Done Done Sent false
5425 Acked Write 3 Done Done Sent false
5426 Acked Write 1 Done Done Sent false
5427 Acked Read 4 Done Done Sent false
5428 Acked Write 4 Done Done Sent false
5429 Acked Write 3 Done Done Sent false
5430 Acked Read 2 Done Done Sent false
5431 Acked Write 9 Done Done Sent false
5432 Acked Read 3 Done Done Sent false
5433 Acked Flush 0 Done Done Sent false
5434 Acked Write 2 Done Done Sent false
5435 Acked Write 8 Done Done Sent false
5436 Acked Write 5 Done Done Sent false
5437 Acked Read 2 Done Done Sent false
5438 Acked Write 10 Done Done Sent false
5439 Acked Write 8 Done Done Sent false
5440 Acked Write 2 Done Done Sent false
5441 Acked Read 1 Done Done Sent false
5442 Acked Read 4 Done Done Sent false
5443 Acked Read 8 Done Done Sent false
5444 Acked Write 5 Done Done Sent false
5445 Acked Write 8 Done Done Sent false
5446 Acked Read 9 Done Done Sent false
5447 Acked Write 6 Done Done Sent false
5448 Acked Read 6 Done Done Sent false
5449 Acked Write 8 Done Done Sent false
5450 Acked Write 5 Done Done Sent false
5451 Acked Write 1 Done Done Sent false
5452 Acked Write 1 Done Done Sent false
5453 Acked Write 3 Done Done Sent false
5454 Acked Read 3 Done Done Sent false
5455 Acked Write 5 Done Done Sent false
5456 Acked Read 9 Done Done Sent false
5457 Acked Write 9 Done Done Sent false
5458 Acked Read 6 Done Done Sent false
5459 Acked Read 6 Done Done Sent false
5460 Acked Read 5 Done Done Sent false
5461 Acked Read 3 Done Done Sent false
5462 Acked Write 3 Done Done Sent false
5463 Acked Read 6 Done Done Sent false
5464 Acked Read 8 Done Done Sent false
5465 Acked Read 9 Done Done Sent false
5466 Acked Read 10 Done Done Sent false
5467 Acked Flush 0 Done Done Sent false
5468 Acked Write 10 Done Done Sent false
5469 Acked Read 6 Done Done Sent false
5470 Acked Write 2 Done Done Sent false
5471 Acked Write 7 Done Done Sent false
5472 Acked Read 1 Done Done Sent false
5473 Acked Write 9 Done Done Sent false
5474 Acked Read 8 Done Done Sent false
5475 Acked Read 4 Done Done Sent false
5476 Acked Flush 0 Done Done Sent false
5477 Acked Write 3 Done Done Sent false
5478 Acked Read 2 Done Done Sent false
5479 Acked Write 3 Done Done Sent false
5480 Acked Read 3 Done Done Sent false
5481 Acked Write 9 Done Done Sent false
5482 Acked Write 5 Done Done Sent false
5483 Acked Write 9 Done Done Sent false
5484 Acked Write 7 Done Done Sent false
5485 Acked Read 10 Done Done Sent false
5486 Acked Read 7 Done Done Sent false
5487 Acked Write 7 Done Done Sent false
5488 Acked Read 10 Done Done Sent false
5489 Acked Read 8 Done Done Sent false
5490 Acked Read 6 Done Done Sent false
5491 Acked Write 10 Done Done Sent false
5492 Acked Read 3 Done Done Sent false
5493 Acked Write 5 Done Done Sent false
5494 Acked Read 2 Done Done Sent false
5495 Acked Write 8 Done Done Sent false
5496 Acked Read 10 Done Done Sent false
5497 Acked Read 6 Done Done Sent false
5498 Acked Flush 0 Done Done Sent false
5499 Acked Read 7 Done Done Sent false
5500 Acked Read 6 Done Done Sent false
5501 Acked Write 7 Done Done Sent false
5502 Acked Write 6 Done Done Sent false
5503 Acked Read 5 Done Done Sent false
5504 Acked Read 6 Done Done Sent false
5505 Acked Write 4 Done Done Sent false
5506 Acked Read 10 Done Done Sent false
5507 Acked Read 1 Done Done Sent false
5508 Acked Read 4 Done Done Sent false
5509 Acked Read 6 Done Done Sent false
5510 Acked Flush 0 Done Done Sent false
5511 Acked Write 5 Done Done Sent false
5512 Acked Write 6 Done Done Sent false
5513 Acked Write 2 Done Done Sent false
5514 Acked Write 3 Done Done Sent false
5515 Acked Write 3 Done Done Sent false
5516 Acked Write 6 Done Done Sent false
5517 Acked Write 3 Done Done Sent false
5518 Acked Write 5 Done Done Sent false
5519 Acked Read 2 Done Done Sent false
5520 Acked Write 10 Done Done Sent false
5521 Acked Write 9 Done Done Sent false
5522 Acked Flush 0 Done Done Sent false
5523 Acked Write 2 Done Done Sent false
5524 Acked Read 5 Done Done Sent false
5525 Acked Read 7 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5522 5522 5224
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5525 5524 5523 5522 5521
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5225 Acked Write 7 Done Done Sent false
5226 Acked Write 6 Done Done Sent false
5227 Acked Read 5 Done Done Sent false
5228 Acked Read 3 Done Done Sent false
5229 Acked Read 7 Done Done Sent false
5230 Acked Write 5 Done Done Sent false
5231 Acked Write 2 Done Done Sent false
5232 Acked Read 7 Done Done Sent false
5233 Acked Write 2 Done Done Sent false
5234 Acked Write 4 Done Done Sent false
5235 Acked Read 3 Done Done Sent false
5236 Acked Flush 0 Done Done Sent false
5237 Acked Flush 0 Done Done Sent false
5238 Acked Read 6 Done Done Sent false
5239 Acked Flush 0 Done Done Sent false
5240 Acked Read 4 Done Done Sent false
5241 Acked Write 9 Done Done Sent false
5242 Acked Read 2 Done Done Sent false
5243 Acked Read 3 Done Done Sent false
5244 Acked Write 4 Done Done Sent false
5245 Acked Read 1 Done Done Sent false
5246 Acked Read 9 Done Done Sent false
5247 Acked Write 8 Done Done Sent false
5248 Acked Read 9 Done Done Sent false
5249 Acked Read 10 Done Done Sent false
5250 Acked Write 6 Done Done Sent false
5251 Acked Write 7 Done Done Sent false
5252 Acked Read 1 Done Done Sent false
5253 Acked Read 2 Done Done Sent false
5254 Acked Write 7 Done Done Sent false
5255 Acked Read 1 Done Done Sent false
5256 Acked Write 10 Done Done Sent false
5257 Acked Read 10 Done Done Sent false
5258 Acked Write 3 Done Done Sent false
5259 Acked Read 8 Done Done Sent false
5260 Acked Read 5 Done Done Sent false
5261 Acked Flush 0 Done Done Sent false
5262 Acked Read 3 Done Done Sent false
5263 Acked Read 8 Done Done Sent false
5264 Acked Read 9 Done Done Sent false
5265 Acked Write 6 Done Done Sent false
5266 Acked Read 9 Done Done Sent false
5267 Acked Flush 0 Done Done Sent false
5268 Acked Read 5 Done Done Sent false
5269 Acked Read 5 Done Done Sent false
5270 Acked Flush 0 Done Done Sent false
5271 Acked Read 1 Done Done Sent false
5272 Acked Write 6 Done Done Sent false
5273 Acked Read 7 Done Done Sent false
5274 Acked Read 5 Done Done Sent false
5275 Acked Write 6 Done Done Sent false
5276 Acked Read 3 Done Done Sent false
5277 Acked Write 6 Done Done Sent false
5278 Acked Write 7 Done Done Sent false
5279 Acked Read 6 Done Done Sent false
5280 Acked Read 2 Done Done Sent false
5281 Acked Read 9 Done Done Sent false
5282 Acked Read 2 Done Done Sent false
5283 Acked Read 3 Done Done Sent false
5284 Acked Flush 0 Done Done Sent false
5285 Acked Write 7 Done Done Sent false
5286 Acked Read 8 Done Done Sent false
5287 Acked Write 8 Done Done Sent false
5288 Acked Read 10 Done Done Sent false
5289 Acked Write 8 Done Done Sent false
5290 Acked Write 5 Done Done Sent false
5291 Acked Write 1 Done Done Sent false
5292 Acked Write 10 Done Done Sent false
5293 Acked Write 2 Done Done Sent false
5294 Acked Write 2 Done Done Sent false
5295 Acked Write 9 Done Done Sent false
5296 Acked Read 10 Done Done Sent false
5297 Acked Read 2 Done Done Sent false
5298 Acked Write 2 Done Done Sent false
5299 Acked Read 2 Done Done Sent false
5300 Acked Write 10 Done Done Sent false
5301 Acked Write 7 Done Done Sent false
5302 Acked Read 1 Done Done Sent false
5303 Acked Read 3 Done Done Sent false
5304 Acked Read 3 Done Done Sent false
5305 Acked Read 10 Done Done Sent false
5306 Acked Read 4 Done Done Sent false
5307 Acked Read 3 Done Done Sent false
5308 Acked Write 5 Done Done Sent false
5309 Acked Read 10 Done Done Sent false
5310 Acked Read 6 Done Done Sent false
5311 Acked Read 1 Done Done Sent false
5312 Acked Write 7 Done Done Sent false
5313 Acked Read 4 Done Done Sent false
5314 Acked Write 3 Done Done Sent false
5315 Acked Read 4 Done Done Sent false
5316 Acked Read 2 Done Done Sent false
5317 Acked Write 2 Done Done Sent false
5318 Acked Write 4 Done Done Sent false
5319 Acked Write 10 Done Done Sent false
5320 Acked Read 1 Done Done Sent false
5321 Acked Write 6 Done Done Sent false
5322 Acked Write 1 Done Done Sent false
5323 Acked Write 8 Done Done Sent false
5324 Acked Flush 0 Done Done Sent false
5325 Acked Write 6 Done Done Sent false
5326 Acked Write 5 Done Done Sent false
5327 Acked Flush 0 Done Done Sent false
5328 Acked Write 4 Done Done Sent false
5329 Acked Read 1 Done Done Sent false
5330 Acked Read 6 Done Done Sent false
5331 Acked Read 8 Done Done Sent false
5332 Acked Write 6 Done Done Sent false
5333 Acked Read 3 Done Done Sent false
5334 Acked Read 3 Done Done Sent false
5335 Acked Write 8 Done Done Sent false
5336 Acked Flush 0 Done Done Sent false
5337 Acked Write 3 Done Done Sent false
5338 Acked Read 3 Done Done Sent false
5339 Acked Read 2 Done Done Sent false
5340 Acked Write 3 Done Done Sent false
5341 Acked Read 6 Done Done Sent false
5342 Acked Read 1 Done Done Sent false
5343 Acked Read 7 Done Done Sent false
5344 Acked Read 2 Done Done Sent false
5345 Acked Write 6 Done Done Sent false
5346 Acked Read 6 Done Done Sent false
5347 Acked Read 2 Done Done Sent false
5348 Acked Write 2 Done Done Sent false
5349 Acked Read 10 Done Done Sent false
5350 Acked Write 5 Done Done Sent false
5351 Acked Read 5 Done Done Sent false
5352 Acked Read 9 Done Done Sent false
5353 Acked Write 10 Done Done Sent false
5354 Acked Write 6 Done Done Sent false
5355 Acked Read 4 Done Done Sent false
5356 Acked Read 7 Done Done Sent false
5357 Acked Read 7 Done Done Sent false
5358 Acked Write 1 Done Done Sent false
5359 Acked Write 2 Done Done Sent false
5360 Acked Flush 0 Done Done Sent false
5361 Acked Write 8 Done Done Sent false
5362 Acked Read 6 Done Done Sent false
5363 Acked Flush 0 Done Done Sent false
5364 Acked Read 9 Done Done Sent false
5365 Acked Write 8 Done Done Sent false
5366 Acked Read 5 Done Done Sent false
5367 Acked Read 5 Done Done Sent false
5368 Acked Write 8 Done Done Sent false
5369 Acked Write 9 Done Done Sent false
5370 Acked Write 7 Done Done Sent false
5371 Acked Read 6 Done Done Sent false
5372 Acked Read 8 Done Done Sent false
5373 Acked Write 7 Done Done Sent false
5374 Acked Write 9 Done Done Sent false
5375 Acked Write 8 Done Done Sent false
5376 Acked Read 9 Done Done Sent false
5377 Acked Read 7 Done Done Sent false
5378 Acked Write 4 Done Done Sent false
5379 Acked Write 10 Done Done Sent false
5380 Acked Write 6 Done Done Sent false
5381 Acked Read 5 Done Done Sent false
5382 Acked Write 7 Done Done Sent false
5383 Acked Read 6 Done Done Sent false
5384 Acked Read 3 Done Done Sent false
5385 Acked Write 7 Done Done Sent false
5386 Acked Read 3 Done Done Sent false
5387 Acked Write 3 Done Done Sent false
5388 Acked Read 8 Done Done Sent false
5389 Acked Read 4 Done Done Sent false
5390 Acked Write 1 Done Done Sent false
5391 Acked Write 5 Done Done Sent false
5392 Acked Read 1 Done Done Sent false
5393 Acked Read 7 Done Done Sent false
5394 Acked Read 4 Done Done Sent false
5395 Acked Write 2 Done Done Sent false
5396 Acked Read 3 Done Done Sent false
5397 Acked Read 10 Done Done Sent false
5398 Acked Read 3 Done Done Sent false
5399 Acked Write 5 Done Done Sent false
5400 Acked Flush 0 Done Done Sent false
5401 Acked Read 2 Done Done Sent false
5402 Acked Read 1 Done Done Sent false
5403 Acked Flush 0 Done Done Sent false
5404 Acked Read 4 Done Done Sent false
5405 Acked Read 4 Done Done Sent false
5406 Acked Read 2 Done Done Sent false
5407 Acked Read 8 Done Done Sent false
5408 Acked Read 4 Done Done Sent false
5409 Acked Read 9 Done Done Sent false
5410 Acked Read 7 Done Done Sent false
5411 Acked Write 1 Done Done Sent false
5412 Acked Write 1 Done Done Sent false
5413 Acked Read 9 Done Done Sent false
5414 Acked Read 4 Done Done Sent false
5415 Acked Write 6 Done Done Sent false
5416 Acked Flush 0 Done Done Sent false
5417 Acked Write 9 Done Done Sent false
5418 Acked Read 6 Done Done Sent false
5419 Acked Read 7 Done Done Sent false
5420 Acked Write 4 Done Done Sent false
5421 Acked Write 4 Done Done Sent false
5422 Acked Read 1 Done Done Sent false
5423 Acked Read 7 Done Done Sent false
5424 Acked Write 10 Done Done Sent false
5425 Acked Write 3 Done Done Sent false
5426 Acked Write 1 Done Done Sent false
5427 Acked Read 4 Done Done Sent false
5428 Acked Write 4 Done Done Sent false
5429 Acked Write 3 Done Done Sent false
5430 Acked Read 2 Done Done Sent false
5431 Acked Write 9 Done Done Sent false
5432 Acked Read 3 Done Done Sent false
5433 Acked Flush 0 Done Done Sent false
5434 Acked Write 2 Done Done Sent false
5435 Acked Write 8 Done Done Sent false
5436 Acked Write 5 Done Done Sent false
5437 Acked Read 2 Done Done Sent false
5438 Acked Write 10 Done Done Sent false
5439 Acked Write 8 Done Done Sent false
5440 Acked Write 2 Done Done Sent false
5441 Acked Read 1 Done Done Sent false
5442 Acked Read 4 Done Done Sent false
5443 Acked Read 8 Done Done Sent false
5444 Acked Write 5 Done Done Sent false
5445 Acked Write 8 Done Done Sent false
5446 Acked Read 9 Done Done Sent false
5447 Acked Write 6 Done Done Sent false
5448 Acked Read 6 Done Done Sent false
5449 Acked Write 8 Done Done Sent false
5450 Acked Write 5 Done Done Sent false
5451 Acked Write 1 Done Done Sent false
5452 Acked Write 1 Done Done Sent false
5453 Acked Write 3 Done Done Sent false
5454 Acked Read 3 Done Done Sent false
5455 Acked Write 5 Done Done Sent false
5456 Acked Read 9 Done Done Sent false
5457 Acked Write 9 Done Done Sent false
5458 Acked Read 6 Done Done Sent false
5459 Acked Read 6 Done Done Sent false
5460 Acked Read 5 Done Done Sent false
5461 Acked Read 3 Done Done Sent false
5462 Acked Write 3 Done Done Sent false
5463 Acked Read 6 Done Done Sent false
5464 Acked Read 8 Done Done Sent false
5465 Acked Read 9 Done Done Sent false
5466 Acked Read 10 Done Done Sent false
5467 Acked Flush 0 Done Done Sent false
5468 Acked Write 10 Done Done Sent false
5469 Acked Read 6 Done Done Sent false
5470 Acked Write 2 Done Done Sent false
5471 Acked Write 7 Done Done Sent false
5472 Acked Read 1 Done Done Sent false
5473 Acked Write 9 Done Done Sent false
5474 Acked Read 8 Done Done Sent false
5475 Acked Read 4 Done Done Sent false
5476 Acked Flush 0 Done Done Sent false
5477 Acked Write 3 Done Done Sent false
5478 Acked Read 2 Done Done Sent false
5479 Acked Write 3 Done Done Sent false
5480 Acked Read 3 Done Done Sent false
5481 Acked Write 9 Done Done Sent false
5482 Acked Write 5 Done Done Sent false
5483 Acked Write 9 Done Done Sent false
5484 Acked Write 7 Done Done Sent false
5485 Acked Read 10 Done Done Sent false
5486 Acked Read 7 Done Done Sent false
5487 Acked Write 7 Done Done Sent false
5488 Acked Read 10 Done Done Sent false
5489 Acked Read 8 Done Done Sent false
5490 Acked Read 6 Done Done Sent false
5491 Acked Write 10 Done Done Sent false
5492 Acked Read 3 Done Done Sent false
5493 Acked Write 5 Done Done Sent false
5494 Acked Read 2 Done Done Sent false
5495 Acked Write 8 Done Done Sent false
5496 Acked Read 10 Done Done Sent false
5497 Acked Read 6 Done Done Sent false
5498 Acked Flush 0 Done Done Sent false
5499 Acked Read 7 Done Done Sent false
5500 Acked Read 6 Done Done Sent false
5501 Acked Write 7 Done Done Sent false
5502 Acked Write 6 Done Done Sent false
5503 Acked Read 5 Done Done Sent false
5504 Acked Read 6 Done Done Sent false
5505 Acked Write 4 Done Done Sent false
5506 Acked Read 10 Done Done Sent false
5507 Acked Read 1 Done Done Sent false
5508 Acked Read 4 Done Done Sent false
5509 Acked Read 6 Done Done Sent false
5510 Acked Flush 0 Done Done Sent false
5511 Acked Write 5 Done Done Sent false
5512 Acked Write 6 Done Done Sent false
5513 Acked Write 2 Done Done Sent false
5514 Acked Write 3 Done Done Sent false
5515 Acked Write 3 Done Done Sent false
5516 Acked Write 6 Done Done Sent false
5517 Acked Write 3 Done Done Sent false
5518 Acked Write 5 Done Done Sent false
5519 Acked Read 2 Done Done Sent false
5520 Acked Write 10 Done Done Sent false
5521 Acked Write 9 Done Done Sent false
5522 Acked Flush 0 Done Done Sent false
5523 Acked Write 2 Done Done Sent false
5524 Acked Read 5 Done Done Sent false
5525 Acked Read 7 Done Done Sent false
5526 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5526 5526 5224
Downstairs last five completed: 5224 5223 5222 5221 5220
Upstairs last five completed: 5526 5525 5524 5523 5522
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.553294918Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61193)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.553428103Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.554825329Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555357247Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555380765Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 5224","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555394453Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 5224","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555615447Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 5224","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555638145Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 5224","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:50.555880126Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5526 5525 5524 5523 5522
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:39:52.622 [8830] Got stop action so:false kr:false
INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a9a37ef1-36a9-4c1c-b5e7-b895f135150a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:52.623223316Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:52.62327374Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:52.623290558Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:39:52.623 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/2, method: GET, req_id: 21c3b959-7133-469a-890a-7b913a4497b7, remote_addr: 127.0.0.1:55047{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:52.623394776Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:39:56.625 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 97dc4fef-f831-48f5-8c3c-41dc0d792140, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 2676 len 16384
002/300 Read block 470 len 16384
003/300 Write block 2035 len 36864 data: 4 4 4 4 3 3 4 6 6
004/300 Read block 1493 len 32768
005/300 Read block 4233 len 28672
006/300 Read block 4486 len 12288
007/300 Flush
008/300 Read block 3565 len 16384
009/300 Read block 4957 len 32768
010/300 Write block 1294 len 20480 data: 4 4 4 5 6
011/300 Write block 4411 len 16384 data: 5 4 5 3
012/300 Read block 2504 len 4096
013/300 Write block 593 len 28672 data: 5 4 3 3 2 4 4
014/300 Read block 799 len 32768
015/300 Read block 3325 len 16384
016/300 Read block 2778 len 28672
017/300 Read block 2716 len 12288
018/300 Read block 2922 len 36864
019/300 Write block 2611 len 4096 data: 5
020/300 Write block 1809 len 36864 data: 5 7 8 7 5 6 8 7 6
021/300 Write block 3104 len 40960 data: 7 6 6 6 7 6 4 3 2 2
022/300 Write block 4649 len 20480 data: 3 2 3 3 3
023/300 Write block 1785 len 4096 data: 5
024/300 Read block 2294 len 16384
025/300 Read block 1675 len 12288
026/300 Read block 1420 len 16384
027/300 Read block 4646 len 20480
028/300 Read block 132 len 24576
029/300 Read block 710 len 16384
030/300 Read block 1020 len 4096
031/300 Read block 776 len 20480
032/300 Read block 723 len 20480
033/300 Flush
034/300 Write block 4030 len 4096 data: 2
035/300 Read block 484 len 4096
036/300 Read block 2386 len 8192
037/300 Read block 3030 len 20480
038/300 Flush
039/300 Write block 2727 len 4096 data: 3
040/300 Read block 3738 len 32768
041/300 Read block 2652 len 12288
042/300 Write block 2867 len 28672 data: 4 3 3 4 3 4 4
043/300 Write block 1374 len 36864 data: 6 7 8 8 7 7 6 5 6
044/300 Read block 2378 len 24576
045/300 Write block 4075 len 4096 data: 2
046/300 Flush
047/300 Read block 2066 len 8192
048/300 Write block 2279 len 12288 data: 2 2 3
049/300 Write block 4738 len 24576 data: 4 4 5 5 4 3
050/300 Read block 706 len 28672
051/300 Flush
052/300 Flush
053/300 Write block 2006 len 16384 data: 3 3 3 3
054/300 Read block 4019 len 4096
055/300 Flush
056/300 Read block 1681 len 12288
057/300 Write block 435 len 24576 data: 2 2 3 3 5 6
058/300 Write block 2678 len 4096 data: 6
059/300 Read block 4974 len 12288
060/300 Write block 4258 len 12288 data: 3 3 4
061/300 Write block 2236 len 40960 data: 4 4 4 5 4 4 4 4 4 5
062/300 Write block 4264 len 32768 data: 5 5 5 4 4 4 4 6
063/300 Write block 4214 len 40960 data: 2 3 3 2 3 3 3 3 3 3
064/300 Read block 2439 len 4096
065/300 Write block 2072 len 36864 data: 3 3 3 3 5 5 5 5 4
066/300 Write block 3061 len 12288 data: 5 5 5
067/300 Read block 3976 len 36864
068/300 Flush
069/300 Write block 3713 len 20480 data: 6 6 6 4 4
070/300 Read block 697 len 28672
071/300 Read block 390 len 40960
072/300 Read block 2377 len 24576
073/300 Read block 1749 len 36864
074/300 Write block 3253 len 36864 data: 4 4 2 3 3 3 3 3 4
075/300 Flush
076/300 Read block 54 len 4096
077/300 Read block 517 len 20480
078/300 Flush
079/300 Write block 1307 len 8192 data: 6 5
080/300 Flush
081/300 Read block 575 len 12288
082/300 Read block 1328 len 20480
083/300 Read block 1261 len 28672
084/300 Read block 1457 len 20480
085/300 Read block 4920 len 16384
086/300 Write block 2713 len 12288 data: 3 3 4
087/300 Write block 3314 len 32768 data: 5 4 4 4 3 3 4 3
088/300 Read block 4624 len 36864
089/300 Write block 642 len 32768 data: 3 2 2 2 2 2 5 5
090/300 Read block 4977 len 24576
091/300 Read block 1053 len 36864
092/300 Read block 971 len 40960
093/300 Write block 361 len 16384 data: 3 3 2 4
094/300 Write block 2303 len 24576 data: 5 5 5 3 4 5
095/300 Write block 1652 len 36864 data: 2 2 3 3 3 3 4 4 4
096/300 Flush
097/300 Write block 4473 len 24576 data: 3 3 5 5 5 5
098/300 Read block 2604 len 40960
099/300 Read block 2068 len 8192
100/300 Write block 2394 len 36864 data: 3 3 3 3 3 3 5 4 4
101/300 Write block 2246 len 28672 data: 5 6 5 4 4 3 2
102/300 Write block 1938 len 24576 data: 2 2 2 3 3 4
103/300 Write block 2407 len 40960 data: 3 3 4 5 4 5 5 6 5 5
104/300 Read block 3769 len 16384
105/300 Read block 1254 len 4096
106/300 Read block 924 len 4096
107/300 Read block 4227 len 32768
108/300 Read block 1561 len 24576
109/300 Write block 3877 len 20480 data: 4 5 4 7 6
110/300 Flush
111/300 Read block 2914 len 4096
112/300 Read block 2984 len 36864
113/300 Write block 4185 len 32768 data: 2 2 2 3 5 7 7 7
114/300 Write block 3707 len 20480 data: 5 7 7 7 8
115/300 Read block 2569 len 4096
116/300 Read block 1918 len 4096
117/300 Write block 422 len 40960 data: 5 5 5 5 5 3 3 4 3 3
118/300 Read block 4694 len 36864
119/300 Read block 390 len 24576
120/300 Write block 4835 len 40960 data: 5 5 5 6 5 4 4 5 4 4
121/300 Read block 2935 len 40960
122/300 Write block 3443 len 4096 data: 2
123/300 Read block 3796 len 12288
124/300 Read block 1435 len 36864
125/300 Read block 1390 len 20480
126/300 Write block 54 len 36864 data: 5 3 3 3 4 4 5 3 4
127/300 Write block 1590 len 36864 data: 2 4 4 4 3 3 4 5 5
128/300 Read block 3629 len 12288
129/300 Write block 1976 len 40960 data: 3 3 4 2 2 3 3 4 4 5
130/300 Write block 613 len 16384 data: 3 2 2 2
131/300 Write block 3757 len 8192 data: 5 6
132/300 Read block 3702 len 36864
133/300 Write block 58 len 32768 data: 5 5 6 4 5 4 5 5
134/300 Read block 3735 len 12288
135/300 Write block 1208 len 4096 data: 5
136/300 Read block 2873 len 32768
137/300 Read block 815 len 40960
138/300 Write block 232 len 36864 data: 2 2 3 4 4 4 4 4 4
139/300 Write block 1957 len 4096 data: 4
140/300 Read block 1654 len 28672
141/300 Flush
142/300 Read block 4618 len 16384
143/300 Write block 608 len 12288 data: 4 4 3
144/300 Flush
145/300 Flush
146/300 Write block 1565 len 40960 data: 2 2 2 2 3 3 4 5 5 4
147/300 Read block 3862 len 24576
148/300 Flush
149/300 Read block 886 len 20480
150/300 Flush
151/300 Read block 1838 len 40960
152/300 Read block 935 len 28672
153/300 Read block 4666 len 12288
154/300 Read block 4432 len 24576
155/300 Write block 3919 len 12288 data: 5 4 5
156/300 Write block 3693 len 20480 data: 5 4 3 4 4
157/300 Flush
158/300 Write block 1378 len 8192 data: 8 8
159/300 Write block 2558 len 28672 data: 4 4 5 5 5 6 5
160/300 Write block 3594 len 16384 data: 2 2 2 3
161/300 Read block 1227 len 20480
162/300 Write block 1370 len 4096 data: 6
163/300 Write block 2711 len 24576 data: 3 3 4 4 5 4
164/300 Flush
165/300 Read block 2815 len 12288
166/300 Write block 2288 len 20480 data: 10 9 9 6 6
167/300 Write block 4564 len 20480 data: 6 5 4 4 5
168/300 Write block 221 len 36864 data: 2 4 4 4 4 5 4 4 4
169/300 Read block 2991 len 4096
170/300 Write block 4524 len 20480 data: 4 3 4 4 4
171/300 Read block 4402 len 36864
172/300 Read block 294 len 36864
173/300 Read block 4107 len 28672
174/300 Write block 402 len 20480 data: 5 5 5 4 4
175/300 Write block 3946 len 12288 data: 2 2 2
176/300 Read block 2601 len 24576
177/300 Read block 1153 len 20480
178/300 Flush
179/300 Read block 557 len 36864
180/300 Read block 2978 len 36864
181/300 Write block 2606 len 8192 data: 5 5
182/300 Write block 1740 len 8192 data: 5 5
183/300 Read block 967 len 16384
184/300 Read block 1158 len 12288
185/300 Write block 2104 len 24576 data: 3 3 3 3 3 3
186/300 Read block 2922 len 12288
187/300 Write block 2982 len 12288 data: 5 6 5
188/300 Read block 155 len 24576
189/300 Write block 588 len 16384 data: 5 5 5 5
190/300 Read block 2535 len 32768
191/300 Write block 1234 len 36864 data: 8 8 8 7 7 7 7 5 3
192/300 Write block 352 len 36864 data: 5 5 5 5 6 5 5 3 3
193/300 Read block 3173 len 40960
194/300 Flush
195/300 Write block 918 len 20480 data: 3 5 5 5 4
196/300 Write block 2941 len 24576 data: 4 4 3 3 3 3
197/300 Write block 23 len 24576 data: 3 4 4 3 3 4
198/300 Read block 2445 len 40960
199/300 Read block 321 len 16384
200/300 Write block 401 len 32768 data: 5 6 6 6 5 5 3 3
201/300 Write block 1235 len 8192 data: 9 9
202/300 Read block 3243 len 40960
203/300 Write block 3003 len 8192 data: 5 4
204/300 Write block 1766 len 8192 data: 3 3
205/300 Write block 2557 len 8192 data: 5 5
206/300 Write block 4320 len 24576 data: 2 3 3 3 3 4
207/300 Read block 999 len 36864
208/300 Write block 153 len 28672 data: 2 3 3 3 4 4 4
209/300 Write block 2532 len 40960 data: 3 3 4 3 3 3 4 4 2 2
210/300 Write block 1640 len 24576 data: 3 4 4 3 3 3
211/300 Read block 3282 len 32768
212/300 Read block 4734 len 4096
213/300 Read block 2132 len 40960
214/300 Write block 2385 len 12288 data: 4 5 5
215/300 Write block 2839 len 12288 data: 5 5 6
216/300 Read block 1656 len 32768
217/300 Read block 4037 len 4096
218/300 Write block 4756 len 4096 data: 4
219/300 Write block 3148 len 16384 data: 4 4 5 6
220/300 Read block 2323 len 24576
221/300 Read block 2531 len 40960
222/300 Read block 2579 len 24576
223/300 Write block 3854 len 32768 data: 5 5 4 4 2 2 3 4
224/300 Read block 858 len 36864
225/300 Read block 330 len 24576
226/300 Write block 1017 len 4096 data: 4
227/300 Read block 3726 len 28672
228/300 Read block 4313 len 36864
229/300 Read block 4367 len 16384
230/300 Write block 4525 len 36864 data: 4 5 5 5 3 3 3 3 3
231/300 Write block 3876 len 4096 data: 4
232/300 Read block 102 len 32768
233/300 Flush
234/300 Read block 4453 len 12288
235/300 Read block 3499 len 32768
236/300 Read block 4295 len 20480
237/300 Write block 699 len 40960 data: 3 3 3 2 2 3 4 4 5 6
238/300 Read block 1203 len 32768
239/300 Write block 3209 len 20480 data: 2 2 4 4 4
240/300 Flush
241/300 Read block 291 len 32768
242/300 Write block 2764 len 20480 data: 2 3 4 5 4
243/300 Read block 1487 len 20480
244/300 Write block 3062 len 8192 data: 6 6
245/300 Write block 136 len 20480 data: 3 4 6 6 4
246/300 Read block 511 len 40960
247/300 Write block 3410 len 8192 data: 4 3
248/300 Read block 2191 len 28672
249/300 Read block 4337 len 8192
250/300 Write block 4937 len 20480 data: 4 5 4 4 4
251/300 Flush
252/300 Write block 1408 len 40960 data: 6 6 5 4 5 5 5 5 5 6
253/300 Read block 1733 len 4096
254/300 Write block 336 len 16384 data: 3 4 4 4
255/300 Flush
256/300 Read block 4805 len 12288
257/300 Read block 2314 len 12288
258/300 Write block 4602 len 8192 data: 4 4
259/300 Flush
260/300 Write block 3208 len 8192 data: 2 3
261/300 Write block 679 len 40960 data: 2 2 3 3 4 4 3 3 2 2
262/300 Read block 3290 len 24576
263/300 Read block 3365 len 4096
264/300 Write block 3321 len 40960 data: 4 3 4 4 5 6 4 4 5 6
265/300 Read block 1656 len 20480
266/300 Write block 2810 len 12288 data: 4 5 5
267/300 Read block 4501 len 12288
268/300 Write block 1591 len 32768 data: 5 5 5 4 4 5 6 6
269/300 Read block 2591 len 24576
270/300 Read block 271 len 28672
271/300 Read block 3987 len 36864
272/300 Write block 462 len 40960 data: 3 4 5 5 4 3 3 3 4 4
273/300 Write block 1781 len 20480 data: 3 2 2 4 6
274/300 Read block 1943 len 40960
275/300 Write block 1904 len 24576 data: 4 4 4 4 3 3
276/300 Write block 704 len 40960 data: 4 5 5 6 7 6 6 7 6 11
277/300 Read block 3376 len 8192
278/300 Read block 2629 len 24576
279/300 Read block 258 len 28672
280/300 Write block 2651 len 24576 data: 6 5 5 5 5 5
281/300 Read block 256 len 24576
282/300 Write block 2599 len 32768 data: 4 5 4 3 3 5 5 6
283/300 Read block 845 len 24576
284/300 Read block 4561 len 8192
285/300 Read block 1527 len 4096
286/300 Read block 3737 len 28672
287/300 Read block 2488 len 12288
288/300 Read block 1234 len 12288
289/300 Read block 4140 len 28672
290/300 Write block 1396 len 20480 data: 4 5 6 7 6
291/300 Read block 1435 len 40960
292/300 Read block 323 len 36864
293/300 Write block 271 len 28672 data: 3 4 4 4 5 5 5
294/300 Read block 2235 len 16384
295/300 Write block 4329 len 12288 data: 4 5 6
296/300 Write block 107 len 12288 data: 3 3 3
297/300 Write block 218 len 36864 data: 3 3 3 3 5 5 5 5 6
298/300 Write block 2828 len 40960 data: 4 4 5 5 6 6 5 3 4 5
299/300 Flush
300/300 Write block 4762 len 40960 data: 4 4 5 5 4 3 3 4 5 6
301/300 Read block 1779 len 32768
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:39:56.864 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0ba6721d-707a-4421-84fd-8573f09ac7ff, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5527 Acked Read 4 Done Done Sent false
5528 Acked Read 4 Done Done Sent false
5529 Acked Write 9 Done Done Sent false
5530 Acked Read 8 Done Done Sent false
5531 Acked Read 7 Done Done Sent false
5532 Acked Read 3 Done Done Sent false
5533 Acked Flush 0 Done Done Sent false
5534 Acked Read 4 Done Done Sent false
5535 Acked Read 8 Done Done Sent false
5536 Acked Write 5 Done Done Sent false
5537 Acked Write 4 Done Done Sent false
5538 Acked Read 1 Done Done Sent false
5539 Acked Write 7 Done Done Sent false
5540 Acked Read 8 Done Done Sent false
5541 Acked Read 4 Done Done Sent false
5542 Acked Read 7 Done Done Sent false
5543 Acked Read 3 Done Done Sent false
5544 Acked Read 9 Done Done Sent false
5545 Acked Write 1 Done Done Sent false
5546 Acked Write 9 Done Done Sent false
5547 Acked Write 10 Done Done Sent false
5548 Acked Write 5 Done Done Sent false
5549 Acked Write 1 Done Done Sent false
5550 Acked Read 4 Done Done Sent false
5551 Acked Read 3 Done Done Sent false
5552 Acked Read 4 Done Done Sent false
5553 Acked Read 5 Done Done Sent false
5554 Acked Read 6 Done Done Sent false
5555 Acked Read 4 Done Done Sent false
5556 Acked Read 1 Done Done Sent false
5557 Acked Read 5 Done Done Sent false
5558 Acked Read 5 Done Done Sent false
5559 Acked Flush 0 Done Done Sent false
5560 Acked Write 1 Done Done Sent false
5561 Acked Read 1 Done Done Sent false
5562 Acked Read 2 Done Done Sent false
5563 Acked Read 5 Done Done Sent false
5564 Acked Flush 0 Done Done Sent false
5565 Acked Write 1 Done Done Sent false
5566 Acked Read 8 Done Done Sent false
5567 Acked Read 3 Done Done Sent false
5568 Acked Write 7 Done Done Sent false
5569 Acked Write 9 Done Done Sent false
5570 Acked Read 6 Done Done Sent false
5571 Acked Write 1 Done Done Sent false
5572 Acked Flush 0 Done Done Sent false
5573 Acked Read 2 Done Done Sent false
5574 Acked Write 3 Done Done Sent false
5575 Acked Write 6 Done Done Sent false
5576 Acked Read 7 Done Done Sent false
5577 Acked Flush 0 Done Done Sent false
5578 Acked Flush 0 Done Done Sent false
5579 Acked Write 4 Done Done Sent false
5580 Acked Read 1 Done Done Sent false
5581 Acked Flush 0 Done Done Sent false
5582 Acked Read 3 Done Done Sent false
5583 Acked Write 6 Done Done Sent false
5584 Acked Write 1 Done Done Sent false
5585 Acked Read 3 Done Done Sent false
5586 Acked Write 3 Done Done Sent false
5587 Acked Write 10 Done Done Sent false
5588 Acked Write 8 Done Done Sent false
5589 Acked Write 10 Done Done Sent false
5590 Acked Read 1 Done Done Sent false
5591 Acked Write 9 Done Done Sent false
5592 Acked Write 3 Done Done Sent false
5593 Acked Read 9 Done Done Sent false
5594 Acked Flush 0 Done Done Sent false
5595 Acked Write 5 Done Done Sent false
5596 Acked Read 7 Done Done Sent false
5597 Acked Read 10 Done Done Sent false
5598 Acked Read 6 Done Done Sent false
5599 Acked Read 9 Done Done Sent false
5600 Acked Write 9 Done Done Sent false
5601 Acked Flush 0 Done Done Sent false
5602 Acked Read 1 Done Done Sent false
5603 Acked Read 5 Done Done Sent false
5604 Acked Flush 0 Done Done Sent false
5605 Acked Write 2 Done Done Sent false
5606 Acked Flush 0 Done Done Sent false
5607 Acked Read 3 Done Done Sent false
5608 Acked Read 5 Done Done Sent false
5609 Acked Read 7 Done Done Sent false
5610 Acked Read 5 Done Done Sent false
5611 Acked Read 4 Done Done Sent false
5612 Acked Write 3 Done Done Sent false
5613 Acked Write 8 Done Done Sent false
5614 Acked Read 9 Done Done Sent false
5615 Acked Write 8 Done Done Sent false
5616 Acked Read 6 Done Done Sent false
5617 Acked Read 9 Done Done Sent false
5618 Acked Read 10 Done Done Sent false
5619 Acked Write 4 Done Done Sent false
5620 Acked Write 6 Done Done Sent false
5621 Acked Write 9 Done Done Sent false
5622 Acked Flush 0 Done Done Sent false
5623 Acked Write 6 Done Done Sent false
5624 Acked Read 10 Done Done Sent false
5625 Acked Read 2 Done Done Sent false
5626 Acked Write 9 Done Done Sent false
5627 Acked Write 7 Done Done Sent false
5628 Acked Write 6 Done Done Sent false
5629 Acked Write 10 Done Done Sent false
5630 Acked Read 4 Done Done Sent false
5631 Acked Read 1 Done Done Sent false
5632 Acked Read 1 Done Done Sent false
5633 Acked Read 8 Done Done Sent false
5634 Acked Read 6 Done Done Sent false
5635 Acked Write 5 Done Done Sent false
5636 Acked Flush 0 Done Done Sent false
5637 Acked Read 1 Done Done Sent false
5638 Acked Read 9 Done Done Sent false
5639 Acked Write 8 Done Done Sent false
5640 Acked Write 5 Done Done Sent false
5641 Acked Read 1 Done Done Sent false
5642 Acked Read 1 Done Done Sent false
5643 Acked Write 10 Done Done Sent false
5644 Acked Read 9 Done Done Sent false
5645 Acked Read 6 Done Done Sent false
5646 Acked Write 10 Done Done Sent false
5647 Acked Read 10 Done Done Sent false
5648 Acked Write 1 Done Done Sent false
5649 Acked Read 3 Done Done Sent false
5650 Acked Read 9 Done Done Sent false
5651 Acked Read 5 Done Done Sent false
5652 Acked Write 9 Done Done Sent false
5653 Acked Write 9 Done Done Sent false
5654 Acked Read 3 Done Done Sent false
5655 Acked Write 10 Done Done Sent false
5656 Acked Write 4 Done Done Sent false
5657 Acked Write 2 Done Done Sent false
5658 Acked Read 9 Done Done Sent false
5659 Acked Write 8 Done Done Sent false
5660 Acked Read 3 Done Done Sent false
5661 Acked Write 1 Done Done Sent false
5662 Acked Read 8 Done Done Sent false
5663 Acked Read 10 Done Done Sent false
5664 Acked Write 9 Done Done Sent false
5665 Acked Write 1 Done Done Sent false
5666 Acked Read 7 Done Done Sent false
5667 Acked Flush 0 Done Done Sent false
5668 Acked Read 4 Done Done Sent false
5669 Acked Write 3 Done Done Sent false
5670 Acked Flush 0 Done Done Sent false
5671 Acked Flush 0 Done Done Sent false
5672 Acked Write 10 Done Done Sent false
5673 Acked Read 6 Done Done Sent false
5674 Acked Flush 0 Done Done Sent false
5675 Acked Read 5 Done Done Sent false
5676 Acked Flush 0 Done Done Sent false
5677 Acked Read 10 Done Done Sent false
5678 Acked Read 7 Done Done Sent false
5679 Acked Read 3 Done Done Sent false
5680 Acked Read 6 Done Done Sent false
5681 Acked Write 3 Done Done Sent false
5682 Acked Write 5 Done Done Sent false
5683 Acked Flush 0 Done Done Sent false
5684 Acked Write 2 Done Done Sent false
5685 Acked Write 7 Done Done Sent false
5686 Acked Write 4 Done Done Sent false
5687 Acked Read 5 Done Done Sent false
5688 Acked Write 1 Done Done Sent false
5689 Acked Write 6 Done Done Sent false
5690 Acked Flush 0 Done Done Sent false
5691 Acked Read 3 Done Done Sent false
5692 Acked Write 5 Done Done Sent false
5693 Acked Write 5 Done Done Sent false
5694 Acked Write 9 Done Done Sent false
5695 Acked Read 1 Done Done Sent false
5696 Acked Write 5 Done Done Sent false
5697 Acked Read 9 Done Done Sent false
5698 Acked Read 9 Done Done Sent false
5699 Acked Read 7 Done Done Sent false
5700 Acked Write 5 Done Done Sent false
5701 Acked Write 3 Done Done Sent false
5702 Acked Read 6 Done Done Sent false
5703 Acked Read 5 Done Done Sent false
5704 Acked Flush 0 Done Done Sent false
5705 Acked Read 9 Done Done Sent false
5706 Acked Read 9 Done Done Sent false
5707 Acked Write 2 Done Done Sent false
5708 Acked Write 2 Done Done Sent false
5709 Acked Read 4 Done Done Sent false
5710 Acked Read 3 Done Done Sent false
5711 Acked Write 6 Done Done Sent false
5712 Acked Read 3 Done Done Sent false
5713 Acked Write 3 Done Done Sent false
5714 Acked Read 6 Done Done Sent false
5715 Acked Write 4 Done Done Sent false
5716 Acked Read 8 Done Done Sent false
5717 Acked Write 9 Done Done Sent false
5718 Acked Write 9 Done Done Sent false
5719 Acked Read 10 Done Done Sent false
5720 Acked Flush 0 Done Done Sent false
5721 Acked Write 5 Done Done Sent false
5722 Acked Write 6 Done Done Sent false
5723 Acked Write 6 Done Done Sent false
5724 Acked Read 10 Done Done Sent false
5725 Acked Read 4 Done Done Sent false
5726 Acked Write 8 Done Done Sent false
5727 Acked Write 2 Done Done Sent false
5728 Acked Read 10 Done Done Sent false
5729 Acked Write 2 Done Done Sent false
5730 Acked Write 2 Done Done Sent false
5731 Acked Write 2 Done Done Sent false
5732 Acked Write 6 Done Done Sent false
5733 Acked Read 9 Done Done Sent false
5734 Acked Write 7 Done Done Sent false
5735 Acked Write 10 Done Done Sent false
5736 Acked Write 6 Done Done Sent false
5737 Acked Read 8 Done Done Sent false
5738 Acked Read 1 Done Done Sent false
5739 Acked Read 10 Done Done Sent false
5740 Acked Write 3 Done Done Sent false
5741 Acked Write 3 Done Done Sent false
5742 Acked Read 8 Done Done Sent false
5743 Acked Read 1 Done Done Sent false
5744 Acked Write 1 Done Done Sent false
5745 Acked Write 4 Done Done Sent false
5746 Acked Read 6 Done Done Sent false
5747 Acked Read 10 Done Done Sent false
5748 Acked Read 6 Done Done Sent false
5749 Acked Write 8 Done Done Sent false
5750 Acked Read 9 Done Done Sent false
5751 Acked Read 6 Done Done Sent false
5752 Acked Write 1 Done Done Sent false
5753 Acked Read 7 Done Done Sent false
5754 Acked Read 9 Done Done Sent false
5755 Acked Read 4 Done Done Sent false
5756 Acked Write 9 Done Done Sent false
5757 Acked Write 1 Done Done Sent false
5758 Acked Read 8 Done Done Sent false
5759 Acked Flush 0 Done Done Sent false
5760 Acked Read 3 Done Done Sent false
5761 Acked Read 8 Done Done Sent false
5762 Acked Read 5 Done Done Sent false
5763 Acked Write 10 Done Done Sent false
5764 Acked Read 8 Done Done Sent false
5765 Acked Write 5 Done Done Sent false
5766 Acked Flush 0 Done Done Sent false
5767 Acked Read 8 Done Done Sent false
5768 Acked Write 5 Done Done Sent false
5769 Acked Read 5 Done Done Sent false
5770 Acked Write 2 Done Done Sent false
5771 Acked Write 5 Done Done Sent false
5772 Acked Read 10 Done Done Sent false
5773 Acked Write 2 Done Done Sent false
5774 Acked Read 7 Done Done Sent false
5775 Acked Read 2 Done Done Sent false
5776 Acked Write 5 Done Done Sent false
5777 Acked Flush 0 Done Done Sent false
5778 Acked Write 10 Done Done Sent false
5779 Acked Read 1 Done Done Sent false
5780 Acked Write 4 Done Done Sent false
5781 Acked Flush 0 Done Done Sent false
5782 Acked Read 3 Done Done Sent false
5783 Acked Read 3 Done Done Sent false
5784 Acked Write 2 Done Done Sent false
5785 Acked Flush 0 Done Done Sent false
5786 Acked Write 2 Done Done Sent false
5787 Acked Write 10 Done Done Sent false
5788 Acked Read 6 Done Done Sent false
5789 Acked Read 1 Done Done Sent false
5790 Acked Write 10 Done Done Sent false
5791 Acked Read 5 Done Done Sent false
5792 Acked Write 3 Done Done Sent false
5793 Acked Read 3 Done Done Sent false
5794 Acked Write 8 Done Done Sent false
5795 Acked Read 6 Done Done Sent false
5796 Acked Read 7 Done Done Sent false
5797 Acked Read 9 Done Done Sent false
5798 Acked Write 10 Done Done Sent false
5799 Acked Write 5 Done Done Sent false
5800 Acked Read 10 Done Done Sent false
5801 Acked Write 6 Done Done Sent false
5802 Acked Write 10 Done Done Sent false
5803 Acked Read 2 Done Done Sent false
5804 Acked Read 6 Done Done Sent false
5805 Acked Read 7 Done Done Sent false
5806 Acked Write 6 Done Done Sent false
5807 Acked Read 6 Done Done Sent false
5808 Acked Write 8 Done Done Sent false
5809 Acked Read 6 Done Done Sent false
5810 Acked Read 2 Done Done Sent false
5811 Acked Read 1 Done Done Sent false
5812 Acked Read 7 Done Done Sent false
5813 Acked Read 3 Done Done Sent false
5814 Acked Read 3 Done Done Sent false
5815 Acked Read 7 Done Done Sent false
5816 Acked Write 5 Done Done Sent false
5817 Acked Read 10 Done Done Sent false
5818 Acked Read 9 Done Done Sent false
5819 Acked Write 7 Done Done Sent false
5820 Acked Read 4 Done Done Sent false
5821 Acked Write 3 Done Done Sent false
5822 Acked Write 3 Done Done Sent false
5823 Acked Write 9 Done Done Sent false
5824 Acked Write 10 Done Done Sent false
5825 Acked Flush 0 Done Done Sent false
5826 Acked Write 10 Done Done Sent false
5827 Acked Read 8 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5825 5825 5526
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5827 5826 5825 5824 5823
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5527 Acked Read 4 Done Done Sent false
5528 Acked Read 4 Done Done Sent false
5529 Acked Write 9 Done Done Sent false
5530 Acked Read 8 Done Done Sent false
5531 Acked Read 7 Done Done Sent false
5532 Acked Read 3 Done Done Sent false
5533 Acked Flush 0 Done Done Sent false
5534 Acked Read 4 Done Done Sent false
5535 Acked Read 8 Done Done Sent false
5536 Acked Write 5 Done Done Sent false
5537 Acked Write 4 Done Done Sent false
5538 Acked Read 1 Done Done Sent false
5539 Acked Write 7 Done Done Sent false
5540 Acked Read 8 Done Done Sent false
5541 Acked Read 4 Done Done Sent false
5542 Acked Read 7 Done Done Sent false
5543 Acked Read 3 Done Done Sent false
5544 Acked Read 9 Done Done Sent false
5545 Acked Write 1 Done Done Sent false
5546 Acked Write 9 Done Done Sent false
5547 Acked Write 10 Done Done Sent false
5548 Acked Write 5 Done Done Sent false
5549 Acked Write 1 Done Done Sent false
5550 Acked Read 4 Done Done Sent false
5551 Acked Read 3 Done Done Sent false
5552 Acked Read 4 Done Done Sent false
5553 Acked Read 5 Done Done Sent false
5554 Acked Read 6 Done Done Sent false
5555 Acked Read 4 Done Done Sent false
5556 Acked Read 1 Done Done Sent false
5557 Acked Read 5 Done Done Sent false
5558 Acked Read 5 Done Done Sent false
5559 Acked Flush 0 Done Done Sent false
5560 Acked Write 1 Done Done Sent false
5561 Acked Read 1 Done Done Sent false
5562 Acked Read 2 Done Done Sent false
5563 Acked Read 5 Done Done Sent false
5564 Acked Flush 0 Done Done Sent false
5565 Acked Write 1 Done Done Sent false
5566 Acked Read 8 Done Done Sent false
5567 Acked Read 3 Done Done Sent false
5568 Acked Write 7 Done Done Sent false
5569 Acked Write 9 Done Done Sent false
5570 Acked Read 6 Done Done Sent false
5571 Acked Write 1 Done Done Sent false
5572 Acked Flush 0 Done Done Sent false
5573 Acked Read 2 Done Done Sent false
5574 Acked Write 3 Done Done Sent false
5575 Acked Write 6 Done Done Sent false
5576 Acked Read 7 Done Done Sent false
5577 Acked Flush 0 Done Done Sent false
5578 Acked Flush 0 Done Done Sent false
5579 Acked Write 4 Done Done Sent false
5580 Acked Read 1 Done Done Sent false
5581 Acked Flush 0 Done Done Sent false
5582 Acked Read 3 Done Done Sent false
5583 Acked Write 6 Done Done Sent false
5584 Acked Write 1 Done Done Sent false
5585 Acked Read 3 Done Done Sent false
5586 Acked Write 3 Done Done Sent false
5587 Acked Write 10 Done Done Sent false
5588 Acked Write 8 Done Done Sent false
5589 Acked Write 10 Done Done Sent false
5590 Acked Read 1 Done Done Sent false
5591 Acked Write 9 Done Done Sent false
5592 Acked Write 3 Done Done Sent false
5593 Acked Read 9 Done Done Sent false
5594 Acked Flush 0 Done Done Sent false
5595 Acked Write 5 Done Done Sent false
5596 Acked Read 7 Done Done Sent false
5597 Acked Read 10 Done Done Sent false
5598 Acked Read 6 Done Done Sent false
5599 Acked Read 9 Done Done Sent false
5600 Acked Write 9 Done Done Sent false
5601 Acked Flush 0 Done Done Sent false
5602 Acked Read 1 Done Done Sent false
5603 Acked Read 5 Done Done Sent false
5604 Acked Flush 0 Done Done Sent false
5605 Acked Write 2 Done Done Sent false
5606 Acked Flush 0 Done Done Sent false
5607 Acked Read 3 Done Done Sent false
5608 Acked Read 5 Done Done Sent false
5609 Acked Read 7 Done Done Sent false
5610 Acked Read 5 Done Done Sent false
5611 Acked Read 4 Done Done Sent false
5612 Acked Write 3 Done Done Sent false
5613 Acked Write 8 Done Done Sent false
5614 Acked Read 9 Done Done Sent false
5615 Acked Write 8 Done Done Sent false
5616 Acked Read 6 Done Done Sent false
5617 Acked Read 9 Done Done Sent false
5618 Acked Read 10 Done Done Sent false
5619 Acked Write 4 Done Done Sent false
5620 Acked Write 6 Done Done Sent false
5621 Acked Write 9 Done Done Sent false
5622 Acked Flush 0 Done Done Sent false
5623 Acked Write 6 Done Done Sent false
5624 Acked Read 10 Done Done Sent false
5625 Acked Read 2 Done Done Sent false
5626 Acked Write 9 Done Done Sent false
5627 Acked Write 7 Done Done Sent false
5628 Acked Write 6 Done Done Sent false
5629 Acked Write 10 Done Done Sent false
5630 Acked Read 4 Done Done Sent false
5631 Acked Read 1 Done Done Sent false
5632 Acked Read 1 Done Done Sent false
5633 Acked Read 8 Done Done Sent false
5634 Acked Read 6 Done Done Sent false
5635 Acked Write 5 Done Done Sent false
5636 Acked Flush 0 Done Done Sent false
5637 Acked Read 1 Done Done Sent false
5638 Acked Read 9 Done Done Sent false
5639 Acked Write 8 Done Done Sent false
5640 Acked Write 5 Done Done Sent false
5641 Acked Read 1 Done Done Sent false
5642 Acked Read 1 Done Done Sent false
5643 Acked Write 10 Done Done Sent false
5644 Acked Read 9 Done Done Sent false
5645 Acked Read 6 Done Done Sent false
5646 Acked Write 10 Done Done Sent false
5647 Acked Read 10 Done Done Sent false
5648 Acked Write 1 Done Done Sent false
5649 Acked Read 3 Done Done Sent false
5650 Acked Read 9 Done Done Sent false
5651 Acked Read 5 Done Done Sent false
5652 Acked Write 9 Done Done Sent false
5653 Acked Write 9 Done Done Sent false
5654 Acked Read 3 Done Done Sent false
5655 Acked Write 10 Done Done Sent false
5656 Acked Write 4 Done Done Sent false
5657 Acked Write 2 Done Done Sent false
5658 Acked Read 9 Done Done Sent false
5659 Acked Write 8 Done Done Sent false
5660 Acked Read 3 Done Done Sent false
5661 Acked Write 1 Done Done Sent false
5662 Acked Read 8 Done Done Sent false
5663 Acked Read 10 Done Done Sent false
5664 Acked Write 9 Done Done Sent false
5665 Acked Write 1 Done Done Sent false
5666 Acked Read 7 Done Done Sent false
5667 Acked Flush 0 Done Done Sent false
5668 Acked Read 4 Done Done Sent false
5669 Acked Write 3 Done Done Sent false
5670 Acked Flush 0 Done Done Sent false
5671 Acked Flush 0 Done Done Sent false
5672 Acked Write 10 Done Done Sent false
5673 Acked Read 6 Done Done Sent false
5674 Acked Flush 0 Done Done Sent false
5675 Acked Read 5 Done Done Sent false
5676 Acked Flush 0 Done Done Sent false
5677 Acked Read 10 Done Done Sent false
5678 Acked Read 7 Done Done Sent false
5679 Acked Read 3 Done Done Sent false
5680 Acked Read 6 Done Done Sent false
5681 Acked Write 3 Done Done Sent false
5682 Acked Write 5 Done Done Sent false
5683 Acked Flush 0 Done Done Sent false
5684 Acked Write 2 Done Done Sent false
5685 Acked Write 7 Done Done Sent false
5686 Acked Write 4 Done Done Sent false
5687 Acked Read 5 Done Done Sent false
5688 Acked Write 1 Done Done Sent false
5689 Acked Write 6 Done Done Sent false
5690 Acked Flush 0 Done Done Sent false
5691 Acked Read 3 Done Done Sent false
5692 Acked Write 5 Done Done Sent false
5693 Acked Write 5 Done Done Sent false
5694 Acked Write 9 Done Done Sent false
5695 Acked Read 1 Done Done Sent false
5696 Acked Write 5 Done Done Sent false
5697 Acked Read 9 Done Done Sent false
5698 Acked Read 9 Done Done Sent false
5699 Acked Read 7 Done Done Sent false
5700 Acked Write 5 Done Done Sent false
5701 Acked Write 3 Done Done Sent false
5702 Acked Read 6 Done Done Sent false
5703 Acked Read 5 Done Done Sent false
5704 Acked Flush 0 Done Done Sent false
5705 Acked Read 9 Done Done Sent false
5706 Acked Read 9 Done Done Sent false
5707 Acked Write 2 Done Done Sent false
5708 Acked Write 2 Done Done Sent false
5709 Acked Read 4 Done Done Sent false
5710 Acked Read 3 Done Done Sent false
5711 Acked Write 6 Done Done Sent false
5712 Acked Read 3 Done Done Sent false
5713 Acked Write 3 Done Done Sent false
5714 Acked Read 6 Done Done Sent false
5715 Acked Write 4 Done Done Sent false
5716 Acked Read 8 Done Done Sent false
5717 Acked Write 9 Done Done Sent false
5718 Acked Write 9 Done Done Sent false
5719 Acked Read 10 Done Done Sent false
5720 Acked Flush 0 Done Done Sent false
5721 Acked Write 5 Done Done Sent false
5722 Acked Write 6 Done Done Sent false
5723 Acked Write 6 Done Done Sent false
5724 Acked Read 10 Done Done Sent false
5725 Acked Read 4 Done Done Sent false
5726 Acked Write 8 Done Done Sent false
5727 Acked Write 2 Done Done Sent false
5728 Acked Read 10 Done Done Sent false
5729 Acked Write 2 Done Done Sent false
5730 Acked Write 2 Done Done Sent false
5731 Acked Write 2 Done Done Sent false
5732 Acked Write 6 Done Done Sent false
5733 Acked Read 9 Done Done Sent false
5734 Acked Write 7 Done Done Sent false
5735 Acked Write 10 Done Done Sent false
5736 Acked Write 6 Done Done Sent false
5737 Acked Read 8 Done Done Sent false
5738 Acked Read 1 Done Done Sent false
5739 Acked Read 10 Done Done Sent false
5740 Acked Write 3 Done Done Sent false
5741 Acked Write 3 Done Done Sent false
5742 Acked Read 8 Done Done Sent false
5743 Acked Read 1 Done Done Sent false
5744 Acked Write 1 Done Done Sent false
5745 Acked Write 4 Done Done Sent false
5746 Acked Read 6 Done Done Sent false
5747 Acked Read 10 Done Done Sent false
5748 Acked Read 6 Done Done Sent false
5749 Acked Write 8 Done Done Sent false
5750 Acked Read 9 Done Done Sent false
5751 Acked Read 6 Done Done Sent false
5752 Acked Write 1 Done Done Sent false
5753 Acked Read 7 Done Done Sent false
5754 Acked Read 9 Done Done Sent false
5755 Acked Read 4 Done Done Sent false
5756 Acked Write 9 Done Done Sent false
5757 Acked Write 1 Done Done Sent false
5758 Acked Read 8 Done Done Sent false
5759 Acked Flush 0 Done Done Sent false
5760 Acked Read 3 Done Done Sent false
5761 Acked Read 8 Done Done Sent false
5762 Acked Read 5 Done Done Sent false
5763 Acked Write 10 Done Done Sent false
5764 Acked Read 8 Done Done Sent false
5765 Acked Write 5 Done Done Sent false
5766 Acked Flush 0 Done Done Sent false
5767 Acked Read 8 Done Done Sent false
5768 Acked Write 5 Done Done Sent false
5769 Acked Read 5 Done Done Sent false
5770 Acked Write 2 Done Done Sent false
5771 Acked Write 5 Done Done Sent false
5772 Acked Read 10 Done Done Sent false
5773 Acked Write 2 Done Done Sent false
5774 Acked Read 7 Done Done Sent false
5775 Acked Read 2 Done Done Sent false
5776 Acked Write 5 Done Done Sent false
5777 Acked Flush 0 Done Done Sent false
5778 Acked Write 10 Done Done Sent false
5779 Acked Read 1 Done Done Sent false
5780 Acked Write 4 Done Done Sent false
5781 Acked Flush 0 Done Done Sent false
5782 Acked Read 3 Done Done Sent false
5783 Acked Read 3 Done Done Sent false
5784 Acked Write 2 Done Done Sent false
5785 Acked Flush 0 Done Done Sent false
5786 Acked Write 2 Done Done Sent false
5787 Acked Write 10 Done Done Sent false
5788 Acked Read 6 Done Done Sent false
5789 Acked Read 1 Done Done Sent false
5790 Acked Write 10 Done Done Sent false
5791 Acked Read 5 Done Done Sent false
5792 Acked Write 3 Done Done Sent false
5793 Acked Read 3 Done Done Sent false
5794 Acked Write 8 Done Done Sent false
5795 Acked Read 6 Done Done Sent false
5796 Acked Read 7 Done Done Sent false
5797 Acked Read 9 Done Done Sent false
5798 Acked Write 10 Done Done Sent false
5799 Acked Write 5 Done Done Sent false
5800 Acked Read 10 Done Done Sent false
5801 Acked Write 6 Done Done Sent false
5802 Acked Write 10 Done Done Sent false
5803 Acked Read 2 Done Done Sent false
5804 Acked Read 6 Done Done Sent false
5805 Acked Read 7 Done Done Sent false
5806 Acked Write 6 Done Done Sent false
5807 Acked Read 6 Done Done Sent false
5808 Acked Write 8 Done Done Sent false
5809 Acked Read 6 Done Done Sent false
5810 Acked Read 2 Done Done Sent false
5811 Acked Read 1 Done Done Sent false
5812 Acked Read 7 Done Done Sent false
5813 Acked Read 3 Done Done Sent false
5814 Acked Read 3 Done Done Sent false
5815 Acked Read 7 Done Done Sent false
5816 Acked Write 5 Done Done Sent false
5817 Acked Read 10 Done Done Sent false
5818 Acked Read 9 Done Done Sent false
5819 Acked Write 7 Done Done Sent false
5820 Acked Read 4 Done Done Sent false
5821 Acked Write 3 Done Done Sent false
5822 Acked Write 3 Done Done Sent false
5823 Acked Write 9 Done Done Sent false
5824 Acked Write 10 Done Done Sent false
5825 Acked Flush 0 Done Done Sent false
5826 Acked Write 10 Done Done Sent false
5827 Acked Read 8 Done Done Sent false
5828 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5828 5828 5526
Downstairs last five completed: 5526 5525 5524 5523 5522
Upstairs last five completed: 5828 5827 5826 5825 5824
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.624653196Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57141)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.624798259Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.625695034Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.626275616Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.626307072Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 5526","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.62632581Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 5526","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.626540685Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 5526","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.626564822Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 5526","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:02.626788476Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 5828 5827 5826 5825 5824
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:40:04.869 INFO request completed, latency_us: 345, response_code: [8810] Got stop action so:false kr:false
204, uri: /stop/cid/0, method: POST, req_id: d4f4cb10-48c8-466c-8e4c-62f5f763f167, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:04.870349513Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:04.870413805Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:04.870434163Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:40:04.870 INFO request completed, latency_us: 142, response_code: 200, uri: /state/cid/0, method: GET, req_id: f26b93fc-3966-4a21-987c-2d40c09065db, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:04.870558838Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:40:08.871 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: 47a63636-649f-428d-81f2-6b1470ee38b3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 3677 len 40960
002/300 Write block 488 len 8192 data: 3 3
003/300 Read block 837 len 16384
004/300 Read block 290 len 12288
005/300 Read block 1797 len 32768
006/300 Write block 2261 len 8192 data: 5 5
007/300 Read block 1831 len 16384
008/300 Write block 42 len 24576 data: 4 5 5 4 5 5
009/300 Read block 4387 len 32768
010/300 Write block 3957 len 12288 data: 4 5 6
011/300 Flush
012/300 Read block 1742 len 24576
013/300 Flush
014/300 Read block 3204 len 16384
015/300 Read block 2426 len 12288
016/300 Read block 4546 len 28672
017/300 Write block 1886 len 16384 data: 5 4 5 6
018/300 Read block 84 len 24576
019/300 Read block 3842 len 32768
020/300 Read block 3350 len 24576
021/300 Read block 2560 len 12288
022/300 Read block 3777 len 20480
023/300 Flush
024/300 Read block 2526 len 12288
025/300 Read block 645 len 32768
026/300 Write block 4177 len 8192 data: 5 5
027/300 Write block 1638 len 8192 data: 4 4
028/300 Flush
029/300 Write block 1683 len 20480 data: 2 2 2 2 2
030/300 Read block 484 len 28672
031/300 Flush
032/300 Write block 1966 len 12288 data: 5 5 5
033/300 Read block 1519 len 24576
034/300 Read block 3824 len 24576
035/300 Write block 3068 len 16384 data: 4 3 2 2
036/300 Read block 3410 len 8192
037/300 Read block 3755 len 32768
038/300 Read block 1468 len 16384
039/300 Write block 2308 len 32768 data: 6 5 5 4 4 3 3 3
040/300 Read block 1116 len 32768
041/300 Write block 3251 len 8192 data: 3 3
042/300 Write block 3597 len 16384 data: 4 3 4 4
043/300 Read block 4271 len 16384
044/300 Write block 1870 len 40960 data: 4 4 4 5 5 5 4 3 3 4
045/300 Read block 2531 len 32768
046/300 Write block 4575 len 12288 data: 3 3 3
047/300 Write block 2700 len 20480 data: 2 2 3 3 2
048/300 Read block 4080 len 8192
049/300 Write block 2684 len 24576 data: 8 7 9 8 6 3
050/300 Write block 1990 len 20480 data: 2 3 3 3 3
051/300 Read block 3657 len 40960
052/300 Read block 4178 len 32768
053/300 Flush
054/300 Read block 3684 len 32768
055/300 Write block 1687 len 32768 data: 3 2 2 3 3 3 3 4
056/300 Read block 1498 len 40960
057/300 Read block 4889 len 20480
058/300 Write block 3642 len 40960 data: 3 3 4 3 3 2 3 4 3 3
059/300 Read block 577 len 16384
060/300 Read block 2340 len 24576
061/300 Write block 1058 len 20480 data: 3 4 4 3 2
062/300 Read block 2024 len 8192
063/300 Read block 4565 len 16384
064/300 Read block 2567 len 20480
065/300 Write block 1206 len 4096 data: 6
066/300 Flush
067/300 Write block 2042 len 28672 data: 7 7 5 5 5 4 4
068/300 Write block 3542 len 40960 data: 3 2 2 2 2 2 2 2 2 2
069/300 Write block 653 len 8192 data: 4 3
070/300 Read block 2288 len 20480
071/300 Read block 4794 len 20480
072/300 Flush
073/300 Read block 105 len 32768
074/300 Write block 3241 len 24576 data: 5 6 6 4 3 4
075/300 Write block 4124 len 28672 data: 3 3 3 4 3 3 3
076/300 Read block 4518 len 16384
077/300 Read block 493 len 36864
078/300 Write block 1933 len 40960 data: 2 3 3 2 2 3 3 3 4 4
079/300 Read block 4676 len 20480
080/300 Read block 1899 len 12288
081/300 Write block 523 len 8192 data: 4 2
082/300 Write block 1163 len 40960 data: 2 3 3 3 4 4 4 4 4 4
083/300 Write block 479 len 20480 data: 5 5 5 5 4
084/300 Write block 1098 len 20480 data: 4 4 4 5 5
085/300 Write block 968 len 36864 data: 3 3 3 3 4 4 5 5 5
086/300 Read block 1936 len 24576
087/300 Write block 3921 len 40960 data: 6 5 4 4 3 4 4 4 3 3
088/300 Read block 441 len 12288
089/300 Write block 2611 len 4096 data: 6
090/300 Write block 4607 len 28672 data: 3 3 3 4 3 3 5
091/300 Write block 2931 len 20480 data: 4 4 5 5 5
092/300 Write block 724 len 28672 data: 5 3 3 2 2 2 2
093/300 Write block 597 len 28672 data: 3 5 5 5 5 6 7
094/300 Read block 3076 len 24576
095/300 Read block 2566 len 32768
096/300 Flush
097/300 Flush
098/300 Read block 3572 len 28672
099/300 Write block 1052 len 12288 data: 2 2 2
100/300 Write block 4292 len 28672 data: 3 3 4 4 2 3 3
101/300 Write block 3178 len 20480 data: 7 7 5 5 5
102/300 Read block 3513 len 32768
103/300 Read block 4348 len 40960
104/300 Write block 4844 len 12288 data: 5 4 4
105/300 Read block 2524 len 4096
106/300 Read block 626 len 12288
107/300 Read block 3280 len 24576
108/300 Read block 4567 len 36864
109/300 Read block 3738 len 8192
110/300 Read block 2222 len 36864
111/300 Write block 44 len 20480 data: 6 5 6 6 6
112/300 Write block 2985 len 32768 data: 5 5 5 4 3 3 3 3
113/300 Read block 4453 len 40960
114/300 Write block 74 len 8192 data: 2 2
115/300 Read block 4917 len 12288
116/300 Read block 4420 len 20480
117/300 Read block 756 len 8192
118/300 Read block 3749 len 24576
119/300 Read block 2127 len 12288
120/300 Read block 1756 len 36864
121/300 Read block 3316 len 16384
122/300 Write block 1110 len 28672 data: 4 5 4 3 4 4 4
123/300 Read block 984 len 40960
124/300 Flush
125/300 Write block 696 len 28672 data: 3 3 3 4 4 4 3
126/300 Read block 4534 len 32768
127/300 Read block 303 len 40960
128/300 Write block 4588 len 40960 data: 2 2 2 2 2 2 3 3 3 3
129/300 Write block 4830 len 16384 data: 5 4 4 4
130/300 Read block 2207 len 16384
131/300 Write block 1982 len 28672 data: 4 5 5 6 5 5 5
132/300 Read block 2180 len 24576
133/300 Read block 2200 len 8192
134/300 Read block 4072 len 4096
135/300 Write block 2632 len 20480 data: 3 5 3 4 4
136/300 Write block 2393 len 12288 data: 4 4 4
137/300 Read block 1855 len 16384
138/300 Flush
139/300 Write block 4613 len 36864 data: 6 4 5 5 5 5 5 4 3
140/300 Write block 4819 len 20480 data: 2 2 3 3 3
141/300 Read block 3081 len 12288
142/300 Read block 4240 len 4096
143/300 Write block 3119 len 24576 data: 3 2 2 3 4 4
144/300 Read block 2355 len 40960
145/300 Read block 395 len 36864
146/300 Write block 1326 len 36864 data: 3 3 3 4 3 3 4 5 4
147/300 Read block 940 len 12288
148/300 Write block 3074 len 36864 data: 3 3 3 3 3 3 3 3 2
149/300 Flush
150/300 Write block 450 len 24576 data: 2 4 4 4 4 3
151/300 Read block 3106 len 20480
152/300 Flush
153/300 Read block 1635 len 16384
154/300 Read block 2072 len 24576
155/300 Write block 2845 len 32768 data: 6 7 5 6 6 5 6 6
156/300 Write block 3650 len 8192 data: 4 4
157/300 Write block 580 len 20480 data: 5 5 5 7 4
158/300 Write block 2507 len 24576 data: 3 5 4 3 4 5
159/300 Flush
160/300 Write block 806 len 28672 data: 2 3 3 4 4 4 4
161/300 Read block 4654 len 28672
162/300 Read block 1866 len 12288
163/300 Write block 3370 len 28672 data: 6 6 7 5 4 3 3
164/300 Write block 2352 len 24576 data: 6 8 6 4 4 2
165/300 Write block 285 len 12288 data: 5 5 4
166/300 Write block 3888 len 12288 data: 4 4 2
167/300 Write block 591 len 36864 data: 6 5 6 5 4 4 4 6 6
168/300 Flush
169/300 Read block 2750 len 12288
170/300 Flush
171/300 Read block 946 len 36864
172/300 Write block 4147 len 40960 data: 3 3 3 3 2 2 3 3 3 3
173/300 Write block 494 len 28672 data: 5 6 6 8 9 9 7
174/300 Read block 1632 len 36864
175/300 Read block 1666 len 40960
176/300 Read block 742 len 16384
177/300 Read block 2889 len 4096
178/300 Write block 4706 len 12288 data: 4 4 5
179/300 Read block 3223 len 4096
180/300 Write block 908 len 28672 data: 2 2 3 3 3 3 3
181/300 Flush
182/300 Write block 4474 len 40960 data: 4 6 6 6 6 3 6 6 6 5
183/300 Read block 3438 len 16384
184/300 Read block 3347 len 20480
185/300 Write block 1737 len 40960 data: 3 4 5 6 6 4 4 4 3 3
186/300 Read block 4570 len 16384
187/300 Flush
188/300 Write block 3606 len 8192 data: 7 8
189/300 Write block 745 len 28672 data: 5 5 5 5 5 4 3
190/300 Read block 3674 len 20480
191/300 Write block 4629 len 12288 data: 3 4 4
192/300 Write block 2860 len 32768 data: 4 4 5 5 5 5 5 5
193/300 Write block 988 len 20480 data: 3 3 2 2 2
194/300 Read block 4712 len 36864
195/300 Read block 1263 len 32768
196/300 Write block 3128 len 12288 data: 2 4 5
197/300 Read block 4338 len 12288
198/300 Read block 1986 len 40960
199/300 Read block 4361 len 40960
200/300 Read block 3976 len 24576
201/300 Flush
202/300 Write block 897 len 40960 data: 2 2 2 2 3 3 3 4 2 2
203/300 Flush
204/300 Read block 1033 len 4096
205/300 Read block 4024 len 4096
206/300 Read block 4725 len 24576
207/300 Flush
208/300 Read block 470 len 28672
209/300 Read block 2776 len 36864
210/300 Read block 2007 len 32768
211/300 Write block 4558 len 28672 data: 4 3 4 4 5 6 7
212/300 Read block 4708 len 4096
213/300 Write block 182 len 20480 data: 5 6 6 6 6
214/300 Read block 4043 len 16384
215/300 Write block 2967 len 36864 data: 3 3 4 3 4 5 5 5 4
216/300 Write block 1661 len 20480 data: 4 5 5 4 3
217/300 Read block 4596 len 36864
218/300 Write block 4744 len 28672 data: 3 4 5 3 3 4 5
219/300 Write block 2631 len 24576 data: 3 4 6 4 5 5
220/300 Read block 441 len 16384
221/300 Read block 3646 len 12288
222/300 Read block 2206 len 8192
223/300 Read block 2376 len 28672
224/300 Write block 459 len 24576 data: 2 4 4 4 5 6
225/300 Read block 2117 len 12288
226/300 Read block 3679 len 28672
227/300 Write block 2960 len 36864 data: 3 3 5 5 4 4 4 4 4
228/300 Write block 784 len 40960 data: 3 2 2 2 2 3 3 3 3 3
229/300 Read block 2016 len 36864
230/300 Read block 2354 len 28672
231/300 Read block 664 len 4096
232/300 Read block 2279 len 20480
233/300 Read block 1288 len 40960
234/300 Read block 97 len 36864
235/300 Write block 3098 len 8192 data: 4 4
236/300 Write block 3971 len 8192 data: 4 3
237/300 Read block 3789 len 16384
238/300 Read block 4741 len 4096
239/300 Read block 4858 len 20480
240/300 Flush
241/300 Read block 122 len 8192
242/300 Read block 3458 len 28672
243/300 Read block 1613 len 4096
244/300 Write block 4836 len 32768 data: 6 6 7 6 5 5 6 5
245/300 Write block 2231 len 4096 data: 4
246/300 Write block 1858 len 36864 data: 6 7 6 5 5 3 3 4 5
247/300 Flush
248/300 Read block 688 len 12288
249/300 Read block 3955 len 24576
250/300 Write block 4985 len 16384 data: 4 4 4 4
251/300 Read block 1126 len 12288
252/300 Flush
253/300 Write block 3936 len 36864 data: 5 6 4 4 4 4 2 2 2
254/300 Read block 24 len 36864
255/300 Write block 3449 len 20480 data: 4 4 5 5 6
256/300 Read block 529 len 20480
257/300 Write block 510 len 8192 data: 4 5
258/300 Write block 3279 len 4096 data: 5
259/300 Read block 4890 len 12288
260/300 Read block 2652 len 20480
261/300 Read block 364 len 20480
262/300 Write block 4445 len 40960 data: 3 4 3 3 3 3 3 3 4 3
263/300 Write block 2371 len 12288 data: 4 4 4
264/300 Flush
265/300 Read block 440 len 32768
266/300 Read block 659 len 32768
267/300 Read block 4384 len 40960
268/300 Read block 1332 len 16384
269/300 Read block 4127 len 40960
270/300 Write block 2763 len 28672 data: 2 3 4 5 6 5 4
271/300 Read block 1178 len 40960
272/300 Write block 809 len 16384 data: 5 5 5 5
273/300 Write block 1446 len 20480 data: 3 3 4 5 5
274/300 Read block 3897 len 36864
275/300 Write block 4805 len 32768 data: 7 4 5 5 4 6 5 3
276/300 Read block 566 len 28672
277/300 Read block 4266 len 16384
278/300 Flush
279/300 Read block 2571 len 24576
280/300 Read block 2266 len 28672
281/300 Read block 1722 len 32768
282/300 Read block 1101 len 36864
283/300 Read block 1652 len 4096
284/300 Write block 1627 len 32768 data: 9 7 6 5 5 5 5 4
285/300 Write block 3131 len 20480 data: 5 5 5 4 5
286/300 Read block 1408 len 24576
287/300 Write block 3142 len 16384 data: 8 9 6 6
288/300 Read block 3766 len 36864
289/300 Read block 1512 len 24576
290/300 Flush
291/300 Write block 573 len 20480 data: 3 3 4 4 4
292/300 Write block 3556 len 12288 data: 4 4 4
293/300 Read block 4206 len 32768
294/300 Read block 4641 len 28672
295/300 Read block 1715 len 28672
296/300 Read block 2231 len 4096
297/300 Write block 1770 len 28672 data: 4 4 5 5 4 4 4
298/300 Read block 4543 len 24576
299/300 Write block 352 len 12288 data: 6 6 6
300/300 Read block 364 len 32768
301/300 Write block 3080 len 20480 data: 4 4 3 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:40:09.119 INFO request completed, latency_us: 252, response_code: 204, uri: /start/cid/0, method: POST, req_id: 22238fbf-e189-4b55-9b96-04d9bf5278e9, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 10 Sent Done Done false
5830 Acked Write 2 Sent Done Done false
5831 Acked Read 4 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 8 Sent Done Done false
5834 Acked Write 2 Sent Done Done false
5835 Acked Read 4 Sent Done Done false
5836 Acked Write 6 Sent Done Done false
5837 Acked Read 8 Sent Done Done false
5838 Acked Write 3 Sent Done Done false
5839 Acked Flush 0 Sent Done Done false
5840 Acked Read 6 Sent Done Done false
5841 Acked Flush 0 Sent Done Done false
5842 Acked Read 4 Sent Done Done false
5843 Acked Read 3 Sent Done Done false
5844 Acked Read 7 Sent Done Done false
5845 Acked Write 4 Sent Done Done false
5846 Acked Read 6 Sent Done Done false
5847 Acked Read 8 Sent Done Done false
5848 Acked Read 6 Sent Done Done false
5849 Acked Read 3 Sent Done Done false
5850 Acked Read 5 Sent Done Done false
5851 Acked Flush 0 Sent Done Done false
5852 Acked Read 3 Sent Done Done false
5853 Acked Read 8 Sent Done Done false
5854 Acked Write 2 Sent Done Done false
5855 Acked Write 2 Sent Done Done false
5856 Acked Flush 0 Sent Done Done false
5857 Acked Write 5 Sent Done Done false
5858 Acked Read 7 Sent Done Done false
5859 Acked Flush 0 Sent Done Done false
5860 Acked Write 3 Sent Done Done false
5861 Acked Read 6 Sent Done Done false
5862 Acked Read 6 Sent Done Done false
5863 Acked Write 4 Sent Done Done false
5864 Acked Read 2 Sent Done Done false
5865 Acked Read 8 Sent Done Done false
5866 Acked Read 4 Sent Done Done false
5867 Acked Write 8 Sent Done Done false
5868 Acked Read 8 Sent Done Done false
5869 Acked Write 2 Sent Done Done false
5870 Acked Write 4 Sent Done Done false
5871 Acked Read 4 Sent Done Done false
5872 Acked Write 10 Sent Done Done false
5873 Acked Read 8 Sent Done Done false
5874 Acked Write 3 Sent Done Done false
5875 Acked Write 5 Sent Done Done false
5876 Acked Read 2 Sent Done Done false
5877 Acked Write 6 Sent Done Done false
5878 Acked Write 5 Sent Done Done false
5879 Acked Read 10 Sent Done Done false
5880 Acked Read 8 Sent Done Done false
5881 Acked Flush 0 Sent Done Done false
5882 Acked Read 8 Sent Done Done false
5883 Acked Write 8 Sent Done Done false
5884 Acked Read 10 Sent Done Done false
5885 Acked Read 5 Sent Done Done false
5886 Acked Write 10 Sent Done Done false
5887 Acked Read 4 Sent Done Done false
5888 Acked Read 6 Sent Done Done false
5889 Acked Write 5 Sent Done Done false
5890 Acked Read 2 Sent Done Done false
5891 Acked Read 4 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Write 1 Sent Done Done false
5894 Acked Flush 0 Sent Done Done false
5895 Acked Write 7 Sent Done Done false
5896 Acked Write 10 Sent Done Done false
5897 Acked Write 2 Sent Done Done false
5898 Acked Read 5 Sent Done Done false
5899 Acked Read 5 Sent Done Done false
5900 Acked Flush 0 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Write 6 Sent Done Done false
5903 Acked Write 7 Sent Done Done false
5904 Acked Read 4 Sent Done Done false
5905 Acked Read 9 Sent Done Done false
5906 Acked Write 10 Sent Done Done false
5907 Acked Read 5 Sent Done Done false
5908 Acked Read 3 Sent Done Done false
5909 Acked Write 2 Sent Done Done false
5910 Acked Write 10 Sent Done Done false
5911 Acked Write 5 Sent Done Done false
5912 Acked Write 5 Sent Done Done false
5913 Acked Write 9 Sent Done Done false
5914 Acked Read 6 Sent Done Done false
5915 Acked Write 10 Sent Done Done false
5916 Acked Read 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Write 7 Sent Done Done false
5919 Acked Write 5 Sent Done Done false
5920 Acked Write 7 Sent Done Done false
5921 Acked Write 7 Sent Done Done false
5922 Acked Read 6 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Flush 0 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Read 7 Sent Done Done false
5927 Acked Write 3 Sent Done Done false
5928 Acked Write 7 Sent Done Done false
5929 Acked Write 5 Sent Done Done false
5930 Acked Read 8 Sent Done Done false
5931 Acked Read 10 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Read 1 Sent Done Done false
5934 Acked Read 3 Sent Done Done false
5935 Acked Read 6 Sent Done Done false
5936 Acked Read 9 Sent Done Done false
5937 Acked Read 2 Sent Done Done false
5938 Acked Read 9 Sent Done Done false
5939 Acked Write 5 Sent Done Done false
5940 Acked Write 8 Sent Done Done false
5941 Acked Read 10 Sent Done Done false
5942 Acked Write 2 Sent Done Done false
5943 Acked Read 3 Sent Done Done false
5944 Acked Read 5 Sent Done Done false
5945 Acked Read 2 Sent Done Done false
5946 Acked Read 6 Sent Done Done false
5947 Acked Read 3 Sent Done Done false
5948 Acked Read 9 Sent Done Done false
5949 Acked Read 4 Sent Done Done false
5950 Acked Write 7 Sent Done Done false
5951 Acked Read 10 Sent Done Done false
5952 Acked Flush 0 Sent Done Done false
5953 Acked Write 7 Sent Done Done false
5954 Acked Read 8 Sent Done Done false
5955 Acked Read 10 Sent Done Done false
5956 Acked Write 10 Sent Done Done false
5957 Acked Write 4 Sent Done Done false
5958 Acked Read 4 Sent Done Done false
5959 Acked Write 7 Sent Done Done false
5960 Acked Read 6 Sent Done Done false
5961 Acked Read 2 Sent Done Done false
5962 Acked Read 1 Sent Done Done false
5963 Acked Write 5 Sent Done Done false
5964 Acked Write 3 Sent Done Done false
5965 Acked Read 4 Sent Done Done false
5966 Acked Flush 0 Sent Done Done false
5967 Acked Write 9 Sent Done Done false
5968 Acked Write 5 Sent Done Done false
5969 Acked Read 3 Sent Done Done false
5970 Acked Read 1 Sent Done Done false
5971 Acked Write 6 Sent Done Done false
5972 Acked Read 10 Sent Done Done false
5973 Acked Read 9 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Read 3 Sent Done Done false
5976 Acked Write 9 Sent Done Done false
5977 Acked Flush 0 Sent Done Done false
5978 Acked Write 6 Sent Done Done false
5979 Acked Read 5 Sent Done Done false
5980 Acked Flush 0 Sent Done Done false
5981 Acked Read 4 Sent Done Done false
5982 Acked Read 6 Sent Done Done false
5983 Acked Write 8 Sent Done Done false
5984 Acked Write 2 Sent Done Done false
5985 Acked Write 5 Sent Done Done false
5986 Acked Write 6 Sent Done Done false
5987 Acked Flush 0 Sent Done Done false
5988 Acked Write 7 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 3 Sent Done Done false
5991 Acked Write 7 Sent Done Done false
5992 Acked Write 6 Sent Done Done false
5993 Acked Write 3 Sent Done Done false
5994 Acked Write 3 Sent Done Done false
5995 Acked Write 9 Sent Done Done false
5996 Acked Flush 0 Sent Done Done false
5997 Acked Read 3 Sent Done Done false
5998 Acked Flush 0 Sent Done Done false
5999 Acked Read 9 Sent Done Done false
6000 Acked Write 10 Sent Done Done false
6001 Acked Write 7 Sent Done Done false
6002 Acked Read 9 Sent Done Done false
6003 Acked Read 10 Sent Done Done false
6004 Acked Read 4 Sent Done Done false
6005 Acked Read 1 Sent Done Done false
6006 Acked Write 3 Sent Done Done false
6007 Acked Read 1 Sent Done Done false
6008 Acked Write 7 Sent Done Done false
6009 Acked Flush 0 Sent Done Done false
6010 Acked Write 10 Sent Done Done false
6011 Acked Read 4 Sent Done Done false
6012 Acked Read 5 Sent Done Done false
6013 Acked Write 10 Sent Done Done false
6014 Acked Read 4 Sent Done Done false
6015 Acked Flush 0 Sent Done Done false
6016 Acked Write 2 Sent Done Done false
6017 Acked Write 7 Sent Done Done false
6018 Acked Read 5 Sent Done Done false
6019 Acked Write 3 Sent Done Done false
6020 Acked Write 8 Sent Done Done false
6021 Acked Write 5 Sent Done Done false
6022 Acked Read 9 Sent Done Done false
6023 Acked Read 8 Sent Done Done false
6024 Acked Write 3 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 10 Sent Done Done false
6027 Acked Read 10 Sent Done Done false
6028 Acked Read 6 Sent Done Done false
6029 Acked Flush 0 Sent Done Done false
6030 Acked Write 10 Sent Done Done false
6031 Acked Flush 0 Sent Done Done false
6032 Acked Read 1 Sent Done Done false
6033 Acked Read 1 Sent Done Done false
6034 Acked Read 6 Sent Done Done false
6035 Acked Flush 0 Sent Done Done false
6036 Acked Read 7 Sent Done Done false
6037 Acked Read 9 Sent Done Done false
6038 Acked Read 8 Sent Done Done false
6039 Acked Write 7 Sent Done Done false
6040 Acked Read 1 Sent Done Done false
6041 Acked Write 5 Sent Done Done false
6042 Acked Read 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 5 Sent Done Done false
6045 Acked Read 9 Sent Done Done false
6046 Acked Write 7 Sent Done Done false
6047 Acked Write 6 Sent Done Done false
6048 Acked Read 4 Sent Done Done false
6049 Acked Read 3 Sent Done Done false
6050 Acked Read 2 Sent Done Done false
6051 Acked Read 7 Sent Done Done false
6052 Acked Write 6 Sent Done Done false
6053 Acked Read 3 Sent Done Done false
6054 Acked Read 7 Sent Done Done false
6055 Acked Write 9 Sent Done Done false
6056 Acked Write 10 Sent Done Done false
6057 Acked Read 9 Sent Done Done false
6058 Acked Read 7 Sent Done Done false
6059 Acked Read 1 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 10 Sent Done Done false
6062 Acked Read 9 Sent Done Done false
6063 Acked Write 2 Sent Done Done false
6064 Acked Write 2 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 1 Sent Done Done false
6067 Acked Read 5 Sent Done Done false
6068 Acked Flush 0 Sent Done Done false
6069 Acked Read 2 Sent Done Done false
6070 Acked Read 7 Sent Done Done false
6071 Acked Read 1 Sent Done Done false
6072 Acked Write 8 Sent Done Done false
6073 Acked Write 1 Sent Done Done false
6074 Acked Write 9 Sent Done Done false
6075 Acked Flush 0 Sent Done Done false
6076 Acked Read 3 Sent Done Done false
6077 Acked Read 6 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Read 3 Sent Done Done false
6080 Acked Flush 0 Sent Done Done false
6081 Acked Write 9 Sent Done Done false
6082 Acked Read 9 Sent Done Done false
6083 Acked Write 5 Sent Done Done false
6084 Acked Read 5 Sent Done Done false
6085 Acked Write 2 Sent Done Done false
6086 Acked Write 1 Sent Done Done false
6087 Acked Read 3 Sent Done Done false
6088 Acked Read 5 Sent Done Done false
6089 Acked Read 5 Sent Done Done false
6090 Acked Write 10 Sent Done Done false
6091 Acked Write 3 Sent Done Done false
6092 Acked Flush 0 Sent Done Done false
6093 Acked Read 8 Sent Done Done false
6094 Acked Read 8 Sent Done Done false
6095 Acked Read 10 Sent Done Done false
6096 Acked Read 4 Sent Done Done false
6097 Acked Read 10 Sent Done Done false
6098 Acked Write 7 Sent Done Done false
6099 Acked Read 10 Sent Done Done false
6100 Acked Write 4 Sent Done Done false
6101 Acked Write 5 Sent Done Done false
6102 Acked Read 9 Sent Done Done false
6103 Acked Write 8 Sent Done Done false
6104 Acked Read 7 Sent Done Done false
6105 Acked Read 4 Sent Done Done false
6106 Acked Flush 0 Sent Done Done false
6107 Acked Read 6 Sent Done Done false
6108 Acked Read 7 Sent Done Done false
6109 Acked Read 8 Sent Done Done false
6110 Acked Read 9 Sent Done Done false
6111 Acked Read 1 Sent Done Done false
6112 Acked Write 8 Sent Done Done false
6113 Acked Write 5 Sent Done Done false
6114 Acked Read 6 Sent Done Done false
6115 Acked Write 4 Sent Done Done false
6116 Acked Read 9 Sent Done Done false
6117 Acked Read 6 Sent Done Done false
6118 Acked Flush 0 Sent Done Done false
6119 Acked Write 5 Sent Done Done false
6120 Acked Write 3 Sent Done Done false
6121 Acked Read 8 Sent Done Done false
6122 Acked Read 7 Sent Done Done false
6123 Acked Read 7 Sent Done Done false
6124 Acked Read 1 Sent Done Done false
6125 Acked Write 7 Sent Done Done false
6126 Acked Read 6 Sent Done Done false
6127 Acked Write 3 Sent Done Done false
6128 Acked Read 8 Sent Done Done false
6129 Acked Write 5 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5828 6118 6118
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6129 6128 6127 6126 6125
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1217)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5829 Acked Read 10 Sent Done Done false
5830 Acked Write 2 Sent Done Done false
5831 Acked Read 4 Sent Done Done false
5832 Acked Read 3 Sent Done Done false
5833 Acked Read 8 Sent Done Done false
5834 Acked Write 2 Sent Done Done false
5835 Acked Read 4 Sent Done Done false
5836 Acked Write 6 Sent Done Done false
5837 Acked Read 8 Sent Done Done false
5838 Acked Write 3 Sent Done Done false
5839 Acked Flush 0 Sent Done Done false
5840 Acked Read 6 Sent Done Done false
5841 Acked Flush 0 Sent Done Done false
5842 Acked Read 4 Sent Done Done false
5843 Acked Read 3 Sent Done Done false
5844 Acked Read 7 Sent Done Done false
5845 Acked Write 4 Sent Done Done false
5846 Acked Read 6 Sent Done Done false
5847 Acked Read 8 Sent Done Done false
5848 Acked Read 6 Sent Done Done false
5849 Acked Read 3 Sent Done Done false
5850 Acked Read 5 Sent Done Done false
5851 Acked Flush 0 Sent Done Done false
5852 Acked Read 3 Sent Done Done false
5853 Acked Read 8 Sent Done Done false
5854 Acked Write 2 Sent Done Done false
5855 Acked Write 2 Sent Done Done false
5856 Acked Flush 0 Sent Done Done false
5857 Acked Write 5 Sent Done Done false
5858 Acked Read 7 Sent Done Done false
5859 Acked Flush 0 Sent Done Done false
5860 Acked Write 3 Sent Done Done false
5861 Acked Read 6 Sent Done Done false
5862 Acked Read 6 Sent Done Done false
5863 Acked Write 4 Sent Done Done false
5864 Acked Read 2 Sent Done Done false
5865 Acked Read 8 Sent Done Done false
5866 Acked Read 4 Sent Done Done false
5867 Acked Write 8 Sent Done Done false
5868 Acked Read 8 Sent Done Done false
5869 Acked Write 2 Sent Done Done false
5870 Acked Write 4 Sent Done Done false
5871 Acked Read 4 Sent Done Done false
5872 Acked Write 10 Sent Done Done false
5873 Acked Read 8 Sent Done Done false
5874 Acked Write 3 Sent Done Done false
5875 Acked Write 5 Sent Done Done false
5876 Acked Read 2 Sent Done Done false
5877 Acked Write 6 Sent Done Done false
5878 Acked Write 5 Sent Done Done false
5879 Acked Read 10 Sent Done Done false
5880 Acked Read 8 Sent Done Done false
5881 Acked Flush 0 Sent Done Done false
5882 Acked Read 8 Sent Done Done false
5883 Acked Write 8 Sent Done Done false
5884 Acked Read 10 Sent Done Done false
5885 Acked Read 5 Sent Done Done false
5886 Acked Write 10 Sent Done Done false
5887 Acked Read 4 Sent Done Done false
5888 Acked Read 6 Sent Done Done false
5889 Acked Write 5 Sent Done Done false
5890 Acked Read 2 Sent Done Done false
5891 Acked Read 4 Sent Done Done false
5892 Acked Read 5 Sent Done Done false
5893 Acked Write 1 Sent Done Done false
5894 Acked Flush 0 Sent Done Done false
5895 Acked Write 7 Sent Done Done false
5896 Acked Write 10 Sent Done Done false
5897 Acked Write 2 Sent Done Done false
5898 Acked Read 5 Sent Done Done false
5899 Acked Read 5 Sent Done Done false
5900 Acked Flush 0 Sent Done Done false
5901 Acked Read 8 Sent Done Done false
5902 Acked Write 6 Sent Done Done false
5903 Acked Write 7 Sent Done Done false
5904 Acked Read 4 Sent Done Done false
5905 Acked Read 9 Sent Done Done false
5906 Acked Write 10 Sent Done Done false
5907 Acked Read 5 Sent Done Done false
5908 Acked Read 3 Sent Done Done false
5909 Acked Write 2 Sent Done Done false
5910 Acked Write 10 Sent Done Done false
5911 Acked Write 5 Sent Done Done false
5912 Acked Write 5 Sent Done Done false
5913 Acked Write 9 Sent Done Done false
5914 Acked Read 6 Sent Done Done false
5915 Acked Write 10 Sent Done Done false
5916 Acked Read 3 Sent Done Done false
5917 Acked Write 1 Sent Done Done false
5918 Acked Write 7 Sent Done Done false
5919 Acked Write 5 Sent Done Done false
5920 Acked Write 7 Sent Done Done false
5921 Acked Write 7 Sent Done Done false
5922 Acked Read 6 Sent Done Done false
5923 Acked Read 8 Sent Done Done false
5924 Acked Flush 0 Sent Done Done false
5925 Acked Flush 0 Sent Done Done false
5926 Acked Read 7 Sent Done Done false
5927 Acked Write 3 Sent Done Done false
5928 Acked Write 7 Sent Done Done false
5929 Acked Write 5 Sent Done Done false
5930 Acked Read 8 Sent Done Done false
5931 Acked Read 10 Sent Done Done false
5932 Acked Write 3 Sent Done Done false
5933 Acked Read 1 Sent Done Done false
5934 Acked Read 3 Sent Done Done false
5935 Acked Read 6 Sent Done Done false
5936 Acked Read 9 Sent Done Done false
5937 Acked Read 2 Sent Done Done false
5938 Acked Read 9 Sent Done Done false
5939 Acked Write 5 Sent Done Done false
5940 Acked Write 8 Sent Done Done false
5941 Acked Read 10 Sent Done Done false
5942 Acked Write 2 Sent Done Done false
5943 Acked Read 3 Sent Done Done false
5944 Acked Read 5 Sent Done Done false
5945 Acked Read 2 Sent Done Done false
5946 Acked Read 6 Sent Done Done false
5947 Acked Read 3 Sent Done Done false
5948 Acked Read 9 Sent Done Done false
5949 Acked Read 4 Sent Done Done false
5950 Acked Write 7 Sent Done Done false
5951 Acked Read 10 Sent Done Done false
5952 Acked Flush 0 Sent Done Done false
5953 Acked Write 7 Sent Done Done false
5954 Acked Read 8 Sent Done Done false
5955 Acked Read 10 Sent Done Done false
5956 Acked Write 10 Sent Done Done false
5957 Acked Write 4 Sent Done Done false
5958 Acked Read 4 Sent Done Done false
5959 Acked Write 7 Sent Done Done false
5960 Acked Read 6 Sent Done Done false
5961 Acked Read 2 Sent Done Done false
5962 Acked Read 1 Sent Done Done false
5963 Acked Write 5 Sent Done Done false
5964 Acked Write 3 Sent Done Done false
5965 Acked Read 4 Sent Done Done false
5966 Acked Flush 0 Sent Done Done false
5967 Acked Write 9 Sent Done Done false
5968 Acked Write 5 Sent Done Done false
5969 Acked Read 3 Sent Done Done false
5970 Acked Read 1 Sent Done Done false
5971 Acked Write 6 Sent Done Done false
5972 Acked Read 10 Sent Done Done false
5973 Acked Read 9 Sent Done Done false
5974 Acked Write 9 Sent Done Done false
5975 Acked Read 3 Sent Done Done false
5976 Acked Write 9 Sent Done Done false
5977 Acked Flush 0 Sent Done Done false
5978 Acked Write 6 Sent Done Done false
5979 Acked Read 5 Sent Done Done false
5980 Acked Flush 0 Sent Done Done false
5981 Acked Read 4 Sent Done Done false
5982 Acked Read 6 Sent Done Done false
5983 Acked Write 8 Sent Done Done false
5984 Acked Write 2 Sent Done Done false
5985 Acked Write 5 Sent Done Done false
5986 Acked Write 6 Sent Done Done false
5987 Acked Flush 0 Sent Done Done false
5988 Acked Write 7 Sent Done Done false
5989 Acked Read 7 Sent Done Done false
5990 Acked Read 3 Sent Done Done false
5991 Acked Write 7 Sent Done Done false
5992 Acked Write 6 Sent Done Done false
5993 Acked Write 3 Sent Done Done false
5994 Acked Write 3 Sent Done Done false
5995 Acked Write 9 Sent Done Done false
5996 Acked Flush 0 Sent Done Done false
5997 Acked Read 3 Sent Done Done false
5998 Acked Flush 0 Sent Done Done false
5999 Acked Read 9 Sent Done Done false
6000 Acked Write 10 Sent Done Done false
6001 Acked Write 7 Sent Done Done false
6002 Acked Read 9 Sent Done Done false
6003 Acked Read 10 Sent Done Done false
6004 Acked Read 4 Sent Done Done false
6005 Acked Read 1 Sent Done Done false
6006 Acked Write 3 Sent Done Done false
6007 Acked Read 1 Sent Done Done false
6008 Acked Write 7 Sent Done Done false
6009 Acked Flush 0 Sent Done Done false
6010 Acked Write 10 Sent Done Done false
6011 Acked Read 4 Sent Done Done false
6012 Acked Read 5 Sent Done Done false
6013 Acked Write 10 Sent Done Done false
6014 Acked Read 4 Sent Done Done false
6015 Acked Flush 0 Sent Done Done false
6016 Acked Write 2 Sent Done Done false
6017 Acked Write 7 Sent Done Done false
6018 Acked Read 5 Sent Done Done false
6019 Acked Write 3 Sent Done Done false
6020 Acked Write 8 Sent Done Done false
6021 Acked Write 5 Sent Done Done false
6022 Acked Read 9 Sent Done Done false
6023 Acked Read 8 Sent Done Done false
6024 Acked Write 3 Sent Done Done false
6025 Acked Read 3 Sent Done Done false
6026 Acked Read 10 Sent Done Done false
6027 Acked Read 10 Sent Done Done false
6028 Acked Read 6 Sent Done Done false
6029 Acked Flush 0 Sent Done Done false
6030 Acked Write 10 Sent Done Done false
6031 Acked Flush 0 Sent Done Done false
6032 Acked Read 1 Sent Done Done false
6033 Acked Read 1 Sent Done Done false
6034 Acked Read 6 Sent Done Done false
6035 Acked Flush 0 Sent Done Done false
6036 Acked Read 7 Sent Done Done false
6037 Acked Read 9 Sent Done Done false
6038 Acked Read 8 Sent Done Done false
6039 Acked Write 7 Sent Done Done false
6040 Acked Read 1 Sent Done Done false
6041 Acked Write 5 Sent Done Done false
6042 Acked Read 4 Sent Done Done false
6043 Acked Write 9 Sent Done Done false
6044 Acked Write 5 Sent Done Done false
6045 Acked Read 9 Sent Done Done false
6046 Acked Write 7 Sent Done Done false
6047 Acked Write 6 Sent Done Done false
6048 Acked Read 4 Sent Done Done false
6049 Acked Read 3 Sent Done Done false
6050 Acked Read 2 Sent Done Done false
6051 Acked Read 7 Sent Done Done false
6052 Acked Write 6 Sent Done Done false
6053 Acked Read 3 Sent Done Done false
6054 Acked Read 7 Sent Done Done false
6055 Acked Write 9 Sent Done Done false
6056 Acked Write 10 Sent Done Done false
6057 Acked Read 9 Sent Done Done false
6058 Acked Read 7 Sent Done Done false
6059 Acked Read 1 Sent Done Done false
6060 Acked Read 5 Sent Done Done false
6061 Acked Read 10 Sent Done Done false
6062 Acked Read 9 Sent Done Done false
6063 Acked Write 2 Sent Done Done false
6064 Acked Write 2 Sent Done Done false
6065 Acked Read 4 Sent Done Done false
6066 Acked Read 1 Sent Done Done false
6067 Acked Read 5 Sent Done Done false
6068 Acked Flush 0 Sent Done Done false
6069 Acked Read 2 Sent Done Done false
6070 Acked Read 7 Sent Done Done false
6071 Acked Read 1 Sent Done Done false
6072 Acked Write 8 Sent Done Done false
6073 Acked Write 1 Sent Done Done false
6074 Acked Write 9 Sent Done Done false
6075 Acked Flush 0 Sent Done Done false
6076 Acked Read 3 Sent Done Done false
6077 Acked Read 6 Sent Done Done false
6078 Acked Write 4 Sent Done Done false
6079 Acked Read 3 Sent Done Done false
6080 Acked Flush 0 Sent Done Done false
6081 Acked Write 9 Sent Done Done false
6082 Acked Read 9 Sent Done Done false
6083 Acked Write 5 Sent Done Done false
6084 Acked Read 5 Sent Done Done false
6085 Acked Write 2 Sent Done Done false
6086 Acked Write 1 Sent Done Done false
6087 Acked Read 3 Sent Done Done false
6088 Acked Read 5 Sent Done Done false
6089 Acked Read 5 Sent Done Done false
6090 Acked Write 10 Sent Done Done false
6091 Acked Write 3 Sent Done Done false
6092 Acked Flush 0 Sent Done Done false
6093 Acked Read 8 Sent Done Done false
6094 Acked Read 8 Sent Done Done false
6095 Acked Read 10 Sent Done Done false
6096 Acked Read 4 Sent Done Done false
6097 Acked Read 10 Sent Done Done false
6098 Acked Write 7 Sent Done Done false
6099 Acked Read 10 Sent Done Done false
6100 Acked Write 4 Sent Done Done false
6101 Acked Write 5 Sent Done Done false
6102 Acked Read 9 Sent Done Done false
6103 Acked Write 8 Sent Done Done false
6104 Acked Read 7 Sent Done Done false
6105 Acked Read 4 Sent Done Done false
6106 Acked Flush 0 Sent Done Done false
6107 Acked Read 6 Sent Done Done false
6108 Acked Read 7 Sent Done Done false
6109 Acked Read 8 Sent Done Done false
6110 Acked Read 9 Sent Done Done false
6111 Acked Read 1 Sent Done Done false
6112 Acked Write 8 Sent Done Done false
6113 Acked Write 5 Sent Done Done false
6114 Acked Read 6 Sent Done Done false
6115 Acked Write 4 Sent Done Done false
6116 Acked Read 9 Sent Done Done false
6117 Acked Read 6 Sent Done Done false
6118 Acked Flush 0 Sent Done Done false
6119 Acked Write 5 Sent Done Done false
6120 Acked Write 3 Sent Done Done false
6121 Acked Read 8 Sent Done Done false
6122 Acked Read 7 Sent Done Done false
6123 Acked Read 7 Sent Done Done false
6124 Acked Read 1 Sent Done Done false
6125 Acked Write 7 Sent Done Done false
6126 Acked Read 6 Sent Done Done false
6127 Acked Write 3 Sent Done Done false
6128 Acked Read 8 Sent Done Done false
6129 Acked Write 5 Sent Done Done false
6130 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5828 6130 6130
Downstairs last five completed: 5828 5827 5826 5825 5824
Upstairs last five completed: 6130 6129 6128 6127 6126
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.871948161Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43221)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.872238777Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.873474922Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874022808Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874049025Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 5828","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874065923Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 5828","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874275969Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 5828","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874297786Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 5828","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.874534249Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6130 6129 6128 6127 6126
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:40:17.124 [8830] Got stop action so:false kr:false
INFO request completed, latency_us: 351, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 38fb13ac-8180-4a7d-b677-e6b7b14fb26b, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:17.12533907Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:40:17.125 INFO request completed, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:17.125397294Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
latency_us: 123, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:17.125443838Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
response_code: 200, uri: /state/cid/2, method{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:17.125486983Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
: GET, req_id: eedd0820-c5ca-478b-a451-edaca50510ea, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:40:21.127 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7b74da59-e323-45f1-8f0e-609b6fdb7b7c, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 1122 len 16384 data: 8 9 8 6
002/300 Write block 2016 len 32768 data: 5 5 5 4 4 4 4 4
003/300 Write block 2484 len 20480 data: 4 3 3 3 4
004/300 Read block 1218 len 12288
005/300 Write block 2660 len 12288 data: 4 4 3
006/300 Read block 4581 len 32768
007/300 Write block 3430 len 40960 data: 8 7 6 6 6 5 5 5 5 4
008/300 Write block 4280 len 24576 data: 6 5 4 4 5 4
009/300 Write block 1580 len 4096 data: 4
010/300 Write block 3052 len 40960 data: 7 6 7 6 5 4 4 4 5 6
011/300 Flush
012/300 Read block 2153 len 32768
013/300 Write block 3275 len 12288 data: 5 5 6
014/300 Write block 1761 len 40960 data: 3 3 3 4 4 4 4 4 4 5
015/300 Write block 2856 len 8192 data: 5 5
016/300 Write block 2156 len 24576 data: 6 7 7 7 6 7
017/300 Write block 1145 len 24576 data: 3 3 4 5 5 3
018/300 Write block 2182 len 28672 data: 7 6 7 5 6 5 2
019/300 Read block 4644 len 4096
020/300 Flush
021/300 Write block 3775 len 24576 data: 8 7 7 6 5 4
022/300 Write block 3809 len 8192 data: 4 4
023/300 Read block 3920 len 28672
024/300 Read block 490 len 4096
025/300 Write block 2252 len 20480 data: 3 3 4 6 7
026/300 Write block 1736 len 40960 data: 4 4 5 6 7 7 5 5 5 4
027/300 Write block 4062 len 40960 data: 4 4 3 3 3 3 3 3 3 3
028/300 Read block 908 len 40960
029/300 Flush
030/300 Read block 2581 len 12288
031/300 Flush
032/300 Read block 3448 len 40960
033/300 Write block 3442 len 24576 data: 3 3 2 2 2 2
034/300 Read block 2065 len 28672
035/300 Read block 4272 len 4096
036/300 Read block 4578 len 16384
037/300 Flush
038/300 Read block 3450 len 36864
039/300 Write block 3553 len 20480 data: 3 3 3 5 5
040/300 Write block 3120 len 40960 data: 3 3 4 5 5 4 3 3 3 5
041/300 Write block 4775 len 16384 data: 6 6 5 4
042/300 Write block 1860 len 16384 data: 7 6 6 4
043/300 Write block 3617 len 12288 data: 4 6 6
044/300 Write block 4505 len 16384 data: 4 4 4 3
045/300 Write block 4110 len 16384 data: 5 6 6 6
046/300 Read block 1472 len 12288
047/300 Write block 214 len 24576 data: 3 3 2 3 4 4
048/300 Read block 767 len 20480
049/300 Read block 3736 len 32768
050/300 Write block 349 len 36864 data: 2 2 4 7 7 7 6 7 6
051/300 Read block 2635 len 32768
052/300 Write block 445 len 16384 data: 3 2 2 2
053/300 Flush
054/300 Flush
055/300 Write block 4797 len 36864 data: 5 4 5 5 5 4 5 5 8
056/300 Read block 598 len 8192
057/300 Read block 2734 len 24576
058/300 Read block 345 len 12288
059/300 Write block 1258 len 24576 data: 3 2 2 2 2 3
060/300 Flush
061/300 Write block 199 len 4096 data: 7
062/300 Write block 2405 len 16384 data: 3 3 4 4
063/300 Flush
064/300 Read block 3338 len 16384
065/300 Write block 848 len 32768 data: 3 3 3 3 2 2 2 2
066/300 Write block 3019 len 4096 data: 3
067/300 Read block 2264 len 28672
068/300 Read block 2276 len 24576
069/300 Write block 51 len 4096 data: 5
070/300 Write block 878 len 20480 data: 3 3 2 2 2
071/300 Flush
072/300 Read block 1331 len 12288
073/300 Write block 670 len 40960 data: 3 3 3 3 3 3 4 4 4 3
074/300 Read block 772 len 40960
075/300 Write block 3600 len 12288 data: 5 3 3
076/300 Read block 2003 len 32768
077/300 Read block 1752 len 36864
078/300 Write block 1529 len 12288 data: 3 4 5
079/300 Write block 859 len 4096 data: 4
080/300 Read block 2972 len 36864
081/300 Write block 2597 len 28672 data: 3 4 5 6 5 4 4
082/300 Flush
083/300 Write block 4879 len 36864 data: 3 3 3 4 4 4 3 4 5
084/300 Read block 2534 len 20480
085/300 Read block 535 len 32768
086/300 Write block 2701 len 28672 data: 3 4 4 3 2 2 2
087/300 Read block 1471 len 40960
088/300 Read block 495 len 40960
089/300 Read block 2214 len 36864
090/300 Read block 4310 len 8192
091/300 Read block 2598 len 36864
092/300 Write block 3184 len 4096 data: 4
093/300 Write block 1063 len 16384 data: 2 2 3 3
094/300 Read block 2311 len 32768
095/300 Read block 1902 len 40960
096/300 Read block 2117 len 4096
097/300 Flush
098/300 Read block 313 len 16384
099/300 Write block 3046 len 32768 data: 5 4 5 5 6 8 8 7
100/300 Flush
101/300 Write block 4826 len 8192 data: 3 3
102/300 Read block 1115 len 32768
103/300 Read block 4032 len 40960
104/300 Flush
105/300 Read block 4829 len 8192
106/300 Write block 2410 len 28672 data: 6 5 6 6 7 6 6
107/300 Read block 4910 len 20480
108/300 Write block 2411 len 20480 data: 6 7 7 8 7
109/300 Read block 4072 len 28672
110/300 Read block 588 len 24576
111/300 Write block 3186 len 28672 data: 2 3 3 3 4 5 5
112/300 Flush
113/300 Read block 408 len 16384
114/300 Read block 3286 len 28672
115/300 Flush
116/300 Write block 687 len 28672 data: 3 3 4 3 3 3 3
117/300 Write block 1444 len 20480 data: 3 3 4 4 5
118/300 Write block 3228 len 40960 data: 3 3 4 4 4 4 4 5 3 3
119/300 Write block 2368 len 28672 data: 3 4 4 5 5 5 4
120/300 Write block 1592 len 36864 data: 6 6 5 5 6 7 7 4 4
121/300 Write block 1015 len 12288 data: 4 3 5
122/300 Read block 1335 len 40960
123/300 Read block 2812 len 36864
124/300 Write block 682 len 36864 data: 4 5 5 4 4 4 4 5 4
125/300 Read block 172 len 16384
126/300 Write block 2053 len 20480 data: 3 3 3 3 3
127/300 Read block 1568 len 28672
128/300 Read block 1488 len 40960
129/300 Read block 2356 len 32768
130/300 Flush
131/300 Write block 2925 len 12288 data: 3 3 4
132/300 Read block 341 len 16384
133/300 Write block 1323 len 32768 data: 3 3 3 4 4 4 5 4
134/300 Write block 4475 len 12288 data: 7 7 7
135/300 Read block 3052 len 32768
136/300 Flush
137/300 Read block 2520 len 20480
138/300 Read block 2438 len 28672
139/300 Read block 3247 len 32768
140/300 Flush
141/300 Read block 381 len 36864
142/300 Read block 4084 len 4096
143/300 Write block 2298 len 12288 data: 3 5 5
144/300 Read block 2168 len 8192
145/300 Write block 462 len 4096 data: 5
146/300 Write block 3331 len 8192 data: 6 7
147/300 Read block 3915 len 8192
148/300 Flush
149/300 Flush
150/300 Read block 3630 len 8192
151/300 Read block 355 len 4096
152/300 Write block 1353 len 12288 data: 6 5 6
153/300 Read block 4623 len 20480
154/300 Write block 226 len 16384 data: 7 5 5 5
155/300 Flush
156/300 Write block 3439 len 28672 data: 5 3 3 4 4 3 3
157/300 Write block 685 len 16384 data: 5 5 5 5
158/300 Write block 3535 len 36864 data: 2 2 3 3 4 4 3 4 3
159/300 Write block 4639 len 24576 data: 4 4 4 3 4 3
160/300 Write block 4335 len 8192 data: 7 9
161/300 Read block 764 len 40960
162/300 Write block 1700 len 32768 data: 3 3 3 3 3 3 3 2
163/300 Read block 27 len 28672
164/300 Read block 3785 len 36864
165/300 Write block 633 len 8192 data: 3 3
166/300 Write block 2780 len 40960 data: 3 3 4 4 4 4 3 2 3 3
167/300 Read block 4954 len 4096
168/300 Read block 1456 len 40960
169/300 Read block 3311 len 8192
170/300 Read block 1994 len 36864
171/300 Read block 1628 len 36864
172/300 Write block 2973 len 8192 data: 6 6
173/300 Write block 1513 len 20480 data: 2 2 3 3 3
174/300 Write block 2339 len 24576 data: 5 6 6 5 5 5
175/300 Flush
176/300 Write block 305 len 12288 data: 4 4 4
177/300 Read block 1227 len 8192
178/300 Read block 4261 len 20480
179/300 Read block 350 len 12288
180/300 Write block 2627 len 16384 data: 3 3 4 3
181/300 Write block 2765 len 32768 data: 5 6 7 6 5 5 5 5
182/300 Write block 1719 len 40960 data: 4 4 4 4 4 3 3 4 4 4
183/300 Read block 1214 len 32768
184/300 Read block 3576 len 12288
185/300 Write block 4797 len 28672 data: 6 5 6 6 6 5 6
186/300 Read block 4366 len 32768
187/300 Write block 578 len 40960 data: 5 5 6 6 6 8 5 4 3 4
188/300 Flush
189/300 Read block 2623 len 28672
190/300 Read block 2196 len 36864
191/300 Write block 4939 len 4096 data: 5
192/300 Read block 4886 len 20480
193/300 Write block 3372 len 32768 data: 8 6 5 4 4 3 3 3
194/300 Read block 4635 len 4096
195/300 Read block 2413 len 24576
196/300 Read block 1092 len 32768
197/300 Write block 2772 len 32768 data: 6 6 5 4 3 3 3 4
198/300 Read block 151 len 16384
199/300 Read block 1149 len 24576
200/300 Read block 1696 len 24576
201/300 Read block 299 len 16384
202/300 Read block 3003 len 20480
203/300 Write block 2782 len 40960 data: 5 5 5 5 4 3 4 4 4 4
204/300 Write block 3744 len 24576 data: 3 3 3 4 5 5
205/300 Flush
206/300 Read block 2528 len 36864
207/300 Write block 361 len 28672 data: 4 4 3 5 4 4 4
208/300 Write block 2678 len 28672 data: 7 7 8 8 8 9 9
209/300 Read block 2126 len 20480
210/300 Read block 3568 len 4096
211/300 Write block 1093 len 12288 data: 2 2 3
212/300 Read block 2837 len 16384
213/300 Write block 866 len 8192 data: 7 7
214/300 Write block 4433 len 16384 data: 3 3 4 3
215/300 Read block 3385 len 28672
216/300 Write block 131 len 24576 data: 7 6 5 4 4 4
217/300 Read block 3235 len 40960
218/300 Flush
219/300 Flush
220/300 Read block 2128 len 8192
221/300 Read block 1412 len 36864
222/300 Read block 1550 len 4096
223/300 Read block 4105 len 24576
224/300 Write block 4260 len 36864 data: 5 4 4 5 6 6 6 5 5
225/300 Write block 1924 len 28672 data: 5 5 4 3 3 3 3
226/300 Read block 1582 len 40960
227/300 Read block 1697 len 20480
228/300 Read block 2866 len 36864
229/300 Write block 4738 len 28672 data: 5 5 6 6 5 4 4
230/300 Read block 2146 len 16384
231/300 Write block 414 len 12288 data: 3 3 3
232/300 Write block 511 len 16384 data: 6 4 4 4
233/300 Write block 2653 len 16384 data: 6 6 6 6
234/300 Write block 1669 len 32768 data: 3 3 3 4 5 7 6 6
235/300 Read block 3484 len 8192
236/300 Read block 1262 len 24576
237/300 Read block 945 len 20480
238/300 Read block 4984 len 36864
239/300 Write block 2404 len 20480 data: 3 4 4 5 5
240/300 Write block 4158 len 40960 data: 4 4 5 7 7 7 7 7 6 5
241/300 Read block 1203 len 24576
242/300 Read block 4958 len 28672
243/300 Read block 3229 len 12288
244/300 Write block 4708 len 16384 data: 6 5 5 4
245/300 Read block 3813 len 32768
246/300 Write block 2282 len 8192 data: 3 4
247/300 Write block 590 len 4096 data: 6
248/300 Flush
249/300 Write block 4119 len 36864 data: 9 8 7 7 5 4 4 4 5
250/300 Read block 3740 len 28672
251/300 Write block 3360 len 32768 data: 4 4 4 3 4 4 4 6
252/300 Read block 3598 len 36864
253/300 Write block 1726 len 8192 data: 5 5
254/300 Read block 1112 len 12288
255/300 Read block 2988 len 20480
256/300 Read block 1197 len 28672
257/300 Read block 501 len 24576
258/300 Read block 4897 len 40960
259/300 Read block 4970 len 28672
260/300 Read block 4513 len 4096
261/300 Write block 1792 len 40960 data: 3 3 4 4 3 3 2 2 2 2
262/300 Read block 2400 len 36864
263/300 Read block 1029 len 24576
264/300 Read block 3466 len 36864
265/300 Write block 653 len 24576 data: 5 4 4 3 3 6
266/300 Flush
267/300 Read block 3227 len 12288
268/300 Read block 147 len 20480
269/300 Write block 4562 len 32768 data: 6 7 8 6 5 5 6 4
270/300 Read block 4440 len 32768
271/300 Write block 1883 len 32768 data: 4 4 4 6 5 6 7 6
272/300 Read block 3469 len 8192
273/300 Write block 2762 len 28672 data: 2 3 4 6 7 8 7
274/300 Write block 4603 len 36864 data: 5 3 3 3 4 4 4 5 4
275/300 Read block 420 len 12288
276/300 Read block 756 len 12288
277/300 Read block 1095 len 16384
278/300 Read block 1001 len 8192
279/300 Write block 4764 len 12288 data: 6 6 5
280/300 Write block 3200 len 24576 data: 3 3 3 2 2 2
281/300 Write block 1075 len 4096 data: 5
282/300 Write block 221 len 28672 data: 4 6 6 6 6 8 6
283/300 Read block 1455 len 20480
284/300 Read block 742 len 16384
285/300 Write block 4706 len 28672 data: 5 5 7 6 6 5 5
286/300 Write block 4786 len 28672 data: 4 4 4 5 7 7 6
287/300 Flush
288/300 Flush
289/300 Write block 1851 len 40960 data: 5 5 5 7 8 7 6 7 8 8
290/300 Write block 4803 len 24576 data: 7 6 9 5 6 6
291/300 Read block 4044 len 32768
292/300 Read block 3309 len 4096
293/300 Read block 1403 len 32768
294/300 Write block 2171 len 12288 data: 5 5 6
295/300 Write block 2370 len 8192 data: 5 6
296/300 Write block 2240 len 36864 data: 5 5 5 5 5 6 6 7 6
297/300 Flush
298/300 Read block 749 len 32768
299/300 Read block 1548 len 12288
300/300 Write block 3653 len 12288 data: 3 3 3
301/300 Read block 1950 len 24576
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:40:21.362 INFO request completed, latency_us: 154, response_code: 204, uri: /start/cid/2, method: POST, req_id: a97192fc-13d6-4f64-8383-5d4d98761464, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Write 4 Done Done Sent false
6132 Acked Write 8 Done Done Sent false
6133 Acked Write 5 Done Done Sent false
6134 Acked Read 3 Done Done Sent false
6135 Acked Write 3 Done Done Sent false
6136 Acked Read 8 Done Done Sent false
6137 Acked Write 10 Done Done Sent false
6138 Acked Write 6 Done Done Sent false
6139 Acked Write 1 Done Done Sent false
6140 Acked Write 10 Done Done Sent false
6141 Acked Flush 0 Done Done Sent false
6142 Acked Read 8 Done Done Sent false
6143 Acked Write 3 Done Done Sent false
6144 Acked Write 10 Done Done Sent false
6145 Acked Write 2 Done Done Sent false
6146 Acked Write 6 Done Done Sent false
6147 Acked Write 6 Done Done Sent false
6148 Acked Write 7 Done Done Sent false
6149 Acked Read 1 Done Done Sent false
6150 Acked Flush 0 Done Done Sent false
6151 Acked Write 6 Done Done Sent false
6152 Acked Write 2 Done Done Sent false
6153 Acked Read 7 Done Done Sent false
6154 Acked Read 1 Done Done Sent false
6155 Acked Write 5 Done Done Sent false
6156 Acked Write 10 Done Done Sent false
6157 Acked Write 10 Done Done Sent false
6158 Acked Read 10 Done Done Sent false
6159 Acked Flush 0 Done Done Sent false
6160 Acked Read 3 Done Done Sent false
6161 Acked Flush 0 Done Done Sent false
6162 Acked Read 10 Done Done Sent false
6163 Acked Write 6 Done Done Sent false
6164 Acked Read 7 Done Done Sent false
6165 Acked Read 1 Done Done Sent false
6166 Acked Read 4 Done Done Sent false
6167 Acked Flush 0 Done Done Sent false
6168 Acked Read 9 Done Done Sent false
6169 Acked Write 5 Done Done Sent false
6170 Acked Write 10 Done Done Sent false
6171 Acked Write 4 Done Done Sent false
6172 Acked Write 4 Done Done Sent false
6173 Acked Write 3 Done Done Sent false
6174 Acked Write 4 Done Done Sent false
6175 Acked Write 4 Done Done Sent false
6176 Acked Read 3 Done Done Sent false
6177 Acked Write 6 Done Done Sent false
6178 Acked Read 5 Done Done Sent false
6179 Acked Read 8 Done Done Sent false
6180 Acked Write 9 Done Done Sent false
6181 Acked Read 8 Done Done Sent false
6182 Acked Write 4 Done Done Sent false
6183 Acked Flush 0 Done Done Sent false
6184 Acked Flush 0 Done Done Sent false
6185 Acked Write 9 Done Done Sent false
6186 Acked Read 2 Done Done Sent false
6187 Acked Read 6 Done Done Sent false
6188 Acked Read 3 Done Done Sent false
6189 Acked Write 6 Done Done Sent false
6190 Acked Flush 0 Done Done Sent false
6191 Acked Write 1 Done Done Sent false
6192 Acked Write 4 Done Done Sent false
6193 Acked Flush 0 Done Done Sent false
6194 Acked Read 4 Done Done Sent false
6195 Acked Write 8 Done Done Sent false
6196 Acked Write 1 Done Done Sent false
6197 Acked Read 7 Done Done Sent false
6198 Acked Read 6 Done Done Sent false
6199 Acked Write 1 Done Done Sent false
6200 Acked Write 5 Done Done Sent false
6201 Acked Flush 0 Done Done Sent false
6202 Acked Read 3 Done Done Sent false
6203 Acked Write 10 Done Done Sent false
6204 Acked Read 10 Done Done Sent false
6205 Acked Write 3 Done Done Sent false
6206 Acked Read 8 Done Done Sent false
6207 Acked Read 9 Done Done Sent false
6208 Acked Write 3 Done Done Sent false
6209 Acked Write 1 Done Done Sent false
6210 Acked Read 9 Done Done Sent false
6211 Acked Write 7 Done Done Sent false
6212 Acked Flush 0 Done Done Sent false
6213 Acked Write 9 Done Done Sent false
6214 Acked Read 5 Done Done Sent false
6215 Acked Read 8 Done Done Sent false
6216 Acked Write 7 Done Done Sent false
6217 Acked Read 10 Done Done Sent false
6218 Acked Read 10 Done Done Sent false
6219 Acked Read 9 Done Done Sent false
6220 Acked Read 2 Done Done Sent false
6221 Acked Read 9 Done Done Sent false
6222 Acked Write 1 Done Done Sent false
6223 Acked Write 4 Done Done Sent false
6224 Acked Read 8 Done Done Sent false
6225 Acked Read 10 Done Done Sent false
6226 Acked Read 1 Done Done Sent false
6227 Acked Flush 0 Done Done Sent false
6228 Acked Read 4 Done Done Sent false
6229 Acked Write 8 Done Done Sent false
6230 Acked Flush 0 Done Done Sent false
6231 Acked Write 2 Done Done Sent false
6232 Acked Read 8 Done Done Sent false
6233 Acked Read 10 Done Done Sent false
6234 Acked Flush 0 Done Done Sent false
6235 Acked Read 2 Done Done Sent false
6236 Acked Write 7 Done Done Sent false
6237 Acked Read 5 Done Done Sent false
6238 Acked Write 5 Done Done Sent false
6239 Acked Read 7 Done Done Sent false
6240 Acked Read 6 Done Done Sent false
6241 Acked Write 7 Done Done Sent false
6242 Acked Flush 0 Done Done Sent false
6243 Acked Read 4 Done Done Sent false
6244 Acked Read 7 Done Done Sent false
6245 Acked Flush 0 Done Done Sent false
6246 Acked Write 7 Done Done Sent false
6247 Acked Write 5 Done Done Sent false
6248 Acked Write 10 Done Done Sent false
6249 Acked Write 7 Done Done Sent false
6250 Acked Write 9 Done Done Sent false
6251 Acked Write 3 Done Done Sent false
6252 Acked Read 10 Done Done Sent false
6253 Acked Read 9 Done Done Sent false
6254 Acked Write 9 Done Done Sent false
6255 Acked Read 4 Done Done Sent false
6256 Acked Write 5 Done Done Sent false
6257 Acked Read 7 Done Done Sent false
6258 Acked Read 10 Done Done Sent false
6259 Acked Read 8 Done Done Sent false
6260 Acked Flush 0 Done Done Sent false
6261 Acked Write 3 Done Done Sent false
6262 Acked Read 4 Done Done Sent false
6263 Acked Write 8 Done Done Sent false
6264 Acked Write 3 Done Done Sent false
6265 Acked Read 8 Done Done Sent false
6266 Acked Flush 0 Done Done Sent false
6267 Acked Read 5 Done Done Sent false
6268 Acked Read 7 Done Done Sent false
6269 Acked Read 8 Done Done Sent false
6270 Acked Flush 0 Done Done Sent false
6271 Acked Read 9 Done Done Sent false
6272 Acked Read 1 Done Done Sent false
6273 Acked Write 3 Done Done Sent false
6274 Acked Read 2 Done Done Sent false
6275 Acked Write 1 Done Done Sent false
6276 Acked Write 2 Done Done Sent false
6277 Acked Read 2 Done Done Sent false
6278 Acked Flush 0 Done Done Sent false
6279 Acked Flush 0 Done Done Sent false
6280 Acked Read 2 Done Done Sent false
6281 Acked Read 1 Done Done Sent false
6282 Acked Write 3 Done Done Sent false
6283 Acked Read 5 Done Done Sent false
6284 Acked Write 4 Done Done Sent false
6285 Acked Flush 0 Done Done Sent false
6286 Acked Write 7 Done Done Sent false
6287 Acked Write 4 Done Done Sent false
6288 Acked Write 9 Done Done Sent false
6289 Acked Write 6 Done Done Sent false
6290 Acked Write 2 Done Done Sent false
6291 Acked Read 10 Done Done Sent false
6292 Acked Write 8 Done Done Sent false
6293 Acked Read 7 Done Done Sent false
6294 Acked Read 9 Done Done Sent false
6295 Acked Write 2 Done Done Sent false
6296 Acked Write 10 Done Done Sent false
6297 Acked Read 1 Done Done Sent false
6298 Acked Read 10 Done Done Sent false
6299 Acked Read 2 Done Done Sent false
6300 Acked Read 9 Done Done Sent false
6301 Acked Read 9 Done Done Sent false
6302 Acked Write 2 Done Done Sent false
6303 Acked Write 5 Done Done Sent false
6304 Acked Write 6 Done Done Sent false
6305 Acked Flush 0 Done Done Sent false
6306 Acked Write 3 Done Done Sent false
6307 Acked Read 2 Done Done Sent false
6308 Acked Read 5 Done Done Sent false
6309 Acked Read 3 Done Done Sent false
6310 Acked Write 4 Done Done Sent false
6311 Acked Write 8 Done Done Sent false
6312 Acked Write 10 Done Done Sent false
6313 Acked Read 8 Done Done Sent false
6314 Acked Read 3 Done Done Sent false
6315 Acked Write 7 Done Done Sent false
6316 Acked Read 8 Done Done Sent false
6317 Acked Write 10 Done Done Sent false
6318 Acked Flush 0 Done Done Sent false
6319 Acked Read 7 Done Done Sent false
6320 Acked Read 9 Done Done Sent false
6321 Acked Write 1 Done Done Sent false
6322 Acked Read 5 Done Done Sent false
6323 Acked Write 8 Done Done Sent false
6324 Acked Read 1 Done Done Sent false
6325 Acked Read 6 Done Done Sent false
6326 Acked Read 8 Done Done Sent false
6327 Acked Write 8 Done Done Sent false
6328 Acked Read 4 Done Done Sent false
6329 Acked Read 6 Done Done Sent false
6330 Acked Read 6 Done Done Sent false
6331 Acked Read 4 Done Done Sent false
6332 Acked Read 5 Done Done Sent false
6333 Acked Write 10 Done Done Sent false
6334 Acked Write 6 Done Done Sent false
6335 Acked Flush 0 Done Done Sent false
6336 Acked Read 9 Done Done Sent false
6337 Acked Write 7 Done Done Sent false
6338 Acked Write 7 Done Done Sent false
6339 Acked Read 5 Done Done Sent false
6340 Acked Read 1 Done Done Sent false
6341 Acked Write 3 Done Done Sent false
6342 Acked Read 4 Done Done Sent false
6343 Acked Write 2 Done Done Sent false
6344 Acked Write 4 Done Done Sent false
6345 Acked Read 7 Done Done Sent false
6346 Acked Write 6 Done Done Sent false
6347 Acked Read 10 Done Done Sent false
6348 Acked Flush 0 Done Done Sent false
6349 Acked Flush 0 Done Done Sent false
6350 Acked Read 2 Done Done Sent false
6351 Acked Read 9 Done Done Sent false
6352 Acked Read 1 Done Done Sent false
6353 Acked Read 6 Done Done Sent false
6354 Acked Write 9 Done Done Sent false
6355 Acked Write 7 Done Done Sent false
6356 Acked Read 10 Done Done Sent false
6357 Acked Read 5 Done Done Sent false
6358 Acked Read 9 Done Done Sent false
6359 Acked Write 7 Done Done Sent false
6360 Acked Read 4 Done Done Sent false
6361 Acked Write 3 Done Done Sent false
6362 Acked Write 4 Done Done Sent false
6363 Acked Write 4 Done Done Sent false
6364 Acked Write 8 Done Done Sent false
6365 Acked Read 2 Done Done Sent false
6366 Acked Read 6 Done Done Sent false
6367 Acked Read 5 Done Done Sent false
6368 Acked Read 9 Done Done Sent false
6369 Acked Write 5 Done Done Sent false
6370 Acked Write 10 Done Done Sent false
6371 Acked Read 6 Done Done Sent false
6372 Acked Read 7 Done Done Sent false
6373 Acked Read 3 Done Done Sent false
6374 Acked Write 4 Done Done Sent false
6375 Acked Read 8 Done Done Sent false
6376 Acked Write 2 Done Done Sent false
6377 Acked Write 1 Done Done Sent false
6378 Acked Flush 0 Done Done Sent false
6379 Acked Write 9 Done Done Sent false
6380 Acked Read 7 Done Done Sent false
6381 Acked Write 8 Done Done Sent false
6382 Acked Read 9 Done Done Sent false
6383 Acked Write 2 Done Done Sent false
6384 Acked Read 3 Done Done Sent false
6385 Acked Read 5 Done Done Sent false
6386 Acked Read 7 Done Done Sent false
6387 Acked Read 6 Done Done Sent false
6388 Acked Read 10 Done Done Sent false
6389 Acked Read 7 Done Done Sent false
6390 Acked Read 1 Done Done Sent false
6391 Acked Write 10 Done Done Sent false
6392 Acked Read 9 Done Done Sent false
6393 Acked Read 6 Done Done Sent false
6394 Acked Read 9 Done Done Sent false
6395 Acked Write 6 Done Done Sent false
6396 Acked Flush 0 Done Done Sent false
6397 Acked Read 3 Done Done Sent false
6398 Acked Read 5 Done Done Sent false
6399 Acked Write 8 Done Done Sent false
6400 Acked Read 8 Done Done Sent false
6401 Acked Write 8 Done Done Sent false
6402 Acked Read 2 Done Done Sent false
6403 Acked Write 7 Done Done Sent false
6404 Acked Write 9 Done Done Sent false
6405 Acked Read 3 Done Done Sent false
6406 Acked Read 3 Done Done Sent false
6407 Acked Read 4 Done Done Sent false
6408 Acked Read 2 Done Done Sent false
6409 Acked Write 3 Done Done Sent false
6410 Acked Write 6 Done Done Sent false
6411 Acked Write 1 Done Done Sent false
6412 Acked Write 7 Done Done Sent false
6413 Acked Read 5 Done Done Sent false
6414 Acked Read 4 Done Done Sent false
6415 Acked Write 7 Done Done Sent false
6416 Acked Write 7 Done Done Sent false
6417 Acked Flush 0 Done Done Sent false
6418 Acked Flush 0 Done Done Sent false
6419 Acked Write 10 Done Done Sent false
6420 Acked Write 6 Done Done Sent false
6421 Acked Read 8 Done Done Sent false
6422 Acked Read 1 Done Done Sent false
6423 Acked Read 8 Done Done Sent false
6424 Acked Write 3 Done Done Sent false
6425 Acked Write 2 Done Done Sent false
6426 Acked Write 9 Done Done Sent false
6427 Acked Flush 0 Done Done Sent false
6428 Acked Read 8 Done Done Sent false
6429 Acked Read 3 Done Done Sent false
6430 Acked Write 3 Done Done Sent false
6431 Acked Read 6 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6427 6427 6130
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6431 6430 6429 6428 6427
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Write 4 Done Done Sent false
6132 Acked Write 8 Done Done Sent false
6133 Acked Write 5 Done Done Sent false
6134 Acked Read 3 Done Done Sent false
6135 Acked Write 3 Done Done Sent false
6136 Acked Read 8 Done Done Sent false
6137 Acked Write 10 Done Done Sent false
6138 Acked Write 6 Done Done Sent false
6139 Acked Write 1 Done Done Sent false
6140 Acked Write 10 Done Done Sent false
6141 Acked Flush 0 Done Done Sent false
6142 Acked Read 8 Done Done Sent false
6143 Acked Write 3 Done Done Sent false
6144 Acked Write 10 Done Done Sent false
6145 Acked Write 2 Done Done Sent false
6146 Acked Write 6 Done Done Sent false
6147 Acked Write 6 Done Done Sent false
6148 Acked Write 7 Done Done Sent false
6149 Acked Read 1 Done Done Sent false
6150 Acked Flush 0 Done Done Sent false
6151 Acked Write 6 Done Done Sent false
6152 Acked Write 2 Done Done Sent false
6153 Acked Read 7 Done Done Sent false
6154 Acked Read 1 Done Done Sent false
6155 Acked Write 5 Done Done Sent false
6156 Acked Write 10 Done Done Sent false
6157 Acked Write 10 Done Done Sent false
6158 Acked Read 10 Done Done Sent false
6159 Acked Flush 0 Done Done Sent false
6160 Acked Read 3 Done Done Sent false
6161 Acked Flush 0 Done Done Sent false
6162 Acked Read 10 Done Done Sent false
6163 Acked Write 6 Done Done Sent false
6164 Acked Read 7 Done Done Sent false
6165 Acked Read 1 Done Done Sent false
6166 Acked Read 4 Done Done Sent false
6167 Acked Flush 0 Done Done Sent false
6168 Acked Read 9 Done Done Sent false
6169 Acked Write 5 Done Done Sent false
6170 Acked Write 10 Done Done Sent false
6171 Acked Write 4 Done Done Sent false
6172 Acked Write 4 Done Done Sent false
6173 Acked Write 3 Done Done Sent false
6174 Acked Write 4 Done Done Sent false
6175 Acked Write 4 Done Done Sent false
6176 Acked Read 3 Done Done Sent false
6177 Acked Write 6 Done Done Sent false
6178 Acked Read 5 Done Done Sent false
6179 Acked Read 8 Done Done Sent false
6180 Acked Write 9 Done Done Sent false
6181 Acked Read 8 Done Done Sent false
6182 Acked Write 4 Done Done Sent false
6183 Acked Flush 0 Done Done Sent false
6184 Acked Flush 0 Done Done Sent false
6185 Acked Write 9 Done Done Sent false
6186 Acked Read 2 Done Done Sent false
6187 Acked Read 6 Done Done Sent false
6188 Acked Read 3 Done Done Sent false
6189 Acked Write 6 Done Done Sent false
6190 Acked Flush 0 Done Done Sent false
6191 Acked Write 1 Done Done Sent false
6192 Acked Write 4 Done Done Sent false
6193 Acked Flush 0 Done Done Sent false
6194 Acked Read 4 Done Done Sent false
6195 Acked Write 8 Done Done Sent false
6196 Acked Write 1 Done Done Sent false
6197 Acked Read 7 Done Done Sent false
6198 Acked Read 6 Done Done Sent false
6199 Acked Write 1 Done Done Sent false
6200 Acked Write 5 Done Done Sent false
6201 Acked Flush 0 Done Done Sent false
6202 Acked Read 3 Done Done Sent false
6203 Acked Write 10 Done Done Sent false
6204 Acked Read 10 Done Done Sent false
6205 Acked Write 3 Done Done Sent false
6206 Acked Read 8 Done Done Sent false
6207 Acked Read 9 Done Done Sent false
6208 Acked Write 3 Done Done Sent false
6209 Acked Write 1 Done Done Sent false
6210 Acked Read 9 Done Done Sent false
6211 Acked Write 7 Done Done Sent false
6212 Acked Flush 0 Done Done Sent false
6213 Acked Write 9 Done Done Sent false
6214 Acked Read 5 Done Done Sent false
6215 Acked Read 8 Done Done Sent false
6216 Acked Write 7 Done Done Sent false
6217 Acked Read 10 Done Done Sent false
6218 Acked Read 10 Done Done Sent false
6219 Acked Read 9 Done Done Sent false
6220 Acked Read 2 Done Done Sent false
6221 Acked Read 9 Done Done Sent false
6222 Acked Write 1 Done Done Sent false
6223 Acked Write 4 Done Done Sent false
6224 Acked Read 8 Done Done Sent false
6225 Acked Read 10 Done Done Sent false
6226 Acked Read 1 Done Done Sent false
6227 Acked Flush 0 Done Done Sent false
6228 Acked Read 4 Done Done Sent false
6229 Acked Write 8 Done Done Sent false
6230 Acked Flush 0 Done Done Sent false
6231 Acked Write 2 Done Done Sent false
6232 Acked Read 8 Done Done Sent false
6233 Acked Read 10 Done Done Sent false
6234 Acked Flush 0 Done Done Sent false
6235 Acked Read 2 Done Done Sent false
6236 Acked Write 7 Done Done Sent false
6237 Acked Read 5 Done Done Sent false
6238 Acked Write 5 Done Done Sent false
6239 Acked Read 7 Done Done Sent false
6240 Acked Read 6 Done Done Sent false
6241 Acked Write 7 Done Done Sent false
6242 Acked Flush 0 Done Done Sent false
6243 Acked Read 4 Done Done Sent false
6244 Acked Read 7 Done Done Sent false
6245 Acked Flush 0 Done Done Sent false
6246 Acked Write 7 Done Done Sent false
6247 Acked Write 5 Done Done Sent false
6248 Acked Write 10 Done Done Sent false
6249 Acked Write 7 Done Done Sent false
6250 Acked Write 9 Done Done Sent false
6251 Acked Write 3 Done Done Sent false
6252 Acked Read 10 Done Done Sent false
6253 Acked Read 9 Done Done Sent false
6254 Acked Write 9 Done Done Sent false
6255 Acked Read 4 Done Done Sent false
6256 Acked Write 5 Done Done Sent false
6257 Acked Read 7 Done Done Sent false
6258 Acked Read 10 Done Done Sent false
6259 Acked Read 8 Done Done Sent false
6260 Acked Flush 0 Done Done Sent false
6261 Acked Write 3 Done Done Sent false
6262 Acked Read 4 Done Done Sent false
6263 Acked Write 8 Done Done Sent false
6264 Acked Write 3 Done Done Sent false
6265 Acked Read 8 Done Done Sent false
6266 Acked Flush 0 Done Done Sent false
6267 Acked Read 5 Done Done Sent false
6268 Acked Read 7 Done Done Sent false
6269 Acked Read 8 Done Done Sent false
6270 Acked Flush 0 Done Done Sent false
6271 Acked Read 9 Done Done Sent false
6272 Acked Read 1 Done Done Sent false
6273 Acked Write 3 Done Done Sent false
6274 Acked Read 2 Done Done Sent false
6275 Acked Write 1 Done Done Sent false
6276 Acked Write 2 Done Done Sent false
6277 Acked Read 2 Done Done Sent false
6278 Acked Flush 0 Done Done Sent false
6279 Acked Flush 0 Done Done Sent false
6280 Acked Read 2 Done Done Sent false
6281 Acked Read 1 Done Done Sent false
6282 Acked Write 3 Done Done Sent false
6283 Acked Read 5 Done Done Sent false
6284 Acked Write 4 Done Done Sent false
6285 Acked Flush 0 Done Done Sent false
6286 Acked Write 7 Done Done Sent false
6287 Acked Write 4 Done Done Sent false
6288 Acked Write 9 Done Done Sent false
6289 Acked Write 6 Done Done Sent false
6290 Acked Write 2 Done Done Sent false
6291 Acked Read 10 Done Done Sent false
6292 Acked Write 8 Done Done Sent false
6293 Acked Read 7 Done Done Sent false
6294 Acked Read 9 Done Done Sent false
6295 Acked Write 2 Done Done Sent false
6296 Acked Write 10 Done Done Sent false
6297 Acked Read 1 Done Done Sent false
6298 Acked Read 10 Done Done Sent false
6299 Acked Read 2 Done Done Sent false
6300 Acked Read 9 Done Done Sent false
6301 Acked Read 9 Done Done Sent false
6302 Acked Write 2 Done Done Sent false
6303 Acked Write 5 Done Done Sent false
6304 Acked Write 6 Done Done Sent false
6305 Acked Flush 0 Done Done Sent false
6306 Acked Write 3 Done Done Sent false
6307 Acked Read 2 Done Done Sent false
6308 Acked Read 5 Done Done Sent false
6309 Acked Read 3 Done Done Sent false
6310 Acked Write 4 Done Done Sent false
6311 Acked Write 8 Done Done Sent false
6312 Acked Write 10 Done Done Sent false
6313 Acked Read 8 Done Done Sent false
6314 Acked Read 3 Done Done Sent false
6315 Acked Write 7 Done Done Sent false
6316 Acked Read 8 Done Done Sent false
6317 Acked Write 10 Done Done Sent false
6318 Acked Flush 0 Done Done Sent false
6319 Acked Read 7 Done Done Sent false
6320 Acked Read 9 Done Done Sent false
6321 Acked Write 1 Done Done Sent false
6322 Acked Read 5 Done Done Sent false
6323 Acked Write 8 Done Done Sent false
6324 Acked Read 1 Done Done Sent false
6325 Acked Read 6 Done Done Sent false
6326 Acked Read 8 Done Done Sent false
6327 Acked Write 8 Done Done Sent false
6328 Acked Read 4 Done Done Sent false
6329 Acked Read 6 Done Done Sent false
6330 Acked Read 6 Done Done Sent false
6331 Acked Read 4 Done Done Sent false
6332 Acked Read 5 Done Done Sent false
6333 Acked Write 10 Done Done Sent false
6334 Acked Write 6 Done Done Sent false
6335 Acked Flush 0 Done Done Sent false
6336 Acked Read 9 Done Done Sent false
6337 Acked Write 7 Done Done Sent false
6338 Acked Write 7 Done Done Sent false
6339 Acked Read 5 Done Done Sent false
6340 Acked Read 1 Done Done Sent false
6341 Acked Write 3 Done Done Sent false
6342 Acked Read 4 Done Done Sent false
6343 Acked Write 2 Done Done Sent false
6344 Acked Write 4 Done Done Sent false
6345 Acked Read 7 Done Done Sent false
6346 Acked Write 6 Done Done Sent false
6347 Acked Read 10 Done Done Sent false
6348 Acked Flush 0 Done Done Sent false
6349 Acked Flush 0 Done Done Sent false
6350 Acked Read 2 Done Done Sent false
6351 Acked Read 9 Done Done Sent false
6352 Acked Read 1 Done Done Sent false
6353 Acked Read 6 Done Done Sent false
6354 Acked Write 9 Done Done Sent false
6355 Acked Write 7 Done Done Sent false
6356 Acked Read 10 Done Done Sent false
6357 Acked Read 5 Done Done Sent false
6358 Acked Read 9 Done Done Sent false
6359 Acked Write 7 Done Done Sent false
6360 Acked Read 4 Done Done Sent false
6361 Acked Write 3 Done Done Sent false
6362 Acked Write 4 Done Done Sent false
6363 Acked Write 4 Done Done Sent false
6364 Acked Write 8 Done Done Sent false
6365 Acked Read 2 Done Done Sent false
6366 Acked Read 6 Done Done Sent false
6367 Acked Read 5 Done Done Sent false
6368 Acked Read 9 Done Done Sent false
6369 Acked Write 5 Done Done Sent false
6370 Acked Write 10 Done Done Sent false
6371 Acked Read 6 Done Done Sent false
6372 Acked Read 7 Done Done Sent false
6373 Acked Read 3 Done Done Sent false
6374 Acked Write 4 Done Done Sent false
6375 Acked Read 8 Done Done Sent false
6376 Acked Write 2 Done Done Sent false
6377 Acked Write 1 Done Done Sent false
6378 Acked Flush 0 Done Done Sent false
6379 Acked Write 9 Done Done Sent false
6380 Acked Read 7 Done Done Sent false
6381 Acked Write 8 Done Done Sent false
6382 Acked Read 9 Done Done Sent false
6383 Acked Write 2 Done Done Sent false
6384 Acked Read 3 Done Done Sent false
6385 Acked Read 5 Done Done Sent false
6386 Acked Read 7 Done Done Sent false
6387 Acked Read 6 Done Done Sent false
6388 Acked Read 10 Done Done Sent false
6389 Acked Read 7 Done Done Sent false
6390 Acked Read 1 Done Done Sent false
6391 Acked Write 10 Done Done Sent false
6392 Acked Read 9 Done Done Sent false
6393 Acked Read 6 Done Done Sent false
6394 Acked Read 9 Done Done Sent false
6395 Acked Write 6 Done Done Sent false
6396 Acked Flush 0 Done Done Sent false
6397 Acked Read 3 Done Done Sent false
6398 Acked Read 5 Done Done Sent false
6399 Acked Write 8 Done Done Sent false
6400 Acked Read 8 Done Done Sent false
6401 Acked Write 8 Done Done Sent false
6402 Acked Read 2 Done Done Sent false
6403 Acked Write 7 Done Done Sent false
6404 Acked Write 9 Done Done Sent false
6405 Acked Read 3 Done Done Sent false
6406 Acked Read 3 Done Done Sent false
6407 Acked Read 4 Done Done Sent false
6408 Acked Read 2 Done Done Sent false
6409 Acked Write 3 Done Done Sent false
6410 Acked Write 6 Done Done Sent false
6411 Acked Write 1 Done Done Sent false
6412 Acked Write 7 Done Done Sent false
6413 Acked Read 5 Done Done Sent false
6414 Acked Read 4 Done Done Sent false
6415 Acked Write 7 Done Done Sent false
6416 Acked Write 7 Done Done Sent false
6417 Acked Flush 0 Done Done Sent false
6418 Acked Flush 0 Done Done Sent false
6419 Acked Write 10 Done Done Sent false
6420 Acked Write 6 Done Done Sent false
6421 Acked Read 8 Done Done Sent false
6422 Acked Read 1 Done Done Sent false
6423 Acked Read 8 Done Done Sent false
6424 Acked Write 3 Done Done Sent false
6425 Acked Write 2 Done Done Sent false
6426 Acked Write 9 Done Done Sent false
6427 Acked Flush 0 Done Done Sent false
6428 Acked Read 8 Done Done Sent false
6429 Acked Read 3 Done Done Sent false
6430 Acked Write 3 Done Done Sent false
6431 Acked Read 6 Done Done Sent false
6432 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6432 6130
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6432 6431 6430 6429 6428
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.126592119Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:27.126714485Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:27.12676011Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:27.126800595Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.126870207Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Write 4 Done Done Sent false
6132 Acked Write 8 Done Done Sent false
6133 Acked Write 5 Done Done Sent false
6134 Acked Read 3 Done Done Sent false
6135 Acked Write 3 Done Done Sent false
6136 Acked Read 8 Done Done Sent false
6137 Acked Write 10 Done Done Sent false
6138 Acked Write 6 Done Done Sent false
6139 Acked Write 1 Done Done Sent false
6140 Acked Write 10 Done Done Sent false
6141 Acked Flush 0 Done Done Sent false
6142 Acked Read 8 Done Done Sent false
6143 Acked Write 3 Done Done Sent false
6144 Acked Write 10 Done Done Sent false
6145 Acked Write 2 Done Done Sent false
6146 Acked Write 6 Done Done Sent false
6147 Acked Write 6 Done Done Sent false
6148 Acked Write 7 Done Done Sent false
6149 Acked Read 1 Done Done Sent false
6150 Acked Flush 0 Done Done Sent false
6151 Acked Write 6 Done Done Sent false
6152 Acked Write 2 Done Done Sent false
6153 Acked Read 7 Done Done Sent false
6154 Acked Read 1 Done Done Sent false
6155 Acked Write 5 Done Done Sent false
6156 Acked Write 10 Done Done Sent false
6157 Acked Write 10 Done Done Sent false
6158 Acked Read 10 Done Done Sent false
6159 Acked Flush 0 Done Done Sent false
6160 Acked Read 3 Done Done Sent false
6161 Acked Flush 0 Done Done Sent false
6162 Acked Read 10 Done Done Sent false
6163 Acked Write 6 Done Done Sent false
6164 Acked Read 7 Done Done Sent false
6165 Acked Read 1 Done Done Sent false
6166 Acked Read 4 Done Done Sent false
6167 Acked Flush 0 Done Done Sent false
6168 Acked Read 9 Done Done Sent false
6169 Acked Write 5 Done Done Sent false
6170 Acked Write 10 Done Done Sent false
6171 Acked Write 4 Done Done Sent false
6172 Acked Write 4 Done Done Sent false
6173 Acked Write 3 Done Done Sent false
6174 Acked Write 4 Done Done Sent false
6175 Acked Write 4 Done Done Sent false
6176 Acked Read 3 Done Done Sent false
6177 Acked Write 6 Done Done Sent false
6178 Acked Read 5 Done Done Sent false
6179 Acked Read 8 Done Done Sent false
6180 Acked Write 9 Done Done Sent false
6181 Acked Read 8 Done Done Sent false
6182 Acked Write 4 Done Done Sent false
6183 Acked Flush 0 Done Done Sent false
6184 Acked Flush 0 Done Done Sent false
6185 Acked Write 9 Done Done Sent false
6186 Acked Read 2 Done Done Sent false
6187 Acked Read 6 Done Done Sent false
6188 Acked Read 3 Done Done Sent false
6189 Acked Write 6 Done Done Sent false
6190 Acked Flush 0 Done Done Sent false
6191 Acked Write 1 Done Done Sent false
6192 Acked Write 4 Done Done Sent false
6193 Acked Flush 0 Done Done Sent false
6194 Acked Read 4 Done Done Sent false
6195 Acked Write 8 Done Done Sent false
6196 Acked Write 1 Done Done Sent false
6197 Acked Read 7 Done Done Sent false
6198 Acked Read 6 Done Done Sent false
6199 Acked Write 1 Done Done Sent false
6200 Acked Write 5 Done Done Sent false
6201 Acked Flush 0 Done Done Sent false
6202 Acked Read 3 Done Done Sent false
6203 Acked Write 10 Done Done Sent false
6204 Acked Read 10 Done Done Sent false
6205 Acked Write 3 Done Done Sent false
6206 Acked Read 8 Done Done Sent false
6207 Acked Read 9 Done Done Sent false
6208 Acked Write 3 Done Done Sent false
6209 Acked Write 1 Done Done Sent false
6210 Acked Read 9 Done Done Sent false
6211 Acked Write 7 Done Done Sent false
6212 Acked Flush 0 Done Done Sent false
6213 Acked Write 9 Done Done Sent false
6214 Acked Read 5 Done Done Sent false
6215 Acked Read 8 Done Done Sent false
6216 Acked Write 7 Done Done Sent false
6217 Acked Read 10 Done Done Sent false
6218 Acked Read 10 Done Done Sent false
6219 Acked Read 9 Done Done Sent false
6220 Acked Read 2 Done Done Sent false
6221 Acked Read 9 Done Done Sent false
6222 Acked Write 1 Done Done Sent false
6223 Acked Write 4 Done Done Sent false
6224 Acked Read 8 Done Done Sent false
6225 Acked Read 10 Done Done Sent false
6226 Acked Read 1 Done Done Sent false
6227 Acked Flush 0 Done Done Sent false
6228 Acked Read 4 Done Done Sent false
6229 Acked Write 8 Done Done Sent false
6230 Acked Flush 0 Done Done Sent false
6231 Acked Write 2 Done Done Sent false
6232 Acked Read 8 Done Done Sent false
6233 Acked Read 10 Done Done Sent false
6234 Acked Flush 0 Done Done Sent false
6235 Acked Read 2 Done Done Sent false
6236 Acked Write 7 Done Done Sent false
6237 Acked Read 5 Done Done Sent false
6238 Acked Write 5 Done Done Sent false
6239 Acked Read 7 Done Done Sent false
6240 Acked Read 6 Done Done Sent false
6241 Acked Write 7 Done Done Sent false
6242 Acked Flush 0 Done Done Sent false
6243 Acked Read 4 Done Done Sent false
6244 Acked Read 7 Done Done Sent false
6245 Acked Flush 0 Done Done Sent false
6246 Acked Write 7 Done Done Sent false
6247 Acked Write 5 Done Done Sent false
6248 Acked Write 10 Done Done Sent false
6249 Acked Write 7 Done Done Sent false
6250 Acked Write 9 Done Done Sent false
6251 Acked Write 3 Done Done Sent false
6252 Acked Read 10 Done Done Sent false
6253 Acked Read 9 Done Done Sent false
6254 Acked Write 9 Done Done Sent false
6255 Acked Read 4 Done Done Sent false
6256 Acked Write 5 Done Done Sent false
6257 Acked Read 7 Done Done Sent false
6258 Acked Read 10 Done Done Sent false
6259 Acked Read 8 Done Done Sent false
6260 Acked Flush 0 Done Done Sent false
6261 Acked Write 3 Done Done Sent false
6262 Acked Read 4 Done Done Sent false
6263 Acked Write 8 Done Done Sent false
6264 Acked Write 3 Done Done Sent false
6265 Acked Read 8 Done Done Sent false
6266 Acked Flush 0 Done Done Sent false
6267 Acked Read 5 Done Done Sent false
6268 Acked Read 7 Done Done Sent false
6269 Acked Read 8 Done Done Sent false
6270 Acked Flush 0 Done Done Sent false
6271 Acked Read 9 Done Done Sent false
6272 Acked Read 1 Done Done Sent false
6273 Acked Write 3 Done Done Sent false
6274 Acked Read 2 Done Done Sent false
6275 Acked Write 1 Done Done Sent false
6276 Acked Write 2 Done Done Sent false
6277 Acked Read 2 Done Done Sent false
6278 Acked Flush 0 Done Done Sent false
6279 Acked Flush 0 Done Done Sent false
6280 Acked Read 2 Done Done Sent false
6281 Acked Read 1 Done Done Sent false
6282 Acked Write 3 Done Done Sent false
6283 Acked Read 5 Done Done Sent false
6284 Acked Write 4 Done Done Sent false
6285 Acked Flush 0 Done Done Sent false
6286 Acked Write 7 Done Done Sent false
6287 Acked Write 4 Done Done Sent false
6288 Acked Write 9 Done Done Sent false
6289 Acked Write 6 Done Done Sent false
6290 Acked Write 2 Done Done Sent false
6291 Acked Read 10 Done Done Sent false
6292 Acked Write 8 Done Done Sent false
6293 Acked Read 7 Done Done Sent false
6294 Acked Read 9 Done Done Sent false
6295 Acked Write 2 Done Done Sent false
6296 Acked Write 10 Done Done Sent false
6297 Acked Read 1 Done Done Sent false
6298 Acked Read 10 Done Done Sent false
6299 Acked Read 2 Done Done Sent false
6300 Acked Read 9 Done Done Sent false
6301 Acked Read 9 Done Done Sent false
6302 Acked Write 2 Done Done Sent false
6303 Acked Write 5 Done Done Sent false
6304 Acked Write 6 Done Done Sent false
6305 Acked Flush 0 Done Done Sent false
6306 Acked Write 3 Done Done Sent false
6307 Acked Read 2 Done Done Sent false
6308 Acked Read 5 Done Done Sent false
6309 Acked Read 3 Done Done Sent false
6310 Acked Write 4 Done Done Sent false
6311 Acked Write 8 Done Done Sent false
6312 Acked Write 10 Done Done Sent false
6313 Acked Read 8 Done Done Sent false
6314 Acked Read 3 Done Done Sent false
6315 Acked Write 7 Done Done Sent false
6316 Acked Read 8 Done Done Sent false
6317 Acked Write 10 Done Done Sent false
6318 Acked Flush 0 Done Done Sent false
6319 Acked Read 7 Done Done Sent false
6320 Acked Read 9 Done Done Sent false
6321 Acked Write 1 Done Done Sent false
6322 Acked Read 5 Done Done Sent false
6323 Acked Write 8 Done Done Sent false
6324 Acked Read 1 Done Done Sent false
6325 Acked Read 6 Done Done Sent false
6326 Acked Read 8 Done Done Sent false
6327 Acked Write 8 Done Done Sent false
6328 Acked Read 4 Done Done Sent false
6329 Acked Read 6 Done Done Sent false
6330 Acked Read 6 Done Done Sent false
6331 Acked Read 4 Done Done Sent false
6332 Acked Read 5 Done Done Sent false
6333 Acked Write 10 Done Done Sent false
6334 Acked Write 6 Done Done Sent false
6335 Acked Flush 0 Done Done Sent false
6336 Acked Read 9 Done Done Sent false
6337 Acked Write 7 Done Done Sent false
6338 Acked Write 7 Done Done Sent false
6339 Acked Read 5 Done Done Sent false
6340 Acked Read 1 Done Done Sent false
6341 Acked Write 3 Done Done Sent false
6342 Acked Read 4 Done Done Sent false
6343 Acked Write 2 Done Done Sent false
6344 Acked Write 4 Done Done Sent false
6345 Acked Read 7 Done Done Sent false
6346 Acked Write 6 Done Done Sent false
6347 Acked Read 10 Done Done Sent false
6348 Acked Flush 0 Done Done Sent false
6349 Acked Flush 0 Done Done Sent false
6350 Acked Read 2 Done Done Sent false
6351 Acked Read 9 Done Done Sent false
6352 Acked Read 1 Done Done Sent false
6353 Acked Read 6 Done Done Sent false
6354 Acked Write 9 Done Done Sent false
6355 Acked Write 7 Done Done Sent false
6356 Acked Read 10 Done Done Sent false
6357 Acked Read 5 Done Done Sent false
6358 Acked Read 9 Done Done Sent false
6359 Acked Write 7 Done Done Sent false
6360 Acked Read 4 Done Done Sent false
6361 Acked Write 3 Done Done Sent false
6362 Acked Write 4 Done Done Sent false
6363 Acked Write 4 Done Done Sent false
6364 Acked Write 8 Done Done Sent false
6365 Acked Read 2 Done Done Sent false
6366 Acked Read 6 Done Done Sent false
6367 Acked Read 5 Done Done Sent false
6368 Acked Read 9 Done Done Sent false
6369 Acked Write 5 Done Done Sent false
6370 Acked Write 10 Done Done Sent false
6371 Acked Read 6 Done Done Sent false
6372 Acked Read 7 Done Done Sent false
6373 Acked Read 3 Done Done Sent false
6374 Acked Write 4 Done Done Sent false
6375 Acked Read 8 Done Done Sent false
6376 Acked Write 2 Done Done Sent false
6377 Acked Write 1 Done Done Sent false
6378 Acked Flush 0 Done Done Sent false
6379 Acked Write 9 Done Done Sent false
6380 Acked Read 7 Done Done Sent false
6381 Acked Write 8 Done Done Sent false
6382 Acked Read 9 Done Done Sent false
6383 Acked Write 2 Done Done Sent false
6384 Acked Read 3 Done Done Sent false
6385 Acked Read 5 Done Done Sent false
6386 Acked Read 7 Done Done Sent false
6387 Acked Read 6 Done Done Sent false
6388 Acked Read 10 Done Done Sent false
6389 Acked Read 7 Done Done Sent false
6390 Acked Read 1 Done Done Sent false
6391 Acked Write 10 Done Done Sent false
6392 Acked Read 9 Done Done Sent false
6393 Acked Read 6 Done Done Sent false
6394 Acked Read 9 Done Done Sent false
6395 Acked Write 6 Done Done Sent false
6396 Acked Flush 0 Done Done Sent false
6397 Acked Read 3 Done Done Sent false
6398 Acked Read 5 Done Done Sent false
6399 Acked Write 8 Done Done Sent false
6400 Acked Read 8 Done Done Sent false
6401 Acked Write 8 Done Done Sent false
6402 Acked Read 2 Done Done Sent false
6403 Acked Write 7 Done Done Sent false
6404 Acked Write 9 Done Done Sent false
6405 Acked Read 3 Done Done Sent false
6406 Acked Read 3 Done Done Sent false
6407 Acked Read 4 Done Done Sent false
6408 Acked Read 2 Done Done Sent false
6409 Acked Write 3 Done Done Sent false
6410 Acked Write 6 Done Done Sent false
6411 Acked Write 1 Done Done Sent false
6412 Acked Write 7 Done Done Sent false
6413 Acked Read 5 Done Done Sent false
6414 Acked Read 4 Done Done Sent false
6415 Acked Write 7 Done Done Sent false
6416 Acked Write 7 Done Done Sent false
6417 Acked Flush 0 Done Done Sent false
6418 Acked Flush 0 Done Done Sent false
6419 Acked Write 10 Done Done Sent false
6420 Acked Write 6 Done Done Sent false
6421 Acked Read 8 Done Done Sent false
6422 Acked Read 1 Done Done Sent false
6423 Acked Read 8 Done Done Sent false
6424 Acked Write 3 Done Done Sent false
6425 Acked Write 2 Done Done Sent false
6426 Acked Write 9 Done Done Sent false
6427 Acked Flush 0 Done Done Sent false
6428 Acked Read 8 Done Done Sent false
6429 Acked Read 3 Done Done Sent false
6430 Acked Write 3 Done Done Sent false
6431 Acked Read 6 Done Done Sent false
6432 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6432 6130
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6432 6431 6430 6429 6428
CLIENT: Up:0 ds:302 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1218)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6131 Acked Write 4 Done Done Sent false
6132 Acked Write 8 Done Done Sent false
6133 Acked Write 5 Done Done Sent false
6134 Acked Read 3 Done Done Sent false
6135 Acked Write 3 Done Done Sent false
6136 Acked Read 8 Done Done Sent false
6137 Acked Write 10 Done Done Sent false
6138 Acked Write 6 Done Done Sent false
6139 Acked Write 1 Done Done Sent false
6140 Acked Write 10 Done Done Sent false
6141 Acked Flush 0 Done Done Sent false
6142 Acked Read 8 Done Done Sent false
6143 Acked Write 3 Done Done Sent false
6144 Acked Write 10 Done Done Sent false
6145 Acked Write 2 Done Done Sent false
6146 Acked Write 6 Done Done Sent false
6147 Acked Write 6 Done Done Sent false
6148 Acked Write 7 Done Done Sent false
6149 Acked Read 1 Done Done Sent false
6150 Acked Flush 0 Done Done Sent false
6151 Acked Write 6 Done Done Sent false
6152 Acked Write 2 Done Done Sent false
6153 Acked Read 7 Done Done Sent false
6154 Acked Read 1 Done Done Sent false
6155 Acked Write 5 Done Done Sent false
6156 Acked Write 10 Done Done Sent false
6157 Acked Write 10 Done Done Sent false
6158 Acked Read 10 Done Done Sent false
6159 Acked Flush 0 Done Done Sent false
6160 Acked Read 3 Done Done Sent false
6161 Acked Flush 0 Done Done Sent false
6162 Acked Read 10 Done Done Sent false
6163 Acked Write 6 Done Done Sent false
6164 Acked Read 7 Done Done Sent false
6165 Acked Read 1 Done Done Sent false
6166 Acked Read 4 Done Done Sent false
6167 Acked Flush 0 Done Done Sent false
6168 Acked Read 9 Done Done Sent false
6169 Acked Write 5 Done Done Sent false
6170 Acked Write 10 Done Done Sent false
6171 Acked Write 4 Done Done Sent false
6172 Acked Write 4 Done Done Sent false
6173 Acked Write 3 Done Done Sent false
6174 Acked Write 4 Done Done Sent false
6175 Acked Write 4 Done Done Sent false
6176 Acked Read 3 Done Done Sent false
6177 Acked Write 6 Done Done Sent false
6178 Acked Read 5 Done Done Sent false
6179 Acked Read 8 Done Done Sent false
6180 Acked Write 9 Done Done Sent false
6181 Acked Read 8 Done Done Sent false
6182 Acked Write 4 Done Done Sent false
6183 Acked Flush 0 Done Done Sent false
6184 Acked Flush 0 Done Done Sent false
6185 Acked Write 9 Done Done Sent false
6186 Acked Read 2 Done Done Sent false
6187 Acked Read 6 Done Done Sent false
6188 Acked Read 3 Done Done Sent false
6189 Acked Write 6 Done Done Sent false
6190 Acked Flush 0 Done Done Sent false
6191 Acked Write 1 Done Done Sent false
6192 Acked Write 4 Done Done Sent false
6193 Acked Flush 0 Done Done Sent false
6194 Acked Read 4 Done Done Sent false
6195 Acked Write 8 Done Done Sent false
6196 Acked Write 1 Done Done Sent false
6197 Acked Read 7 Done Done Sent false
6198 Acked Read 6 Done Done Sent false
6199 Acked Write 1 Done Done Sent false
6200 Acked Write 5 Done Done Sent false
6201 Acked Flush 0 Done Done Sent false
6202 Acked Read 3 Done Done Sent false
6203 Acked Write 10 Done Done Sent false
6204 Acked Read 10 Done Done Sent false
6205 Acked Write 3 Done Done Sent false
6206 Acked Read 8 Done Done Sent false
6207 Acked Read 9 Done Done Sent false
6208 Acked Write 3 Done Done Sent false
6209 Acked Write 1 Done Done Sent false
6210 Acked Read 9 Done Done Sent false
6211 Acked Write 7 Done Done Sent false
6212 Acked Flush 0 Done Done Sent false
6213 Acked Write 9 Done Done Sent false
6214 Acked Read 5 Done Done Sent false
6215 Acked Read 8 Done Done Sent false
6216 Acked Write 7 Done Done Sent false
6217 Acked Read 10 Done Done Sent false
6218 Acked Read 10 Done Done Sent false
6219 Acked Read 9 Done Done Sent false
6220 Acked Read 2 Done Done Sent false
6221 Acked Read 9 Done Done Sent false
6222 Acked Write 1 Done Done Sent false
6223 Acked Write 4 Done Done Sent false
6224 Acked Read 8 Done Done Sent false
6225 Acked Read 10 Done Done Sent false
6226 Acked Read 1 Done Done Sent false
6227 Acked Flush 0 Done Done Sent false
6228 Acked Read 4 Done Done Sent false
6229 Acked Write 8 Done Done Sent false
6230 Acked Flush 0 Done Done Sent false
6231 Acked Write 2 Done Done Sent false
6232 Acked Read 8 Done Done Sent false
6233 Acked Read 10 Done Done Sent false
6234 Acked Flush 0 Done Done Sent false
6235 Acked Read 2 Done Done Sent false
6236 Acked Write 7 Done Done Sent false
6237 Acked Read 5 Done Done Sent false
6238 Acked Write 5 Done Done Sent false
6239 Acked Read 7 Done Done Sent false
6240 Acked Read 6 Done Done Sent false
6241 Acked Write 7 Done Done Sent false
6242 Acked Flush 0 Done Done Sent false
6243 Acked Read 4 Done Done Sent false
6244 Acked Read 7 Done Done Sent false
6245 Acked Flush 0 Done Done Sent false
6246 Acked Write 7 Done Done Sent false
6247 Acked Write 5 Done Done Sent false
6248 Acked Write 10 Done Done Sent false
6249 Acked Write 7 Done Done Sent false
6250 Acked Write 9 Done Done Sent false
6251 Acked Write 3 Done Done Sent false
6252 Acked Read 10 Done Done Sent false
6253 Acked Read 9 Done Done Sent false
6254 Acked Write 9 Done Done Sent false
6255 Acked Read 4 Done Done Sent false
6256 Acked Write 5 Done Done Sent false
6257 Acked Read 7 Done Done Sent false
6258 Acked Read 10 Done Done Sent false
6259 Acked Read 8 Done Done Sent false
6260 Acked Flush 0 Done Done Sent false
6261 Acked Write 3 Done Done Sent false
6262 Acked Read 4 Done Done Sent false
6263 Acked Write 8 Done Done Sent false
6264 Acked Write 3 Done Done Sent false
6265 Acked Read 8 Done Done Sent false
6266 Acked Flush 0 Done Done Sent false
6267 Acked Read 5 Done Done Sent false
6268 Acked Read 7 Done Done Sent false
6269 Acked Read 8 Done Done Sent false
6270 Acked Flush 0 Done Done Sent false
6271 Acked Read 9 Done Done Sent false
6272 Acked Read 1 Done Done Sent false
6273 Acked Write 3 Done Done Sent false
6274 Acked Read 2 Done Done Sent false
6275 Acked Write 1 Done Done Sent false
6276 Acked Write 2 Done Done Sent false
6277 Acked Read 2 Done Done Sent false
6278 Acked Flush 0 Done Done Sent false
6279 Acked Flush 0 Done Done Sent false
6280 Acked Read 2 Done Done Sent false
6281 Acked Read 1 Done Done Sent false
6282 Acked Write 3 Done Done Sent false
6283 Acked Read 5 Done Done Sent false
6284 Acked Write 4 Done Done Sent false
6285 Acked Flush 0 Done Done Sent false
6286 Acked Write 7 Done Done Sent false
6287 Acked Write 4 Done Done Sent false
6288 Acked Write 9 Done Done Sent false
6289 Acked Write 6 Done Done Sent false
6290 Acked Write 2 Done Done Sent false
6291 Acked Read 10 Done Done Sent false
6292 Acked Write 8 Done Done Sent false
6293 Acked Read 7 Done Done Sent false
6294 Acked Read 9 Done Done Sent false
6295 Acked Write 2 Done Done Sent false
6296 Acked Write 10 Done Done Sent false
6297 Acked Read 1 Done Done Sent false
6298 Acked Read 10 Done Done Sent false
6299 Acked Read 2 Done Done Sent false
6300 Acked Read 9 Done Done Sent false
6301 Acked Read 9 Done Done Sent false
6302 Acked Write 2 Done Done Sent false
6303 Acked Write 5 Done Done Sent false
6304 Acked Write 6 Done Done Sent false
6305 Acked Flush 0 Done Done Sent false
6306 Acked Write 3 Done Done Sent false
6307 Acked Read 2 Done Done Sent false
6308 Acked Read 5 Done Done Sent false
6309 Acked Read 3 Done Done Sent false
6310 Acked Write 4 Done Done Sent false
6311 Acked Write 8 Done Done Sent false
6312 Acked Write 10 Done Done Sent false
6313 Acked Read 8 Done Done Sent false
6314 Acked Read 3 Done Done Sent false
6315 Acked Write 7 Done Done Sent false
6316 Acked Read 8 Done Done Sent false
6317 Acked Write 10 Done Done Sent false
6318 Acked Flush 0 Done Done Sent false
6319 Acked Read 7 Done Done Sent false
6320 Acked Read 9 Done Done Sent false
6321 Acked Write 1 Done Done Sent false
6322 Acked Read 5 Done Done Sent false
6323 Acked Write 8 Done Done Sent false
6324 Acked Read 1 Done Done Sent false
6325 Acked Read 6 Done Done Sent false
6326 Acked Read 8 Done Done Sent false
6327 Acked Write 8 Done Done Sent false
6328 Acked Read 4 Done Done Sent false
6329 Acked Read 6 Done Done Sent false
6330 Acked Read 6 Done Done Sent false
6331 Acked Read 4 Done Done Sent false
6332 Acked Read 5 Done Done Sent false
6333 Acked Write 10 Done Done Sent false
6334 Acked Write 6 Done Done Sent false
6335 Acked Flush 0 Done Done Sent false
6336 Acked Read 9 Done Done Sent false
6337 Acked Write 7 Done Done Sent false
6338 Acked Write 7 Done Done Sent false
6339 Acked Read 5 Done Done Sent false
6340 Acked Read 1 Done Done Sent false
6341 Acked Write 3 Done Done Sent false
6342 Acked Read 4 Done Done Sent false
6343 Acked Write 2 Done Done Sent false
6344 Acked Write 4 Done Done Sent false
6345 Acked Read 7 Done Done Sent false
6346 Acked Write 6 Done Done Sent false
6347 Acked Read 10 Done Done Sent false
6348 Acked Flush 0 Done Done Sent false
6349 Acked Flush 0 Done Done Sent false
6350 Acked Read 2 Done Done Sent false
6351 Acked Read 9 Done Done Sent false
6352 Acked Read 1 Done Done Sent false
6353 Acked Read 6 Done Done Sent false
6354 Acked Write 9 Done Done Sent false
6355 Acked Write 7 Done Done Sent false
6356 Acked Read 10 Done Done Sent false
6357 Acked Read 5 Done Done Sent false
6358 Acked Read 9 Done Done Sent false
6359 Acked Write 7 Done Done Sent false
6360 Acked Read 4 Done Done Sent false
6361 Acked Write 3 Done Done Sent false
6362 Acked Write 4 Done Done Sent false
6363 Acked Write 4 Done Done Sent false
6364 Acked Write 8 Done Done Sent false
6365 Acked Read 2 Done Done Sent false
6366 Acked Read 6 Done Done Sent false
6367 Acked Read 5 Done Done Sent false
6368 Acked Read 9 Done Done Sent false
6369 Acked Write 5 Done Done Sent false
6370 Acked Write 10 Done Done Sent false
6371 Acked Read 6 Done Done Sent false
6372 Acked Read 7 Done Done Sent false
6373 Acked Read 3 Done Done Sent false
6374 Acked Write 4 Done Done Sent false
6375 Acked Read 8 Done Done Sent false
6376 Acked Write 2 Done Done Sent false
6377 Acked Write 1 Done Done Sent false
6378 Acked Flush 0 Done Done Sent false
6379 Acked Write 9 Done Done Sent false
6380 Acked Read 7 Done Done Sent false
6381 Acked Write 8 Done Done Sent false
6382 Acked Read 9 Done Done Sent false
6383 Acked Write 2 Done Done Sent false
6384 Acked Read 3 Done Done Sent false
6385 Acked Read 5 Done Done Sent false
6386 Acked Read 7 Done Done Sent false
6387 Acked Read 6 Done Done Sent false
6388 Acked Read 10 Done Done Sent false
6389 Acked Read 7 Done Done Sent false
6390 Acked Read 1 Done Done Sent false
6391 Acked Write 10 Done Done Sent false
6392 Acked Read 9 Done Done Sent false
6393 Acked Read 6 Done Done Sent false
6394 Acked Read 9 Done Done Sent false
6395 Acked Write 6 Done Done Sent false
6396 Acked Flush 0 Done Done Sent false
6397 Acked Read 3 Done Done Sent false
6398 Acked Read 5 Done Done Sent false
6399 Acked Write 8 Done Done Sent false
6400 Acked Read 8 Done Done Sent false
6401 Acked Write 8 Done Done Sent false
6402 Acked Read 2 Done Done Sent false
6403 Acked Write 7 Done Done Sent false
6404 Acked Write 9 Done Done Sent false
6405 Acked Read 3 Done Done Sent false
6406 Acked Read 3 Done Done Sent false
6407 Acked Read 4 Done Done Sent false
6408 Acked Read 2 Done Done Sent false
6409 Acked Write 3 Done Done Sent false
6410 Acked Write 6 Done Done Sent false
6411 Acked Write 1 Done Done Sent false
6412 Acked Write 7 Done Done Sent false
6413 Acked Read 5 Done Done Sent false
6414 Acked Read 4 Done Done Sent false
6415 Acked Write 7 Done Done Sent false
6416 Acked Write 7 Done Done Sent false
6417 Acked Flush 0 Done Done Sent false
6418 Acked Flush 0 Done Done Sent false
6419 Acked Write 10 Done Done Sent false
6420 Acked Write 6 Done Done Sent false
6421 Acked Read 8 Done Done Sent false
6422 Acked Read 1 Done Done Sent false
6423 Acked Read 8 Done Done Sent false
6424 Acked Write 3 Done Done Sent false
6425 Acked Write 2 Done Done Sent false
6426 Acked Write 9 Done Done Sent false
6427 Acked Flush 0 Done Done Sent false
6428 Acked Read 8 Done Done Sent false
6429 Acked Read 3 Done Done Sent false
6430 Acked Write 3 Done Done Sent false
6431 Acked Read 6 Done Done Sent false
6432 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6432 6130
Downstairs last five completed: 6130 6129 6128 6127 6126
Upstairs last five completed: 6432 6431 6430 6429 6428
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.128120136Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42397)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.128282497Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.129388878Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.130087806Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.130118333Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 6130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.130133341Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 6130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.13031807Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 6130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.130346626Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 6130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.130557711Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6432 6431 6430 6429 6428
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:40:37.372 INFO request completed, latency_us: 374, response_code: 204[8810] Got stop action so:false kr:false
, uri: /stop/cid/0, method: POST, req_id: 8af69870-9e89-4f71-bedc-a4a08527a99a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:37.373580344Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:40:37.373 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/0, method: GET, req_id: abb59965-ffda-42d4-a17f-7286920ffd84, remote_addr: 127.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:37.373644727Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:37.373672114Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:37.373719318Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:40:41.375 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8d4949b7-4a94-45b0-b178-1bff3e1c7b46, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 1771 len 12288 data: 5 6 6
002/300 Read block 4302 len 28672
003/300 Read block 1818 len 24576
004/300 Read block 1731 len 12288
005/300 Flush
006/300 Read block 1428 len 40960
007/300 Read block 2680 len 20480
008/300 Flush
009/300 Read block 1950 len 20480
010/300 Read block 2933 len 20480
011/300 Write block 233 len 12288 data: 3 4 5
012/300 Read block 3932 len 36864
013/300 Read block 917 len 40960
014/300 Write block 3175 len 4096 data: 5
015/300 Read block 501 len 36864
016/300 Write block 1323 len 12288 data: 4 4 4
017/300 Read block 898 len 24576
018/300 Read block 2536 len 40960
019/300 Write block 4910 len 4096 data: 3
020/300 Read block 2505 len 8192
021/300 Read block 1061 len 16384
022/300 Flush
023/300 Write block 4244 len 4096 data: 4
024/300 Read block 2853 len 32768
025/300 Read block 2842 len 32768
026/300 Read block 3059 len 36864
027/300 Write block 2418 len 32768 data: 4 4 4 2 2 3 3 5
028/300 Read block 1758 len 8192
029/300 Read block 2656 len 36864
030/300 Write block 4159 len 40960 data: 5 6 8 8 8 8 8 7 6 5
031/300 Write block 4973 len 40960 data: 4 4 3 2 2 2 3 2 3 4
032/300 Flush
033/300 Read block 2366 len 40960
034/300 Write block 4954 len 12288 data: 4 4 3
035/300 Read block 4766 len 24576
036/300 Read block 1691 len 40960
037/300 Write block 338 len 24576 data: 5 5 4 4 4 4
038/300 Read block 2476 len 40960
039/300 Read block 4497 len 4096
040/300 Read block 2443 len 4096
041/300 Read block 608 len 36864
042/300 Read block 2536 len 20480
043/300 Write block 369 len 28672 data: 3 2 4 5 5 5 5
044/300 Read block 380 len 8192
045/300 Read block 529 len 32768
046/300 Read block 1164 len 28672
047/300 Read block 3756 len 16384
048/300 Read block 1142 len 32768
049/300 Write block 3433 len 24576 data: 7 7 6 6 6 6
050/300 Read block 831 len 4096
051/300 Read block 396 len 40960
052/300 Read block 1344 len 16384
053/300 Read block 3385 len 28672
054/300 Read block 4106 len 24576
055/300 Flush
056/300 Read block 1280 len 8192
057/300 Flush
058/300 Read block 4634 len 8192
059/300 Write block 205 len 28672 data: 5 5 5 5 5 4 4
060/300 Write block 1218 len 20480 data: 4 5 4 5 5
061/300 Write block 4074 len 40960 data: 2 3 2 3 3 2 3 3 3 3
062/300 Read block 3543 len 28672
063/300 Write block 1956 len 12288 data: 4 5 5
064/300 Read block 482 len 24576
065/300 Read block 3444 len 16384
066/300 Write block 3863 len 28672 data: 4 3 3 3 4 4 4
067/300 Read block 4257 len 20480
068/300 Read block 109 len 24576
069/300 Write block 4753 len 12288 data: 4 4 5
070/300 Write block 4401 len 4096 data: 2
071/300 Read block 1448 len 40960
072/300 Read block 333 len 8192
073/300 Write block 2291 len 8192 data: 7 7
074/300 Read block 1157 len 40960
075/300 Write block 1197 len 36864 data: 5 5 5 4 4 4 5 6 6
076/300 Read block 4982 len 4096
077/300 Write block 253 len 4096 data: 3
078/300 Flush
079/300 Read block 1953 len 28672
080/300 Read block 4885 len 16384
081/300 Flush
082/300 Read block 4963 len 32768
083/300 Write block 2235 len 12288 data: 3 5 5
084/300 Read block 1833 len 24576
085/300 Flush
086/300 Read block 1260 len 8192
087/300 Read block 1717 len 12288
088/300 Read block 3109 len 8192
089/300 Flush
090/300 Write block 1600 len 16384 data: 5 4 4 4
091/300 Read block 2014 len 28672
092/300 Write block 4498 len 16384 data: 5 5 5 4
093/300 Read block 4633 len 24576
094/300 Read block 3940 len 4096
095/300 Write block 2144 len 12288 data: 6 5 5
096/300 Read block 4642 len 8192
097/300 Read block 1949 len 32768
098/300 Read block 4241 len 36864
099/300 Write block 4255 len 28672 data: 4 5 4 4 4 6 5
100/300 Read block 4302 len 4096
101/300 Write block 742 len 32768 data: 5 6 5 6 6 6 6 6
102/300 Flush
103/300 Flush
104/300 Write block 1312 len 36864 data: 5 4 5 5 5 5 5 5 5
105/300 Read block 3309 len 40960
106/300 Read block 3109 len 8192
107/300 Read block 3374 len 12288
108/300 Read block 1382 len 36864
109/300 Read block 3715 len 36864
110/300 Flush
111/300 Write block 2722 len 8192 data: 5 3
112/300 Read block 3156 len 20480
113/300 Read block 2783 len 28672
114/300 Read block 4659 len 24576
115/300 Write block 843 len 24576 data: 3 3 3 4 3 4
116/300 Write block 3162 len 36864 data: 4 4 3 2 2 2 2 3 3
117/300 Write block 4677 len 20480 data: 2 2 2 5 6
118/300 Flush
119/300 Flush
120/300 Read block 1012 len 32768
121/300 Read block 4042 len 28672
122/300 Read block 2628 len 24576
123/300 Read block 3393 len 20480
124/300 Write block 1619 len 16384 data: 5 4 3 3
125/300 Flush
126/300 Read block 803 len 28672
127/300 Write block 2562 len 8192 data: 6 7
128/300 Read block 3523 len 4096
129/300 Read block 674 len 32768
130/300 Read block 3063 len 36864
131/300 Write block 512 len 40960 data: 5 5 5 5 4 4 4 5 5 4
132/300 Write block 1598 len 12288 data: 8 5 6
133/300 Read block 1090 len 20480
134/300 Flush
135/300 Read block 1384 len 8192
136/300 Write block 3128 len 4096 data: 4
137/300 Read block 60 len 12288
138/300 Write block 4777 len 40960 data: 6 5 3 2 2 3 3 3 4 5
139/300 Write block 2188 len 24576 data: 3 2 3 3 3 4
140/300 Read block 290 len 40960
141/300 Read block 2507 len 20480
142/300 Read block 4416 len 20480
143/300 Read block 3075 len 12288
144/300 Write block 4339 len 12288 data: 6 3 3
145/300 Read block 1025 len 40960
146/300 Read block 1450 len 32768
147/300 Flush
148/300 Write block 496 len 16384 data: 7 9 10 10
149/300 Write block 885 len 36864 data: 2 2 2 2 2 3 4 4 3
150/300 Read block 4735 len 12288
151/300 Flush
152/300 Read block 3869 len 32768
153/300 Read block 4000 len 8192
154/300 Read block 389 len 36864
155/300 Write block 3525 len 20480 data: 5 5 4 4 3
156/300 Read block 4981 len 4096
157/300 Read block 224 len 16384
158/300 Read block 346 len 4096
159/300 Read block 3563 len 16384
160/300 Read block 3771 len 16384
161/300 Read block 3539 len 28672
162/300 Read block 187 len 24576
163/300 Write block 2008 len 36864 data: 4 4 4 6 4 5 5 5 6
164/300 Write block 3450 len 32768 data: 5 6 6 7 6 8 7 7
165/300 Write block 166 len 28672 data: 5 5 7 7 7 7 5
166/300 Read block 3867 len 40960
167/300 Read block 3676 len 24576
168/300 Read block 749 len 28672
169/300 Read block 3431 len 28672
170/300 Flush
171/300 Write block 4 len 12288 data: 2 2 2
172/300 Write block 2824 len 40960 data: 3 3 3 3 5 5 6 6 7 7
173/300 Write block 3580 len 8192 data: 2 2
174/300 Write block 168 len 16384 data: 8 8 8 8
175/300 Read block 2572 len 4096
176/300 Write block 2620 len 36864 data: 4 5 5 5 6 4 3 4 4
177/300 Flush
178/300 Write block 2255 len 24576 data: 7 8 6 6 6 6
179/300 Flush
180/300 Read block 2458 len 20480
181/300 Read block 1666 len 8192
182/300 Read block 2929 len 20480
183/300 Write block 252 len 40960 data: 3 4 3 3 3 3 3 4 4 4
184/300 Read block 2518 len 36864
185/300 Write block 406 len 36864 data: 6 4 4 5 5 4 4 4 4
186/300 Flush
187/300 Read block 4140 len 12288
188/300 Read block 4443 len 24576
189/300 Read block 4528 len 12288
190/300 Write block 4368 len 16384 data: 3 3 2 2
191/300 Read block 1 len 20480
192/300 Write block 3549 len 40960 data: 3 3 3 2 4 4 4 6 6 5
193/300 Write block 1371 len 40960 data: 5 4 6 7 8 9 9 9 9 7
194/300 Write block 134 len 20480 data: 5 5 5 5 7
195/300 Read block 1814 len 24576
196/300 Flush
197/300 Read block 397 len 4096
198/300 Read block 1247 len 4096
199/300 Read block 4117 len 8192
200/300 Write block 2608 len 8192 data: 5 4
201/300 Flush
202/300 Read block 4949 len 8192
203/300 Write block 1400 len 28672 data: 7 6 5 5 4 7 7
204/300 Write block 4835 len 4096 data: 6
205/300 Write block 4197 len 12288 data: 7 5 4
206/300 Read block 3683 len 40960
207/300 Flush
208/300 Write block 2617 len 36864 data: 3 3 4 5 6 6 6 7 5
209/300 Read block 10 len 4096
210/300 Read block 4630 len 32768
211/300 Write block 4632 len 32768 data: 4 5 4 3 3 3 4 5
212/300 Write block 256 len 32768 data: 4 4 4 5 5 5 4 4
213/300 Flush
214/300 Read block 1982 len 8192
215/300 Read block 695 len 32768
216/300 Read block 1110 len 4096
217/300 Read block 2839 len 12288
218/300 Write block 4860 len 12288 data: 5 6 5
219/300 Write block 3748 len 16384 data: 6 6 4 3
220/300 Write block 1113 len 36864 data: 4 5 5 5 6 6 4 5 6
221/300 Read block 1410 len 12288
222/300 Read block 2779 len 20480
223/300 Read block 570 len 4096
224/300 Write block 1019 len 12288 data: 6 6 5
225/300 Read block 3354 len 4096
226/300 Read block 1635 len 36864
227/300 Read block 3405 len 16384
228/300 Write block 702 len 8192 data: 4 3
229/300 Read block 626 len 8192
230/300 Write block 2377 len 12288 data: 7 9 5
231/300 Write block 4273 len 32768 data: 4 4 7 6 6 6 6 7
232/300 Read block 4998 len 8192
233/300 Write block 2408 len 16384 data: 6 5 7 7
234/300 Read block 2049 len 20480
235/300 Read block 4762 len 28672
236/300 Read block 2218 len 40960
237/300 Read block 2397 len 32768
238/300 Write block 3646 len 28672 data: 4 3 4 5 5 5 3
239/300 Write block 4686 len 4096 data: 4
240/300 Read block 1606 len 8192
241/300 Read block 2533 len 32768
242/300 Read block 2551 len 24576
243/300 Read block 1938 len 8192
244/300 Read block 1766 len 8192
245/300 Read block 4703 len 28672
246/300 Read block 2235 len 8192
247/300 Write block 2133 len 24576 data: 4 4 2 3 3 3
248/300 Write block 1710 len 40960 data: 2 2 2 3 3 4 4 4 3 5
249/300 Read block 3553 len 40960
250/300 Write block 2278 len 28672 data: 3 3 3 4 4 5 4
251/300 Write block 3987 len 8192 data: 3 3
252/300 Write block 1878 len 4096 data: 4
253/300 Read block 470 len 32768
254/300 Read block 3806 len 4096
255/300 Write block 3444 len 24576 data: 4 4 3 3 2 5
256/300 Read block 4061 len 24576
257/300 Write block 3557 len 12288 data: 7 6 6
258/300 Flush
259/300 Read block 3653 len 40960
260/300 Write block 4777 len 20480 data: 7 6 4 3 3
261/300 Read block 3244 len 8192
262/300 Flush
263/300 Write block 2252 len 28672 data: 4 4 5 8 9 7 7
264/300 Flush
265/300 Read block 3718 len 40960
266/300 Read block 2049 len 28672
267/300 Read block 227 len 24576
268/300 Read block 3715 len 24576
269/300 Read block 133 len 32768
270/300 Write block 3830 len 24576 data: 5 4 4 4 3 2
271/300 Read block 4915 len 24576
272/300 Read block 1723 len 40960
273/300 Read block 3027 len 28672
274/300 Write block 1620 len 24576 data: 5 4 4 4 5 5
275/300 Read block 3586 len 20480
276/300 Write block 257 len 24576 data: 5 5 6 6 6 5
277/300 Write block 937 len 20480 data: 3 3 2 2 3
278/300 Write block 4358 len 28672 data: 3 3 2 3 3 4 4
279/300 Read block 1566 len 36864
280/300 Read block 1657 len 32768
281/300 Read block 1921 len 36864
282/300 Read block 4962 len 28672
283/300 Read block 4603 len 40960
284/300 Write block 2152 len 28672 data: 3 3 5 5 7 8 8
285/300 Write block 252 len 28672 data: 4 5 4 4 5 6 6
286/300 Read block 1289 len 40960
287/300 Flush
288/300 Flush
289/300 Write block 279 len 32768 data: 8 10 11 9 7 6 6 6
290/300 Read block 2339 len 28672
291/300 Flush
292/300 Read block 3729 len 4096
293/300 Read block 4371 len 12288
294/300 Write block 602 len 4096 data: 7
295/300 Write block 4718 len 8192 data: 6 3
296/300 Write block 4867 len 12288 data: 6 5 6
297/300 Read block 1337 len 32768
298/300 Read block 1734 len 4096
299/300 Flush
300/300 Read block 4266 len 12288
301/300 Read block 4599 len 28672
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:40:41.622 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/0, method: POST, req_id: 00035e63-56ef-4e12-b691-405374461798, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Sent Done Done false
6434 Acked Read 7 Sent Done Done false
6435 Acked Read 6 Sent Done Done false
6436 Acked Read 3 Sent Done Done false
6437 Acked Flush 0 Sent Done Done false
6438 Acked Read 10 Sent Done Done false
6439 Acked Read 5 Sent Done Done false
6440 Acked Flush 0 Sent Done Done false
6441 Acked Read 5 Sent Done Done false
6442 Acked Read 5 Sent Done Done false
6443 Acked Write 3 Sent Done Done false
6444 Acked Read 9 Sent Done Done false
6445 Acked Read 10 Sent Done Done false
6446 Acked Write 1 Sent Done Done false
6447 Acked Read 9 Sent Done Done false
6448 Acked Write 3 Sent Done Done false
6449 Acked Read 6 Sent Done Done false
6450 Acked Read 10 Sent Done Done false
6451 Acked Write 1 Sent Done Done false
6452 Acked Read 2 Sent Done Done false
6453 Acked Read 4 Sent Done Done false
6454 Acked Flush 0 Sent Done Done false
6455 Acked Write 1 Sent Done Done false
6456 Acked Read 8 Sent Done Done false
6457 Acked Read 8 Sent Done Done false
6458 Acked Read 9 Sent Done Done false
6459 Acked Write 8 Sent Done Done false
6460 Acked Read 2 Sent Done Done false
6461 Acked Read 9 Sent Done Done false
6462 Acked Write 10 Sent Done Done false
6463 Acked Write 10 Sent Done Done false
6464 Acked Flush 0 Sent Done Done false
6465 Acked Read 10 Sent Done Done false
6466 Acked Write 3 Sent Done Done false
6467 Acked Read 6 Sent Done Done false
6468 Acked Read 10 Sent Done Done false
6469 Acked Write 6 Sent Done Done false
6470 Acked Read 10 Sent Done Done false
6471 Acked Read 1 Sent Done Done false
6472 Acked Read 1 Sent Done Done false
6473 Acked Read 9 Sent Done Done false
6474 Acked Read 5 Sent Done Done false
6475 Acked Write 7 Sent Done Done false
6476 Acked Read 2 Sent Done Done false
6477 Acked Read 8 Sent Done Done false
6478 Acked Read 7 Sent Done Done false
6479 Acked Read 4 Sent Done Done false
6480 Acked Read 8 Sent Done Done false
6481 Acked Write 6 Sent Done Done false
6482 Acked Read 1 Sent Done Done false
6483 Acked Read 10 Sent Done Done false
6484 Acked Read 4 Sent Done Done false
6485 Acked Read 7 Sent Done Done false
6486 Acked Read 6 Sent Done Done false
6487 Acked Flush 0 Sent Done Done false
6488 Acked Read 2 Sent Done Done false
6489 Acked Flush 0 Sent Done Done false
6490 Acked Read 2 Sent Done Done false
6491 Acked Write 7 Sent Done Done false
6492 Acked Write 5 Sent Done Done false
6493 Acked Write 10 Sent Done Done false
6494 Acked Read 7 Sent Done Done false
6495 Acked Write 3 Sent Done Done false
6496 Acked Read 6 Sent Done Done false
6497 Acked Read 4 Sent Done Done false
6498 Acked Write 7 Sent Done Done false
6499 Acked Read 5 Sent Done Done false
6500 Acked Read 6 Sent Done Done false
6501 Acked Write 3 Sent Done Done false
6502 Acked Write 1 Sent Done Done false
6503 Acked Read 10 Sent Done Done false
6504 Acked Read 2 Sent Done Done false
6505 Acked Write 2 Sent Done Done false
6506 Acked Read 10 Sent Done Done false
6507 Acked Write 9 Sent Done Done false
6508 Acked Read 1 Sent Done Done false
6509 Acked Write 1 Sent Done Done false
6510 Acked Flush 0 Sent Done Done false
6511 Acked Read 7 Sent Done Done false
6512 Acked Read 4 Sent Done Done false
6513 Acked Flush 0 Sent Done Done false
6514 Acked Read 8 Sent Done Done false
6515 Acked Write 3 Sent Done Done false
6516 Acked Read 6 Sent Done Done false
6517 Acked Flush 0 Sent Done Done false
6518 Acked Read 2 Sent Done Done false
6519 Acked Read 3 Sent Done Done false
6520 Acked Read 2 Sent Done Done false
6521 Acked Flush 0 Sent Done Done false
6522 Acked Write 4 Sent Done Done false
6523 Acked Read 7 Sent Done Done false
6524 Acked Write 4 Sent Done Done false
6525 Acked Read 6 Sent Done Done false
6526 Acked Read 1 Sent Done Done false
6527 Acked Write 3 Sent Done Done false
6528 Acked Read 2 Sent Done Done false
6529 Acked Read 8 Sent Done Done false
6530 Acked Read 9 Sent Done Done false
6531 Acked Write 7 Sent Done Done false
6532 Acked Read 1 Sent Done Done false
6533 Acked Write 8 Sent Done Done false
6534 Acked Flush 0 Sent Done Done false
6535 Acked Flush 0 Sent Done Done false
6536 Acked Write 9 Sent Done Done false
6537 Acked Read 10 Sent Done Done false
6538 Acked Read 2 Sent Done Done false
6539 Acked Read 3 Sent Done Done false
6540 Acked Read 9 Sent Done Done false
6541 Acked Read 9 Sent Done Done false
6542 Acked Flush 0 Sent Done Done false
6543 Acked Write 2 Sent Done Done false
6544 Acked Read 5 Sent Done Done false
6545 Acked Read 7 Sent Done Done false
6546 Acked Read 6 Sent Done Done false
6547 Acked Write 6 Sent Done Done false
6548 Acked Write 9 Sent Done Done false
6549 Acked Write 5 Sent Done Done false
6550 Acked Flush 0 Sent Done Done false
6551 Acked Flush 0 Sent Done Done false
6552 Acked Read 8 Sent Done Done false
6553 Acked Read 7 Sent Done Done false
6554 Acked Read 6 Sent Done Done false
6555 Acked Read 5 Sent Done Done false
6556 Acked Write 4 Sent Done Done false
6557 Acked Flush 0 Sent Done Done false
6558 Acked Read 7 Sent Done Done false
6559 Acked Write 2 Sent Done Done false
6560 Acked Read 1 Sent Done Done false
6561 Acked Read 8 Sent Done Done false
6562 Acked Read 9 Sent Done Done false
6563 Acked Write 10 Sent Done Done false
6564 Acked Write 3 Sent Done Done false
6565 Acked Read 5 Sent Done Done false
6566 Acked Flush 0 Sent Done Done false
6567 Acked Read 2 Sent Done Done false
6568 Acked Write 1 Sent Done Done false
6569 Acked Read 3 Sent Done Done false
6570 Acked Write 10 Sent Done Done false
6571 Acked Write 6 Sent Done Done false
6572 Acked Read 10 Sent Done Done false
6573 Acked Read 5 Sent Done Done false
6574 Acked Read 5 Sent Done Done false
6575 Acked Read 3 Sent Done Done false
6576 Acked Write 3 Sent Done Done false
6577 Acked Read 10 Sent Done Done false
6578 Acked Read 8 Sent Done Done false
6579 Acked Flush 0 Sent Done Done false
6580 Acked Write 4 Sent Done Done false
6581 Acked Write 9 Sent Done Done false
6582 Acked Read 3 Sent Done Done false
6583 Acked Flush 0 Sent Done Done false
6584 Acked Read 8 Sent Done Done false
6585 Acked Read 2 Sent Done Done false
6586 Acked Read 9 Sent Done Done false
6587 Acked Write 5 Sent Done Done false
6588 Acked Read 1 Sent Done Done false
6589 Acked Read 4 Sent Done Done false
6590 Acked Read 1 Sent Done Done false
6591 Acked Read 4 Sent Done Done false
6592 Acked Read 4 Sent Done Done false
6593 Acked Read 7 Sent Done Done false
6594 Acked Read 6 Sent Done Done false
6595 Acked Write 9 Sent Done Done false
6596 Acked Write 8 Sent Done Done false
6597 Acked Write 7 Sent Done Done false
6598 Acked Read 10 Sent Done Done false
6599 Acked Read 6 Sent Done Done false
6600 Acked Read 7 Sent Done Done false
6601 Acked Read 7 Sent Done Done false
6602 Acked Flush 0 Sent Done Done false
6603 Acked Write 3 Sent Done Done false
6604 Acked Write 10 Sent Done Done false
6605 Acked Write 2 Sent Done Done false
6606 Acked Write 4 Sent Done Done false
6607 Acked Read 1 Sent Done Done false
6608 Acked Write 9 Sent Done Done false
6609 Acked Flush 0 Sent Done Done false
6610 Acked Write 6 Sent Done Done false
6611 Acked Flush 0 Sent Done Done false
6612 Acked Read 5 Sent Done Done false
6613 Acked Read 2 Sent Done Done false
6614 Acked Read 5 Sent Done Done false
6615 Acked Write 10 Sent Done Done false
6616 Acked Read 9 Sent Done Done false
6617 Acked Write 9 Sent Done Done false
6618 Acked Flush 0 Sent Done Done false
6619 Acked Read 3 Sent Done Done false
6620 Acked Read 6 Sent Done Done false
6621 Acked Read 3 Sent Done Done false
6622 Acked Write 4 Sent Done Done false
6623 Acked Read 5 Sent Done Done false
6624 Acked Write 10 Sent Done Done false
6625 Acked Write 10 Sent Done Done false
6626 Acked Write 5 Sent Done Done false
6627 Acked Read 6 Sent Done Done false
6628 Acked Flush 0 Sent Done Done false
6629 Acked Read 1 Sent Done Done false
6630 Acked Read 1 Sent Done Done false
6631 Acked Read 2 Sent Done Done false
6632 Acked Write 2 Sent Done Done false
6633 Acked Flush 0 Sent Done Done false
6634 Acked Read 2 Sent Done Done false
6635 Acked Write 7 Sent Done Done false
6636 Acked Write 1 Sent Done Done false
6637 Acked Write 3 Sent Done Done false
6638 Acked Read 10 Sent Done Done false
6639 Acked Flush 0 Sent Done Done false
6640 Acked Write 9 Sent Done Done false
6641 Acked Read 1 Sent Done Done false
6642 Acked Read 8 Sent Done Done false
6643 Acked Write 8 Sent Done Done false
6644 Acked Write 8 Sent Done Done false
6645 Acked Flush 0 Sent Done Done false
6646 Acked Read 2 Sent Done Done false
6647 Acked Read 8 Sent Done Done false
6648 Acked Read 1 Sent Done Done false
6649 Acked Read 3 Sent Done Done false
6650 Acked Write 3 Sent Done Done false
6651 Acked Write 4 Sent Done Done false
6652 Acked Write 9 Sent Done Done false
6653 Acked Read 3 Sent Done Done false
6654 Acked Read 5 Sent Done Done false
6655 Acked Read 1 Sent Done Done false
6656 Acked Write 3 Sent Done Done false
6657 Acked Read 1 Sent Done Done false
6658 Acked Read 9 Sent Done Done false
6659 Acked Read 4 Sent Done Done false
6660 Acked Write 2 Sent Done Done false
6661 Acked Read 2 Sent Done Done false
6662 Acked Write 3 Sent Done Done false
6663 Acked Write 8 Sent Done Done false
6664 Acked Read 2 Sent Done Done false
6665 Acked Write 4 Sent Done Done false
6666 Acked Read 5 Sent Done Done false
6667 Acked Read 7 Sent Done Done false
6668 Acked Read 10 Sent Done Done false
6669 Acked Read 8 Sent Done Done false
6670 Acked Write 7 Sent Done Done false
6671 Acked Write 1 Sent Done Done false
6672 Acked Read 2 Sent Done Done false
6673 Acked Read 8 Sent Done Done false
6674 Acked Read 6 Sent Done Done false
6675 Acked Read 2 Sent Done Done false
6676 Acked Read 2 Sent Done Done false
6677 Acked Read 7 Sent Done Done false
6678 Acked Read 2 Sent Done Done false
6679 Acked Write 6 Sent Done Done false
6680 Acked Write 10 Sent Done Done false
6681 Acked Read 10 Sent Done Done false
6682 Acked Write 7 Sent Done Done false
6683 Acked Write 2 Sent Done Done false
6684 Acked Write 1 Sent Done Done false
6685 Acked Read 8 Sent Done Done false
6686 Acked Read 1 Sent Done Done false
6687 Acked Write 6 Sent Done Done false
6688 Acked Read 6 Sent Done Done false
6689 Acked Write 3 Sent Done Done false
6690 Acked Flush 0 Sent Done Done false
6691 Acked Read 10 Sent Done Done false
6692 Acked Write 5 Sent Done Done false
6693 Acked Read 2 Sent Done Done false
6694 Acked Flush 0 Sent Done Done false
6695 Acked Write 7 Sent Done Done false
6696 Acked Flush 0 Sent Done Done false
6697 Acked Read 10 Sent Done Done false
6698 Acked Read 7 Sent Done Done false
6699 Acked Read 6 Sent Done Done false
6700 Acked Read 6 Sent Done Done false
6701 Acked Read 8 Sent Done Done false
6702 Acked Write 6 Sent Done Done false
6703 Acked Read 6 Sent Done Done false
6704 Acked Read 10 Sent Done Done false
6705 Acked Read 7 Sent Done Done false
6706 Acked Write 6 Sent Done Done false
6707 Acked Read 5 Sent Done Done false
6708 Acked Write 6 Sent Done Done false
6709 Acked Write 5 Sent Done Done false
6710 Acked Write 7 Sent Done Done false
6711 Acked Read 9 Sent Done Done false
6712 Acked Read 8 Sent Done Done false
6713 Acked Read 9 Sent Done Done false
6714 Acked Read 7 Sent Done Done false
6715 Acked Read 10 Sent Done Done false
6716 Acked Write 7 Sent Done Done false
6717 Acked Write 7 Sent Done Done false
6718 Acked Read 10 Sent Done Done false
6719 Acked Flush 0 Sent Done Done false
6720 Acked Flush 0 Sent Done Done false
6721 Acked Write 8 Sent Done Done false
6722 Acked Read 7 Sent Done Done false
6723 Acked Flush 0 Sent Done Done false
6724 Acked Read 1 Sent Done Done false
6725 Acked Read 3 Sent Done Done false
6726 Acked Write 1 Sent Done Done false
6727 Acked Write 2 Sent Done Done false
6728 Acked Write 3 Sent Done Done false
6729 Acked Read 8 Sent Done Done false
6730 Acked Read 1 Sent Done Done false
6731 Acked Flush 0 Sent Done Done false
6732 Acked Read 3 Sent Done Done false
6733 Acked Read 7 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6731 6731
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6733 6732 6731 6730 6729
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Sent Done Done false
6434 Acked Read 7 Sent Done Done false
6435 Acked Read 6 Sent Done Done false
6436 Acked Read 3 Sent Done Done false
6437 Acked Flush 0 Sent Done Done false
6438 Acked Read 10 Sent Done Done false
6439 Acked Read 5 Sent Done Done false
6440 Acked Flush 0 Sent Done Done false
6441 Acked Read 5 Sent Done Done false
6442 Acked Read 5 Sent Done Done false
6443 Acked Write 3 Sent Done Done false
6444 Acked Read 9 Sent Done Done false
6445 Acked Read 10 Sent Done Done false
6446 Acked Write 1 Sent Done Done false
6447 Acked Read 9 Sent Done Done false
6448 Acked Write 3 Sent Done Done false
6449 Acked Read 6 Sent Done Done false
6450 Acked Read 10 Sent Done Done false
6451 Acked Write 1 Sent Done Done false
6452 Acked Read 2 Sent Done Done false
6453 Acked Read 4 Sent Done Done false
6454 Acked Flush 0 Sent Done Done false
6455 Acked Write 1 Sent Done Done false
6456 Acked Read 8 Sent Done Done false
6457 Acked Read 8 Sent Done Done false
6458 Acked Read 9 Sent Done Done false
6459 Acked Write 8 Sent Done Done false
6460 Acked Read 2 Sent Done Done false
6461 Acked Read 9 Sent Done Done false
6462 Acked Write 10 Sent Done Done false
6463 Acked Write 10 Sent Done Done false
6464 Acked Flush 0 Sent Done Done false
6465 Acked Read 10 Sent Done Done false
6466 Acked Write 3 Sent Done Done false
6467 Acked Read 6 Sent Done Done false
6468 Acked Read 10 Sent Done Done false
6469 Acked Write 6 Sent Done Done false
6470 Acked Read 10 Sent Done Done false
6471 Acked Read 1 Sent Done Done false
6472 Acked Read 1 Sent Done Done false
6473 Acked Read 9 Sent Done Done false
6474 Acked Read 5 Sent Done Done false
6475 Acked Write 7 Sent Done Done false
6476 Acked Read 2 Sent Done Done false
6477 Acked Read 8 Sent Done Done false
6478 Acked Read 7 Sent Done Done false
6479 Acked Read 4 Sent Done Done false
6480 Acked Read 8 Sent Done Done false
6481 Acked Write 6 Sent Done Done false
6482 Acked Read 1 Sent Done Done false
6483 Acked Read 10 Sent Done Done false
6484 Acked Read 4 Sent Done Done false
6485 Acked Read 7 Sent Done Done false
6486 Acked Read 6 Sent Done Done false
6487 Acked Flush 0 Sent Done Done false
6488 Acked Read 2 Sent Done Done false
6489 Acked Flush 0 Sent Done Done false
6490 Acked Read 2 Sent Done Done false
6491 Acked Write 7 Sent Done Done false
6492 Acked Write 5 Sent Done Done false
6493 Acked Write 10 Sent Done Done false
6494 Acked Read 7 Sent Done Done false
6495 Acked Write 3 Sent Done Done false
6496 Acked Read 6 Sent Done Done false
6497 Acked Read 4 Sent Done Done false
6498 Acked Write 7 Sent Done Done false
6499 Acked Read 5 Sent Done Done false
6500 Acked Read 6 Sent Done Done false
6501 Acked Write 3 Sent Done Done false
6502 Acked Write 1 Sent Done Done false
6503 Acked Read 10 Sent Done Done false
6504 Acked Read 2 Sent Done Done false
6505 Acked Write 2 Sent Done Done false
6506 Acked Read 10 Sent Done Done false
6507 Acked Write 9 Sent Done Done false
6508 Acked Read 1 Sent Done Done false
6509 Acked Write 1 Sent Done Done false
6510 Acked Flush 0 Sent Done Done false
6511 Acked Read 7 Sent Done Done false
6512 Acked Read 4 Sent Done Done false
6513 Acked Flush 0 Sent Done Done false
6514 Acked Read 8 Sent Done Done false
6515 Acked Write 3 Sent Done Done false
6516 Acked Read 6 Sent Done Done false
6517 Acked Flush 0 Sent Done Done false
6518 Acked Read 2 Sent Done Done false
6519 Acked Read 3 Sent Done Done false
6520 Acked Read 2 Sent Done Done false
6521 Acked Flush 0 Sent Done Done false
6522 Acked Write 4 Sent Done Done false
6523 Acked Read 7 Sent Done Done false
6524 Acked Write 4 Sent Done Done false
6525 Acked Read 6 Sent Done Done false
6526 Acked Read 1 Sent Done Done false
6527 Acked Write 3 Sent Done Done false
6528 Acked Read 2 Sent Done Done false
6529 Acked Read 8 Sent Done Done false
6530 Acked Read 9 Sent Done Done false
6531 Acked Write 7 Sent Done Done false
6532 Acked Read 1 Sent Done Done false
6533 Acked Write 8 Sent Done Done false
6534 Acked Flush 0 Sent Done Done false
6535 Acked Flush 0 Sent Done Done false
6536 Acked Write 9 Sent Done Done false
6537 Acked Read 10 Sent Done Done false
6538 Acked Read 2 Sent Done Done false
6539 Acked Read 3 Sent Done Done false
6540 Acked Read 9 Sent Done Done false
6541 Acked Read 9 Sent Done Done false
6542 Acked Flush 0 Sent Done Done false
6543 Acked Write 2 Sent Done Done false
6544 Acked Read 5 Sent Done Done false
6545 Acked Read 7 Sent Done Done false
6546 Acked Read 6 Sent Done Done false
6547 Acked Write 6 Sent Done Done false
6548 Acked Write 9 Sent Done Done false
6549 Acked Write 5 Sent Done Done false
6550 Acked Flush 0 Sent Done Done false
6551 Acked Flush 0 Sent Done Done false
6552 Acked Read 8 Sent Done Done false
6553 Acked Read 7 Sent Done Done false
6554 Acked Read 6 Sent Done Done false
6555 Acked Read 5 Sent Done Done false
6556 Acked Write 4 Sent Done Done false
6557 Acked Flush 0 Sent Done Done false
6558 Acked Read 7 Sent Done Done false
6559 Acked Write 2 Sent Done Done false
6560 Acked Read 1 Sent Done Done false
6561 Acked Read 8 Sent Done Done false
6562 Acked Read 9 Sent Done Done false
6563 Acked Write 10 Sent Done Done false
6564 Acked Write 3 Sent Done Done false
6565 Acked Read 5 Sent Done Done false
6566 Acked Flush 0 Sent Done Done false
6567 Acked Read 2 Sent Done Done false
6568 Acked Write 1 Sent Done Done false
6569 Acked Read 3 Sent Done Done false
6570 Acked Write 10 Sent Done Done false
6571 Acked Write 6 Sent Done Done false
6572 Acked Read 10 Sent Done Done false
6573 Acked Read 5 Sent Done Done false
6574 Acked Read 5 Sent Done Done false
6575 Acked Read 3 Sent Done Done false
6576 Acked Write 3 Sent Done Done false
6577 Acked Read 10 Sent Done Done false
6578 Acked Read 8 Sent Done Done false
6579 Acked Flush 0 Sent Done Done false
6580 Acked Write 4 Sent Done Done false
6581 Acked Write 9 Sent Done Done false
6582 Acked Read 3 Sent Done Done false
6583 Acked Flush 0 Sent Done Done false
6584 Acked Read 8 Sent Done Done false
6585 Acked Read 2 Sent Done Done false
6586 Acked Read 9 Sent Done Done false
6587 Acked Write 5 Sent Done Done false
6588 Acked Read 1 Sent Done Done false
6589 Acked Read 4 Sent Done Done false
6590 Acked Read 1 Sent Done Done false
6591 Acked Read 4 Sent Done Done false
6592 Acked Read 4 Sent Done Done false
6593 Acked Read 7 Sent Done Done false
6594 Acked Read 6 Sent Done Done false
6595 Acked Write 9 Sent Done Done false
6596 Acked Write 8 Sent Done Done false
6597 Acked Write 7 Sent Done Done false
6598 Acked Read 10 Sent Done Done false
6599 Acked Read 6 Sent Done Done false
6600 Acked Read 7 Sent Done Done false
6601 Acked Read 7 Sent Done Done false
6602 Acked Flush 0 Sent Done Done false
6603 Acked Write 3 Sent Done Done false
6604 Acked Write 10 Sent Done Done false
6605 Acked Write 2 Sent Done Done false
6606 Acked Write 4 Sent Done Done false
6607 Acked Read 1 Sent Done Done false
6608 Acked Write 9 Sent Done Done false
6609 Acked Flush 0 Sent Done Done false
6610 Acked Write 6 Sent Done Done false
6611 Acked Flush 0 Sent Done Done false
6612 Acked Read 5 Sent Done Done false
6613 Acked Read 2 Sent Done Done false
6614 Acked Read 5 Sent Done Done false
6615 Acked Write 10 Sent Done Done false
6616 Acked Read 9 Sent Done Done false
6617 Acked Write 9 Sent Done Done false
6618 Acked Flush 0 Sent Done Done false
6619 Acked Read 3 Sent Done Done false
6620 Acked Read 6 Sent Done Done false
6621 Acked Read 3 Sent Done Done false
6622 Acked Write 4 Sent Done Done false
6623 Acked Read 5 Sent Done Done false
6624 Acked Write 10 Sent Done Done false
6625 Acked Write 10 Sent Done Done false
6626 Acked Write 5 Sent Done Done false
6627 Acked Read 6 Sent Done Done false
6628 Acked Flush 0 Sent Done Done false
6629 Acked Read 1 Sent Done Done false
6630 Acked Read 1 Sent Done Done false
6631 Acked Read 2 Sent Done Done false
6632 Acked Write 2 Sent Done Done false
6633 Acked Flush 0 Sent Done Done false
6634 Acked Read 2 Sent Done Done false
6635 Acked Write 7 Sent Done Done false
6636 Acked Write 1 Sent Done Done false
6637 Acked Write 3 Sent Done Done false
6638 Acked Read 10 Sent Done Done false
6639 Acked Flush 0 Sent Done Done false
6640 Acked Write 9 Sent Done Done false
6641 Acked Read 1 Sent Done Done false
6642 Acked Read 8 Sent Done Done false
6643 Acked Write 8 Sent Done Done false
6644 Acked Write 8 Sent Done Done false
6645 Acked Flush 0 Sent Done Done false
6646 Acked Read 2 Sent Done Done false
6647 Acked Read 8 Sent Done Done false
6648 Acked Read 1 Sent Done Done false
6649 Acked Read 3 Sent Done Done false
6650 Acked Write 3 Sent Done Done false
6651 Acked Write 4 Sent Done Done false
6652 Acked Write 9 Sent Done Done false
6653 Acked Read 3 Sent Done Done false
6654 Acked Read 5 Sent Done Done false
6655 Acked Read 1 Sent Done Done false
6656 Acked Write 3 Sent Done Done false
6657 Acked Read 1 Sent Done Done false
6658 Acked Read 9 Sent Done Done false
6659 Acked Read 4 Sent Done Done false
6660 Acked Write 2 Sent Done Done false
6661 Acked Read 2 Sent Done Done false
6662 Acked Write 3 Sent Done Done false
6663 Acked Write 8 Sent Done Done false
6664 Acked Read 2 Sent Done Done false
6665 Acked Write 4 Sent Done Done false
6666 Acked Read 5 Sent Done Done false
6667 Acked Read 7 Sent Done Done false
6668 Acked Read 10 Sent Done Done false
6669 Acked Read 8 Sent Done Done false
6670 Acked Write 7 Sent Done Done false
6671 Acked Write 1 Sent Done Done false
6672 Acked Read 2 Sent Done Done false
6673 Acked Read 8 Sent Done Done false
6674 Acked Read 6 Sent Done Done false
6675 Acked Read 2 Sent Done Done false
6676 Acked Read 2 Sent Done Done false
6677 Acked Read 7 Sent Done Done false
6678 Acked Read 2 Sent Done Done false
6679 Acked Write 6 Sent Done Done false
6680 Acked Write 10 Sent Done Done false
6681 Acked Read 10 Sent Done Done false
6682 Acked Write 7 Sent Done Done false
6683 Acked Write 2 Sent Done Done false
6684 Acked Write 1 Sent Done Done false
6685 Acked Read 8 Sent Done Done false
6686 Acked Read 1 Sent Done Done false
6687 Acked Write 6 Sent Done Done false
6688 Acked Read 6 Sent Done Done false
6689 Acked Write 3 Sent Done Done false
6690 Acked Flush 0 Sent Done Done false
6691 Acked Read 10 Sent Done Done false
6692 Acked Write 5 Sent Done Done false
6693 Acked Read 2 Sent Done Done false
6694 Acked Flush 0 Sent Done Done false
6695 Acked Write 7 Sent Done Done false
6696 Acked Flush 0 Sent Done Done false
6697 Acked Read 10 Sent Done Done false
6698 Acked Read 7 Sent Done Done false
6699 Acked Read 6 Sent Done Done false
6700 Acked Read 6 Sent Done Done false
6701 Acked Read 8 Sent Done Done false
6702 Acked Write 6 Sent Done Done false
6703 Acked Read 6 Sent Done Done false
6704 Acked Read 10 Sent Done Done false
6705 Acked Read 7 Sent Done Done false
6706 Acked Write 6 Sent Done Done false
6707 Acked Read 5 Sent Done Done false
6708 Acked Write 6 Sent Done Done false
6709 Acked Write 5 Sent Done Done false
6710 Acked Write 7 Sent Done Done false
6711 Acked Read 9 Sent Done Done false
6712 Acked Read 8 Sent Done Done false
6713 Acked Read 9 Sent Done Done false
6714 Acked Read 7 Sent Done Done false
6715 Acked Read 10 Sent Done Done false
6716 Acked Write 7 Sent Done Done false
6717 Acked Write 7 Sent Done Done false
6718 Acked Read 10 Sent Done Done false
6719 Acked Flush 0 Sent Done Done false
6720 Acked Flush 0 Sent Done Done false
6721 Acked Write 8 Sent Done Done false
6722 Acked Read 7 Sent Done Done false
6723 Acked Flush 0 Sent Done Done false
6724 Acked Read 1 Sent Done Done false
6725 Acked Read 3 Sent Done Done false
6726 Acked Write 1 Sent Done Done false
6727 Acked Write 2 Sent Done Done false
6728 Acked Write 3 Sent Done Done false
6729 Acked Read 8 Sent Done Done false
6730 Acked Read 1 Sent Done Done false
6731 Acked Flush 0 Sent Done Done false
6732 Acked Read 3 Sent Done Done false
6733 Acked Read 7 Sent Done Done false
6734 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6734 6734
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6734 6733 6732 6731 6730
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:47.374341051Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:47.374440239Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:47.374493603Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:47.374532338Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:47.374582652Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Sent Done Done false
6434 Acked Read 7 Sent Done Done false
6435 Acked Read 6 Sent Done Done false
6436 Acked Read 3 Sent Done Done false
6437 Acked Flush 0 Sent Done Done false
6438 Acked Read 10 Sent Done Done false
6439 Acked Read 5 Sent Done Done false
6440 Acked Flush 0 Sent Done Done false
6441 Acked Read 5 Sent Done Done false
6442 Acked Read 5 Sent Done Done false
6443 Acked Write 3 Sent Done Done false
6444 Acked Read 9 Sent Done Done false
6445 Acked Read 10 Sent Done Done false
6446 Acked Write 1 Sent Done Done false
6447 Acked Read 9 Sent Done Done false
6448 Acked Write 3 Sent Done Done false
6449 Acked Read 6 Sent Done Done false
6450 Acked Read 10 Sent Done Done false
6451 Acked Write 1 Sent Done Done false
6452 Acked Read 2 Sent Done Done false
6453 Acked Read 4 Sent Done Done false
6454 Acked Flush 0 Sent Done Done false
6455 Acked Write 1 Sent Done Done false
6456 Acked Read 8 Sent Done Done false
6457 Acked Read 8 Sent Done Done false
6458 Acked Read 9 Sent Done Done false
6459 Acked Write 8 Sent Done Done false
6460 Acked Read 2 Sent Done Done false
6461 Acked Read 9 Sent Done Done false
6462 Acked Write 10 Sent Done Done false
6463 Acked Write 10 Sent Done Done false
6464 Acked Flush 0 Sent Done Done false
6465 Acked Read 10 Sent Done Done false
6466 Acked Write 3 Sent Done Done false
6467 Acked Read 6 Sent Done Done false
6468 Acked Read 10 Sent Done Done false
6469 Acked Write 6 Sent Done Done false
6470 Acked Read 10 Sent Done Done false
6471 Acked Read 1 Sent Done Done false
6472 Acked Read 1 Sent Done Done false
6473 Acked Read 9 Sent Done Done false
6474 Acked Read 5 Sent Done Done false
6475 Acked Write 7 Sent Done Done false
6476 Acked Read 2 Sent Done Done false
6477 Acked Read 8 Sent Done Done false
6478 Acked Read 7 Sent Done Done false
6479 Acked Read 4 Sent Done Done false
6480 Acked Read 8 Sent Done Done false
6481 Acked Write 6 Sent Done Done false
6482 Acked Read 1 Sent Done Done false
6483 Acked Read 10 Sent Done Done false
6484 Acked Read 4 Sent Done Done false
6485 Acked Read 7 Sent Done Done false
6486 Acked Read 6 Sent Done Done false
6487 Acked Flush 0 Sent Done Done false
6488 Acked Read 2 Sent Done Done false
6489 Acked Flush 0 Sent Done Done false
6490 Acked Read 2 Sent Done Done false
6491 Acked Write 7 Sent Done Done false
6492 Acked Write 5 Sent Done Done false
6493 Acked Write 10 Sent Done Done false
6494 Acked Read 7 Sent Done Done false
6495 Acked Write 3 Sent Done Done false
6496 Acked Read 6 Sent Done Done false
6497 Acked Read 4 Sent Done Done false
6498 Acked Write 7 Sent Done Done false
6499 Acked Read 5 Sent Done Done false
6500 Acked Read 6 Sent Done Done false
6501 Acked Write 3 Sent Done Done false
6502 Acked Write 1 Sent Done Done false
6503 Acked Read 10 Sent Done Done false
6504 Acked Read 2 Sent Done Done false
6505 Acked Write 2 Sent Done Done false
6506 Acked Read 10 Sent Done Done false
6507 Acked Write 9 Sent Done Done false
6508 Acked Read 1 Sent Done Done false
6509 Acked Write 1 Sent Done Done false
6510 Acked Flush 0 Sent Done Done false
6511 Acked Read 7 Sent Done Done false
6512 Acked Read 4 Sent Done Done false
6513 Acked Flush 0 Sent Done Done false
6514 Acked Read 8 Sent Done Done false
6515 Acked Write 3 Sent Done Done false
6516 Acked Read 6 Sent Done Done false
6517 Acked Flush 0 Sent Done Done false
6518 Acked Read 2 Sent Done Done false
6519 Acked Read 3 Sent Done Done false
6520 Acked Read 2 Sent Done Done false
6521 Acked Flush 0 Sent Done Done false
6522 Acked Write 4 Sent Done Done false
6523 Acked Read 7 Sent Done Done false
6524 Acked Write 4 Sent Done Done false
6525 Acked Read 6 Sent Done Done false
6526 Acked Read 1 Sent Done Done false
6527 Acked Write 3 Sent Done Done false
6528 Acked Read 2 Sent Done Done false
6529 Acked Read 8 Sent Done Done false
6530 Acked Read 9 Sent Done Done false
6531 Acked Write 7 Sent Done Done false
6532 Acked Read 1 Sent Done Done false
6533 Acked Write 8 Sent Done Done false
6534 Acked Flush 0 Sent Done Done false
6535 Acked Flush 0 Sent Done Done false
6536 Acked Write 9 Sent Done Done false
6537 Acked Read 10 Sent Done Done false
6538 Acked Read 2 Sent Done Done false
6539 Acked Read 3 Sent Done Done false
6540 Acked Read 9 Sent Done Done false
6541 Acked Read 9 Sent Done Done false
6542 Acked Flush 0 Sent Done Done false
6543 Acked Write 2 Sent Done Done false
6544 Acked Read 5 Sent Done Done false
6545 Acked Read 7 Sent Done Done false
6546 Acked Read 6 Sent Done Done false
6547 Acked Write 6 Sent Done Done false
6548 Acked Write 9 Sent Done Done false
6549 Acked Write 5 Sent Done Done false
6550 Acked Flush 0 Sent Done Done false
6551 Acked Flush 0 Sent Done Done false
6552 Acked Read 8 Sent Done Done false
6553 Acked Read 7 Sent Done Done false
6554 Acked Read 6 Sent Done Done false
6555 Acked Read 5 Sent Done Done false
6556 Acked Write 4 Sent Done Done false
6557 Acked Flush 0 Sent Done Done false
6558 Acked Read 7 Sent Done Done false
6559 Acked Write 2 Sent Done Done false
6560 Acked Read 1 Sent Done Done false
6561 Acked Read 8 Sent Done Done false
6562 Acked Read 9 Sent Done Done false
6563 Acked Write 10 Sent Done Done false
6564 Acked Write 3 Sent Done Done false
6565 Acked Read 5 Sent Done Done false
6566 Acked Flush 0 Sent Done Done false
6567 Acked Read 2 Sent Done Done false
6568 Acked Write 1 Sent Done Done false
6569 Acked Read 3 Sent Done Done false
6570 Acked Write 10 Sent Done Done false
6571 Acked Write 6 Sent Done Done false
6572 Acked Read 10 Sent Done Done false
6573 Acked Read 5 Sent Done Done false
6574 Acked Read 5 Sent Done Done false
6575 Acked Read 3 Sent Done Done false
6576 Acked Write 3 Sent Done Done false
6577 Acked Read 10 Sent Done Done false
6578 Acked Read 8 Sent Done Done false
6579 Acked Flush 0 Sent Done Done false
6580 Acked Write 4 Sent Done Done false
6581 Acked Write 9 Sent Done Done false
6582 Acked Read 3 Sent Done Done false
6583 Acked Flush 0 Sent Done Done false
6584 Acked Read 8 Sent Done Done false
6585 Acked Read 2 Sent Done Done false
6586 Acked Read 9 Sent Done Done false
6587 Acked Write 5 Sent Done Done false
6588 Acked Read 1 Sent Done Done false
6589 Acked Read 4 Sent Done Done false
6590 Acked Read 1 Sent Done Done false
6591 Acked Read 4 Sent Done Done false
6592 Acked Read 4 Sent Done Done false
6593 Acked Read 7 Sent Done Done false
6594 Acked Read 6 Sent Done Done false
6595 Acked Write 9 Sent Done Done false
6596 Acked Write 8 Sent Done Done false
6597 Acked Write 7 Sent Done Done false
6598 Acked Read 10 Sent Done Done false
6599 Acked Read 6 Sent Done Done false
6600 Acked Read 7 Sent Done Done false
6601 Acked Read 7 Sent Done Done false
6602 Acked Flush 0 Sent Done Done false
6603 Acked Write 3 Sent Done Done false
6604 Acked Write 10 Sent Done Done false
6605 Acked Write 2 Sent Done Done false
6606 Acked Write 4 Sent Done Done false
6607 Acked Read 1 Sent Done Done false
6608 Acked Write 9 Sent Done Done false
6609 Acked Flush 0 Sent Done Done false
6610 Acked Write 6 Sent Done Done false
6611 Acked Flush 0 Sent Done Done false
6612 Acked Read 5 Sent Done Done false
6613 Acked Read 2 Sent Done Done false
6614 Acked Read 5 Sent Done Done false
6615 Acked Write 10 Sent Done Done false
6616 Acked Read 9 Sent Done Done false
6617 Acked Write 9 Sent Done Done false
6618 Acked Flush 0 Sent Done Done false
6619 Acked Read 3 Sent Done Done false
6620 Acked Read 6 Sent Done Done false
6621 Acked Read 3 Sent Done Done false
6622 Acked Write 4 Sent Done Done false
6623 Acked Read 5 Sent Done Done false
6624 Acked Write 10 Sent Done Done false
6625 Acked Write 10 Sent Done Done false
6626 Acked Write 5 Sent Done Done false
6627 Acked Read 6 Sent Done Done false
6628 Acked Flush 0 Sent Done Done false
6629 Acked Read 1 Sent Done Done false
6630 Acked Read 1 Sent Done Done false
6631 Acked Read 2 Sent Done Done false
6632 Acked Write 2 Sent Done Done false
6633 Acked Flush 0 Sent Done Done false
6634 Acked Read 2 Sent Done Done false
6635 Acked Write 7 Sent Done Done false
6636 Acked Write 1 Sent Done Done false
6637 Acked Write 3 Sent Done Done false
6638 Acked Read 10 Sent Done Done false
6639 Acked Flush 0 Sent Done Done false
6640 Acked Write 9 Sent Done Done false
6641 Acked Read 1 Sent Done Done false
6642 Acked Read 8 Sent Done Done false
6643 Acked Write 8 Sent Done Done false
6644 Acked Write 8 Sent Done Done false
6645 Acked Flush 0 Sent Done Done false
6646 Acked Read 2 Sent Done Done false
6647 Acked Read 8 Sent Done Done false
6648 Acked Read 1 Sent Done Done false
6649 Acked Read 3 Sent Done Done false
6650 Acked Write 3 Sent Done Done false
6651 Acked Write 4 Sent Done Done false
6652 Acked Write 9 Sent Done Done false
6653 Acked Read 3 Sent Done Done false
6654 Acked Read 5 Sent Done Done false
6655 Acked Read 1 Sent Done Done false
6656 Acked Write 3 Sent Done Done false
6657 Acked Read 1 Sent Done Done false
6658 Acked Read 9 Sent Done Done false
6659 Acked Read 4 Sent Done Done false
6660 Acked Write 2 Sent Done Done false
6661 Acked Read 2 Sent Done Done false
6662 Acked Write 3 Sent Done Done false
6663 Acked Write 8 Sent Done Done false
6664 Acked Read 2 Sent Done Done false
6665 Acked Write 4 Sent Done Done false
6666 Acked Read 5 Sent Done Done false
6667 Acked Read 7 Sent Done Done false
6668 Acked Read 10 Sent Done Done false
6669 Acked Read 8 Sent Done Done false
6670 Acked Write 7 Sent Done Done false
6671 Acked Write 1 Sent Done Done false
6672 Acked Read 2 Sent Done Done false
6673 Acked Read 8 Sent Done Done false
6674 Acked Read 6 Sent Done Done false
6675 Acked Read 2 Sent Done Done false
6676 Acked Read 2 Sent Done Done false
6677 Acked Read 7 Sent Done Done false
6678 Acked Read 2 Sent Done Done false
6679 Acked Write 6 Sent Done Done false
6680 Acked Write 10 Sent Done Done false
6681 Acked Read 10 Sent Done Done false
6682 Acked Write 7 Sent Done Done false
6683 Acked Write 2 Sent Done Done false
6684 Acked Write 1 Sent Done Done false
6685 Acked Read 8 Sent Done Done false
6686 Acked Read 1 Sent Done Done false
6687 Acked Write 6 Sent Done Done false
6688 Acked Read 6 Sent Done Done false
6689 Acked Write 3 Sent Done Done false
6690 Acked Flush 0 Sent Done Done false
6691 Acked Read 10 Sent Done Done false
6692 Acked Write 5 Sent Done Done false
6693 Acked Read 2 Sent Done Done false
6694 Acked Flush 0 Sent Done Done false
6695 Acked Write 7 Sent Done Done false
6696 Acked Flush 0 Sent Done Done false
6697 Acked Read 10 Sent Done Done false
6698 Acked Read 7 Sent Done Done false
6699 Acked Read 6 Sent Done Done false
6700 Acked Read 6 Sent Done Done false
6701 Acked Read 8 Sent Done Done false
6702 Acked Write 6 Sent Done Done false
6703 Acked Read 6 Sent Done Done false
6704 Acked Read 10 Sent Done Done false
6705 Acked Read 7 Sent Done Done false
6706 Acked Write 6 Sent Done Done false
6707 Acked Read 5 Sent Done Done false
6708 Acked Write 6 Sent Done Done false
6709 Acked Write 5 Sent Done Done false
6710 Acked Write 7 Sent Done Done false
6711 Acked Read 9 Sent Done Done false
6712 Acked Read 8 Sent Done Done false
6713 Acked Read 9 Sent Done Done false
6714 Acked Read 7 Sent Done Done false
6715 Acked Read 10 Sent Done Done false
6716 Acked Write 7 Sent Done Done false
6717 Acked Write 7 Sent Done Done false
6718 Acked Read 10 Sent Done Done false
6719 Acked Flush 0 Sent Done Done false
6720 Acked Flush 0 Sent Done Done false
6721 Acked Write 8 Sent Done Done false
6722 Acked Read 7 Sent Done Done false
6723 Acked Flush 0 Sent Done Done false
6724 Acked Read 1 Sent Done Done false
6725 Acked Read 3 Sent Done Done false
6726 Acked Write 1 Sent Done Done false
6727 Acked Write 2 Sent Done Done false
6728 Acked Write 3 Sent Done Done false
6729 Acked Read 8 Sent Done Done false
6730 Acked Read 1 Sent Done Done false
6731 Acked Flush 0 Sent Done Done false
6732 Acked Read 3 Sent Done Done false
6733 Acked Read 7 Sent Done Done false
6734 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6734 6734
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6734 6733 6732 6731 6730
CLIENT: Up:0 ds:302 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6433 Acked Write 3 Sent Done Done false
6434 Acked Read 7 Sent Done Done false
6435 Acked Read 6 Sent Done Done false
6436 Acked Read 3 Sent Done Done false
6437 Acked Flush 0 Sent Done Done false
6438 Acked Read 10 Sent Done Done false
6439 Acked Read 5 Sent Done Done false
6440 Acked Flush 0 Sent Done Done false
6441 Acked Read 5 Sent Done Done false
6442 Acked Read 5 Sent Done Done false
6443 Acked Write 3 Sent Done Done false
6444 Acked Read 9 Sent Done Done false
6445 Acked Read 10 Sent Done Done false
6446 Acked Write 1 Sent Done Done false
6447 Acked Read 9 Sent Done Done false
6448 Acked Write 3 Sent Done Done false
6449 Acked Read 6 Sent Done Done false
6450 Acked Read 10 Sent Done Done false
6451 Acked Write 1 Sent Done Done false
6452 Acked Read 2 Sent Done Done false
6453 Acked Read 4 Sent Done Done false
6454 Acked Flush 0 Sent Done Done false
6455 Acked Write 1 Sent Done Done false
6456 Acked Read 8 Sent Done Done false
6457 Acked Read 8 Sent Done Done false
6458 Acked Read 9 Sent Done Done false
6459 Acked Write 8 Sent Done Done false
6460 Acked Read 2 Sent Done Done false
6461 Acked Read 9 Sent Done Done false
6462 Acked Write 10 Sent Done Done false
6463 Acked Write 10 Sent Done Done false
6464 Acked Flush 0 Sent Done Done false
6465 Acked Read 10 Sent Done Done false
6466 Acked Write 3 Sent Done Done false
6467 Acked Read 6 Sent Done Done false
6468 Acked Read 10 Sent Done Done false
6469 Acked Write 6 Sent Done Done false
6470 Acked Read 10 Sent Done Done false
6471 Acked Read 1 Sent Done Done false
6472 Acked Read 1 Sent Done Done false
6473 Acked Read 9 Sent Done Done false
6474 Acked Read 5 Sent Done Done false
6475 Acked Write 7 Sent Done Done false
6476 Acked Read 2 Sent Done Done false
6477 Acked Read 8 Sent Done Done false
6478 Acked Read 7 Sent Done Done false
6479 Acked Read 4 Sent Done Done false
6480 Acked Read 8 Sent Done Done false
6481 Acked Write 6 Sent Done Done false
6482 Acked Read 1 Sent Done Done false
6483 Acked Read 10 Sent Done Done false
6484 Acked Read 4 Sent Done Done false
6485 Acked Read 7 Sent Done Done false
6486 Acked Read 6 Sent Done Done false
6487 Acked Flush 0 Sent Done Done false
6488 Acked Read 2 Sent Done Done false
6489 Acked Flush 0 Sent Done Done false
6490 Acked Read 2 Sent Done Done false
6491 Acked Write 7 Sent Done Done false
6492 Acked Write 5 Sent Done Done false
6493 Acked Write 10 Sent Done Done false
6494 Acked Read 7 Sent Done Done false
6495 Acked Write 3 Sent Done Done false
6496 Acked Read 6 Sent Done Done false
6497 Acked Read 4 Sent Done Done false
6498 Acked Write 7 Sent Done Done false
6499 Acked Read 5 Sent Done Done false
6500 Acked Read 6 Sent Done Done false
6501 Acked Write 3 Sent Done Done false
6502 Acked Write 1 Sent Done Done false
6503 Acked Read 10 Sent Done Done false
6504 Acked Read 2 Sent Done Done false
6505 Acked Write 2 Sent Done Done false
6506 Acked Read 10 Sent Done Done false
6507 Acked Write 9 Sent Done Done false
6508 Acked Read 1 Sent Done Done false
6509 Acked Write 1 Sent Done Done false
6510 Acked Flush 0 Sent Done Done false
6511 Acked Read 7 Sent Done Done false
6512 Acked Read 4 Sent Done Done false
6513 Acked Flush 0 Sent Done Done false
6514 Acked Read 8 Sent Done Done false
6515 Acked Write 3 Sent Done Done false
6516 Acked Read 6 Sent Done Done false
6517 Acked Flush 0 Sent Done Done false
6518 Acked Read 2 Sent Done Done false
6519 Acked Read 3 Sent Done Done false
6520 Acked Read 2 Sent Done Done false
6521 Acked Flush 0 Sent Done Done false
6522 Acked Write 4 Sent Done Done false
6523 Acked Read 7 Sent Done Done false
6524 Acked Write 4 Sent Done Done false
6525 Acked Read 6 Sent Done Done false
6526 Acked Read 1 Sent Done Done false
6527 Acked Write 3 Sent Done Done false
6528 Acked Read 2 Sent Done Done false
6529 Acked Read 8 Sent Done Done false
6530 Acked Read 9 Sent Done Done false
6531 Acked Write 7 Sent Done Done false
6532 Acked Read 1 Sent Done Done false
6533 Acked Write 8 Sent Done Done false
6534 Acked Flush 0 Sent Done Done false
6535 Acked Flush 0 Sent Done Done false
6536 Acked Write 9 Sent Done Done false
6537 Acked Read 10 Sent Done Done false
6538 Acked Read 2 Sent Done Done false
6539 Acked Read 3 Sent Done Done false
6540 Acked Read 9 Sent Done Done false
6541 Acked Read 9 Sent Done Done false
6542 Acked Flush 0 Sent Done Done false
6543 Acked Write 2 Sent Done Done false
6544 Acked Read 5 Sent Done Done false
6545 Acked Read 7 Sent Done Done false
6546 Acked Read 6 Sent Done Done false
6547 Acked Write 6 Sent Done Done false
6548 Acked Write 9 Sent Done Done false
6549 Acked Write 5 Sent Done Done false
6550 Acked Flush 0 Sent Done Done false
6551 Acked Flush 0 Sent Done Done false
6552 Acked Read 8 Sent Done Done false
6553 Acked Read 7 Sent Done Done false
6554 Acked Read 6 Sent Done Done false
6555 Acked Read 5 Sent Done Done false
6556 Acked Write 4 Sent Done Done false
6557 Acked Flush 0 Sent Done Done false
6558 Acked Read 7 Sent Done Done false
6559 Acked Write 2 Sent Done Done false
6560 Acked Read 1 Sent Done Done false
6561 Acked Read 8 Sent Done Done false
6562 Acked Read 9 Sent Done Done false
6563 Acked Write 10 Sent Done Done false
6564 Acked Write 3 Sent Done Done false
6565 Acked Read 5 Sent Done Done false
6566 Acked Flush 0 Sent Done Done false
6567 Acked Read 2 Sent Done Done false
6568 Acked Write 1 Sent Done Done false
6569 Acked Read 3 Sent Done Done false
6570 Acked Write 10 Sent Done Done false
6571 Acked Write 6 Sent Done Done false
6572 Acked Read 10 Sent Done Done false
6573 Acked Read 5 Sent Done Done false
6574 Acked Read 5 Sent Done Done false
6575 Acked Read 3 Sent Done Done false
6576 Acked Write 3 Sent Done Done false
6577 Acked Read 10 Sent Done Done false
6578 Acked Read 8 Sent Done Done false
6579 Acked Flush 0 Sent Done Done false
6580 Acked Write 4 Sent Done Done false
6581 Acked Write 9 Sent Done Done false
6582 Acked Read 3 Sent Done Done false
6583 Acked Flush 0 Sent Done Done false
6584 Acked Read 8 Sent Done Done false
6585 Acked Read 2 Sent Done Done false
6586 Acked Read 9 Sent Done Done false
6587 Acked Write 5 Sent Done Done false
6588 Acked Read 1 Sent Done Done false
6589 Acked Read 4 Sent Done Done false
6590 Acked Read 1 Sent Done Done false
6591 Acked Read 4 Sent Done Done false
6592 Acked Read 4 Sent Done Done false
6593 Acked Read 7 Sent Done Done false
6594 Acked Read 6 Sent Done Done false
6595 Acked Write 9 Sent Done Done false
6596 Acked Write 8 Sent Done Done false
6597 Acked Write 7 Sent Done Done false
6598 Acked Read 10 Sent Done Done false
6599 Acked Read 6 Sent Done Done false
6600 Acked Read 7 Sent Done Done false
6601 Acked Read 7 Sent Done Done false
6602 Acked Flush 0 Sent Done Done false
6603 Acked Write 3 Sent Done Done false
6604 Acked Write 10 Sent Done Done false
6605 Acked Write 2 Sent Done Done false
6606 Acked Write 4 Sent Done Done false
6607 Acked Read 1 Sent Done Done false
6608 Acked Write 9 Sent Done Done false
6609 Acked Flush 0 Sent Done Done false
6610 Acked Write 6 Sent Done Done false
6611 Acked Flush 0 Sent Done Done false
6612 Acked Read 5 Sent Done Done false
6613 Acked Read 2 Sent Done Done false
6614 Acked Read 5 Sent Done Done false
6615 Acked Write 10 Sent Done Done false
6616 Acked Read 9 Sent Done Done false
6617 Acked Write 9 Sent Done Done false
6618 Acked Flush 0 Sent Done Done false
6619 Acked Read 3 Sent Done Done false
6620 Acked Read 6 Sent Done Done false
6621 Acked Read 3 Sent Done Done false
6622 Acked Write 4 Sent Done Done false
6623 Acked Read 5 Sent Done Done false
6624 Acked Write 10 Sent Done Done false
6625 Acked Write 10 Sent Done Done false
6626 Acked Write 5 Sent Done Done false
6627 Acked Read 6 Sent Done Done false
6628 Acked Flush 0 Sent Done Done false
6629 Acked Read 1 Sent Done Done false
6630 Acked Read 1 Sent Done Done false
6631 Acked Read 2 Sent Done Done false
6632 Acked Write 2 Sent Done Done false
6633 Acked Flush 0 Sent Done Done false
6634 Acked Read 2 Sent Done Done false
6635 Acked Write 7 Sent Done Done false
6636 Acked Write 1 Sent Done Done false
6637 Acked Write 3 Sent Done Done false
6638 Acked Read 10 Sent Done Done false
6639 Acked Flush 0 Sent Done Done false
6640 Acked Write 9 Sent Done Done false
6641 Acked Read 1 Sent Done Done false
6642 Acked Read 8 Sent Done Done false
6643 Acked Write 8 Sent Done Done false
6644 Acked Write 8 Sent Done Done false
6645 Acked Flush 0 Sent Done Done false
6646 Acked Read 2 Sent Done Done false
6647 Acked Read 8 Sent Done Done false
6648 Acked Read 1 Sent Done Done false
6649 Acked Read 3 Sent Done Done false
6650 Acked Write 3 Sent Done Done false
6651 Acked Write 4 Sent Done Done false
6652 Acked Write 9 Sent Done Done false
6653 Acked Read 3 Sent Done Done false
6654 Acked Read 5 Sent Done Done false
6655 Acked Read 1 Sent Done Done false
6656 Acked Write 3 Sent Done Done false
6657 Acked Read 1 Sent Done Done false
6658 Acked Read 9 Sent Done Done false
6659 Acked Read 4 Sent Done Done false
6660 Acked Write 2 Sent Done Done false
6661 Acked Read 2 Sent Done Done false
6662 Acked Write 3 Sent Done Done false
6663 Acked Write 8 Sent Done Done false
6664 Acked Read 2 Sent Done Done false
6665 Acked Write 4 Sent Done Done false
6666 Acked Read 5 Sent Done Done false
6667 Acked Read 7 Sent Done Done false
6668 Acked Read 10 Sent Done Done false
6669 Acked Read 8 Sent Done Done false
6670 Acked Write 7 Sent Done Done false
6671 Acked Write 1 Sent Done Done false
6672 Acked Read 2 Sent Done Done false
6673 Acked Read 8 Sent Done Done false
6674 Acked Read 6 Sent Done Done false
6675 Acked Read 2 Sent Done Done false
6676 Acked Read 2 Sent Done Done false
6677 Acked Read 7 Sent Done Done false
6678 Acked Read 2 Sent Done Done false
6679 Acked Write 6 Sent Done Done false
6680 Acked Write 10 Sent Done Done false
6681 Acked Read 10 Sent Done Done false
6682 Acked Write 7 Sent Done Done false
6683 Acked Write 2 Sent Done Done false
6684 Acked Write 1 Sent Done Done false
6685 Acked Read 8 Sent Done Done false
6686 Acked Read 1 Sent Done Done false
6687 Acked Write 6 Sent Done Done false
6688 Acked Read 6 Sent Done Done false
6689 Acked Write 3 Sent Done Done false
6690 Acked Flush 0 Sent Done Done false
6691 Acked Read 10 Sent Done Done false
6692 Acked Write 5 Sent Done Done false
6693 Acked Read 2 Sent Done Done false
6694 Acked Flush 0 Sent Done Done false
6695 Acked Write 7 Sent Done Done false
6696 Acked Flush 0 Sent Done Done false
6697 Acked Read 10 Sent Done Done false
6698 Acked Read 7 Sent Done Done false
6699 Acked Read 6 Sent Done Done false
6700 Acked Read 6 Sent Done Done false
6701 Acked Read 8 Sent Done Done false
6702 Acked Write 6 Sent Done Done false
6703 Acked Read 6 Sent Done Done false
6704 Acked Read 10 Sent Done Done false
6705 Acked Read 7 Sent Done Done false
6706 Acked Write 6 Sent Done Done false
6707 Acked Read 5 Sent Done Done false
6708 Acked Write 6 Sent Done Done false
6709 Acked Write 5 Sent Done Done false
6710 Acked Write 7 Sent Done Done false
6711 Acked Read 9 Sent Done Done false
6712 Acked Read 8 Sent Done Done false
6713 Acked Read 9 Sent Done Done false
6714 Acked Read 7 Sent Done Done false
6715 Acked Read 10 Sent Done Done false
6716 Acked Write 7 Sent Done Done false
6717 Acked Write 7 Sent Done Done false
6718 Acked Read 10 Sent Done Done false
6719 Acked Flush 0 Sent Done Done false
6720 Acked Flush 0 Sent Done Done false
6721 Acked Write 8 Sent Done Done false
6722 Acked Read 7 Sent Done Done false
6723 Acked Flush 0 Sent Done Done false
6724 Acked Read 1 Sent Done Done false
6725 Acked Read 3 Sent Done Done false
6726 Acked Write 1 Sent Done Done false
6727 Acked Write 2 Sent Done Done false
6728 Acked Write 3 Sent Done Done false
6729 Acked Read 8 Sent Done Done false
6730 Acked Read 1 Sent Done Done false
6731 Acked Flush 0 Sent Done Done false
6732 Acked Read 3 Sent Done Done false
6733 Acked Read 7 Sent Done Done false
6734 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6432 6734 6734
Downstairs last five completed: 6432 6431 6430 6429 6428
Upstairs last five completed: 6734 6733 6732 6731 6730
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.375567185Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65232)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.375760772Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.37688989Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.377418329Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.377441926Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 6432","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.377458304Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 6432","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.377670809Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 6432","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.377694037Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 6432","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.37792289Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 6734 6733 6732 6731 6730
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
[8820] Got stop action so:false kr:false
Jan 27 19:40:57.632 INFO request completed, latency_us: 579, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e886db29-46ca-45f9-ac40-321efaa74c34, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.632999226Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.633128531Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.633165647Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.633253546Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:40:57.633 INFO request completed, latency_us: 129, response_code: 200, uri: /state/cid/1, method: GET, req_id: 365e158b-774d-4aee-9f4e-bde42ca4abec, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:41:01.635 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/1, method: GET, req_id: 97c49dd6-cbce-4c7a-a188-7d41318c02ae, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 967 len 32768
002/300 Write block 4822 len 20480 data: 4 4 4 4 4
003/300 Read block 1459 len 36864
004/300 Read block 1393 len 8192
005/300 Read block 1681 len 24576
006/300 Read block 2225 len 16384
007/300 Read block 143 len 4096
008/300 Read block 105 len 28672
009/300 Flush
010/300 Write block 878 len 40960 data: 4 4 3 3 3 2 2 3 3 3
011/300 Read block 4800 len 4096
012/300 Read block 3137 len 24576
013/300 Read block 4826 len 36864
014/300 Write block 2773 len 8192 data: 7 6
015/300 Read block 4810 len 24576
016/300 Flush
017/300 Write block 3873 len 32768 data: 5 5 4 5 5 6 5 8
018/300 Read block 879 len 16384
019/300 Flush
020/300 Flush
021/300 Write block 1305 len 16384 data: 6 6 7 6
022/300 Read block 1475 len 16384
023/300 Write block 3581 len 28672 data: 3 3 3 2 2 2 2
024/300 Write block 997 len 36864 data: 3 3 3 4 4 3 4 5 4
025/300 Write block 2071 len 16384 data: 2 4 4 4
026/300 Read block 1365 len 24576
027/300 Write block 2246 len 20480 data: 7 8 7 5 5
028/300 Write block 287 len 4096 data: 5
029/300 Write block 4403 len 12288 data: 2 2 3
030/300 Flush
031/300 Read block 3186 len 28672
032/300 Read block 1828 len 32768
033/300 Write block 3131 len 20480 data: 6 6 6 5 6
034/300 Read block 4474 len 16384
035/300 Read block 2751 len 8192
036/300 Read block 1172 len 40960
037/300 Read block 1374 len 8192
038/300 Write block 597 len 4096 data: 5
039/300 Read block 2557 len 36864
040/300 Write block 4299 len 4096 data: 3
041/300 Write block 2964 len 20480 data: 5 5 5 5 5
042/300 Flush
043/300 Read block 2560 len 24576
044/300 Write block 4644 len 24576 data: 4 3 4 4 3 4
045/300 Read block 4485 len 16384
046/300 Read block 3662 len 36864
047/300 Flush
048/300 Write block 4503 len 12288 data: 6 5 5
049/300 Read block 4848 len 12288
050/300 Write block 4588 len 28672 data: 3 3 3 3 3 3 4
051/300 Read block 794 len 24576
052/300 Write block 35 len 16384 data: 3 3 4 4
053/300 Write block 351 len 4096 data: 5
054/300 Write block 1879 len 40960 data: 5 3 3 3 5 5 5 7 6 7
055/300 Read block 649 len 28672
056/300 Write block 1915 len 28672 data: 2 2 2 2 2 3 3
057/300 Read block 664 len 12288
058/300 Write block 2023 len 40960 data: 5 4 3 5 5 5 5 5 4 5
059/300 Read block 3958 len 12288
060/300 Write block 3208 len 36864 data: 3 4 3 5 5 5 3 6 6
061/300 Flush
062/300 Write block 49 len 16384 data: 6 6 6 5
063/300 Write block 2429 len 20480 data: 5 4 3 2 2
064/300 Read block 1255 len 28672
065/300 Read block 3971 len 4096
066/300 Flush
067/300 Read block 939 len 12288
068/300 Write block 303 len 32768 data: 5 4 5 5 5 4 4 3
069/300 Read block 3468 len 28672
070/300 Read block 4537 len 24576
071/300 Read block 3777 len 12288
072/300 Flush
073/300 Read block 478 len 40960
074/300 Read block 3937 len 24576
075/300 Read block 1251 len 8192
076/300 Read block 3942 len 16384
077/300 Write block 4591 len 28672 data: 4 4 4 5 4 4 4
078/300 Read block 649 len 24576
079/300 Flush
080/300 Write block 675 len 28672 data: 4 5 5 5 4 3 4
081/300 Read block 1120 len 24576
082/300 Read block 494 len 16384
083/300 Read block 4548 len 16384
084/300 Read block 2555 len 16384
085/300 Write block 145 len 4096 data: 3
086/300 Write block 2436 len 36864 data: 2 5 5 5 6 8 6 6 6
087/300 Read block 3430 len 16384
088/300 Write block 2424 len 8192 data: 4 6
089/300 Read block 692 len 20480
090/300 Write block 2457 len 32768 data: 3 2 3 3 2 2 3 2
091/300 Write block 786 len 24576 data: 3 3 3 4 4 4
092/300 Read block 1201 len 40960
093/300 Read block 3935 len 4096
094/300 Write block 2825 len 8192 data: 4 4
095/300 Read block 2283 len 20480
096/300 Write block 4302 len 4096 data: 4
097/300 Write block 2692 len 8192 data: 3 3
098/300 Flush
099/300 Read block 163 len 16384
100/300 Read block 1454 len 40960
101/300 Write block 1899 len 32768 data: 2 2 2 2 3 5 5 5
102/300 Read block 3638 len 4096
103/300 Flush
104/300 Read block 3242 len 8192
105/300 Flush
106/300 Read block 3023 len 20480
107/300 Write block 2788 len 12288 data: 5 5 5
108/300 Write block 4446 len 24576 data: 5 4 4 4 4 4
109/300 Read block 2063 len 12288
110/300 Write block 501 len 32768 data: 8 7 4 3 5 5 4 3
111/300 Read block 390 len 12288
112/300 Read block 2085 len 4096
113/300 Write block 1632 len 20480 data: 6 6 5 5 4
114/300 Read block 101 len 32768
115/300 Flush
116/300 Read block 4054 len 28672
117/300 Write block 2768 len 40960 data: 8 6 6 6 7 8 7 5 4 4
118/300 Read block 2400 len 4096
119/300 Write block 677 len 40960 data: 6 6 5 4 5 5 6 6 6 6
120/300 Write block 1355 len 20480 data: 7 6 6 8 8
121/300 Write block 4892 len 24576 data: 3 3 3 2 3 4
122/300 Read block 2839 len 32768
123/300 Flush
124/300 Read block 589 len 36864
125/300 Write block 2907 len 28672 data: 3 3 3 3 3 3 3
126/300 Write block 1389 len 12288 data: 5 5 6
127/300 Read block 4851 len 32768
128/300 Read block 3933 len 20480
129/300 Write block 3011 len 32768 data: 3 3 2 2 2 2 2 2
130/300 Write block 903 len 16384 data: 4 5 3 3
131/300 Write block 3319 len 36864 data: 4 5 5 4 5 5 6 7 5
132/300 Write block 2596 len 4096 data: 4
133/300 Write block 1457 len 16384 data: 7 8 6 5
134/300 Read block 429 len 12288
135/300 Read block 2568 len 12288
136/300 Write block 4531 len 32768 data: 4 4 4 4 4 3 2 2
137/300 Read block 4485 len 36864
138/300 Read block 2575 len 4096
139/300 Read block 506 len 12288
140/300 Read block 108 len 24576
141/300 Read block 241 len 28672
142/300 Read block 3734 len 28672
143/300 Write block 713 len 40960 data: 12 10 9 8 6 5 6 6 6 6
144/300 Read block 928 len 40960
145/300 Write block 4586 len 4096 data: 2
146/300 Read block 4800 len 16384
147/300 Write block 4904 len 24576 data: 3 3 2 3 4 3
148/300 Write block 892 len 4096 data: 5
149/300 Read block 3205 len 12288
150/300 Read block 184 len 40960
151/300 Write block 3011 len 12288 data: 4 4 3
152/300 Write block 2708 len 24576 data: 2 3 4 4 4 5
153/300 Read block 3961 len 32768
154/300 Read block 3733 len 28672
155/300 Read block 3875 len 8192
156/300 Flush
157/300 Read block 4629 len 8192
158/300 Read block 2865 len 12288
159/300 Read block 1190 len 20480
160/300 Read block 2075 len 16384
161/300 Read block 1118 len 24576
162/300 Read block 1625 len 28672
163/300 Read block 315 len 12288
164/300 Flush
165/300 Read block 3885 len 16384
166/300 Read block 2320 len 4096
167/300 Read block 3225 len 24576
168/300 Write block 1096 len 28672 data: 3 3 5 5 5 6 6
169/300 Write block 162 len 12288 data: 3 3 4
170/300 Write block 2607 len 16384 data: 6 6 5 4
171/300 Flush
172/300 Write block 4933 len 4096 data: 4
173/300 Read block 4881 len 40960
174/300 Read block 4722 len 40960
175/300 Write block 116 len 32768 data: 7 6 6 5 5 5 6 6
176/300 Read block 1552 len 36864
177/300 Write block 1019 len 28672 data: 7 7 6 4 3 2 2
178/300 Read block 3495 len 4096
179/300 Write block 2587 len 20480 data: 3 3 3 4 4
180/300 Write block 3047 len 16384 data: 5 6 6 7
181/300 Write block 930 len 12288 data: 5 5 4
182/300 Write block 255 len 12288 data: 5 6 7
183/300 Write block 205 len 32768 data: 6 6 6 6 6 5 5 3
184/300 Read block 4906 len 12288
185/300 Read block 362 len 12288
186/300 Read block 3033 len 28672
187/300 Read block 4019 len 28672
188/300 Write block 4772 len 12288 data: 4 5 6
189/300 Flush
190/300 Read block 321 len 12288
191/300 Write block 3174 len 16384 data: 5 6 7 7
192/300 Read block 2446 len 36864
193/300 Read block 1759 len 36864
194/300 Read block 3871 len 20480
195/300 Read block 4468 len 4096
196/300 Flush
197/300 Read block 2309 len 28672
198/300 Read block 1788 len 40960
199/300 Read block 1411 len 8192
200/300 Read block 4948 len 24576
201/300 Read block 2542 len 16384
202/300 Write block 1934 len 16384 data: 4 4 3 3
203/300 Read block 3476 len 24576
204/300 Write block 2671 len 16384 data: 4 5 6 6
205/300 Write block 813 len 28672 data: 3 3 4 4 4 4 6
206/300 Flush
207/300 Write block 2564 len 24576 data: 6 6 6 3 3 3
208/300 Read block 1798 len 20480
209/300 Write block 2157 len 4096 data: 9
210/300 Write block 286 len 16384 data: 7 6 6 5
211/300 Read block 906 len 36864
212/300 Write block 2750 len 28672 data: 3 6 5 5 4 4 4
213/300 Read block 2585 len 12288
214/300 Write block 2065 len 16384 data: 3 3 3 4
215/300 Read block 3259 len 32768
216/300 Write block 4658 len 12288 data: 5 4 2
217/300 Write block 720 len 4096 data: 7
218/300 Flush
219/300 Read block 1406 len 12288
220/300 Write block 2518 len 16384 data: 3 2 2 2
221/300 Write block 3608 len 12288 data: 8 6 5
222/300 Write block 184 len 32768 data: 7 7 7 5 3 3 3 3
223/300 Read block 1712 len 24576
224/300 Write block 2407 len 16384 data: 6 7 6 8
225/300 Read block 4583 len 16384
226/300 Write block 1464 len 20480 data: 5 5 5 5 4
227/300 Write block 3123 len 28672 data: 6 6 5 4 4 5 6
228/300 Read block 272 len 40960
229/300 Read block 3322 len 24576
230/300 Read block 2007 len 12288
231/300 Read block 2903 len 28672
232/300 Flush
233/300 Write block 2929 len 24576 data: 6 5 5 5 6 6
234/300 Read block 771 len 40960
235/300 Write block 4735 len 20480 data: 6 7 7 6 6
236/300 Read block 3731 len 12288
237/300 Read block 575 len 28672
238/300 Read block 1175 len 8192
239/300 Read block 857 len 8192
240/300 Read block 516 len 36864
241/300 Read block 4049 len 28672
242/300 Read block 2849 len 12288
243/300 Read block 4498 len 40960
244/300 Read block 1631 len 24576
245/300 Write block 757 len 8192 data: 2 3
246/300 Flush
247/300 Write block 644 len 20480 data: 3 3 3 3 6
248/300 Write block 3712 len 36864 data: 7 7 7 7 5 5 4 3 3
249/300 Write block 228 len 8192 data: 6 6
250/300 Read block 3045 len 20480
251/300 Flush
252/300 Flush
253/300 Read block 4762 len 8192
254/300 Read block 1443 len 36864
255/300 Write block 3004 len 28672 data: 5 4 4 4 4 4 3
256/300 Read block 1730 len 12288
257/300 Read block 1019 len 28672
258/300 Write block 3226 len 24576 data: 3 3 4 4 5 5
259/300 Write block 3810 len 32768 data: 5 4 3 3 3 2 2 3
260/300 Flush
261/300 Write block 2490 len 16384 data: 4 4 4 5
262/300 Read block 789 len 24576
263/300 Read block 32 len 40960
264/300 Read block 3395 len 16384
265/300 Read block 1617 len 36864
266/300 Write block 2803 len 4096 data: 4
267/300 Read block 250 len 28672
268/300 Flush
269/300 Read block 3620 len 40960
270/300 Read block 3918 len 32768
271/300 Read block 4781 len 16384
272/300 Flush
273/300 Read block 2346 len 4096
274/300 Write block 3541 len 12288 data: 4 5 4
275/300 Read block 4683 len 8192
276/300 Write block 3106 len 8192 data: 7 7
277/300 Write block 2349 len 40960 data: 5 6 6 7 9 7 5 5 3 2
278/300 Write block 4865 len 24576 data: 5 5 7 6 7 6
279/300 Write block 645 len 16384 data: 4 4 4 7
280/300 Read block 3626 len 4096
281/300 Read block 1515 len 4096
282/300 Write block 4190 len 40960 data: 8 8 8 7 7 7 8 8 6 5
283/300 Read block 2824 len 36864
284/300 Read block 4716 len 12288
285/300 Flush
286/300 Write block 1683 len 16384 data: 3 3 3 3
287/300 Read block 500 len 4096
288/300 Flush
289/300 Flush
290/300 Flush
291/300 Read block 2504 len 4096
292/300 Read block 1080 len 32768
293/300 Write block 1651 len 24576 data: 2 3 3 4 4 4
294/300 Write block 2870 len 20480 data: 5 4 5 5 4
295/300 Flush
296/300 Flush
297/300 Write block 1197 len 8192 data: 6 6
298/300 Read block 4435 len 20480
299/300 Read block 274 len 16384
300/300 Read block 1763 len 16384
301/300 Write block 1596 len 32768 data: 7 8 9 6 7 5 5 5
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:41:01.867 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/1, method: POST, req_id: e742cd8a-f9af-45c0-99c1-e6739e7cc972, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6735 Acked Read 8 Done Sent Done false
6736 Acked Write 5 Done Sent Done false
6737 Acked Read 9 Done Sent Done false
6738 Acked Read 2 Done Sent Done false
6739 Acked Read 6 Done Sent Done false
6740 Acked Read 4 Done Sent Done false
6741 Acked Read 1 Done Sent Done false
6742 Acked Read 7 Done Sent Done false
6743 Acked Flush 0 Done Sent Done false
6744 Acked Write 10 Done Sent Done false
6745 Acked Read 1 Done Sent Done false
6746 Acked Read 6 Done Sent Done false
6747 Acked Read 9 Done Sent Done false
6748 Acked Write 2 Done Sent Done false
6749 Acked Read 6 Done Sent Done false
6750 Acked Flush 0 Done Sent Done false
6751 Acked Write 8 Done Sent Done false
6752 Acked Read 4 Done Sent Done false
6753 Acked Flush 0 Done Sent Done false
6754 Acked Flush 0 Done Sent Done false
6755 Acked Write 4 Done Sent Done false
6756 Acked Read 4 Done Sent Done false
6757 Acked Write 7 Done Sent Done false
6758 Acked Write 9 Done Sent Done false
6759 Acked Write 4 Done Sent Done false
6760 Acked Read 6 Done Sent Done false
6761 Acked Write 5 Done Sent Done false
6762 Acked Write 1 Done Sent Done false
6763 Acked Write 3 Done Sent Done false
6764 Acked Flush 0 Done Sent Done false
6765 Acked Read 7 Done Sent Done false
6766 Acked Read 8 Done Sent Done false
6767 Acked Write 5 Done Sent Done false
6768 Acked Read 4 Done Sent Done false
6769 Acked Read 2 Done Sent Done false
6770 Acked Read 10 Done Sent Done false
6771 Acked Read 2 Done Sent Done false
6772 Acked Write 1 Done Sent Done false
6773 Acked Read 9 Done Sent Done false
6774 Acked Write 1 Done Sent Done false
6775 Acked Write 5 Done Sent Done false
6776 Acked Flush 0 Done Sent Done false
6777 Acked Read 6 Done Sent Done false
6778 Acked Write 6 Done Sent Done false
6779 Acked Read 4 Done Sent Done false
6780 Acked Read 9 Done Sent Done false
6781 Acked Flush 0 Done Sent Done false
6782 Acked Write 3 Done Sent Done false
6783 Acked Read 3 Done Sent Done false
6784 Acked Write 7 Done Sent Done false
6785 Acked Read 6 Done Sent Done false
6786 Acked Write 4 Done Sent Done false
6787 Acked Write 1 Done Sent Done false
6788 Acked Write 10 Done Sent Done false
6789 Acked Read 7 Done Sent Done false
6790 Acked Write 7 Done Sent Done false
6791 Acked Read 3 Done Sent Done false
6792 Acked Write 10 Done Sent Done false
6793 Acked Read 3 Done Sent Done false
6794 Acked Write 9 Done Sent Done false
6795 Acked Flush 0 Done Sent Done false
6796 Acked Write 4 Done Sent Done false
6797 Acked Write 5 Done Sent Done false
6798 Acked Read 7 Done Sent Done false
6799 Acked Read 1 Done Sent Done false
6800 Acked Flush 0 Done Sent Done false
6801 Acked Read 3 Done Sent Done false
6802 Acked Write 8 Done Sent Done false
6803 Acked Read 7 Done Sent Done false
6804 Acked Read 6 Done Sent Done false
6805 Acked Read 3 Done Sent Done false
6806 Acked Flush 0 Done Sent Done false
6807 Acked Read 10 Done Sent Done false
6808 Acked Read 6 Done Sent Done false
6809 Acked Read 2 Done Sent Done false
6810 Acked Read 4 Done Sent Done false
6811 Acked Write 7 Done Sent Done false
6812 Acked Read 6 Done Sent Done false
6813 Acked Flush 0 Done Sent Done false
6814 Acked Write 7 Done Sent Done false
6815 Acked Read 6 Done Sent Done false
6816 Acked Read 4 Done Sent Done false
6817 Acked Read 4 Done Sent Done false
6818 Acked Read 4 Done Sent Done false
6819 Acked Write 1 Done Sent Done false
6820 Acked Write 9 Done Sent Done false
6821 Acked Read 4 Done Sent Done false
6822 Acked Write 2 Done Sent Done false
6823 Acked Read 5 Done Sent Done false
6824 Acked Write 8 Done Sent Done false
6825 Acked Write 6 Done Sent Done false
6826 Acked Read 10 Done Sent Done false
6827 Acked Read 1 Done Sent Done false
6828 Acked Write 2 Done Sent Done false
6829 Acked Read 5 Done Sent Done false
6830 Acked Write 1 Done Sent Done false
6831 Acked Write 2 Done Sent Done false
6832 Acked Flush 0 Done Sent Done false
6833 Acked Read 4 Done Sent Done false
6834 Acked Read 10 Done Sent Done false
6835 Acked Write 8 Done Sent Done false
6836 Acked Read 1 Done Sent Done false
6837 Acked Flush 0 Done Sent Done false
6838 Acked Read 2 Done Sent Done false
6839 Acked Flush 0 Done Sent Done false
6840 Acked Read 5 Done Sent Done false
6841 Acked Write 3 Done Sent Done false
6842 Acked Write 6 Done Sent Done false
6843 Acked Read 3 Done Sent Done false
6844 Acked Write 8 Done Sent Done false
6845 Acked Read 3 Done Sent Done false
6846 Acked Read 1 Done Sent Done false
6847 Acked Write 5 Done Sent Done false
6848 Acked Read 8 Done Sent Done false
6849 Acked Flush 0 Done Sent Done false
6850 Acked Read 7 Done Sent Done false
6851 Acked Write 10 Done Sent Done false
6852 Acked Read 1 Done Sent Done false
6853 Acked Write 10 Done Sent Done false
6854 Acked Write 5 Done Sent Done false
6855 Acked Write 6 Done Sent Done false
6856 Acked Read 8 Done Sent Done false
6857 Acked Flush 0 Done Sent Done false
6858 Acked Read 9 Done Sent Done false
6859 Acked Write 7 Done Sent Done false
6860 Acked Write 3 Done Sent Done false
6861 Acked Read 8 Done Sent Done false
6862 Acked Read 5 Done Sent Done false
6863 Acked Write 8 Done Sent Done false
6864 Acked Write 4 Done Sent Done false
6865 Acked Write 9 Done Sent Done false
6866 Acked Write 1 Done Sent Done false
6867 Acked Write 4 Done Sent Done false
6868 Acked Read 3 Done Sent Done false
6869 Acked Read 3 Done Sent Done false
6870 Acked Write 8 Done Sent Done false
6871 Acked Read 9 Done Sent Done false
6872 Acked Read 1 Done Sent Done false
6873 Acked Read 3 Done Sent Done false
6874 Acked Read 6 Done Sent Done false
6875 Acked Read 7 Done Sent Done false
6876 Acked Read 7 Done Sent Done false
6877 Acked Write 10 Done Sent Done false
6878 Acked Read 10 Done Sent Done false
6879 Acked Write 1 Done Sent Done false
6880 Acked Read 4 Done Sent Done false
6881 Acked Write 6 Done Sent Done false
6882 Acked Write 1 Done Sent Done false
6883 Acked Read 3 Done Sent Done false
6884 Acked Read 10 Done Sent Done false
6885 Acked Write 3 Done Sent Done false
6886 Acked Write 6 Done Sent Done false
6887 Acked Read 8 Done Sent Done false
6888 Acked Read 7 Done Sent Done false
6889 Acked Read 2 Done Sent Done false
6890 Acked Flush 0 Done Sent Done false
6891 Acked Read 2 Done Sent Done false
6892 Acked Read 3 Done Sent Done false
6893 Acked Read 5 Done Sent Done false
6894 Acked Read 4 Done Sent Done false
6895 Acked Read 6 Done Sent Done false
6896 Acked Read 7 Done Sent Done false
6897 Acked Read 3 Done Sent Done false
6898 Acked Flush 0 Done Sent Done false
6899 Acked Read 4 Done Sent Done false
6900 Acked Read 1 Done Sent Done false
6901 Acked Read 6 Done Sent Done false
6902 Acked Write 7 Done Sent Done false
6903 Acked Write 3 Done Sent Done false
6904 Acked Write 4 Done Sent Done false
6905 Acked Flush 0 Done Sent Done false
6906 Acked Write 1 Done Sent Done false
6907 Acked Read 10 Done Sent Done false
6908 Acked Read 10 Done Sent Done false
6909 Acked Write 8 Done Sent Done false
6910 Acked Read 9 Done Sent Done false
6911 Acked Write 7 Done Sent Done false
6912 Acked Read 1 Done Sent Done false
6913 Acked Write 5 Done Sent Done false
6914 Acked Write 4 Done Sent Done false
6915 Acked Write 3 Done Sent Done false
6916 Acked Write 3 Done Sent Done false
6917 Acked Write 8 Done Sent Done false
6918 Acked Read 3 Done Sent Done false
6919 Acked Read 3 Done Sent Done false
6920 Acked Read 7 Done Sent Done false
6921 Acked Read 7 Done Sent Done false
6922 Acked Write 3 Done Sent Done false
6923 Acked Flush 0 Done Sent Done false
6924 Acked Read 3 Done Sent Done false
6925 Acked Write 4 Done Sent Done false
6926 Acked Read 9 Done Sent Done false
6927 Acked Read 9 Done Sent Done false
6928 Acked Read 5 Done Sent Done false
6929 Acked Read 1 Done Sent Done false
6930 Acked Flush 0 Done Sent Done false
6931 Acked Read 7 Done Sent Done false
6932 Acked Read 10 Done Sent Done false
6933 Acked Read 2 Done Sent Done false
6934 Acked Read 6 Done Sent Done false
6935 Acked Read 4 Done Sent Done false
6936 Acked Write 4 Done Sent Done false
6937 Acked Read 6 Done Sent Done false
6938 Acked Write 4 Done Sent Done false
6939 Acked Write 7 Done Sent Done false
6940 Acked Flush 0 Done Sent Done false
6941 Acked Write 6 Done Sent Done false
6942 Acked Read 5 Done Sent Done false
6943 Acked Write 1 Done Sent Done false
6944 Acked Write 4 Done Sent Done false
6945 Acked Read 9 Done Sent Done false
6946 Acked Write 7 Done Sent Done false
6947 Acked Read 3 Done Sent Done false
6948 Acked Write 4 Done Sent Done false
6949 Acked Read 8 Done Sent Done false
6950 Acked Write 3 Done Sent Done false
6951 Acked Write 1 Done Sent Done false
6952 Acked Flush 0 Done Sent Done false
6953 Acked Read 3 Done Sent Done false
6954 Acked Write 4 Done Sent Done false
6955 Acked Write 3 Done Sent Done false
6956 Acked Write 8 Done Sent Done false
6957 Acked Read 6 Done Sent Done false
6958 Acked Write 4 Done Sent Done false
6959 Acked Read 4 Done Sent Done false
6960 Acked Write 5 Done Sent Done false
6961 Acked Write 7 Done Sent Done false
6962 Acked Read 10 Done Sent Done false
6963 Acked Read 6 Done Sent Done false
6964 Acked Read 3 Done Sent Done false
6965 Acked Read 7 Done Sent Done false
6966 Acked Flush 0 Done Sent Done false
6967 Acked Write 6 Done Sent Done false
6968 Acked Read 10 Done Sent Done false
6969 Acked Write 5 Done Sent Done false
6970 Acked Read 3 Done Sent Done false
6971 Acked Read 7 Done Sent Done false
6972 Acked Read 2 Done Sent Done false
6973 Acked Read 2 Done Sent Done false
6974 Acked Read 9 Done Sent Done false
6975 Acked Read 7 Done Sent Done false
6976 Acked Read 3 Done Sent Done false
6977 Acked Read 10 Done Sent Done false
6978 Acked Read 6 Done Sent Done false
6979 Acked Write 2 Done Sent Done false
6980 Acked Flush 0 Done Sent Done false
6981 Acked Write 5 Done Sent Done false
6982 Acked Write 9 Done Sent Done false
6983 Acked Write 2 Done Sent Done false
6984 Acked Read 5 Done Sent Done false
6985 Acked Flush 0 Done Sent Done false
6986 Acked Flush 0 Done Sent Done false
6987 Acked Read 2 Done Sent Done false
6988 Acked Read 9 Done Sent Done false
6989 Acked Write 7 Done Sent Done false
6990 Acked Read 3 Done Sent Done false
6991 Acked Read 7 Done Sent Done false
6992 Acked Write 6 Done Sent Done false
6993 Acked Write 8 Done Sent Done false
6994 Acked Flush 0 Done Sent Done false
6995 Acked Write 4 Done Sent Done false
6996 Acked Read 6 Done Sent Done false
6997 Acked Read 10 Done Sent Done false
6998 Acked Read 4 Done Sent Done false
6999 Acked Read 9 Done Sent Done false
7000 Acked Write 1 Done Sent Done false
7001 Acked Read 7 Done Sent Done false
7002 Acked Flush 0 Done Sent Done false
7003 Acked Read 10 Done Sent Done false
7004 Acked Read 8 Done Sent Done false
7005 Acked Read 4 Done Sent Done false
7006 Acked Flush 0 Done Sent Done false
7007 Acked Read 1 Done Sent Done false
7008 Acked Write 3 Done Sent Done false
7009 Acked Read 2 Done Sent Done false
7010 Acked Write 2 Done Sent Done false
7011 Acked Write 10 Done Sent Done false
7012 Acked Write 6 Done Sent Done false
7013 Acked Write 4 Done Sent Done false
7014 Acked Read 1 Done Sent Done false
7015 Acked Read 1 Done Sent Done false
7016 Acked Write 10 Done Sent Done false
7017 Acked Read 9 Done Sent Done false
7018 Acked Read 3 Done Sent Done false
7019 Acked Flush 0 Done Sent Done false
7020 Acked Write 4 Done Sent Done false
7021 Acked Read 1 Done Sent Done false
7022 Acked Flush 0 Done Sent Done false
7023 Acked Flush 0 Done Sent Done false
7024 Acked Flush 0 Done Sent Done false
7025 Acked Read 1 Done Sent Done false
7026 Acked Read 8 Done Sent Done false
7027 Acked Write 6 Done Sent Done false
7028 Acked Write 5 Done Sent Done false
7029 Acked Flush 0 Done Sent Done false
7030 Acked Flush 0 Done Sent Done false
7031 Acked Write 2 Done Sent Done false
7032 Acked Read 5 Done Sent Done false
7033 Acked Read 4 Done Sent Done false
7034 Acked Read 4 Done Sent Done false
7035 Acked Write 8 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7030 6734 7030
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 7035 7034 7033 7032 7031
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1220)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6735 Acked Read 8 Done Sent Done false
6736 Acked Write 5 Done Sent Done false
6737 Acked Read 9 Done Sent Done false
6738 Acked Read 2 Done Sent Done false
6739 Acked Read 6 Done Sent Done false
6740 Acked Read 4 Done Sent Done false
6741 Acked Read 1 Done Sent Done false
6742 Acked Read 7 Done Sent Done false
6743 Acked Flush 0 Done Sent Done false
6744 Acked Write 10 Done Sent Done false
6745 Acked Read 1 Done Sent Done false
6746 Acked Read 6 Done Sent Done false
6747 Acked Read 9 Done Sent Done false
6748 Acked Write 2 Done Sent Done false
6749 Acked Read 6 Done Sent Done false
6750 Acked Flush 0 Done Sent Done false
6751 Acked Write 8 Done Sent Done false
6752 Acked Read 4 Done Sent Done false
6753 Acked Flush 0 Done Sent Done false
6754 Acked Flush 0 Done Sent Done false
6755 Acked Write 4 Done Sent Done false
6756 Acked Read 4 Done Sent Done false
6757 Acked Write 7 Done Sent Done false
6758 Acked Write 9 Done Sent Done false
6759 Acked Write 4 Done Sent Done false
6760 Acked Read 6 Done Sent Done false
6761 Acked Write 5 Done Sent Done false
6762 Acked Write 1 Done Sent Done false
6763 Acked Write 3 Done Sent Done false
6764 Acked Flush 0 Done Sent Done false
6765 Acked Read 7 Done Sent Done false
6766 Acked Read 8 Done Sent Done false
6767 Acked Write 5 Done Sent Done false
6768 Acked Read 4 Done Sent Done false
6769 Acked Read 2 Done Sent Done false
6770 Acked Read 10 Done Sent Done false
6771 Acked Read 2 Done Sent Done false
6772 Acked Write 1 Done Sent Done false
6773 Acked Read 9 Done Sent Done false
6774 Acked Write 1 Done Sent Done false
6775 Acked Write 5 Done Sent Done false
6776 Acked Flush 0 Done Sent Done false
6777 Acked Read 6 Done Sent Done false
6778 Acked Write 6 Done Sent Done false
6779 Acked Read 4 Done Sent Done false
6780 Acked Read 9 Done Sent Done false
6781 Acked Flush 0 Done Sent Done false
6782 Acked Write 3 Done Sent Done false
6783 Acked Read 3 Done Sent Done false
6784 Acked Write 7 Done Sent Done false
6785 Acked Read 6 Done Sent Done false
6786 Acked Write 4 Done Sent Done false
6787 Acked Write 1 Done Sent Done false
6788 Acked Write 10 Done Sent Done false
6789 Acked Read 7 Done Sent Done false
6790 Acked Write 7 Done Sent Done false
6791 Acked Read 3 Done Sent Done false
6792 Acked Write 10 Done Sent Done false
6793 Acked Read 3 Done Sent Done false
6794 Acked Write 9 Done Sent Done false
6795 Acked Flush 0 Done Sent Done false
6796 Acked Write 4 Done Sent Done false
6797 Acked Write 5 Done Sent Done false
6798 Acked Read 7 Done Sent Done false
6799 Acked Read 1 Done Sent Done false
6800 Acked Flush 0 Done Sent Done false
6801 Acked Read 3 Done Sent Done false
6802 Acked Write 8 Done Sent Done false
6803 Acked Read 7 Done Sent Done false
6804 Acked Read 6 Done Sent Done false
6805 Acked Read 3 Done Sent Done false
6806 Acked Flush 0 Done Sent Done false
6807 Acked Read 10 Done Sent Done false
6808 Acked Read 6 Done Sent Done false
6809 Acked Read 2 Done Sent Done false
6810 Acked Read 4 Done Sent Done false
6811 Acked Write 7 Done Sent Done false
6812 Acked Read 6 Done Sent Done false
6813 Acked Flush 0 Done Sent Done false
6814 Acked Write 7 Done Sent Done false
6815 Acked Read 6 Done Sent Done false
6816 Acked Read 4 Done Sent Done false
6817 Acked Read 4 Done Sent Done false
6818 Acked Read 4 Done Sent Done false
6819 Acked Write 1 Done Sent Done false
6820 Acked Write 9 Done Sent Done false
6821 Acked Read 4 Done Sent Done false
6822 Acked Write 2 Done Sent Done false
6823 Acked Read 5 Done Sent Done false
6824 Acked Write 8 Done Sent Done false
6825 Acked Write 6 Done Sent Done false
6826 Acked Read 10 Done Sent Done false
6827 Acked Read 1 Done Sent Done false
6828 Acked Write 2 Done Sent Done false
6829 Acked Read 5 Done Sent Done false
6830 Acked Write 1 Done Sent Done false
6831 Acked Write 2 Done Sent Done false
6832 Acked Flush 0 Done Sent Done false
6833 Acked Read 4 Done Sent Done false
6834 Acked Read 10 Done Sent Done false
6835 Acked Write 8 Done Sent Done false
6836 Acked Read 1 Done Sent Done false
6837 Acked Flush 0 Done Sent Done false
6838 Acked Read 2 Done Sent Done false
6839 Acked Flush 0 Done Sent Done false
6840 Acked Read 5 Done Sent Done false
6841 Acked Write 3 Done Sent Done false
6842 Acked Write 6 Done Sent Done false
6843 Acked Read 3 Done Sent Done false
6844 Acked Write 8 Done Sent Done false
6845 Acked Read 3 Done Sent Done false
6846 Acked Read 1 Done Sent Done false
6847 Acked Write 5 Done Sent Done false
6848 Acked Read 8 Done Sent Done false
6849 Acked Flush 0 Done Sent Done false
6850 Acked Read 7 Done Sent Done false
6851 Acked Write 10 Done Sent Done false
6852 Acked Read 1 Done Sent Done false
6853 Acked Write 10 Done Sent Done false
6854 Acked Write 5 Done Sent Done false
6855 Acked Write 6 Done Sent Done false
6856 Acked Read 8 Done Sent Done false
6857 Acked Flush 0 Done Sent Done false
6858 Acked Read 9 Done Sent Done false
6859 Acked Write 7 Done Sent Done false
6860 Acked Write 3 Done Sent Done false
6861 Acked Read 8 Done Sent Done false
6862 Acked Read 5 Done Sent Done false
6863 Acked Write 8 Done Sent Done false
6864 Acked Write 4 Done Sent Done false
6865 Acked Write 9 Done Sent Done false
6866 Acked Write 1 Done Sent Done false
6867 Acked Write 4 Done Sent Done false
6868 Acked Read 3 Done Sent Done false
6869 Acked Read 3 Done Sent Done false
6870 Acked Write 8 Done Sent Done false
6871 Acked Read 9 Done Sent Done false
6872 Acked Read 1 Done Sent Done false
6873 Acked Read 3 Done Sent Done false
6874 Acked Read 6 Done Sent Done false
6875 Acked Read 7 Done Sent Done false
6876 Acked Read 7 Done Sent Done false
6877 Acked Write 10 Done Sent Done false
6878 Acked Read 10 Done Sent Done false
6879 Acked Write 1 Done Sent Done false
6880 Acked Read 4 Done Sent Done false
6881 Acked Write 6 Done Sent Done false
6882 Acked Write 1 Done Sent Done false
6883 Acked Read 3 Done Sent Done false
6884 Acked Read 10 Done Sent Done false
6885 Acked Write 3 Done Sent Done false
6886 Acked Write 6 Done Sent Done false
6887 Acked Read 8 Done Sent Done false
6888 Acked Read 7 Done Sent Done false
6889 Acked Read 2 Done Sent Done false
6890 Acked Flush 0 Done Sent Done false
6891 Acked Read 2 Done Sent Done false
6892 Acked Read 3 Done Sent Done false
6893 Acked Read 5 Done Sent Done false
6894 Acked Read 4 Done Sent Done false
6895 Acked Read 6 Done Sent Done false
6896 Acked Read 7 Done Sent Done false
6897 Acked Read 3 Done Sent Done false
6898 Acked Flush 0 Done Sent Done false
6899 Acked Read 4 Done Sent Done false
6900 Acked Read 1 Done Sent Done false
6901 Acked Read 6 Done Sent Done false
6902 Acked Write 7 Done Sent Done false
6903 Acked Write 3 Done Sent Done false
6904 Acked Write 4 Done Sent Done false
6905 Acked Flush 0 Done Sent Done false
6906 Acked Write 1 Done Sent Done false
6907 Acked Read 10 Done Sent Done false
6908 Acked Read 10 Done Sent Done false
6909 Acked Write 8 Done Sent Done false
6910 Acked Read 9 Done Sent Done false
6911 Acked Write 7 Done Sent Done false
6912 Acked Read 1 Done Sent Done false
6913 Acked Write 5 Done Sent Done false
6914 Acked Write 4 Done Sent Done false
6915 Acked Write 3 Done Sent Done false
6916 Acked Write 3 Done Sent Done false
6917 Acked Write 8 Done Sent Done false
6918 Acked Read 3 Done Sent Done false
6919 Acked Read 3 Done Sent Done false
6920 Acked Read 7 Done Sent Done false
6921 Acked Read 7 Done Sent Done false
6922 Acked Write 3 Done Sent Done false
6923 Acked Flush 0 Done Sent Done false
6924 Acked Read 3 Done Sent Done false
6925 Acked Write 4 Done Sent Done false
6926 Acked Read 9 Done Sent Done false
6927 Acked Read 9 Done Sent Done false
6928 Acked Read 5 Done Sent Done false
6929 Acked Read 1 Done Sent Done false
6930 Acked Flush 0 Done Sent Done false
6931 Acked Read 7 Done Sent Done false
6932 Acked Read 10 Done Sent Done false
6933 Acked Read 2 Done Sent Done false
6934 Acked Read 6 Done Sent Done false
6935 Acked Read 4 Done Sent Done false
6936 Acked Write 4 Done Sent Done false
6937 Acked Read 6 Done Sent Done false
6938 Acked Write 4 Done Sent Done false
6939 Acked Write 7 Done Sent Done false
6940 Acked Flush 0 Done Sent Done false
6941 Acked Write 6 Done Sent Done false
6942 Acked Read 5 Done Sent Done false
6943 Acked Write 1 Done Sent Done false
6944 Acked Write 4 Done Sent Done false
6945 Acked Read 9 Done Sent Done false
6946 Acked Write 7 Done Sent Done false
6947 Acked Read 3 Done Sent Done false
6948 Acked Write 4 Done Sent Done false
6949 Acked Read 8 Done Sent Done false
6950 Acked Write 3 Done Sent Done false
6951 Acked Write 1 Done Sent Done false
6952 Acked Flush 0 Done Sent Done false
6953 Acked Read 3 Done Sent Done false
6954 Acked Write 4 Done Sent Done false
6955 Acked Write 3 Done Sent Done false
6956 Acked Write 8 Done Sent Done false
6957 Acked Read 6 Done Sent Done false
6958 Acked Write 4 Done Sent Done false
6959 Acked Read 4 Done Sent Done false
6960 Acked Write 5 Done Sent Done false
6961 Acked Write 7 Done Sent Done false
6962 Acked Read 10 Done Sent Done false
6963 Acked Read 6 Done Sent Done false
6964 Acked Read 3 Done Sent Done false
6965 Acked Read 7 Done Sent Done false
6966 Acked Flush 0 Done Sent Done false
6967 Acked Write 6 Done Sent Done false
6968 Acked Read 10 Done Sent Done false
6969 Acked Write 5 Done Sent Done false
6970 Acked Read 3 Done Sent Done false
6971 Acked Read 7 Done Sent Done false
6972 Acked Read 2 Done Sent Done false
6973 Acked Read 2 Done Sent Done false
6974 Acked Read 9 Done Sent Done false
6975 Acked Read 7 Done Sent Done false
6976 Acked Read 3 Done Sent Done false
6977 Acked Read 10 Done Sent Done false
6978 Acked Read 6 Done Sent Done false
6979 Acked Write 2 Done Sent Done false
6980 Acked Flush 0 Done Sent Done false
6981 Acked Write 5 Done Sent Done false
6982 Acked Write 9 Done Sent Done false
6983 Acked Write 2 Done Sent Done false
6984 Acked Read 5 Done Sent Done false
6985 Acked Flush 0 Done Sent Done false
6986 Acked Flush 0 Done Sent Done false
6987 Acked Read 2 Done Sent Done false
6988 Acked Read 9 Done Sent Done false
6989 Acked Write 7 Done Sent Done false
6990 Acked Read 3 Done Sent Done false
6991 Acked Read 7 Done Sent Done false
6992 Acked Write 6 Done Sent Done false
6993 Acked Write 8 Done Sent Done false
6994 Acked Flush 0 Done Sent Done false
6995 Acked Write 4 Done Sent Done false
6996 Acked Read 6 Done Sent Done false
6997 Acked Read 10 Done Sent Done false
6998 Acked Read 4 Done Sent Done false
6999 Acked Read 9 Done Sent Done false
7000 Acked Write 1 Done Sent Done false
7001 Acked Read 7 Done Sent Done false
7002 Acked Flush 0 Done Sent Done false
7003 Acked Read 10 Done Sent Done false
7004 Acked Read 8 Done Sent Done false
7005 Acked Read 4 Done Sent Done false
7006 Acked Flush 0 Done Sent Done false
7007 Acked Read 1 Done Sent Done false
7008 Acked Write 3 Done Sent Done false
7009 Acked Read 2 Done Sent Done false
7010 Acked Write 2 Done Sent Done false
7011 Acked Write 10 Done Sent Done false
7012 Acked Write 6 Done Sent Done false
7013 Acked Write 4 Done Sent Done false
7014 Acked Read 1 Done Sent Done false
7015 Acked Read 1 Done Sent Done false
7016 Acked Write 10 Done Sent Done false
7017 Acked Read 9 Done Sent Done false
7018 Acked Read 3 Done Sent Done false
7019 Acked Flush 0 Done Sent Done false
7020 Acked Write 4 Done Sent Done false
7021 Acked Read 1 Done Sent Done false
7022 Acked Flush 0 Done Sent Done false
7023 Acked Flush 0 Done Sent Done false
7024 Acked Flush 0 Done Sent Done false
7025 Acked Read 1 Done Sent Done false
7026 Acked Read 8 Done Sent Done false
7027 Acked Write 6 Done Sent Done false
7028 Acked Write 5 Done Sent Done false
7029 Acked Flush 0 Done Sent Done false
7030 Acked Flush 0 Done Sent Done false
7031 Acked Write 2 Done Sent Done false
7032 Acked Read 5 Done Sent Done false
7033 Acked Read 4 Done Sent Done false
7034 Acked Read 4 Done Sent Done false
7035 Acked Write 8 Done Sent Done false
7036 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7036 6734 7036
Downstairs last five completed: 6734 6733 6732 6731 6730
Upstairs last five completed: 7036 7035 7034 7033 7032
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.634256756Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52788)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.634384091Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.635648574Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.636257343Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.636292808Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 6734","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.636312166Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 6734","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.636513723Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 6734","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.63653743Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 6734","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:07.636761554Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7036 7035 7034 7033 7032
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:41:09.872 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 055c4d6a-2a9d-4816-8c32-82d4dbc5b4cc, [8830] Got stop action so:false kr:false
remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:41:09.873 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5317fb54-546f-4339-b615-db0c98957eb3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:09.873658189Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:09.87373027Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:09.873756407Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:09.87381522Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:41:13.876 INFO request completed, latency_us: 319, response_code: 200, uri: /state/cid/2, method: GET, req_id: b5df2c67-f99e-4c87-b67a-c1ae0ae3e606, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 4880 len 16384
002/300 Read block 3042 len 20480
003/300 Read block 2134 len 4096
004/300 Write block 4268 len 24576 data: 6 5 5 7 5 5
005/300 Read block 91 len 4096
006/300 Read block 4825 len 4096
007/300 Read block 2538 len 16384
008/300 Read block 1158 len 28672
009/300 Read block 3517 len 40960
010/300 Write block 1807 len 8192 data: 5 5
011/300 Flush
012/300 Write block 2198 len 4096 data: 5
013/300 Read block 3901 len 4096
014/300 Read block 2075 len 12288
015/300 Write block 3960 len 36864 data: 5 5 6 7 6 8 6 6 6
016/300 Write block 4135 len 16384 data: 4 4 4 3
017/300 Flush
018/300 Write block 1339 len 24576 data: 5 4 3 5 5 6
019/300 Read block 2151 len 4096
020/300 Flush
021/300 Read block 4898 len 12288
022/300 Read block 4959 len 16384
023/300 Read block 3856 len 28672
024/300 Read block 634 len 8192
025/300 Read block 4774 len 24576
026/300 Flush
027/300 Write block 4924 len 32768 data: 7 7 6 7 7 6 5 5
028/300 Write block 1214 len 20480 data: 3 4 4 4 5
029/300 Write block 2489 len 16384 data: 4 5 5 5
030/300 Write block 2506 len 16384 data: 3 4 6 5
031/300 Read block 972 len 4096
032/300 Read block 1367 len 24576
033/300 Write block 2555 len 8192 data: 4 4
034/300 Read block 1074 len 20480
035/300 Write block 3866 len 4096 data: 4
036/300 Read block 554 len 20480
037/300 Read block 4530 len 40960
038/300 Write block 503 len 40960 data: 5 4 6 6 5 4 3 5 7 6
039/300 Read block 4341 len 4096
040/300 Write block 2251 len 24576 data: 4 5 5 6 9 10
041/300 Read block 2153 len 12288
042/300 Read block 3254 len 12288
043/300 Write block 4655 len 28672 data: 4 5 5 6 5 3 2
044/300 Write block 1820 len 4096 data: 6
045/300 Write block 1028 len 12288 data: 3 5 5
046/300 Read block 2609 len 4096
047/300 Read block 404 len 32768
048/300 Write block 3174 len 32768 data: 6 7 8 8 8 8 6 6
049/300 Read block 138 len 4096
050/300 Read block 1216 len 40960
051/300 Read block 2315 len 8192
052/300 Write block 3715 len 16384 data: 8 6 6 5
053/300 Read block 3143 len 24576
054/300 Write block 3122 len 28672 data: 5 7 7 6 5 5 6
055/300 Flush
056/300 Read block 3877 len 24576
057/300 Read block 979 len 36864
058/300 Read block 1090 len 16384
059/300 Flush
060/300 Read block 1178 len 16384
061/300 Write block 3068 len 20480 data: 5 4 3 3 3
062/300 Read block 1947 len 24576
063/300 Write block 2884 len 12288 data: 2 4 4
064/300 Read block 4392 len 4096
065/300 Read block 4328 len 40960
066/300 Write block 2105 len 24576 data: 4 4 4 4 4 3
067/300 Write block 3993 len 28672 data: 3 3 2 2 2 2 2
068/300 Read block 2618 len 24576
069/300 Read block 474 len 12288
070/300 Flush
071/300 Read block 2886 len 32768
072/300 Write block 4130 len 32768 data: 4 3 3 4 5 5 5 5
073/300 Read block 1878 len 28672
074/300 Write block 4529 len 36864 data: 4 4 5 5 5 5 5 4 3
075/300 Read block 2370 len 20480
076/300 Read block 3536 len 16384
077/300 Write block 920 len 24576 data: 6 6 5 3 4 4
078/300 Read block 3131 len 24576
079/300 Write block 1573 len 40960 data: 6 5 5 5 4 4 3 5 4 4
080/300 Write block 1223 len 24576 data: 6 6 5 4 3 3
081/300 Write block 864 len 20480 data: 6 5 8 8 8
082/300 Read block 1165 len 20480
083/300 Write block 4007 len 40960 data: 3 5 6 7 5 4 5 3 3 3
084/300 Read block 3988 len 4096
085/300 Write block 2380 len 24576 data: 5 4 3 3 3 5
086/300 Read block 3646 len 40960
087/300 Read block 4315 len 32768
088/300 Write block 3576 len 24576 data: 3 2 2 2 3 4
089/300 Read block 3049 len 28672
090/300 Read block 464 len 4096
091/300 Write block 3627 len 32768 data: 2 3 3 3 3 3 4 4
092/300 Write block 4905 len 36864 data: 4 3 4 5 4 4 4 4 4
093/300 Write block 31 len 8192 data: 2 2
094/300 Write block 3506 len 20480 data: 3 3 3 4 4
095/300 Read block 386 len 4096
096/300 Write block 3786 len 36864 data: 3 3 3 3 3 3 3 3 2
097/300 Write block 2950 len 28672 data: 4 5 6 6 6 5 5
098/300 Read block 3883 len 24576
099/300 Write block 3550 len 16384 data: 4 4 3 5
100/300 Write block 2479 len 8192 data: 2 2
101/300 Read block 2660 len 12288
102/300 Read block 4066 len 16384
103/300 Read block 795 len 40960
104/300 Read block 145 len 4096
105/300 Read block 4022 len 28672
106/300 Write block 3750 len 12288 data: 5 4 4
107/300 Write block 2610 len 40960 data: 5 7 5 5 4 3 3 4 4 5
108/300 Write block 1594 len 28672 data: 6 6 8 9 10 7 8
109/300 Write block 644 len 4096 data: 4
110/300 Read block 1550 len 12288
111/300 Write block 3101 len 32768 data: 4 4 6 8 7 8 8 8
112/300 Read block 2607 len 36864
113/300 Write block 4000 len 16384 data: 2 3 3 3
114/300 Write block 4031 len 28672 data: 2 2 2 4 5 6 6
115/300 Read block 3400 len 24576
116/300 Read block 2564 len 20480
117/300 Write block 3614 len 4096 data: 4
118/300 Read block 2865 len 32768
119/300 Write block 1918 len 36864 data: 3 3 4 4 3 4 6 6 5
120/300 Read block 4552 len 36864
121/300 Write block 960 len 24576 data: 5 5 4 4 4 3
122/300 Write block 4485 len 4096 data: 4
123/300 Read block 1013 len 12288
124/300 Read block 4241 len 8192
125/300 Write block 1167 len 16384 data: 5 5 5 5
126/300 Flush
127/300 Write block 2050 len 32768 data: 5 5 3 4 4 4 4 4
128/300 Read block 3568 len 8192
129/300 Write block 400 len 24576 data: 5 6 7 7 7 6
130/300 Write block 116 len 8192 data: 8 7
131/300 Read block 4759 len 28672
132/300 Read block 326 len 36864
133/300 Flush
134/300 Write block 4636 len 12288 data: 4 4 5
135/300 Read block 1096 len 40960
136/300 Read block 4681 len 8192
137/300 Write block 738 len 32768 data: 4 4 4 4 6 7 6 7
138/300 Read block 3362 len 40960
139/300 Write block 1814 len 32768 data: 7 9 8 7 5 5 7 6
140/300 Flush
141/300 Write block 1709 len 12288 data: 2 3 3
142/300 Write block 4987 len 24576 data: 5 5 6 5 4 4
143/300 Write block 954 len 36864 data: 3 3 6 6 5 5 6 6 5
144/300 Flush
145/300 Write block 1362 len 32768 data: 5 5 6 6 6 6 5 5
146/300 Read block 4579 len 28672
147/300 Read block 2094 len 12288
148/300 Read block 195 len 16384
149/300 Write block 3098 len 20480 data: 5 5 4 5 5
150/300 Write block 3836 len 36864 data: 4 4 5 5 6 7 8 7 6
151/300 Read block 392 len 4096
152/300 Write block 2275 len 32768 data: 3 3 3 4 4 4 5 5
153/300 Read block 3203 len 4096
154/300 Write block 2011 len 12288 data: 7 5 6
155/300 Write block 3337 len 16384 data: 6 5 5 5
156/300 Write block 4825 len 28672 data: 5 5 4 2 5 6 5
157/300 Read block 830 len 8192
158/300 Read block 1049 len 4096
159/300 Read block 3823 len 16384
160/300 Flush
161/300 Write block 1176 len 12288 data: 2 2 2
162/300 Read block 4099 len 16384
163/300 Read block 1019 len 36864
164/300 Read block 1558 len 16384
165/300 Write block 509 len 16384 data: 4 6 8 7
166/300 Flush
167/300 Write block 2302 len 24576 data: 5 6 6 6 4 5
168/300 Read block 1195 len 8192
169/300 Read block 787 len 12288
170/300 Read block 1068 len 32768
171/300 Read block 3546 len 4096
172/300 Read block 1657 len 12288
173/300 Read block 3007 len 40960
174/300 Write block 325 len 32768 data: 5 4 4 4 4 3 3 3
175/300 Read block 1532 len 16384
176/300 Read block 1590 len 24576
177/300 Write block 858 len 32768 data: 4 5 6 7 7 6 7 6
178/300 Write block 1504 len 20480 data: 2 2 2 3 3
179/300 Write block 2391 len 12288 data: 7 6 5
180/300 Read block 400 len 12288
181/300 Write block 182 len 12288 data: 6 7 8
182/300 Write block 381 len 28672 data: 2 2 2 2 2 2 2
183/300 Read block 1527 len 16384
184/300 Read block 2540 len 28672
185/300 Read block 255 len 12288
186/300 Read block 1703 len 28672
187/300 Read block 1099 len 40960
188/300 Write block 2839 len 8192 data: 6 6
189/300 Read block 2287 len 4096
190/300 Write block 1729 len 20480 data: 5 4 3 2 2
191/300 Flush
192/300 Read block 3083 len 36864
193/300 Write block 3802 len 4096 data: 2
194/300 Write block 4247 len 16384 data: 7 7 7 8
195/300 Flush
196/300 Read block 369 len 8192
197/300 Read block 4468 len 40960
198/300 Write block 4576 len 4096 data: 4
199/300 Read block 2519 len 24576
200/300 Read block 123 len 8192
201/300 Read block 3400 len 4096
202/300 Read block 306 len 40960
203/300 Read block 3634 len 28672
204/300 Read block 4123 len 28672
205/300 Read block 3408 len 4096
206/300 Write block 1219 len 8192 data: 6 5
207/300 Write block 91 len 20480 data: 4 4 3 4 4
208/300 Read block 4591 len 12288
209/300 Read block 1437 len 32768
210/300 Write block 4526 len 36864 data: 6 6 6 5 5 6 6 6 6
211/300 Read block 4194 len 20480
212/300 Read block 2589 len 8192
213/300 Flush
214/300 Write block 3679 len 12288 data: 3 3 2
215/300 Read block 2458 len 28672
216/300 Read block 1068 len 24576
217/300 Write block 138 len 4096 data: 8
218/300 Read block 2474 len 20480
219/300 Read block 3132 len 12288
220/300 Read block 3867 len 36864
221/300 Flush
222/300 Read block 3323 len 4096
223/300 Read block 3297 len 36864
224/300 Write block 4909 len 32768 data: 5 5 5 5 5 4 4 5
225/300 Write block 1581 len 12288 data: 5 5 4
226/300 Write block 796 len 36864 data: 6 6 5 4 5 4 5 4 3
227/300 Write block 942 len 8192 data: 4 3
228/300 Read block 1438 len 4096
229/300 Flush
230/300 Read block 2011 len 40960
231/300 Read block 4491 len 32768
232/300 Read block 720 len 40960
233/300 Read block 1657 len 24576
234/300 Read block 2516 len 32768
235/300 Read block 4267 len 8192
236/300 Write block 554 len 28672 data: 7 4 3 4 4 5 5
237/300 Read block 1643 len 40960
238/300 Read block 4091 len 40960
239/300 Read block 1712 len 12288
240/300 Flush
241/300 Read block 3317 len 4096
242/300 Read block 2730 len 40960
243/300 Read block 907 len 16384
244/300 Flush
245/300 Read block 1396 len 32768
246/300 Flush
247/300 Read block 1254 len 20480
248/300 Write block 4614 len 40960 data: 5 6 6 6 6 6 5 4 3 3
249/300 Write block 180 len 4096 data: 5
250/300 Read block 4264 len 8192
251/300 Write block 3621 len 24576 data: 7 6 7 7 4 3
252/300 Read block 2988 len 8192
253/300 Read block 2180 len 36864
254/300 Read block 1987 len 36864
255/300 Read block 3093 len 24576
256/300 Write block 1863 len 36864 data: 5 4 5 6 4 3 3 5 5
257/300 Write block 2586 len 28672 data: 3 4 4 4 5 5 5
258/300 Read block 4728 len 16384
259/300 Read block 1423 len 24576
260/300 Flush
261/300 Flush
262/300 Write block 2811 len 12288 data: 6 6 5
263/300 Flush
264/300 Write block 1973 len 8192 data: 3 3
265/300 Read block 1285 len 24576
266/300 Write block 2729 len 28672 data: 4 4 4 4 3 3 3
267/300 Write block 3645 len 20480 data: 4 5 4 5 6
268/300 Write block 936 len 12288 data: 3 4 4
269/300 Read block 4607 len 16384
270/300 Write block 454 len 24576 data: 5 4 3 3 3 3
271/300 Write block 3207 len 4096 data: 2
272/300 Write block 2748 len 12288 data: 3 3 4
273/300 Read block 1877 len 20480
274/300 Flush
275/300 Write block 684 len 32768 data: 7 7 7 6 6 6 5 4
276/300 Read block 3557 len 24576
277/300 Flush
278/300 Write block 4736 len 8192 data: 8 8
279/300 Read block 683 len 24576
280/300 Read block 3022 len 4096
281/300 Read block 3608 len 32768
282/300 Write block 666 len 36864 data: 3 3 3 3 4 4 4 4 4
283/300 Write block 2094 len 12288 data: 2 2 3
284/300 Write block 1172 len 8192 data: 5 3
285/300 Write block 4327 len 12288 data: 3 3 5
286/300 Write block 2153 len 16384 data: 4 6 6 8
287/300 Read block 4642 len 16384
288/300 Read block 3838 len 8192
289/300 Write block 272 len 28672 data: 5 5 5 6 6 6 7
290/300 Read block 4656 len 20480
291/300 Write block 2370 len 24576 data: 6 7 6 6 5 6
292/300 Write block 4599 len 36864 data: 3 3 4 5 6 4 4 4 5
293/300 Write block 4415 len 12288 data: 3 4 4
294/300 Flush
295/300 Read block 214 len 4096
296/300 Read block 2870 len 32768
297/300 Flush
298/300 Read block 2661 len 8192
299/300 Read block 4607 len 12288
300/300 Read block 6 len 40960
301/300 Flush
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:41:14.109 INFO request completed, latency_us: 162, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8463033e-a51b-4464-b5dd-7215b10c4a63, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 4 Done Done Sent false
7038 Acked Read 5 Done Done Sent false
7039 Acked Read 1 Done Done Sent false
7040 Acked Write 6 Done Done Sent false
7041 Acked Read 1 Done Done Sent false
7042 Acked Read 1 Done Done Sent false
7043 Acked Read 4 Done Done Sent false
7044 Acked Read 7 Done Done Sent false
7045 Acked Read 10 Done Done Sent false
7046 Acked Write 2 Done Done Sent false
7047 Acked Flush 0 Done Done Sent false
7048 Acked Write 1 Done Done Sent false
7049 Acked Read 1 Done Done Sent false
7050 Acked Read 3 Done Done Sent false
7051 Acked Write 9 Done Done Sent false
7052 Acked Write 4 Done Done Sent false
7053 Acked Flush 0 Done Done Sent false
7054 Acked Write 6 Done Done Sent false
7055 Acked Read 1 Done Done Sent false
7056 Acked Flush 0 Done Done Sent false
7057 Acked Read 3 Done Done Sent false
7058 Acked Read 4 Done Done Sent false
7059 Acked Read 7 Done Done Sent false
7060 Acked Read 2 Done Done Sent false
7061 Acked Read 6 Done Done Sent false
7062 Acked Flush 0 Done Done Sent false
7063 Acked Write 8 Done Done Sent false
7064 Acked Write 5 Done Done Sent false
7065 Acked Write 4 Done Done Sent false
7066 Acked Write 4 Done Done Sent false
7067 Acked Read 1 Done Done Sent false
7068 Acked Read 6 Done Done Sent false
7069 Acked Write 2 Done Done Sent false
7070 Acked Read 5 Done Done Sent false
7071 Acked Write 1 Done Done Sent false
7072 Acked Read 5 Done Done Sent false
7073 Acked Read 10 Done Done Sent false
7074 Acked Write 10 Done Done Sent false
7075 Acked Read 1 Done Done Sent false
7076 Acked Write 6 Done Done Sent false
7077 Acked Read 3 Done Done Sent false
7078 Acked Read 3 Done Done Sent false
7079 Acked Write 7 Done Done Sent false
7080 Acked Write 1 Done Done Sent false
7081 Acked Write 3 Done Done Sent false
7082 Acked Read 1 Done Done Sent false
7083 Acked Read 8 Done Done Sent false
7084 Acked Write 8 Done Done Sent false
7085 Acked Read 1 Done Done Sent false
7086 Acked Read 10 Done Done Sent false
7087 Acked Read 2 Done Done Sent false
7088 Acked Write 4 Done Done Sent false
7089 Acked Read 6 Done Done Sent false
7090 Acked Write 7 Done Done Sent false
7091 Acked Flush 0 Done Done Sent false
7092 Acked Read 6 Done Done Sent false
7093 Acked Read 9 Done Done Sent false
7094 Acked Read 4 Done Done Sent false
7095 Acked Flush 0 Done Done Sent false
7096 Acked Read 4 Done Done Sent false
7097 Acked Write 5 Done Done Sent false
7098 Acked Read 6 Done Done Sent false
7099 Acked Write 3 Done Done Sent false
7100 Acked Read 1 Done Done Sent false
7101 Acked Read 10 Done Done Sent false
7102 Acked Write 6 Done Done Sent false
7103 Acked Write 7 Done Done Sent false
7104 Acked Read 6 Done Done Sent false
7105 Acked Read 3 Done Done Sent false
7106 Acked Flush 0 Done Done Sent false
7107 Acked Read 8 Done Done Sent false
7108 Acked Write 8 Done Done Sent false
7109 Acked Read 7 Done Done Sent false
7110 Acked Write 9 Done Done Sent false
7111 Acked Read 5 Done Done Sent false
7112 Acked Read 4 Done Done Sent false
7113 Acked Write 6 Done Done Sent false
7114 Acked Read 6 Done Done Sent false
7115 Acked Write 10 Done Done Sent false
7116 Acked Write 6 Done Done Sent false
7117 Acked Write 5 Done Done Sent false
7118 Acked Read 5 Done Done Sent false
7119 Acked Write 10 Done Done Sent false
7120 Acked Read 1 Done Done Sent false
7121 Acked Write 6 Done Done Sent false
7122 Acked Read 10 Done Done Sent false
7123 Acked Read 8 Done Done Sent false
7124 Acked Write 6 Done Done Sent false
7125 Acked Read 7 Done Done Sent false
7126 Acked Read 1 Done Done Sent false
7127 Acked Write 8 Done Done Sent false
7128 Acked Write 9 Done Done Sent false
7129 Acked Write 2 Done Done Sent false
7130 Acked Write 5 Done Done Sent false
7131 Acked Read 1 Done Done Sent false
7132 Acked Write 9 Done Done Sent false
7133 Acked Write 7 Done Done Sent false
7134 Acked Read 6 Done Done Sent false
7135 Acked Write 4 Done Done Sent false
7136 Acked Write 2 Done Done Sent false
7137 Acked Read 3 Done Done Sent false
7138 Acked Read 4 Done Done Sent false
7139 Acked Read 10 Done Done Sent false
7140 Acked Read 1 Done Done Sent false
7141 Acked Read 7 Done Done Sent false
7142 Acked Write 3 Done Done Sent false
7143 Acked Write 10 Done Done Sent false
7144 Acked Write 7 Done Done Sent false
7145 Acked Write 1 Done Done Sent false
7146 Acked Read 3 Done Done Sent false
7147 Acked Write 8 Done Done Sent false
7148 Acked Read 9 Done Done Sent false
7149 Acked Write 4 Done Done Sent false
7150 Acked Write 7 Done Done Sent false
7151 Acked Read 6 Done Done Sent false
7152 Acked Read 5 Done Done Sent false
7153 Acked Write 1 Done Done Sent false
7154 Acked Read 8 Done Done Sent false
7155 Acked Write 9 Done Done Sent false
7156 Acked Read 9 Done Done Sent false
7157 Acked Write 6 Done Done Sent false
7158 Acked Write 1 Done Done Sent false
7159 Acked Read 3 Done Done Sent false
7160 Acked Read 2 Done Done Sent false
7161 Acked Write 4 Done Done Sent false
7162 Acked Flush 0 Done Done Sent false
7163 Acked Write 8 Done Done Sent false
7164 Acked Read 2 Done Done Sent false
7165 Acked Write 6 Done Done Sent false
7166 Acked Write 2 Done Done Sent false
7167 Acked Read 7 Done Done Sent false
7168 Acked Read 9 Done Done Sent false
7169 Acked Flush 0 Done Done Sent false
7170 Acked Write 3 Done Done Sent false
7171 Acked Read 10 Done Done Sent false
7172 Acked Read 2 Done Done Sent false
7173 Acked Write 8 Done Done Sent false
7174 Acked Read 10 Done Done Sent false
7175 Acked Write 8 Done Done Sent false
7176 Acked Flush 0 Done Done Sent false
7177 Acked Write 3 Done Done Sent false
7178 Acked Write 6 Done Done Sent false
7179 Acked Write 9 Done Done Sent false
7180 Acked Flush 0 Done Done Sent false
7181 Acked Write 8 Done Done Sent false
7182 Acked Read 7 Done Done Sent false
7183 Acked Read 3 Done Done Sent false
7184 Acked Read 4 Done Done Sent false
7185 Acked Write 5 Done Done Sent false
7186 Acked Write 9 Done Done Sent false
7187 Acked Read 1 Done Done Sent false
7188 Acked Write 8 Done Done Sent false
7189 Acked Read 1 Done Done Sent false
7190 Acked Write 3 Done Done Sent false
7191 Acked Write 4 Done Done Sent false
7192 Acked Write 7 Done Done Sent false
7193 Acked Read 2 Done Done Sent false
7194 Acked Read 1 Done Done Sent false
7195 Acked Read 4 Done Done Sent false
7196 Acked Flush 0 Done Done Sent false
7197 Acked Write 3 Done Done Sent false
7198 Acked Read 4 Done Done Sent false
7199 Acked Read 9 Done Done Sent false
7200 Acked Read 4 Done Done Sent false
7201 Acked Write 4 Done Done Sent false
7202 Acked Flush 0 Done Done Sent false
7203 Acked Write 6 Done Done Sent false
7204 Acked Read 2 Done Done Sent false
7205 Acked Read 3 Done Done Sent false
7206 Acked Read 8 Done Done Sent false
7207 Acked Read 1 Done Done Sent false
7208 Acked Read 3 Done Done Sent false
7209 Acked Read 10 Done Done Sent false
7210 Acked Write 8 Done Done Sent false
7211 Acked Read 4 Done Done Sent false
7212 Acked Read 6 Done Done Sent false
7213 Acked Write 8 Done Done Sent false
7214 Acked Write 5 Done Done Sent false
7215 Acked Write 3 Done Done Sent false
7216 Acked Read 3 Done Done Sent false
7217 Acked Write 3 Done Done Sent false
7218 Acked Write 7 Done Done Sent false
7219 Acked Read 4 Done Done Sent false
7220 Acked Read 7 Done Done Sent false
7221 Acked Read 3 Done Done Sent false
7222 Acked Read 7 Done Done Sent false
7223 Acked Read 10 Done Done Sent false
7224 Acked Write 2 Done Done Sent false
7225 Acked Read 1 Done Done Sent false
7226 Acked Write 5 Done Done Sent false
7227 Acked Flush 0 Done Done Sent false
7228 Acked Read 9 Done Done Sent false
7229 Acked Write 1 Done Done Sent false
7230 Acked Write 4 Done Done Sent false
7231 Acked Flush 0 Done Done Sent false
7232 Acked Read 2 Done Done Sent false
7233 Acked Read 10 Done Done Sent false
7234 Acked Write 1 Done Done Sent false
7235 Acked Read 6 Done Done Sent false
7236 Acked Read 2 Done Done Sent false
7237 Acked Read 1 Done Done Sent false
7238 Acked Read 10 Done Done Sent false
7239 Acked Read 7 Done Done Sent false
7240 Acked Read 7 Done Done Sent false
7241 Acked Read 1 Done Done Sent false
7242 Acked Write 2 Done Done Sent false
7243 Acked Write 5 Done Done Sent false
7244 Acked Read 3 Done Done Sent false
7245 Acked Read 8 Done Done Sent false
7246 Acked Write 9 Done Done Sent false
7247 Acked Read 5 Done Done Sent false
7248 Acked Read 2 Done Done Sent false
7249 Acked Flush 0 Done Done Sent false
7250 Acked Write 3 Done Done Sent false
7251 Acked Read 7 Done Done Sent false
7252 Acked Read 6 Done Done Sent false
7253 Acked Write 1 Done Done Sent false
7254 Acked Read 5 Done Done Sent false
7255 Acked Read 3 Done Done Sent false
7256 Acked Read 9 Done Done Sent false
7257 Acked Flush 0 Done Done Sent false
7258 Acked Read 1 Done Done Sent false
7259 Acked Read 9 Done Done Sent false
7260 Acked Write 8 Done Done Sent false
7261 Acked Write 3 Done Done Sent false
7262 Acked Write 9 Done Done Sent false
7263 Acked Write 2 Done Done Sent false
7264 Acked Read 1 Done Done Sent false
7265 Acked Flush 0 Done Done Sent false
7266 Acked Read 10 Done Done Sent false
7267 Acked Read 8 Done Done Sent false
7268 Acked Read 10 Done Done Sent false
7269 Acked Read 6 Done Done Sent false
7270 Acked Read 8 Done Done Sent false
7271 Acked Read 2 Done Done Sent false
7272 Acked Write 7 Done Done Sent false
7273 Acked Read 10 Done Done Sent false
7274 Acked Read 10 Done Done Sent false
7275 Acked Read 3 Done Done Sent false
7276 Acked Flush 0 Done Done Sent false
7277 Acked Read 1 Done Done Sent false
7278 Acked Read 10 Done Done Sent false
7279 Acked Read 4 Done Done Sent false
7280 Acked Flush 0 Done Done Sent false
7281 Acked Read 8 Done Done Sent false
7282 Acked Flush 0 Done Done Sent false
7283 Acked Read 5 Done Done Sent false
7284 Acked Write 10 Done Done Sent false
7285 Acked Write 1 Done Done Sent false
7286 Acked Read 2 Done Done Sent false
7287 Acked Write 6 Done Done Sent false
7288 Acked Read 2 Done Done Sent false
7289 Acked Read 9 Done Done Sent false
7290 Acked Read 9 Done Done Sent false
7291 Acked Read 6 Done Done Sent false
7292 Acked Write 9 Done Done Sent false
7293 Acked Write 7 Done Done Sent false
7294 Acked Read 4 Done Done Sent false
7295 Acked Read 6 Done Done Sent false
7296 Acked Flush 0 Done Done Sent false
7297 Acked Flush 0 Done Done Sent false
7298 Acked Write 3 Done Done Sent false
7299 Acked Flush 0 Done Done Sent false
7300 Acked Write 2 Done Done Sent false
7301 Acked Read 6 Done Done Sent false
7302 Acked Write 7 Done Done Sent false
7303 Acked Write 5 Done Done Sent false
7304 Acked Write 3 Done Done Sent false
7305 Acked Read 4 Done Done Sent false
7306 Acked Write 6 Done Done Sent false
7307 Acked Write 1 Done Done Sent false
7308 Acked Write 3 Done Done Sent false
7309 Acked Read 5 Done Done Sent false
7310 Acked Flush 0 Done Done Sent false
7311 Acked Write 8 Done Done Sent false
7312 Acked Read 6 Done Done Sent false
7313 Acked Flush 0 Done Done Sent false
7314 Acked Write 2 Done Done Sent false
7315 Acked Read 6 Done Done Sent false
7316 Acked Read 1 Done Done Sent false
7317 Acked Read 8 Done Done Sent false
7318 Acked Write 9 Done Done Sent false
7319 Acked Write 3 Done Done Sent false
7320 Acked Write 2 Done Done Sent false
7321 Acked Write 3 Done Done Sent false
7322 Acked Write 4 Done Done Sent false
7323 Acked Read 4 Done Done Sent false
7324 Acked Read 2 Done Done Sent false
7325 Acked Write 7 Done Done Sent false
7326 Acked Read 5 Done Done Sent false
7327 Acked Write 6 Done Done Sent false
7328 Acked Write 9 Done Done Sent false
7329 Acked Write 3 Done Done Sent false
7330 Acked Flush 0 Done Done Sent false
7331 Acked Read 1 Done Done Sent false
7332 Acked Read 8 Done Done Sent false
7333 Acked Flush 0 Done Done Sent false
7334 Acked Read 2 Done Done Sent false
7335 Acked Read 3 Done Done Sent false
7336 Acked Read 10 Done Done Sent false
7337 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7337 7337 7036
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7037 Acked Read 4 Done Done Sent false
7038 Acked Read 5 Done Done Sent false
7039 Acked Read 1 Done Done Sent false
7040 Acked Write 6 Done Done Sent false
7041 Acked Read 1 Done Done Sent false
7042 Acked Read 1 Done Done Sent false
7043 Acked Read 4 Done Done Sent false
7044 Acked Read 7 Done Done Sent false
7045 Acked Read 10 Done Done Sent false
7046 Acked Write 2 Done Done Sent false
7047 Acked Flush 0 Done Done Sent false
7048 Acked Write 1 Done Done Sent false
7049 Acked Read 1 Done Done Sent false
7050 Acked Read 3 Done Done Sent false
7051 Acked Write 9 Done Done Sent false
7052 Acked Write 4 Done Done Sent false
7053 Acked Flush 0 Done Done Sent false
7054 Acked Write 6 Done Done Sent false
7055 Acked Read 1 Done Done Sent false
7056 Acked Flush 0 Done Done Sent false
7057 Acked Read 3 Done Done Sent false
7058 Acked Read 4 Done Done Sent false
7059 Acked Read 7 Done Done Sent false
7060 Acked Read 2 Done Done Sent false
7061 Acked Read 6 Done Done Sent false
7062 Acked Flush 0 Done Done Sent false
7063 Acked Write 8 Done Done Sent false
7064 Acked Write 5 Done Done Sent false
7065 Acked Write 4 Done Done Sent false
7066 Acked Write 4 Done Done Sent false
7067 Acked Read 1 Done Done Sent false
7068 Acked Read 6 Done Done Sent false
7069 Acked Write 2 Done Done Sent false
7070 Acked Read 5 Done Done Sent false
7071 Acked Write 1 Done Done Sent false
7072 Acked Read 5 Done Done Sent false
7073 Acked Read 10 Done Done Sent false
7074 Acked Write 10 Done Done Sent false
7075 Acked Read 1 Done Done Sent false
7076 Acked Write 6 Done Done Sent false
7077 Acked Read 3 Done Done Sent false
7078 Acked Read 3 Done Done Sent false
7079 Acked Write 7 Done Done Sent false
7080 Acked Write 1 Done Done Sent false
7081 Acked Write 3 Done Done Sent false
7082 Acked Read 1 Done Done Sent false
7083 Acked Read 8 Done Done Sent false
7084 Acked Write 8 Done Done Sent false
7085 Acked Read 1 Done Done Sent false
7086 Acked Read 10 Done Done Sent false
7087 Acked Read 2 Done Done Sent false
7088 Acked Write 4 Done Done Sent false
7089 Acked Read 6 Done Done Sent false
7090 Acked Write 7 Done Done Sent false
7091 Acked Flush 0 Done Done Sent false
7092 Acked Read 6 Done Done Sent false
7093 Acked Read 9 Done Done Sent false
7094 Acked Read 4 Done Done Sent false
7095 Acked Flush 0 Done Done Sent false
7096 Acked Read 4 Done Done Sent false
7097 Acked Write 5 Done Done Sent false
7098 Acked Read 6 Done Done Sent false
7099 Acked Write 3 Done Done Sent false
7100 Acked Read 1 Done Done Sent false
7101 Acked Read 10 Done Done Sent false
7102 Acked Write 6 Done Done Sent false
7103 Acked Write 7 Done Done Sent false
7104 Acked Read 6 Done Done Sent false
7105 Acked Read 3 Done Done Sent false
7106 Acked Flush 0 Done Done Sent false
7107 Acked Read 8 Done Done Sent false
7108 Acked Write 8 Done Done Sent false
7109 Acked Read 7 Done Done Sent false
7110 Acked Write 9 Done Done Sent false
7111 Acked Read 5 Done Done Sent false
7112 Acked Read 4 Done Done Sent false
7113 Acked Write 6 Done Done Sent false
7114 Acked Read 6 Done Done Sent false
7115 Acked Write 10 Done Done Sent false
7116 Acked Write 6 Done Done Sent false
7117 Acked Write 5 Done Done Sent false
7118 Acked Read 5 Done Done Sent false
7119 Acked Write 10 Done Done Sent false
7120 Acked Read 1 Done Done Sent false
7121 Acked Write 6 Done Done Sent false
7122 Acked Read 10 Done Done Sent false
7123 Acked Read 8 Done Done Sent false
7124 Acked Write 6 Done Done Sent false
7125 Acked Read 7 Done Done Sent false
7126 Acked Read 1 Done Done Sent false
7127 Acked Write 8 Done Done Sent false
7128 Acked Write 9 Done Done Sent false
7129 Acked Write 2 Done Done Sent false
7130 Acked Write 5 Done Done Sent false
7131 Acked Read 1 Done Done Sent false
7132 Acked Write 9 Done Done Sent false
7133 Acked Write 7 Done Done Sent false
7134 Acked Read 6 Done Done Sent false
7135 Acked Write 4 Done Done Sent false
7136 Acked Write 2 Done Done Sent false
7137 Acked Read 3 Done Done Sent false
7138 Acked Read 4 Done Done Sent false
7139 Acked Read 10 Done Done Sent false
7140 Acked Read 1 Done Done Sent false
7141 Acked Read 7 Done Done Sent false
7142 Acked Write 3 Done Done Sent false
7143 Acked Write 10 Done Done Sent false
7144 Acked Write 7 Done Done Sent false
7145 Acked Write 1 Done Done Sent false
7146 Acked Read 3 Done Done Sent false
7147 Acked Write 8 Done Done Sent false
7148 Acked Read 9 Done Done Sent false
7149 Acked Write 4 Done Done Sent false
7150 Acked Write 7 Done Done Sent false
7151 Acked Read 6 Done Done Sent false
7152 Acked Read 5 Done Done Sent false
7153 Acked Write 1 Done Done Sent false
7154 Acked Read 8 Done Done Sent false
7155 Acked Write 9 Done Done Sent false
7156 Acked Read 9 Done Done Sent false
7157 Acked Write 6 Done Done Sent false
7158 Acked Write 1 Done Done Sent false
7159 Acked Read 3 Done Done Sent false
7160 Acked Read 2 Done Done Sent false
7161 Acked Write 4 Done Done Sent false
7162 Acked Flush 0 Done Done Sent false
7163 Acked Write 8 Done Done Sent false
7164 Acked Read 2 Done Done Sent false
7165 Acked Write 6 Done Done Sent false
7166 Acked Write 2 Done Done Sent false
7167 Acked Read 7 Done Done Sent false
7168 Acked Read 9 Done Done Sent false
7169 Acked Flush 0 Done Done Sent false
7170 Acked Write 3 Done Done Sent false
7171 Acked Read 10 Done Done Sent false
7172 Acked Read 2 Done Done Sent false
7173 Acked Write 8 Done Done Sent false
7174 Acked Read 10 Done Done Sent false
7175 Acked Write 8 Done Done Sent false
7176 Acked Flush 0 Done Done Sent false
7177 Acked Write 3 Done Done Sent false
7178 Acked Write 6 Done Done Sent false
7179 Acked Write 9 Done Done Sent false
7180 Acked Flush 0 Done Done Sent false
7181 Acked Write 8 Done Done Sent false
7182 Acked Read 7 Done Done Sent false
7183 Acked Read 3 Done Done Sent false
7184 Acked Read 4 Done Done Sent false
7185 Acked Write 5 Done Done Sent false
7186 Acked Write 9 Done Done Sent false
7187 Acked Read 1 Done Done Sent false
7188 Acked Write 8 Done Done Sent false
7189 Acked Read 1 Done Done Sent false
7190 Acked Write 3 Done Done Sent false
7191 Acked Write 4 Done Done Sent false
7192 Acked Write 7 Done Done Sent false
7193 Acked Read 2 Done Done Sent false
7194 Acked Read 1 Done Done Sent false
7195 Acked Read 4 Done Done Sent false
7196 Acked Flush 0 Done Done Sent false
7197 Acked Write 3 Done Done Sent false
7198 Acked Read 4 Done Done Sent false
7199 Acked Read 9 Done Done Sent false
7200 Acked Read 4 Done Done Sent false
7201 Acked Write 4 Done Done Sent false
7202 Acked Flush 0 Done Done Sent false
7203 Acked Write 6 Done Done Sent false
7204 Acked Read 2 Done Done Sent false
7205 Acked Read 3 Done Done Sent false
7206 Acked Read 8 Done Done Sent false
7207 Acked Read 1 Done Done Sent false
7208 Acked Read 3 Done Done Sent false
7209 Acked Read 10 Done Done Sent false
7210 Acked Write 8 Done Done Sent false
7211 Acked Read 4 Done Done Sent false
7212 Acked Read 6 Done Done Sent false
7213 Acked Write 8 Done Done Sent false
7214 Acked Write 5 Done Done Sent false
7215 Acked Write 3 Done Done Sent false
7216 Acked Read 3 Done Done Sent false
7217 Acked Write 3 Done Done Sent false
7218 Acked Write 7 Done Done Sent false
7219 Acked Read 4 Done Done Sent false
7220 Acked Read 7 Done Done Sent false
7221 Acked Read 3 Done Done Sent false
7222 Acked Read 7 Done Done Sent false
7223 Acked Read 10 Done Done Sent false
7224 Acked Write 2 Done Done Sent false
7225 Acked Read 1 Done Done Sent false
7226 Acked Write 5 Done Done Sent false
7227 Acked Flush 0 Done Done Sent false
7228 Acked Read 9 Done Done Sent false
7229 Acked Write 1 Done Done Sent false
7230 Acked Write 4 Done Done Sent false
7231 Acked Flush 0 Done Done Sent false
7232 Acked Read 2 Done Done Sent false
7233 Acked Read 10 Done Done Sent false
7234 Acked Write 1 Done Done Sent false
7235 Acked Read 6 Done Done Sent false
7236 Acked Read 2 Done Done Sent false
7237 Acked Read 1 Done Done Sent false
7238 Acked Read 10 Done Done Sent false
7239 Acked Read 7 Done Done Sent false
7240 Acked Read 7 Done Done Sent false
7241 Acked Read 1 Done Done Sent false
7242 Acked Write 2 Done Done Sent false
7243 Acked Write 5 Done Done Sent false
7244 Acked Read 3 Done Done Sent false
7245 Acked Read 8 Done Done Sent false
7246 Acked Write 9 Done Done Sent false
7247 Acked Read 5 Done Done Sent false
7248 Acked Read 2 Done Done Sent false
7249 Acked Flush 0 Done Done Sent false
7250 Acked Write 3 Done Done Sent false
7251 Acked Read 7 Done Done Sent false
7252 Acked Read 6 Done Done Sent false
7253 Acked Write 1 Done Done Sent false
7254 Acked Read 5 Done Done Sent false
7255 Acked Read 3 Done Done Sent false
7256 Acked Read 9 Done Done Sent false
7257 Acked Flush 0 Done Done Sent false
7258 Acked Read 1 Done Done Sent false
7259 Acked Read 9 Done Done Sent false
7260 Acked Write 8 Done Done Sent false
7261 Acked Write 3 Done Done Sent false
7262 Acked Write 9 Done Done Sent false
7263 Acked Write 2 Done Done Sent false
7264 Acked Read 1 Done Done Sent false
7265 Acked Flush 0 Done Done Sent false
7266 Acked Read 10 Done Done Sent false
7267 Acked Read 8 Done Done Sent false
7268 Acked Read 10 Done Done Sent false
7269 Acked Read 6 Done Done Sent false
7270 Acked Read 8 Done Done Sent false
7271 Acked Read 2 Done Done Sent false
7272 Acked Write 7 Done Done Sent false
7273 Acked Read 10 Done Done Sent false
7274 Acked Read 10 Done Done Sent false
7275 Acked Read 3 Done Done Sent false
7276 Acked Flush 0 Done Done Sent false
7277 Acked Read 1 Done Done Sent false
7278 Acked Read 10 Done Done Sent false
7279 Acked Read 4 Done Done Sent false
7280 Acked Flush 0 Done Done Sent false
7281 Acked Read 8 Done Done Sent false
7282 Acked Flush 0 Done Done Sent false
7283 Acked Read 5 Done Done Sent false
7284 Acked Write 10 Done Done Sent false
7285 Acked Write 1 Done Done Sent false
7286 Acked Read 2 Done Done Sent false
7287 Acked Write 6 Done Done Sent false
7288 Acked Read 2 Done Done Sent false
7289 Acked Read 9 Done Done Sent false
7290 Acked Read 9 Done Done Sent false
7291 Acked Read 6 Done Done Sent false
7292 Acked Write 9 Done Done Sent false
7293 Acked Write 7 Done Done Sent false
7294 Acked Read 4 Done Done Sent false
7295 Acked Read 6 Done Done Sent false
7296 Acked Flush 0 Done Done Sent false
7297 Acked Flush 0 Done Done Sent false
7298 Acked Write 3 Done Done Sent false
7299 Acked Flush 0 Done Done Sent false
7300 Acked Write 2 Done Done Sent false
7301 Acked Read 6 Done Done Sent false
7302 Acked Write 7 Done Done Sent false
7303 Acked Write 5 Done Done Sent false
7304 Acked Write 3 Done Done Sent false
7305 Acked Read 4 Done Done Sent false
7306 Acked Write 6 Done Done Sent false
7307 Acked Write 1 Done Done Sent false
7308 Acked Write 3 Done Done Sent false
7309 Acked Read 5 Done Done Sent false
7310 Acked Flush 0 Done Done Sent false
7311 Acked Write 8 Done Done Sent false
7312 Acked Read 6 Done Done Sent false
7313 Acked Flush 0 Done Done Sent false
7314 Acked Write 2 Done Done Sent false
7315 Acked Read 6 Done Done Sent false
7316 Acked Read 1 Done Done Sent false
7317 Acked Read 8 Done Done Sent false
7318 Acked Write 9 Done Done Sent false
7319 Acked Write 3 Done Done Sent false
7320 Acked Write 2 Done Done Sent false
7321 Acked Write 3 Done Done Sent false
7322 Acked Write 4 Done Done Sent false
7323 Acked Read 4 Done Done Sent false
7324 Acked Read 2 Done Done Sent false
7325 Acked Write 7 Done Done Sent false
7326 Acked Read 5 Done Done Sent false
7327 Acked Write 6 Done Done Sent false
7328 Acked Write 9 Done Done Sent false
7329 Acked Write 3 Done Done Sent false
7330 Acked Flush 0 Done Done Sent false
7331 Acked Read 1 Done Done Sent false
7332 Acked Read 8 Done Done Sent false
7333 Acked Flush 0 Done Done Sent false
7334 Acked Read 2 Done Done Sent false
7335 Acked Read 3 Done Done Sent false
7336 Acked Read 10 Done Done Sent false
7337 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7337 7337 7036
Downstairs last five completed: 7036 7035 7034 7033 7032
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.874544171Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35952)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.874860604Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.875897893Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.876470646Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.876505992Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 7036","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.8765252Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 7036","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.876730636Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 301 jobs since flush 7036","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.876754423Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 301 jobs since flush: 7036","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.876989725Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7337 7336 7335 7334 7333
Upstairs last five completed: 7337 7336 7335 7334 7333
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
[8820] Got stop action so:false kr:false
Jan 27 19:41:22.116 INFO request completed, latency_us: 360, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3bd20715-5187-4381-af2c-d8bc343479ff, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:22.116788631Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:41:22.116 INFO request completed, latency_us: 133, response_code{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:22.116863333Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
: 200, uri: /state/cid/1, method: GET, req_id: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:22.116896319Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
274f2289-c467-4fca-933a-623153329301, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:22.116981259Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:41:26.119 INFO request completed, latency_us: 328, response_code: 200, uri: /state/cid/1, method: GET, req_id: bf907f22-54e0-4b40-b839-e29ae35b5399, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 390 len 4096
003/300 Read block 1495 len 32768
004/300 Write block 778 len 8192 data: 5 4
005/300 Flush
006/300 Read block 2278 len 4096
007/300 Write block 4925 len 16384 data: 8 7 8 8
008/300 Read block 1034 len 8192
009/300 Flush
010/300 Read block 1728 len 24576
011/300 Write block 1358 len 28672 data: 9 9 7 7 6 6 7
012/300 Write block 608 len 28672 data: 5 5 4 3 3 4 3
013/300 Read block 4628 len 8192
014/300 Read block 4523 len 4096
015/300 Write block 48 len 36864 data: 7 7 7 7 6 4 6 4 4
016/300 Write block 1612 len 16384 data: 4 4 4 4
017/300 Write block 1951 len 4096 data: 4
018/300 Write block 327 len 24576 data: 5 5 5 4 4 4
019/300 Write block 4672 len 24576 data: 4 4 3 3 3 3
020/300 Write block 906 len 12288 data: 4 2 3
021/300 Read block 2488 len 36864
022/300 Read block 1005 len 12288
023/300 Write block 4680 len 12288 data: 6 7 5
024/300 Write block 3000 len 32768 data: 5 5 6 6 6 5 5 5
025/300 Write block 122 len 40960 data: 7 7 7 4 4 4 4 5 6 8
026/300 Read block 775 len 16384
027/300 Read block 1706 len 20480
028/300 Read block 283 len 12288
029/300 Read block 1785 len 16384
030/300 Read block 1989 len 8192
031/300 Write block 1183 len 20480 data: 2 3 4 3 3
032/300 Write block 1847 len 28672 data: 4 5 5 5 6 6 6
033/300 Write block 1254 len 40960 data: 4 3 4 3 4 3 3 3 3 4
034/300 Flush
035/300 Read block 4882 len 40960
036/300 Write block 2514 len 8192 data: 5 5
037/300 Write block 1403 len 24576 data: 6 5 8 8 7 7
038/300 Flush
039/300 Read block 4922 len 32768
040/300 Read block 1074 len 12288
041/300 Read block 1767 len 16384
042/300 Read block 4211 len 40960
043/300 Read block 3520 len 4096
044/300 Write block 4713 len 28672 data: 6 6 5 5 5 7 4
045/300 Flush
046/300 Write block 1941 len 36864 data: 5 5 5 4 4 4 4 4 3
047/300 Write block 4867 len 20480 data: 8 7 8 7 5
048/300 Read block 3187 len 28672
049/300 Write block 955 len 32768 data: 4 7 7 6 6 7 7 6
050/300 Read block 2121 len 16384
051/300 Write block 1143 len 32768 data: 3 2 4 4 5 6 6 4
052/300 Write block 1093 len 32768 data: 3 3 4 4 4 6 6 6
053/300 Read block 2124 len 36864
054/300 Read block 3769 len 40960
055/300 Write block 396 len 8192 data: 5 7
056/300 Write block 1516 len 12288 data: 4 4 2
057/300 Write block 627 len 32768 data: 4 4 4 5 4 4 4 4
058/300 Read block 3750 len 4096
059/300 Read block 3337 len 32768
060/300 Read block 1770 len 28672
061/300 Read block 311 len 12288
062/300 Read block 4440 len 12288
063/300 Read block 3719 len 16384
064/300 Write block 2144 len 20480 data: 7 6 6 4 3
065/300 Read block 2952 len 12288
066/300 Flush
067/300 Read block 2814 len 28672
068/300 Read block 1158 len 40960
069/300 Read block 3525 len 4096
070/300 Write block 856 len 24576 data: 3 3 5 6 7 8
071/300 Read block 1134 len 36864
072/300 Write block 918 len 40960 data: 4 6 7 7 6 4 5 5 4 3
073/300 Read block 2575 len 16384
074/300 Write block 2282 len 36864 data: 6 6 5 6 7 8 11 10 10
075/300 Read block 811 len 16384
076/300 Read block 3006 len 40960
077/300 Write block 3355 len 12288 data: 3 3 3
078/300 Write block 2628 len 28672 data: 5 5 4 4 5 7 5
079/300 Write block 4683 len 20480 data: 5 4 4 5 4
080/300 Read block 1387 len 32768
081/300 Write block 1303 len 12288 data: 7 7 7
082/300 Read block 1178 len 4096
083/300 Write block 2516 len 20480 data: 5 4 4 3 3
084/300 Write block 3924 len 36864 data: 5 4 5 5 5 4 4 5 5
085/300 Read block 3228 len 36864
086/300 Write block 914 len 32768 data: 4 2 3 3 5 7 8 8
087/300 Read block 2712 len 32768
088/300 Read block 38 len 4096
089/300 Read block 194 len 16384
090/300 Write block 4369 len 12288 data: 4 3 3
091/300 Read block 1037 len 32768
092/300 Read block 1 len 20480
093/300 Flush
094/300 Write block 1088 len 28672 data: 2 3 4 4 2 4 4
095/300 Write block 470 len 28672 data: 5 5 3 3 3 4 4
096/300 Read block 1042 len 20480
097/300 Read block 756 len 20480
098/300 Read block 234 len 24576
099/300 Read block 1882 len 28672
100/300 Write block 539 len 20480 data: 3 2 2 3 3
101/300 Write block 4297 len 36864 data: 4 4 4 3 3 5 4 5 4
102/300 Write block 1966 len 8192 data: 6 6
103/300 Flush
104/300 Flush
105/300 Read block 144 len 40960
106/300 Write block 1496 len 8192 data: 2 2
107/300 Read block 4280 len 32768
108/300 Read block 4959 len 4096
109/300 Write block 3798 len 12288 data: 2 2 2
110/300 Write block 1840 len 8192 data: 3 3
111/300 Read block 1854 len 36864
112/300 Read block 1283 len 20480
113/300 Read block 2798 len 40960
114/300 Read block 0 len 16384
115/300 Read block 2745 len 4096
116/300 Read block 4889 len 32768
117/300 Write block 799 len 20480 data: 5 6 5 6 5
118/300 Write block 1139 len 4096 data: 4
119/300 Read block 4386 len 12288
120/300 Write block 2813 len 16384 data: 6 5 4 3
121/300 Write block 1231 len 4096 data: 3
122/300 Read block 4034 len 20480
123/300 Read block 4336 len 40960
124/300 Read block 2714 len 20480
125/300 Write block 1577 len 24576 data: 5 5 4 6 6 6
126/300 Write block 2826 len 32768 data: 5 4 6 6 7 7 8 8
127/300 Read block 114 len 12288
128/300 Read block 628 len 16384
129/300 Flush
130/300 Read block 2509 len 20480
131/300 Read block 193 len 4096
132/300 Read block 2392 len 8192
133/300 Read block 3147 len 36864
134/300 Read block 2409 len 36864
135/300 Read block 4971 len 8192
136/300 Read block 4422 len 8192
137/300 Write block 1621 len 8192 data: 5 5
138/300 Read block 1305 len 12288
139/300 Read block 3872 len 24576
140/300 Write block 1629 len 8192 data: 7 6
141/300 Write block 4314 len 32768 data: 2 2 2 2 2 2 3 4
142/300 Write block 2520 len 24576 data: 4 3 3 3 4 3
143/300 Read block 554 len 28672
144/300 Write block 614 len 24576 data: 4 3 3 2 2 3
145/300 Flush
146/300 Write block 1120 len 36864 data: 6 7 9 10 9 7 7 8 8
147/300 Flush
148/300 Write block 3692 len 4096 data: 5
149/300 Read block 257 len 4096
150/300 Write block 1475 len 40960 data: 4 4 3 4 6 6 6 6 4 3
151/300 Write block 181 len 16384 data: 5 7 8 9
152/300 Read block 231 len 40960
153/300 Read block 626 len 28672
154/300 Write block 1135 len 12288 data: 4 3 4
155/300 Write block 165 len 40960 data: 4 6 6 9 9 9 9 6 4 4
156/300 Write block 1978 len 8192 data: 5 3
157/300 Write block 4526 len 28672 data: 7 7 7 6 6 7 7
158/300 Read block 2438 len 32768
159/300 Read block 2941 len 8192
160/300 Read block 597 len 4096
161/300 Write block 3672 len 24576 data: 4 4 4 3 3 3
162/300 Write block 502 len 28672 data: 8 6 5 7 7 6 5
163/300 Write block 2550 len 40960 data: 4 4 4 4 4 5 5 6 6 5
164/300 Flush
165/300 Read block 1252 len 16384
166/300 Read block 2590 len 40960
167/300 Flush
168/300 Read block 3915 len 32768
169/300 Read block 345 len 4096
170/300 Read block 915 len 12288
171/300 Read block 400 len 36864
172/300 Write block 2660 len 28672 data: 5 5 4 5 5 5 4
173/300 Read block 4657 len 36864
174/300 Write block 111 len 8192 data: 2 2
175/300 Read block 3492 len 4096
176/300 Write block 2365 len 40960 data: 3 3 3 4 5 7 8 7 7 6
177/300 Read block 4309 len 4096
178/300 Write block 1297 len 28672 data: 6 7 6 5 7 6 8
179/300 Read block 1711 len 32768
180/300 Write block 4473 len 36864 data: 4 5 8 8 8 7 4 7 7
181/300 Read block 4273 len 40960
182/300 Read block 1601 len 32768
183/300 Write block 2619 len 24576 data: 6 6 7 7 7 8
184/300 Read block 2909 len 16384
185/300 Read block 3384 len 24576
186/300 Flush
187/300 Read block 2193 len 32768
188/300 Write block 1531 len 24576 data: 6 5 4 4 4 4
189/300 Write block 2010 len 8192 data: 5 8
190/300 Flush
191/300 Read block 78 len 36864
192/300 Write block 2959 len 28672 data: 3 4 4 6 6 6 6
193/300 Read block 4164 len 28672
194/300 Write block 3084 len 28672 data: 3 2 2 2 3 2 3
195/300 Read block 4499 len 8192
196/300 Flush
197/300 Read block 2026 len 12288
198/300 Read block 2692 len 32768
199/300 Flush
200/300 Read block 102 len 24576
201/300 Read block 265 len 4096
202/300 Read block 2641 len 20480
203/300 Read block 3274 len 12288
204/300 Read block 2767 len 20480
205/300 Read block 3805 len 24576
206/300 Read block 2148 len 8192
207/300 Read block 4519 len 8192
208/300 Read block 1451 len 24576
209/300 Read block 4267 len 24576
210/300 Flush
211/300 Write block 2758 len 24576 data: 3 3 3 3 3 4
212/300 Write block 4347 len 16384 data: 5 7 8 7
213/300 Read block 4316 len 32768
214/300 Read block 1733 len 12288
215/300 Read block 778 len 28672
216/300 Write block 1498 len 40960 data: 2 2 2 2 2 2 3 3 3 4
217/300 Read block 4367 len 28672
218/300 Flush
219/300 Flush
220/300 Read block 343 len 8192
221/300 Write block 3720 len 20480 data: 4 5 4 4 5
222/300 Write block 3804 len 36864 data: 2 3 3 3 3 5 6 5 4
223/300 Read block 3749 len 4096
224/300 Read block 1468 len 12288
225/300 Write block 3175 len 32768 data: 8 9 9 9 9 7 7 6
226/300 Read block 4508 len 4096
227/300 Read block 418 len 16384
228/300 Read block 4948 len 28672
229/300 Read block 282 len 36864
230/300 Read block 2640 len 20480
231/300 Write block 3762 len 28672 data: 7 4 4 5 5 6 6
232/300 Write block 4494 len 28672 data: 6 6 4 4 6 6 6
233/300 Write block 2401 len 36864 data: 5 5 4 4 5 5 7 8 7
234/300 Read block 1464 len 12288
235/300 Read block 2633 len 16384
236/300 Read block 2250 len 4096
237/300 Write block 1390 len 40960 data: 6 7 5 4 6 5 5 6 7 8
238/300 Write block 2180 len 8192 data: 5 5
239/300 Flush
240/300 Read block 3488 len 32768
241/300 Write block 2042 len 28672 data: 8 8 6 6 6 5 5
242/300 Write block 3888 len 32768 data: 5 5 3 2 2 4 5 5
243/300 Write block 4658 len 8192 data: 7 6
244/300 Write block 500 len 12288 data: 8 9 9
245/300 Read block 493 len 8192
246/300 Read block 3455 len 16384
247/300 Read block 2337 len 28672
248/300 Read block 3587 len 16384
249/300 Write block 1725 len 36864 data: 4 6 6 5 6 5 4 3 3
250/300 Read block 2754 len 20480
251/300 Read block 2984 len 40960
252/300 Write block 1357 len 12288 data: 7 10 10
253/300 Read block 1596 len 40960
254/300 Write block 2709 len 40960 data: 4 5 5 5 6 5 6 5 4 7
255/300 Flush
256/300 Read block 4497 len 32768
257/300 Flush
258/300 Read block 2398 len 20480
259/300 Write block 4991 len 28672 data: 5 5 5 4 5 5 4
260/300 Read block 4847 len 36864
261/300 Read block 3964 len 16384
262/300 Write block 4397 len 32768 data: 2 2 2 2 3 2 3 3
263/300 Read block 66 len 36864
264/300 Read block 3760 len 36864
265/300 Flush
266/300 Read block 3312 len 24576
267/300 Write block 2475 len 28672 data: 5 5 4 2 3 3 2
268/300 Read block 4365 len 20480
269/300 Read block 592 len 40960
270/300 Read block 1183 len 28672
271/300 Flush
272/300 Read block 6 len 4096
273/300 Read block 3475 len 16384
274/300 Flush
275/300 Read block 2828 len 20480
276/300 Read block 3206 len 32768
277/300 Write block 1066 len 32768 data: 4 3 5 4 5 5 7 6
278/300 Write block 4348 len 20480 data: 8 9 8 7 7
279/300 Read block 4058 len 20480
280/300 Write block 4167 len 28672 data: 7 6 5 5 4 4 6
281/300 Read block 292 len 20480
282/300 Write block 3221 len 28672 data: 6 7 6 6 5 4 4
283/300 Read block 4948 len 32768
284/300 Read block 3785 len 8192
285/300 Write block 599 len 28672 data: 7 6 6 8 8 7 6
286/300 Write block 1542 len 12288 data: 5 7 7
287/300 Write block 1539 len 12288 data: 5 5 5
288/300 Read block 2178 len 32768
289/300 Read block 3057 len 20480
290/300 Read block 160 len 28672
291/300 Flush
292/300 Flush
293/300 Write block 1195 len 16384 data: 4 6 7 7
294/300 Write block 2191 len 8192 data: 4 4
295/300 Read block 4767 len 40960
296/300 Write block 3386 len 28672 data: 3 4 5 6 5 5 4
297/300 Write block 900 len 36864 data: 3 4 4 5 6 4 5 3 4
298/300 Flush
299/300 Read block 3599 len 20480
300/300 Write block 295 len 20480 data: 5 5 5 5 5
301/300 Read block 1023 len 40960
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:41:26.360 INFO request completed, latency_us: 169, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2944104e-a640-4f71-9c7f-02530711e025, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7338 Acked Flush 0 Done Sent Done false
7339 Acked Read 1 Done Sent Done false
7340 Acked Read 8 Done Sent Done false
7341 Acked Write 2 Done Sent Done false
7342 Acked Flush 0 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Write 4 Done Sent Done false
7345 Acked Read 2 Done Sent Done false
7346 Acked Flush 0 Done Sent Done false
7347 Acked Read 6 Done Sent Done false
7348 Acked Write 7 Done Sent Done false
7349 Acked Write 7 Done Sent Done false
7350 Acked Read 2 Done Sent Done false
7351 Acked Read 1 Done Sent Done false
7352 Acked Write 9 Done Sent Done false
7353 Acked Write 4 Done Sent Done false
7354 Acked Write 1 Done Sent Done false
7355 Acked Write 6 Done Sent Done false
7356 Acked Write 6 Done Sent Done false
7357 Acked Write 3 Done Sent Done false
7358 Acked Read 9 Done Sent Done false
7359 Acked Read 3 Done Sent Done false
7360 Acked Write 3 Done Sent Done false
7361 Acked Write 8 Done Sent Done false
7362 Acked Write 10 Done Sent Done false
7363 Acked Read 4 Done Sent Done false
7364 Acked Read 5 Done Sent Done false
7365 Acked Read 3 Done Sent Done false
7366 Acked Read 4 Done Sent Done false
7367 Acked Read 2 Done Sent Done false
7368 Acked Write 5 Done Sent Done false
7369 Acked Write 7 Done Sent Done false
7370 Acked Write 10 Done Sent Done false
7371 Acked Flush 0 Done Sent Done false
7372 Acked Read 10 Done Sent Done false
7373 Acked Write 2 Done Sent Done false
7374 Acked Write 6 Done Sent Done false
7375 Acked Flush 0 Done Sent Done false
7376 Acked Read 8 Done Sent Done false
7377 Acked Read 3 Done Sent Done false
7378 Acked Read 4 Done Sent Done false
7379 Acked Read 10 Done Sent Done false
7380 Acked Read 1 Done Sent Done false
7381 Acked Write 7 Done Sent Done false
7382 Acked Flush 0 Done Sent Done false
7383 Acked Write 9 Done Sent Done false
7384 Acked Write 5 Done Sent Done false
7385 Acked Read 7 Done Sent Done false
7386 Acked Write 8 Done Sent Done false
7387 Acked Read 4 Done Sent Done false
7388 Acked Write 8 Done Sent Done false
7389 Acked Write 8 Done Sent Done false
7390 Acked Read 9 Done Sent Done false
7391 Acked Read 10 Done Sent Done false
7392 Acked Write 2 Done Sent Done false
7393 Acked Write 3 Done Sent Done false
7394 Acked Write 8 Done Sent Done false
7395 Acked Read 1 Done Sent Done false
7396 Acked Read 8 Done Sent Done false
7397 Acked Read 7 Done Sent Done false
7398 Acked Read 3 Done Sent Done false
7399 Acked Read 3 Done Sent Done false
7400 Acked Read 4 Done Sent Done false
7401 Acked Write 5 Done Sent Done false
7402 Acked Read 3 Done Sent Done false
7403 Acked Flush 0 Done Sent Done false
7404 Acked Read 7 Done Sent Done false
7405 Acked Read 10 Done Sent Done false
7406 Acked Read 1 Done Sent Done false
7407 Acked Write 6 Done Sent Done false
7408 Acked Read 9 Done Sent Done false
7409 Acked Write 10 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 9 Done Sent Done false
7412 Acked Read 4 Done Sent Done false
7413 Acked Read 10 Done Sent Done false
7414 Acked Write 3 Done Sent Done false
7415 Acked Write 7 Done Sent Done false
7416 Acked Write 5 Done Sent Done false
7417 Acked Read 8 Done Sent Done false
7418 Acked Write 3 Done Sent Done false
7419 Acked Read 1 Done Sent Done false
7420 Acked Write 5 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 9 Done Sent Done false
7423 Acked Write 8 Done Sent Done false
7424 Acked Read 8 Done Sent Done false
7425 Acked Read 1 Done Sent Done false
7426 Acked Read 4 Done Sent Done false
7427 Acked Write 3 Done Sent Done false
7428 Acked Read 8 Done Sent Done false
7429 Acked Read 5 Done Sent Done false
7430 Acked Flush 0 Done Sent Done false
7431 Acked Write 7 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 5 Done Sent Done false
7434 Acked Read 5 Done Sent Done false
7435 Acked Read 6 Done Sent Done false
7436 Acked Read 7 Done Sent Done false
7437 Acked Write 5 Done Sent Done false
7438 Acked Write 9 Done Sent Done false
7439 Acked Write 2 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Flush 0 Done Sent Done false
7442 Acked Read 10 Done Sent Done false
7443 Acked Write 2 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 1 Done Sent Done false
7446 Acked Write 3 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 9 Done Sent Done false
7449 Acked Read 5 Done Sent Done false
7450 Acked Read 10 Done Sent Done false
7451 Acked Read 4 Done Sent Done false
7452 Acked Read 1 Done Sent Done false
7453 Acked Read 8 Done Sent Done false
7454 Acked Write 5 Done Sent Done false
7455 Acked Write 1 Done Sent Done false
7456 Acked Read 3 Done Sent Done false
7457 Acked Write 4 Done Sent Done false
7458 Acked Write 1 Done Sent Done false
7459 Acked Read 5 Done Sent Done false
7460 Acked Read 10 Done Sent Done false
7461 Acked Read 5 Done Sent Done false
7462 Acked Write 6 Done Sent Done false
7463 Acked Write 8 Done Sent Done false
7464 Acked Read 3 Done Sent Done false
7465 Acked Read 4 Done Sent Done false
7466 Acked Flush 0 Done Sent Done false
7467 Acked Read 5 Done Sent Done false
7468 Acked Read 1 Done Sent Done false
7469 Acked Read 2 Done Sent Done false
7470 Acked Read 9 Done Sent Done false
7471 Acked Read 9 Done Sent Done false
7472 Acked Read 2 Done Sent Done false
7473 Acked Read 2 Done Sent Done false
7474 Acked Write 2 Done Sent Done false
7475 Acked Read 3 Done Sent Done false
7476 Acked Read 6 Done Sent Done false
7477 Acked Write 2 Done Sent Done false
7478 Acked Write 8 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 7 Done Sent Done false
7481 Acked Write 6 Done Sent Done false
7482 Acked Flush 0 Done Sent Done false
7483 Acked Write 9 Done Sent Done false
7484 Acked Flush 0 Done Sent Done false
7485 Acked Write 1 Done Sent Done false
7486 Acked Read 1 Done Sent Done false
7487 Acked Write 10 Done Sent Done false
7488 Acked Write 4 Done Sent Done false
7489 Acked Read 10 Done Sent Done false
7490 Acked Read 7 Done Sent Done false
7491 Acked Write 3 Done Sent Done false
7492 Acked Write 10 Done Sent Done false
7493 Acked Write 2 Done Sent Done false
7494 Acked Write 7 Done Sent Done false
7495 Acked Read 8 Done Sent Done false
7496 Acked Read 2 Done Sent Done false
7497 Acked Read 1 Done Sent Done false
7498 Acked Write 6 Done Sent Done false
7499 Acked Write 7 Done Sent Done false
7500 Acked Write 10 Done Sent Done false
7501 Acked Flush 0 Done Sent Done false
7502 Acked Read 4 Done Sent Done false
7503 Acked Read 10 Done Sent Done false
7504 Acked Flush 0 Done Sent Done false
7505 Acked Read 8 Done Sent Done false
7506 Acked Read 1 Done Sent Done false
7507 Acked Read 3 Done Sent Done false
7508 Acked Read 9 Done Sent Done false
7509 Acked Write 7 Done Sent Done false
7510 Acked Read 9 Done Sent Done false
7511 Acked Write 2 Done Sent Done false
7512 Acked Read 1 Done Sent Done false
7513 Acked Write 10 Done Sent Done false
7514 Acked Read 1 Done Sent Done false
7515 Acked Write 7 Done Sent Done false
7516 Acked Read 8 Done Sent Done false
7517 Acked Write 9 Done Sent Done false
7518 Acked Read 10 Done Sent Done false
7519 Acked Read 8 Done Sent Done false
7520 Acked Write 6 Done Sent Done false
7521 Acked Read 4 Done Sent Done false
7522 Acked Read 6 Done Sent Done false
7523 Acked Flush 0 Done Sent Done false
7524 Acked Read 8 Done Sent Done false
7525 Acked Write 6 Done Sent Done false
7526 Acked Write 2 Done Sent Done false
7527 Acked Flush 0 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Write 7 Done Sent Done false
7530 Acked Read 7 Done Sent Done false
7531 Acked Write 7 Done Sent Done false
7532 Acked Read 2 Done Sent Done false
7533 Acked Flush 0 Done Sent Done false
7534 Acked Read 3 Done Sent Done false
7535 Acked Read 8 Done Sent Done false
7536 Acked Flush 0 Done Sent Done false
7537 Acked Read 6 Done Sent Done false
7538 Acked Read 1 Done Sent Done false
7539 Acked Read 5 Done Sent Done false
7540 Acked Read 3 Done Sent Done false
7541 Acked Read 5 Done Sent Done false
7542 Acked Read 6 Done Sent Done false
7543 Acked Read 2 Done Sent Done false
7544 Acked Read 2 Done Sent Done false
7545 Acked Read 6 Done Sent Done false
7546 Acked Read 6 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Write 6 Done Sent Done false
7549 Acked Write 4 Done Sent Done false
7550 Acked Read 8 Done Sent Done false
7551 Acked Read 3 Done Sent Done false
7552 Acked Read 7 Done Sent Done false
7553 Acked Write 10 Done Sent Done false
7554 Acked Read 7 Done Sent Done false
7555 Acked Flush 0 Done Sent Done false
7556 Acked Flush 0 Done Sent Done false
7557 Acked Read 2 Done Sent Done false
7558 Acked Write 5 Done Sent Done false
7559 Acked Write 9 Done Sent Done false
7560 Acked Read 1 Done Sent Done false
7561 Acked Read 3 Done Sent Done false
7562 Acked Write 8 Done Sent Done false
7563 Acked Read 1 Done Sent Done false
7564 Acked Read 4 Done Sent Done false
7565 Acked Read 7 Done Sent Done false
7566 Acked Read 9 Done Sent Done false
7567 Acked Read 5 Done Sent Done false
7568 Acked Write 7 Done Sent Done false
7569 Acked Write 7 Done Sent Done false
7570 Acked Write 9 Done Sent Done false
7571 Acked Read 3 Done Sent Done false
7572 Acked Read 4 Done Sent Done false
7573 Acked Read 1 Done Sent Done false
7574 Acked Write 10 Done Sent Done false
7575 Acked Write 2 Done Sent Done false
7576 Acked Flush 0 Done Sent Done false
7577 Acked Read 8 Done Sent Done false
7578 Acked Write 7 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 2 Done Sent Done false
7581 Acked Write 3 Done Sent Done false
7582 Acked Read 2 Done Sent Done false
7583 Acked Read 4 Done Sent Done false
7584 Acked Read 7 Done Sent Done false
7585 Acked Read 4 Done Sent Done false
7586 Acked Write 9 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 3 Done Sent Done false
7590 Acked Read 10 Done Sent Done false
7591 Acked Write 10 Done Sent Done false
7592 Acked Flush 0 Done Sent Done false
7593 Acked Read 8 Done Sent Done false
7594 Acked Flush 0 Done Sent Done false
7595 Acked Read 5 Done Sent Done false
7596 Acked Write 7 Done Sent Done false
7597 Acked Read 9 Done Sent Done false
7598 Acked Read 4 Done Sent Done false
7599 Acked Write 8 Done Sent Done false
7600 Acked Read 9 Done Sent Done false
7601 Acked Read 9 Done Sent Done false
7602 Acked Flush 0 Done Sent Done false
7603 Acked Read 6 Done Sent Done false
7604 Acked Write 7 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 10 Done Sent Done false
7607 Acked Read 7 Done Sent Done false
7608 Acked Flush 0 Done Sent Done false
7609 Acked Read 1 Done Sent Done false
7610 Acked Read 4 Done Sent Done false
7611 Acked Flush 0 Done Sent Done false
7612 Acked Read 5 Done Sent Done false
7613 Acked Read 8 Done Sent Done false
7614 Acked Write 8 Done Sent Done false
7615 Acked Write 5 Done Sent Done false
7616 Acked Read 5 Done Sent Done false
7617 Acked Write 7 Done Sent Done false
7618 Acked Read 5 Done Sent Done false
7619 Acked Write 7 Done Sent Done false
7620 Acked Read 8 Done Sent Done false
7621 Acked Read 2 Done Sent Done false
7622 Acked Write 7 Done Sent Done false
7623 Acked Write 3 Done Sent Done false
7624 Acked Write 3 Done Sent Done false
7625 Acked Read 8 Done Sent Done false
7626 Acked Read 5 Done Sent Done false
7627 Acked Read 7 Done Sent Done false
7628 Acked Flush 0 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Write 4 Done Sent Done false
7631 Acked Write 2 Done Sent Done false
7632 Acked Read 10 Done Sent Done false
7633 Acked Write 7 Done Sent Done false
7634 Acked Write 9 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Read 5 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 10 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7635 7337 7635
Downstairs last five completed: 7337 7336 7335 7334 7333
Upstairs last five completed: 7638 7637 7636 7635 7634
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7338 Acked Flush 0 Done Sent Done false
7339 Acked Read 1 Done Sent Done false
7340 Acked Read 8 Done Sent Done false
7341 Acked Write 2 Done Sent Done false
7342 Acked Flush 0 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Write 4 Done Sent Done false
7345 Acked Read 2 Done Sent Done false
7346 Acked Flush 0 Done Sent Done false
7347 Acked Read 6 Done Sent Done false
7348 Acked Write 7 Done Sent Done false
7349 Acked Write 7 Done Sent Done false
7350 Acked Read 2 Done Sent Done false
7351 Acked Read 1 Done Sent Done false
7352 Acked Write 9 Done Sent Done false
7353 Acked Write 4 Done Sent Done false
7354 Acked Write 1 Done Sent Done false
7355 Acked Write 6 Done Sent Done false
7356 Acked Write 6 Done Sent Done false
7357 Acked Write 3 Done Sent Done false
7358 Acked Read 9 Done Sent Done false
7359 Acked Read 3 Done Sent Done false
7360 Acked Write 3 Done Sent Done false
7361 Acked Write 8 Done Sent Done false
7362 Acked Write 10 Done Sent Done false
7363 Acked Read 4 Done Sent Done false
7364 Acked Read 5 Done Sent Done false
7365 Acked Read 3 Done Sent Done false
7366 Acked Read 4 Done Sent Done false
7367 Acked Read 2 Done Sent Done false
7368 Acked Write 5 Done Sent Done false
7369 Acked Write 7 Done Sent Done false
7370 Acked Write 10 Done Sent Done false
7371 Acked Flush 0 Done Sent Done false
7372 Acked Read 10 Done Sent Done false
7373 Acked Write 2 Done Sent Done false
7374 Acked Write 6 Done Sent Done false
7375 Acked Flush 0 Done Sent Done false
7376 Acked Read 8 Done Sent Done false
7377 Acked Read 3 Done Sent Done false
7378 Acked Read 4 Done Sent Done false
7379 Acked Read 10 Done Sent Done false
7380 Acked Read 1 Done Sent Done false
7381 Acked Write 7 Done Sent Done false
7382 Acked Flush 0 Done Sent Done false
7383 Acked Write 9 Done Sent Done false
7384 Acked Write 5 Done Sent Done false
7385 Acked Read 7 Done Sent Done false
7386 Acked Write 8 Done Sent Done false
7387 Acked Read 4 Done Sent Done false
7388 Acked Write 8 Done Sent Done false
7389 Acked Write 8 Done Sent Done false
7390 Acked Read 9 Done Sent Done false
7391 Acked Read 10 Done Sent Done false
7392 Acked Write 2 Done Sent Done false
7393 Acked Write 3 Done Sent Done false
7394 Acked Write 8 Done Sent Done false
7395 Acked Read 1 Done Sent Done false
7396 Acked Read 8 Done Sent Done false
7397 Acked Read 7 Done Sent Done false
7398 Acked Read 3 Done Sent Done false
7399 Acked Read 3 Done Sent Done false
7400 Acked Read 4 Done Sent Done false
7401 Acked Write 5 Done Sent Done false
7402 Acked Read 3 Done Sent Done false
7403 Acked Flush 0 Done Sent Done false
7404 Acked Read 7 Done Sent Done false
7405 Acked Read 10 Done Sent Done false
7406 Acked Read 1 Done Sent Done false
7407 Acked Write 6 Done Sent Done false
7408 Acked Read 9 Done Sent Done false
7409 Acked Write 10 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 9 Done Sent Done false
7412 Acked Read 4 Done Sent Done false
7413 Acked Read 10 Done Sent Done false
7414 Acked Write 3 Done Sent Done false
7415 Acked Write 7 Done Sent Done false
7416 Acked Write 5 Done Sent Done false
7417 Acked Read 8 Done Sent Done false
7418 Acked Write 3 Done Sent Done false
7419 Acked Read 1 Done Sent Done false
7420 Acked Write 5 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 9 Done Sent Done false
7423 Acked Write 8 Done Sent Done false
7424 Acked Read 8 Done Sent Done false
7425 Acked Read 1 Done Sent Done false
7426 Acked Read 4 Done Sent Done false
7427 Acked Write 3 Done Sent Done false
7428 Acked Read 8 Done Sent Done false
7429 Acked Read 5 Done Sent Done false
7430 Acked Flush 0 Done Sent Done false
7431 Acked Write 7 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 5 Done Sent Done false
7434 Acked Read 5 Done Sent Done false
7435 Acked Read 6 Done Sent Done false
7436 Acked Read 7 Done Sent Done false
7437 Acked Write 5 Done Sent Done false
7438 Acked Write 9 Done Sent Done false
7439 Acked Write 2 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Flush 0 Done Sent Done false
7442 Acked Read 10 Done Sent Done false
7443 Acked Write 2 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 1 Done Sent Done false
7446 Acked Write 3 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 9 Done Sent Done false
7449 Acked Read 5 Done Sent Done false
7450 Acked Read 10 Done Sent Done false
7451 Acked Read 4 Done Sent Done false
7452 Acked Read 1 Done Sent Done false
7453 Acked Read 8 Done Sent Done false
7454 Acked Write 5 Done Sent Done false
7455 Acked Write 1 Done Sent Done false
7456 Acked Read 3 Done Sent Done false
7457 Acked Write 4 Done Sent Done false
7458 Acked Write 1 Done Sent Done false
7459 Acked Read 5 Done Sent Done false
7460 Acked Read 10 Done Sent Done false
7461 Acked Read 5 Done Sent Done false
7462 Acked Write 6 Done Sent Done false
7463 Acked Write 8 Done Sent Done false
7464 Acked Read 3 Done Sent Done false
7465 Acked Read 4 Done Sent Done false
7466 Acked Flush 0 Done Sent Done false
7467 Acked Read 5 Done Sent Done false
7468 Acked Read 1 Done Sent Done false
7469 Acked Read 2 Done Sent Done false
7470 Acked Read 9 Done Sent Done false
7471 Acked Read 9 Done Sent Done false
7472 Acked Read 2 Done Sent Done false
7473 Acked Read 2 Done Sent Done false
7474 Acked Write 2 Done Sent Done false
7475 Acked Read 3 Done Sent Done false
7476 Acked Read 6 Done Sent Done false
7477 Acked Write 2 Done Sent Done false
7478 Acked Write 8 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 7 Done Sent Done false
7481 Acked Write 6 Done Sent Done false
7482 Acked Flush 0 Done Sent Done false
7483 Acked Write 9 Done Sent Done false
7484 Acked Flush 0 Done Sent Done false
7485 Acked Write 1 Done Sent Done false
7486 Acked Read 1 Done Sent Done false
7487 Acked Write 10 Done Sent Done false
7488 Acked Write 4 Done Sent Done false
7489 Acked Read 10 Done Sent Done false
7490 Acked Read 7 Done Sent Done false
7491 Acked Write 3 Done Sent Done false
7492 Acked Write 10 Done Sent Done false
7493 Acked Write 2 Done Sent Done false
7494 Acked Write 7 Done Sent Done false
7495 Acked Read 8 Done Sent Done false
7496 Acked Read 2 Done Sent Done false
7497 Acked Read 1 Done Sent Done false
7498 Acked Write 6 Done Sent Done false
7499 Acked Write 7 Done Sent Done false
7500 Acked Write 10 Done Sent Done false
7501 Acked Flush 0 Done Sent Done false
7502 Acked Read 4 Done Sent Done false
7503 Acked Read 10 Done Sent Done false
7504 Acked Flush 0 Done Sent Done false
7505 Acked Read 8 Done Sent Done false
7506 Acked Read 1 Done Sent Done false
7507 Acked Read 3 Done Sent Done false
7508 Acked Read 9 Done Sent Done false
7509 Acked Write 7 Done Sent Done false
7510 Acked Read 9 Done Sent Done false
7511 Acked Write 2 Done Sent Done false
7512 Acked Read 1 Done Sent Done false
7513 Acked Write 10 Done Sent Done false
7514 Acked Read 1 Done Sent Done false
7515 Acked Write 7 Done Sent Done false
7516 Acked Read 8 Done Sent Done false
7517 Acked Write 9 Done Sent Done false
7518 Acked Read 10 Done Sent Done false
7519 Acked Read 8 Done Sent Done false
7520 Acked Write 6 Done Sent Done false
7521 Acked Read 4 Done Sent Done false
7522 Acked Read 6 Done Sent Done false
7523 Acked Flush 0 Done Sent Done false
7524 Acked Read 8 Done Sent Done false
7525 Acked Write 6 Done Sent Done false
7526 Acked Write 2 Done Sent Done false
7527 Acked Flush 0 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Write 7 Done Sent Done false
7530 Acked Read 7 Done Sent Done false
7531 Acked Write 7 Done Sent Done false
7532 Acked Read 2 Done Sent Done false
7533 Acked Flush 0 Done Sent Done false
7534 Acked Read 3 Done Sent Done false
7535 Acked Read 8 Done Sent Done false
7536 Acked Flush 0 Done Sent Done false
7537 Acked Read 6 Done Sent Done false
7538 Acked Read 1 Done Sent Done false
7539 Acked Read 5 Done Sent Done false
7540 Acked Read 3 Done Sent Done false
7541 Acked Read 5 Done Sent Done false
7542 Acked Read 6 Done Sent Done false
7543 Acked Read 2 Done Sent Done false
7544 Acked Read 2 Done Sent Done false
7545 Acked Read 6 Done Sent Done false
7546 Acked Read 6 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Write 6 Done Sent Done false
7549 Acked Write 4 Done Sent Done false
7550 Acked Read 8 Done Sent Done false
7551 Acked Read 3 Done Sent Done false
7552 Acked Read 7 Done Sent Done false
7553 Acked Write 10 Done Sent Done false
7554 Acked Read 7 Done Sent Done false
7555 Acked Flush 0 Done Sent Done false
7556 Acked Flush 0 Done Sent Done false
7557 Acked Read 2 Done Sent Done false
7558 Acked Write 5 Done Sent Done false
7559 Acked Write 9 Done Sent Done false
7560 Acked Read 1 Done Sent Done false
7561 Acked Read 3 Done Sent Done false
7562 Acked Write 8 Done Sent Done false
7563 Acked Read 1 Done Sent Done false
7564 Acked Read 4 Done Sent Done false
7565 Acked Read 7 Done Sent Done false
7566 Acked Read 9 Done Sent Done false
7567 Acked Read 5 Done Sent Done false
7568 Acked Write 7 Done Sent Done false
7569 Acked Write 7 Done Sent Done false
7570 Acked Write 9 Done Sent Done false
7571 Acked Read 3 Done Sent Done false
7572 Acked Read 4 Done Sent Done false
7573 Acked Read 1 Done Sent Done false
7574 Acked Write 10 Done Sent Done false
7575 Acked Write 2 Done Sent Done false
7576 Acked Flush 0 Done Sent Done false
7577 Acked Read 8 Done Sent Done false
7578 Acked Write 7 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 2 Done Sent Done false
7581 Acked Write 3 Done Sent Done false
7582 Acked Read 2 Done Sent Done false
7583 Acked Read 4 Done Sent Done false
7584 Acked Read 7 Done Sent Done false
7585 Acked Read 4 Done Sent Done false
7586 Acked Write 9 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 3 Done Sent Done false
7590 Acked Read 10 Done Sent Done false
7591 Acked Write 10 Done Sent Done false
7592 Acked Flush 0 Done Sent Done false
7593 Acked Read 8 Done Sent Done false
7594 Acked Flush 0 Done Sent Done false
7595 Acked Read 5 Done Sent Done false
7596 Acked Write 7 Done Sent Done false
7597 Acked Read 9 Done Sent Done false
7598 Acked Read 4 Done Sent Done false
7599 Acked Write 8 Done Sent Done false
7600 Acked Read 9 Done Sent Done false
7601 Acked Read 9 Done Sent Done false
7602 Acked Flush 0 Done Sent Done false
7603 Acked Read 6 Done Sent Done false
7604 Acked Write 7 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 10 Done Sent Done false
7607 Acked Read 7 Done Sent Done false
7608 Acked Flush 0 Done Sent Done false
7609 Acked Read 1 Done Sent Done false
7610 Acked Read 4 Done Sent Done false
7611 Acked Flush 0 Done Sent Done false
7612 Acked Read 5 Done Sent Done false
7613 Acked Read 8 Done Sent Done false
7614 Acked Write 8 Done Sent Done false
7615 Acked Write 5 Done Sent Done false
7616 Acked Read 5 Done Sent Done false
7617 Acked Write 7 Done Sent Done false
7618 Acked Read 5 Done Sent Done false
7619 Acked Write 7 Done Sent Done false
7620 Acked Read 8 Done Sent Done false
7621 Acked Read 2 Done Sent Done false
7622 Acked Write 7 Done Sent Done false
7623 Acked Write 3 Done Sent Done false
7624 Acked Write 3 Done Sent Done false
7625 Acked Read 8 Done Sent Done false
7626 Acked Read 5 Done Sent Done false
7627 Acked Read 7 Done Sent Done false
7628 Acked Flush 0 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Write 4 Done Sent Done false
7631 Acked Write 2 Done Sent Done false
7632 Acked Read 10 Done Sent Done false
7633 Acked Write 7 Done Sent Done false
7634 Acked Write 9 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Read 5 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 10 Done Sent Done false
7639 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7639 7337 7639
Downstairs last five completed: 7337 7336 7335 7334 7333
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.118779884Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.118840907Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.118859045Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.118874733Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.118952514Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7338 Acked Flush 0 Done Sent Done false
7339 Acked Read 1 Done Sent Done false
7340 Acked Read 8 Done Sent Done false
7341 Acked Write 2 Done Sent Done false
7342 Acked Flush 0 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Write 4 Done Sent Done false
7345 Acked Read 2 Done Sent Done false
7346 Acked Flush 0 Done Sent Done false
7347 Acked Read 6 Done Sent Done false
7348 Acked Write 7 Done Sent Done false
7349 Acked Write 7 Done Sent Done false
7350 Acked Read 2 Done Sent Done false
7351 Acked Read 1 Done Sent Done false
7352 Acked Write 9 Done Sent Done false
7353 Acked Write 4 Done Sent Done false
7354 Acked Write 1 Done Sent Done false
7355 Acked Write 6 Done Sent Done false
7356 Acked Write 6 Done Sent Done false
7357 Acked Write 3 Done Sent Done false
7358 Acked Read 9 Done Sent Done false
7359 Acked Read 3 Done Sent Done false
7360 Acked Write 3 Done Sent Done false
7361 Acked Write 8 Done Sent Done false
7362 Acked Write 10 Done Sent Done false
7363 Acked Read 4 Done Sent Done false
7364 Acked Read 5 Done Sent Done false
7365 Acked Read 3 Done Sent Done false
7366 Acked Read 4 Done Sent Done false
7367 Acked Read 2 Done Sent Done false
7368 Acked Write 5 Done Sent Done false
7369 Acked Write 7 Done Sent Done false
7370 Acked Write 10 Done Sent Done false
7371 Acked Flush 0 Done Sent Done false
7372 Acked Read 10 Done Sent Done false
7373 Acked Write 2 Done Sent Done false
7374 Acked Write 6 Done Sent Done false
7375 Acked Flush 0 Done Sent Done false
7376 Acked Read 8 Done Sent Done false
7377 Acked Read 3 Done Sent Done false
7378 Acked Read 4 Done Sent Done false
7379 Acked Read 10 Done Sent Done false
7380 Acked Read 1 Done Sent Done false
7381 Acked Write 7 Done Sent Done false
7382 Acked Flush 0 Done Sent Done false
7383 Acked Write 9 Done Sent Done false
7384 Acked Write 5 Done Sent Done false
7385 Acked Read 7 Done Sent Done false
7386 Acked Write 8 Done Sent Done false
7387 Acked Read 4 Done Sent Done false
7388 Acked Write 8 Done Sent Done false
7389 Acked Write 8 Done Sent Done false
7390 Acked Read 9 Done Sent Done false
7391 Acked Read 10 Done Sent Done false
7392 Acked Write 2 Done Sent Done false
7393 Acked Write 3 Done Sent Done false
7394 Acked Write 8 Done Sent Done false
7395 Acked Read 1 Done Sent Done false
7396 Acked Read 8 Done Sent Done false
7397 Acked Read 7 Done Sent Done false
7398 Acked Read 3 Done Sent Done false
7399 Acked Read 3 Done Sent Done false
7400 Acked Read 4 Done Sent Done false
7401 Acked Write 5 Done Sent Done false
7402 Acked Read 3 Done Sent Done false
7403 Acked Flush 0 Done Sent Done false
7404 Acked Read 7 Done Sent Done false
7405 Acked Read 10 Done Sent Done false
7406 Acked Read 1 Done Sent Done false
7407 Acked Write 6 Done Sent Done false
7408 Acked Read 9 Done Sent Done false
7409 Acked Write 10 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 9 Done Sent Done false
7412 Acked Read 4 Done Sent Done false
7413 Acked Read 10 Done Sent Done false
7414 Acked Write 3 Done Sent Done false
7415 Acked Write 7 Done Sent Done false
7416 Acked Write 5 Done Sent Done false
7417 Acked Read 8 Done Sent Done false
7418 Acked Write 3 Done Sent Done false
7419 Acked Read 1 Done Sent Done false
7420 Acked Write 5 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 9 Done Sent Done false
7423 Acked Write 8 Done Sent Done false
7424 Acked Read 8 Done Sent Done false
7425 Acked Read 1 Done Sent Done false
7426 Acked Read 4 Done Sent Done false
7427 Acked Write 3 Done Sent Done false
7428 Acked Read 8 Done Sent Done false
7429 Acked Read 5 Done Sent Done false
7430 Acked Flush 0 Done Sent Done false
7431 Acked Write 7 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 5 Done Sent Done false
7434 Acked Read 5 Done Sent Done false
7435 Acked Read 6 Done Sent Done false
7436 Acked Read 7 Done Sent Done false
7437 Acked Write 5 Done Sent Done false
7438 Acked Write 9 Done Sent Done false
7439 Acked Write 2 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Flush 0 Done Sent Done false
7442 Acked Read 10 Done Sent Done false
7443 Acked Write 2 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 1 Done Sent Done false
7446 Acked Write 3 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 9 Done Sent Done false
7449 Acked Read 5 Done Sent Done false
7450 Acked Read 10 Done Sent Done false
7451 Acked Read 4 Done Sent Done false
7452 Acked Read 1 Done Sent Done false
7453 Acked Read 8 Done Sent Done false
7454 Acked Write 5 Done Sent Done false
7455 Acked Write 1 Done Sent Done false
7456 Acked Read 3 Done Sent Done false
7457 Acked Write 4 Done Sent Done false
7458 Acked Write 1 Done Sent Done false
7459 Acked Read 5 Done Sent Done false
7460 Acked Read 10 Done Sent Done false
7461 Acked Read 5 Done Sent Done false
7462 Acked Write 6 Done Sent Done false
7463 Acked Write 8 Done Sent Done false
7464 Acked Read 3 Done Sent Done false
7465 Acked Read 4 Done Sent Done false
7466 Acked Flush 0 Done Sent Done false
7467 Acked Read 5 Done Sent Done false
7468 Acked Read 1 Done Sent Done false
7469 Acked Read 2 Done Sent Done false
7470 Acked Read 9 Done Sent Done false
7471 Acked Read 9 Done Sent Done false
7472 Acked Read 2 Done Sent Done false
7473 Acked Read 2 Done Sent Done false
7474 Acked Write 2 Done Sent Done false
7475 Acked Read 3 Done Sent Done false
7476 Acked Read 6 Done Sent Done false
7477 Acked Write 2 Done Sent Done false
7478 Acked Write 8 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 7 Done Sent Done false
7481 Acked Write 6 Done Sent Done false
7482 Acked Flush 0 Done Sent Done false
7483 Acked Write 9 Done Sent Done false
7484 Acked Flush 0 Done Sent Done false
7485 Acked Write 1 Done Sent Done false
7486 Acked Read 1 Done Sent Done false
7487 Acked Write 10 Done Sent Done false
7488 Acked Write 4 Done Sent Done false
7489 Acked Read 10 Done Sent Done false
7490 Acked Read 7 Done Sent Done false
7491 Acked Write 3 Done Sent Done false
7492 Acked Write 10 Done Sent Done false
7493 Acked Write 2 Done Sent Done false
7494 Acked Write 7 Done Sent Done false
7495 Acked Read 8 Done Sent Done false
7496 Acked Read 2 Done Sent Done false
7497 Acked Read 1 Done Sent Done false
7498 Acked Write 6 Done Sent Done false
7499 Acked Write 7 Done Sent Done false
7500 Acked Write 10 Done Sent Done false
7501 Acked Flush 0 Done Sent Done false
7502 Acked Read 4 Done Sent Done false
7503 Acked Read 10 Done Sent Done false
7504 Acked Flush 0 Done Sent Done false
7505 Acked Read 8 Done Sent Done false
7506 Acked Read 1 Done Sent Done false
7507 Acked Read 3 Done Sent Done false
7508 Acked Read 9 Done Sent Done false
7509 Acked Write 7 Done Sent Done false
7510 Acked Read 9 Done Sent Done false
7511 Acked Write 2 Done Sent Done false
7512 Acked Read 1 Done Sent Done false
7513 Acked Write 10 Done Sent Done false
7514 Acked Read 1 Done Sent Done false
7515 Acked Write 7 Done Sent Done false
7516 Acked Read 8 Done Sent Done false
7517 Acked Write 9 Done Sent Done false
7518 Acked Read 10 Done Sent Done false
7519 Acked Read 8 Done Sent Done false
7520 Acked Write 6 Done Sent Done false
7521 Acked Read 4 Done Sent Done false
7522 Acked Read 6 Done Sent Done false
7523 Acked Flush 0 Done Sent Done false
7524 Acked Read 8 Done Sent Done false
7525 Acked Write 6 Done Sent Done false
7526 Acked Write 2 Done Sent Done false
7527 Acked Flush 0 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Write 7 Done Sent Done false
7530 Acked Read 7 Done Sent Done false
7531 Acked Write 7 Done Sent Done false
7532 Acked Read 2 Done Sent Done false
7533 Acked Flush 0 Done Sent Done false
7534 Acked Read 3 Done Sent Done false
7535 Acked Read 8 Done Sent Done false
7536 Acked Flush 0 Done Sent Done false
7537 Acked Read 6 Done Sent Done false
7538 Acked Read 1 Done Sent Done false
7539 Acked Read 5 Done Sent Done false
7540 Acked Read 3 Done Sent Done false
7541 Acked Read 5 Done Sent Done false
7542 Acked Read 6 Done Sent Done false
7543 Acked Read 2 Done Sent Done false
7544 Acked Read 2 Done Sent Done false
7545 Acked Read 6 Done Sent Done false
7546 Acked Read 6 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Write 6 Done Sent Done false
7549 Acked Write 4 Done Sent Done false
7550 Acked Read 8 Done Sent Done false
7551 Acked Read 3 Done Sent Done false
7552 Acked Read 7 Done Sent Done false
7553 Acked Write 10 Done Sent Done false
7554 Acked Read 7 Done Sent Done false
7555 Acked Flush 0 Done Sent Done false
7556 Acked Flush 0 Done Sent Done false
7557 Acked Read 2 Done Sent Done false
7558 Acked Write 5 Done Sent Done false
7559 Acked Write 9 Done Sent Done false
7560 Acked Read 1 Done Sent Done false
7561 Acked Read 3 Done Sent Done false
7562 Acked Write 8 Done Sent Done false
7563 Acked Read 1 Done Sent Done false
7564 Acked Read 4 Done Sent Done false
7565 Acked Read 7 Done Sent Done false
7566 Acked Read 9 Done Sent Done false
7567 Acked Read 5 Done Sent Done false
7568 Acked Write 7 Done Sent Done false
7569 Acked Write 7 Done Sent Done false
7570 Acked Write 9 Done Sent Done false
7571 Acked Read 3 Done Sent Done false
7572 Acked Read 4 Done Sent Done false
7573 Acked Read 1 Done Sent Done false
7574 Acked Write 10 Done Sent Done false
7575 Acked Write 2 Done Sent Done false
7576 Acked Flush 0 Done Sent Done false
7577 Acked Read 8 Done Sent Done false
7578 Acked Write 7 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 2 Done Sent Done false
7581 Acked Write 3 Done Sent Done false
7582 Acked Read 2 Done Sent Done false
7583 Acked Read 4 Done Sent Done false
7584 Acked Read 7 Done Sent Done false
7585 Acked Read 4 Done Sent Done false
7586 Acked Write 9 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 3 Done Sent Done false
7590 Acked Read 10 Done Sent Done false
7591 Acked Write 10 Done Sent Done false
7592 Acked Flush 0 Done Sent Done false
7593 Acked Read 8 Done Sent Done false
7594 Acked Flush 0 Done Sent Done false
7595 Acked Read 5 Done Sent Done false
7596 Acked Write 7 Done Sent Done false
7597 Acked Read 9 Done Sent Done false
7598 Acked Read 4 Done Sent Done false
7599 Acked Write 8 Done Sent Done false
7600 Acked Read 9 Done Sent Done false
7601 Acked Read 9 Done Sent Done false
7602 Acked Flush 0 Done Sent Done false
7603 Acked Read 6 Done Sent Done false
7604 Acked Write 7 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 10 Done Sent Done false
7607 Acked Read 7 Done Sent Done false
7608 Acked Flush 0 Done Sent Done false
7609 Acked Read 1 Done Sent Done false
7610 Acked Read 4 Done Sent Done false
7611 Acked Flush 0 Done Sent Done false
7612 Acked Read 5 Done Sent Done false
7613 Acked Read 8 Done Sent Done false
7614 Acked Write 8 Done Sent Done false
7615 Acked Write 5 Done Sent Done false
7616 Acked Read 5 Done Sent Done false
7617 Acked Write 7 Done Sent Done false
7618 Acked Read 5 Done Sent Done false
7619 Acked Write 7 Done Sent Done false
7620 Acked Read 8 Done Sent Done false
7621 Acked Read 2 Done Sent Done false
7622 Acked Write 7 Done Sent Done false
7623 Acked Write 3 Done Sent Done false
7624 Acked Write 3 Done Sent Done false
7625 Acked Read 8 Done Sent Done false
7626 Acked Read 5 Done Sent Done false
7627 Acked Read 7 Done Sent Done false
7628 Acked Flush 0 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Write 4 Done Sent Done false
7631 Acked Write 2 Done Sent Done false
7632 Acked Read 10 Done Sent Done false
7633 Acked Write 7 Done Sent Done false
7634 Acked Write 9 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Read 5 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 10 Done Sent Done false
7639 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7639 7337 7639
Downstairs last five completed: 7337 7336 7335 7334 7333
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:302 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1222)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7338 Acked Flush 0 Done Sent Done false
7339 Acked Read 1 Done Sent Done false
7340 Acked Read 8 Done Sent Done false
7341 Acked Write 2 Done Sent Done false
7342 Acked Flush 0 Done Sent Done false
7343 Acked Read 1 Done Sent Done false
7344 Acked Write 4 Done Sent Done false
7345 Acked Read 2 Done Sent Done false
7346 Acked Flush 0 Done Sent Done false
7347 Acked Read 6 Done Sent Done false
7348 Acked Write 7 Done Sent Done false
7349 Acked Write 7 Done Sent Done false
7350 Acked Read 2 Done Sent Done false
7351 Acked Read 1 Done Sent Done false
7352 Acked Write 9 Done Sent Done false
7353 Acked Write 4 Done Sent Done false
7354 Acked Write 1 Done Sent Done false
7355 Acked Write 6 Done Sent Done false
7356 Acked Write 6 Done Sent Done false
7357 Acked Write 3 Done Sent Done false
7358 Acked Read 9 Done Sent Done false
7359 Acked Read 3 Done Sent Done false
7360 Acked Write 3 Done Sent Done false
7361 Acked Write 8 Done Sent Done false
7362 Acked Write 10 Done Sent Done false
7363 Acked Read 4 Done Sent Done false
7364 Acked Read 5 Done Sent Done false
7365 Acked Read 3 Done Sent Done false
7366 Acked Read 4 Done Sent Done false
7367 Acked Read 2 Done Sent Done false
7368 Acked Write 5 Done Sent Done false
7369 Acked Write 7 Done Sent Done false
7370 Acked Write 10 Done Sent Done false
7371 Acked Flush 0 Done Sent Done false
7372 Acked Read 10 Done Sent Done false
7373 Acked Write 2 Done Sent Done false
7374 Acked Write 6 Done Sent Done false
7375 Acked Flush 0 Done Sent Done false
7376 Acked Read 8 Done Sent Done false
7377 Acked Read 3 Done Sent Done false
7378 Acked Read 4 Done Sent Done false
7379 Acked Read 10 Done Sent Done false
7380 Acked Read 1 Done Sent Done false
7381 Acked Write 7 Done Sent Done false
7382 Acked Flush 0 Done Sent Done false
7383 Acked Write 9 Done Sent Done false
7384 Acked Write 5 Done Sent Done false
7385 Acked Read 7 Done Sent Done false
7386 Acked Write 8 Done Sent Done false
7387 Acked Read 4 Done Sent Done false
7388 Acked Write 8 Done Sent Done false
7389 Acked Write 8 Done Sent Done false
7390 Acked Read 9 Done Sent Done false
7391 Acked Read 10 Done Sent Done false
7392 Acked Write 2 Done Sent Done false
7393 Acked Write 3 Done Sent Done false
7394 Acked Write 8 Done Sent Done false
7395 Acked Read 1 Done Sent Done false
7396 Acked Read 8 Done Sent Done false
7397 Acked Read 7 Done Sent Done false
7398 Acked Read 3 Done Sent Done false
7399 Acked Read 3 Done Sent Done false
7400 Acked Read 4 Done Sent Done false
7401 Acked Write 5 Done Sent Done false
7402 Acked Read 3 Done Sent Done false
7403 Acked Flush 0 Done Sent Done false
7404 Acked Read 7 Done Sent Done false
7405 Acked Read 10 Done Sent Done false
7406 Acked Read 1 Done Sent Done false
7407 Acked Write 6 Done Sent Done false
7408 Acked Read 9 Done Sent Done false
7409 Acked Write 10 Done Sent Done false
7410 Acked Read 4 Done Sent Done false
7411 Acked Write 9 Done Sent Done false
7412 Acked Read 4 Done Sent Done false
7413 Acked Read 10 Done Sent Done false
7414 Acked Write 3 Done Sent Done false
7415 Acked Write 7 Done Sent Done false
7416 Acked Write 5 Done Sent Done false
7417 Acked Read 8 Done Sent Done false
7418 Acked Write 3 Done Sent Done false
7419 Acked Read 1 Done Sent Done false
7420 Acked Write 5 Done Sent Done false
7421 Acked Write 9 Done Sent Done false
7422 Acked Read 9 Done Sent Done false
7423 Acked Write 8 Done Sent Done false
7424 Acked Read 8 Done Sent Done false
7425 Acked Read 1 Done Sent Done false
7426 Acked Read 4 Done Sent Done false
7427 Acked Write 3 Done Sent Done false
7428 Acked Read 8 Done Sent Done false
7429 Acked Read 5 Done Sent Done false
7430 Acked Flush 0 Done Sent Done false
7431 Acked Write 7 Done Sent Done false
7432 Acked Write 7 Done Sent Done false
7433 Acked Read 5 Done Sent Done false
7434 Acked Read 5 Done Sent Done false
7435 Acked Read 6 Done Sent Done false
7436 Acked Read 7 Done Sent Done false
7437 Acked Write 5 Done Sent Done false
7438 Acked Write 9 Done Sent Done false
7439 Acked Write 2 Done Sent Done false
7440 Acked Flush 0 Done Sent Done false
7441 Acked Flush 0 Done Sent Done false
7442 Acked Read 10 Done Sent Done false
7443 Acked Write 2 Done Sent Done false
7444 Acked Read 8 Done Sent Done false
7445 Acked Read 1 Done Sent Done false
7446 Acked Write 3 Done Sent Done false
7447 Acked Write 2 Done Sent Done false
7448 Acked Read 9 Done Sent Done false
7449 Acked Read 5 Done Sent Done false
7450 Acked Read 10 Done Sent Done false
7451 Acked Read 4 Done Sent Done false
7452 Acked Read 1 Done Sent Done false
7453 Acked Read 8 Done Sent Done false
7454 Acked Write 5 Done Sent Done false
7455 Acked Write 1 Done Sent Done false
7456 Acked Read 3 Done Sent Done false
7457 Acked Write 4 Done Sent Done false
7458 Acked Write 1 Done Sent Done false
7459 Acked Read 5 Done Sent Done false
7460 Acked Read 10 Done Sent Done false
7461 Acked Read 5 Done Sent Done false
7462 Acked Write 6 Done Sent Done false
7463 Acked Write 8 Done Sent Done false
7464 Acked Read 3 Done Sent Done false
7465 Acked Read 4 Done Sent Done false
7466 Acked Flush 0 Done Sent Done false
7467 Acked Read 5 Done Sent Done false
7468 Acked Read 1 Done Sent Done false
7469 Acked Read 2 Done Sent Done false
7470 Acked Read 9 Done Sent Done false
7471 Acked Read 9 Done Sent Done false
7472 Acked Read 2 Done Sent Done false
7473 Acked Read 2 Done Sent Done false
7474 Acked Write 2 Done Sent Done false
7475 Acked Read 3 Done Sent Done false
7476 Acked Read 6 Done Sent Done false
7477 Acked Write 2 Done Sent Done false
7478 Acked Write 8 Done Sent Done false
7479 Acked Write 6 Done Sent Done false
7480 Acked Read 7 Done Sent Done false
7481 Acked Write 6 Done Sent Done false
7482 Acked Flush 0 Done Sent Done false
7483 Acked Write 9 Done Sent Done false
7484 Acked Flush 0 Done Sent Done false
7485 Acked Write 1 Done Sent Done false
7486 Acked Read 1 Done Sent Done false
7487 Acked Write 10 Done Sent Done false
7488 Acked Write 4 Done Sent Done false
7489 Acked Read 10 Done Sent Done false
7490 Acked Read 7 Done Sent Done false
7491 Acked Write 3 Done Sent Done false
7492 Acked Write 10 Done Sent Done false
7493 Acked Write 2 Done Sent Done false
7494 Acked Write 7 Done Sent Done false
7495 Acked Read 8 Done Sent Done false
7496 Acked Read 2 Done Sent Done false
7497 Acked Read 1 Done Sent Done false
7498 Acked Write 6 Done Sent Done false
7499 Acked Write 7 Done Sent Done false
7500 Acked Write 10 Done Sent Done false
7501 Acked Flush 0 Done Sent Done false
7502 Acked Read 4 Done Sent Done false
7503 Acked Read 10 Done Sent Done false
7504 Acked Flush 0 Done Sent Done false
7505 Acked Read 8 Done Sent Done false
7506 Acked Read 1 Done Sent Done false
7507 Acked Read 3 Done Sent Done false
7508 Acked Read 9 Done Sent Done false
7509 Acked Write 7 Done Sent Done false
7510 Acked Read 9 Done Sent Done false
7511 Acked Write 2 Done Sent Done false
7512 Acked Read 1 Done Sent Done false
7513 Acked Write 10 Done Sent Done false
7514 Acked Read 1 Done Sent Done false
7515 Acked Write 7 Done Sent Done false
7516 Acked Read 8 Done Sent Done false
7517 Acked Write 9 Done Sent Done false
7518 Acked Read 10 Done Sent Done false
7519 Acked Read 8 Done Sent Done false
7520 Acked Write 6 Done Sent Done false
7521 Acked Read 4 Done Sent Done false
7522 Acked Read 6 Done Sent Done false
7523 Acked Flush 0 Done Sent Done false
7524 Acked Read 8 Done Sent Done false
7525 Acked Write 6 Done Sent Done false
7526 Acked Write 2 Done Sent Done false
7527 Acked Flush 0 Done Sent Done false
7528 Acked Read 9 Done Sent Done false
7529 Acked Write 7 Done Sent Done false
7530 Acked Read 7 Done Sent Done false
7531 Acked Write 7 Done Sent Done false
7532 Acked Read 2 Done Sent Done false
7533 Acked Flush 0 Done Sent Done false
7534 Acked Read 3 Done Sent Done false
7535 Acked Read 8 Done Sent Done false
7536 Acked Flush 0 Done Sent Done false
7537 Acked Read 6 Done Sent Done false
7538 Acked Read 1 Done Sent Done false
7539 Acked Read 5 Done Sent Done false
7540 Acked Read 3 Done Sent Done false
7541 Acked Read 5 Done Sent Done false
7542 Acked Read 6 Done Sent Done false
7543 Acked Read 2 Done Sent Done false
7544 Acked Read 2 Done Sent Done false
7545 Acked Read 6 Done Sent Done false
7546 Acked Read 6 Done Sent Done false
7547 Acked Flush 0 Done Sent Done false
7548 Acked Write 6 Done Sent Done false
7549 Acked Write 4 Done Sent Done false
7550 Acked Read 8 Done Sent Done false
7551 Acked Read 3 Done Sent Done false
7552 Acked Read 7 Done Sent Done false
7553 Acked Write 10 Done Sent Done false
7554 Acked Read 7 Done Sent Done false
7555 Acked Flush 0 Done Sent Done false
7556 Acked Flush 0 Done Sent Done false
7557 Acked Read 2 Done Sent Done false
7558 Acked Write 5 Done Sent Done false
7559 Acked Write 9 Done Sent Done false
7560 Acked Read 1 Done Sent Done false
7561 Acked Read 3 Done Sent Done false
7562 Acked Write 8 Done Sent Done false
7563 Acked Read 1 Done Sent Done false
7564 Acked Read 4 Done Sent Done false
7565 Acked Read 7 Done Sent Done false
7566 Acked Read 9 Done Sent Done false
7567 Acked Read 5 Done Sent Done false
7568 Acked Write 7 Done Sent Done false
7569 Acked Write 7 Done Sent Done false
7570 Acked Write 9 Done Sent Done false
7571 Acked Read 3 Done Sent Done false
7572 Acked Read 4 Done Sent Done false
7573 Acked Read 1 Done Sent Done false
7574 Acked Write 10 Done Sent Done false
7575 Acked Write 2 Done Sent Done false
7576 Acked Flush 0 Done Sent Done false
7577 Acked Read 8 Done Sent Done false
7578 Acked Write 7 Done Sent Done false
7579 Acked Write 8 Done Sent Done false
7580 Acked Write 2 Done Sent Done false
7581 Acked Write 3 Done Sent Done false
7582 Acked Read 2 Done Sent Done false
7583 Acked Read 4 Done Sent Done false
7584 Acked Read 7 Done Sent Done false
7585 Acked Read 4 Done Sent Done false
7586 Acked Write 9 Done Sent Done false
7587 Acked Read 5 Done Sent Done false
7588 Acked Read 10 Done Sent Done false
7589 Acked Write 3 Done Sent Done false
7590 Acked Read 10 Done Sent Done false
7591 Acked Write 10 Done Sent Done false
7592 Acked Flush 0 Done Sent Done false
7593 Acked Read 8 Done Sent Done false
7594 Acked Flush 0 Done Sent Done false
7595 Acked Read 5 Done Sent Done false
7596 Acked Write 7 Done Sent Done false
7597 Acked Read 9 Done Sent Done false
7598 Acked Read 4 Done Sent Done false
7599 Acked Write 8 Done Sent Done false
7600 Acked Read 9 Done Sent Done false
7601 Acked Read 9 Done Sent Done false
7602 Acked Flush 0 Done Sent Done false
7603 Acked Read 6 Done Sent Done false
7604 Acked Write 7 Done Sent Done false
7605 Acked Read 5 Done Sent Done false
7606 Acked Read 10 Done Sent Done false
7607 Acked Read 7 Done Sent Done false
7608 Acked Flush 0 Done Sent Done false
7609 Acked Read 1 Done Sent Done false
7610 Acked Read 4 Done Sent Done false
7611 Acked Flush 0 Done Sent Done false
7612 Acked Read 5 Done Sent Done false
7613 Acked Read 8 Done Sent Done false
7614 Acked Write 8 Done Sent Done false
7615 Acked Write 5 Done Sent Done false
7616 Acked Read 5 Done Sent Done false
7617 Acked Write 7 Done Sent Done false
7618 Acked Read 5 Done Sent Done false
7619 Acked Write 7 Done Sent Done false
7620 Acked Read 8 Done Sent Done false
7621 Acked Read 2 Done Sent Done false
7622 Acked Write 7 Done Sent Done false
7623 Acked Write 3 Done Sent Done false
7624 Acked Write 3 Done Sent Done false
7625 Acked Read 8 Done Sent Done false
7626 Acked Read 5 Done Sent Done false
7627 Acked Read 7 Done Sent Done false
7628 Acked Flush 0 Done Sent Done false
7629 Acked Flush 0 Done Sent Done false
7630 Acked Write 4 Done Sent Done false
7631 Acked Write 2 Done Sent Done false
7632 Acked Read 10 Done Sent Done false
7633 Acked Write 7 Done Sent Done false
7634 Acked Write 9 Done Sent Done false
7635 Acked Flush 0 Done Sent Done false
7636 Acked Read 5 Done Sent Done false
7637 Acked Write 5 Done Sent Done false
7638 Acked Read 10 Done Sent Done false
7639 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7639 7337 7639
Downstairs last five completed: 7337 7336 7335 7334 7333
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.120602826Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33987)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.120819541Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.122053617Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.122666866Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.122687123Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 7337","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.122700672Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 7337","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.122954162Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 7337","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.12297287Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 7337","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.123256147Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7639 7638 7637 7636 7635
Upstairs last five completed: 7639 7638 7637 7636 7635
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Jan 27 19:41:42.369 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 205b0e3d-6fb2-4683-a876-e934a730ba04, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:42.369801988Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:42.369844553Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:41:42.369 INFO request completed, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:42.36987047Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
latency_us: 129, response_code: 200, uri: /state/cid/2, method: GET, req_id: 625089a1-cddc-46bd-8001-2eb77fe3d7fe, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:42.369949211Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:41:46.373 INFO request completed, latency_us: 335, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9afaddf4-4d77-4f63-8dcc-e2d46d45b9b2, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 212 len 28672
002/300 Read block 600 len 12288
003/300 Read block 246 len 12288
004/300 Read block 4945 len 28672
005/300 Read block 738 len 12288
006/300 Write block 4466 len 24576 data: 3 3 4 4 4 4
007/300 Read block 4179 len 12288
008/300 Write block 2376 len 12288 data: 7 8 10
009/300 Write block 3509 len 24576 data: 5 5 3 3 2 3
010/300 Write block 1024 len 28672 data: 3 3 2 2 4 6 6
011/300 Read block 50 len 20480
012/300 Read block 2211 len 36864
013/300 Write block 2786 len 32768 data: 5 4 6 6 6 5 4 6
014/300 Write block 1597 len 8192 data: 10 11
015/300 Read block 4773 len 40960
016/300 Write block 1795 len 16384 data: 5 4 4 3
017/300 Read block 639 len 16384
018/300 Read block 329 len 32768
019/300 Write block 467 len 24576 data: 4 4 4 6 6 4
020/300 Read block 1655 len 4096
021/300 Read block 4993 len 28672
022/300 Read block 407 len 4096
023/300 Write block 738 len 16384 data: 5 5 5 5
024/300 Read block 4463 len 24576
025/300 Write block 3509 len 4096 data: 6
026/300 Read block 1841 len 40960
027/300 Flush
028/300 Write block 4168 len 8192 data: 7 6
029/300 Write block 2188 len 4096 data: 4
030/300 Read block 4708 len 16384
031/300 Flush
032/300 Read block 3409 len 20480
033/300 Read block 813 len 40960
034/300 Read block 763 len 24576
035/300 Write block 1704 len 20480 data: 4 4 4 3 3
036/300 Write block 2908 len 4096 data: 4
037/300 Read block 4331 len 36864
038/300 Write block 4966 len 32768 data: 3 5 6 6 7 7 5 5
039/300 Read block 4235 len 4096
040/300 Read block 4662 len 32768
041/300 Read block 4001 len 8192
042/300 Flush
043/300 Read block 3088 len 20480
044/300 Write block 4008 len 8192 data: 6 7
045/300 Write block 4582 len 20480 data: 2 2 2 2 3
046/300 Write block 3686 len 28672 data: 4 5 6 6 6 6 6
047/300 Read block 3885 len 40960
048/300 Write block 2151 len 12288 data: 3 4 5
049/300 Write block 344 len 32768 data: 2 2 3 3 3 3 3 6
050/300 Read block 4215 len 32768
051/300 Write block 3763 len 20480 data: 5 5 6 6 7
052/300 Read block 4223 len 36864
053/300 Read block 2969 len 8192
054/300 Read block 4923 len 24576
055/300 Read block 1529 len 36864
056/300 Read block 1279 len 28672
057/300 Write block 4776 len 16384 data: 7 8 7 5
058/300 Read block 4344 len 32768
059/300 Read block 2436 len 36864
060/300 Write block 990 len 16384 data: 3 3 3 2
061/300 Flush
062/300 Flush
063/300 Read block 2379 len 12288
064/300 Read block 775 len 4096
065/300 Read block 556 len 8192
066/300 Read block 1052 len 20480
067/300 Read block 3902 len 8192
068/300 Write block 2297 len 8192 data: 2 4
069/300 Read block 4035 len 4096
070/300 Write block 3019 len 16384 data: 4 6 5 5
071/300 Write block 1207 len 36864 data: 6 6 5 4 3 3 3 4 5
072/300 Read block 1144 len 20480
073/300 Write block 1160 len 4096 data: 2
074/300 Read block 4719 len 8192
075/300 Flush
076/300 Write block 2436 len 20480 data: 3 6 6 6 7
077/300 Write block 3432 len 16384 data: 7 8 8 7
078/300 Write block 4264 len 12288 data: 7 7 7
079/300 Read block 4593 len 8192
080/300 Read block 4860 len 12288
081/300 Write block 802 len 32768 data: 7 6 4 2 3 4 4 6
082/300 Write block 3169 len 32768 data: 4 4 4 5 5 7 9 10
083/300 Write block 2135 len 4096 data: 3
084/300 Read block 4021 len 40960
085/300 Read block 1508 len 32768
086/300 Read block 842 len 4096
087/300 Read block 4328 len 4096
088/300 Write block 2210 len 8192 data: 4 4
089/300 Read block 1180 len 8192
090/300 Write block 1057 len 32768 data: 3 4 5 5 4 3 3 3
091/300 Flush
092/300 Write block 623 len 28672 data: 5 4 3 3 5 5 5
093/300 Flush
094/300 Read block 1392 len 12288
095/300 Write block 1491 len 20480 data: 2 2 2 2 2
096/300 Flush
097/300 Read block 4877 len 20480
098/300 Write block 2199 len 24576 data: 5 4 7 6 7 5
099/300 Read block 4832 len 4096
100/300 Write block 3992 len 32768 data: 3 4 4 3 3 3 3 3
101/300 Write block 2935 len 32768 data: 6 7 5 4 4 4 5 5
102/300 Write block 946 len 20480 data: 4 4 3 3 3
103/300 Write block 3422 len 40960 data: 3 3 4 5 9 8 8 8 9 8
104/300 Read block 4873 len 28672
105/300 Read block 3173 len 8192
106/300 Write block 4349 len 28672 data: 10 9 8 8 6 5 6
107/300 Read block 4161 len 4096
108/300 Write block 519 len 40960 data: 6 6 5 4 5 3 2 2 2 3
109/300 Write block 987 len 24576 data: 4 4 4 4 4 4
110/300 Read block 2389 len 40960
111/300 Write block 1294 len 40960 data: 5 5 5 7 8 7 6 8 7 9
112/300 Write block 4035 len 24576 data: 6 7 7 4 4 4
113/300 Write block 422 len 4096 data: 6
114/300 Write block 1526 len 16384 data: 4 5 3 4
115/300 Read block 1059 len 24576
116/300 Read block 4 len 32768
117/300 Read block 4361 len 20480
118/300 Write block 2579 len 40960 data: 4 4 3 2 2 2 2 4 5 5
119/300 Write block 665 len 16384 data: 4 4 4 4
120/300 Write block 245 len 40960 data: 2 3 2 2 3 3 3 5 6 5
121/300 Flush
122/300 Write block 4405 len 32768 data: 4 4 5 5 4 4 6 5
123/300 Write block 2426 len 12288 data: 4 4 5
124/300 Read block 3235 len 32768
125/300 Write block 2559 len 36864 data: 6 6 6 7 8 7 7 7 4
126/300 Read block 3031 len 28672
127/300 Write block 1942 len 8192 data: 6 6
128/300 Read block 162 len 24576
129/300 Write block 591 len 28672 data: 7 6 7 6 5 5 6
130/300 Read block 2296 len 24576
131/300 Write block 1425 len 4096 data: 5
132/300 Write block 1423 len 20480 data: 4 4 6 5 5
133/300 Write block 1971 len 24576 data: 3 3 4 4 3 4
134/300 Read block 3007 len 16384
135/300 Write block 1040 len 36864 data: 3 3 4 4 3 3 3 4 3
136/300 Flush
137/300 Write block 2585 len 12288 data: 3 5 6
138/300 Read block 4004 len 40960
139/300 Flush
140/300 Read block 1584 len 12288
141/300 Read block 2283 len 40960
142/300 Flush
143/300 Read block 4664 len 16384
144/300 Flush
145/300 Write block 34 len 4096 data: 2
146/300 Read block 1850 len 20480
147/300 Read block 1130 len 8192
148/300 Write block 3271 len 12288 data: 5 6 6
149/300 Read block 1718 len 20480
150/300 Write block 1563 len 12288 data: 2 2 3
151/300 Write block 1887 len 24576 data: 7 8 8 7 5 5
152/300 Write block 1095 len 12288 data: 5 5 5
153/300 Flush
154/300 Write block 4841 len 40960 data: 6 7 6 6 5 5 2 2 2 2
155/300 Read block 2119 len 4096
156/300 Read block 728 len 28672
157/300 Read block 209 len 28672
158/300 Read block 1301 len 16384
159/300 Write block 4490 len 28672 data: 4 4 6 6 7 7 5
160/300 Read block 3523 len 4096
161/300 Read block 3559 len 12288
162/300 Read block 3409 len 20480
163/300 Write block 2614 len 36864 data: 5 4 4 5 5 7 7 8 8
164/300 Read block 3077 len 4096
165/300 Write block 959 len 40960 data: 7 8 8 7 5 5 4 2 3 4
166/300 Read block 1675 len 40960
167/300 Read block 2626 len 36864
168/300 Read block 4852 len 40960
169/300 Read block 2161 len 24576
170/300 Write block 2154 len 20480 data: 7 7 9 10 9
171/300 Read block 1218 len 4096
172/300 Flush
173/300 Write block 4807 len 24576 data: 7 7 5 7 6 4
174/300 Write block 3779 len 24576 data: 6 5 4 3 3 3
175/300 Read block 2848 len 28672
176/300 Flush
177/300 Write block 1220 len 36864 data: 6 6 6 7 7 6 5 4 4
178/300 Read block 2742 len 16384
179/300 Write block 891 len 24576 data: 5 6 4 3 3 2
180/300 Read block 1945 len 28672
181/300 Read block 1679 len 24576
182/300 Read block 348 len 28672
183/300 Read block 1499 len 28672
184/300 Read block 1177 len 40960
185/300 Write block 1480 len 12288 data: 7 7 7
186/300 Read block 1177 len 20480
187/300 Read block 481 len 36864
188/300 Write block 4229 len 12288 data: 4 4 4
189/300 Read block 3486 len 8192
190/300 Write block 3739 len 16384 data: 3 3 2 3
191/300 Read block 2713 len 16384
192/300 Read block 1896 len 16384
193/300 Read block 44 len 4096
194/300 Read block 2411 len 4096
195/300 Write block 654 len 36864 data: 5 5 4 4 7 6 5 5 4
196/300 Write block 1676 len 28672 data: 7 4 4 4 3 3 2
197/300 Write block 766 len 4096 data: 4
198/300 Flush
199/300 Write block 3361 len 24576 data: 5 5 4 5 5 5
200/300 Read block 4173 len 16384
201/300 Write block 2529 len 8192 data: 4 4
202/300 Write block 803 len 12288 data: 7 5 3
203/300 Flush
204/300 Read block 2134 len 24576
205/300 Read block 1874 len 32768
206/300 Write block 2739 len 12288 data: 3 3 2
207/300 Read block 4546 len 24576
208/300 Write block 2295 len 20480 data: 3 3 3 5 6
209/300 Read block 3524 len 28672
210/300 Flush
211/300 Write block 1983 len 20480 data: 6 6 7 6 6
212/300 Read block 52 len 12288
213/300 Read block 2713 len 28672
214/300 Read block 2399 len 12288
215/300 Flush
216/300 Read block 4068 len 16384
217/300 Write block 3604 len 40960 data: 5 5 8 9 9 7 6 4 3 3
218/300 Read block 1477 len 32768
219/300 Read block 2563 len 12288
220/300 Read block 3221 len 20480
221/300 Write block 826 len 16384 data: 3 3 5 5
222/300 Read block 1603 len 28672
223/300 Read block 710 len 8192
224/300 Write block 1130 len 40960 data: 7 6 6 5 5 5 4 5 4 5
225/300 Write block 363 len 24576 data: 4 6 5 5 5 4
226/300 Read block 4565 len 24576
227/300 Read block 2061 len 20480
228/300 Write block 3359 len 32768 data: 4 5 6 6 5 6 6 6
229/300 Read block 1441 len 20480
230/300 Read block 2414 len 32768
231/300 Write block 4264 len 32768 data: 8 8 8 6 7 6 6 8
232/300 Read block 2849 len 32768
233/300 Read block 2093 len 36864
234/300 Write block 1283 len 28672 data: 3 3 2 2 2 2 2
235/300 Read block 716 len 20480
236/300 Read block 939 len 28672
237/300 Write block 523 len 8192 data: 6 4
238/300 Write block 2929 len 36864 data: 7 6 6 6 7 7 7 8 6
239/300 Write block 60 len 12288 data: 7 5 6
240/300 Write block 3812 len 4096 data: 5
241/300 Write block 4466 len 20480 data: 4 4 5 5 5
242/300 Write block 3588 len 12288 data: 2 2 2
243/300 Read block 2478 len 36864
244/300 Read block 4260 len 12288
245/300 Flush
246/300 Read block 1047 len 8192
247/300 Read block 3790 len 12288
248/300 Read block 1089 len 8192
249/300 Read block 145 len 28672
250/300 Read block 1042 len 28672
251/300 Write block 4295 len 20480 data: 5 3 5 5 5
252/300 Flush
253/300 Flush
254/300 Read block 3198 len 16384
255/300 Write block 928 len 12288 data: 4 4 6
256/300 Read block 657 len 32768
257/300 Read block 2645 len 8192
258/300 Read block 4033 len 4096
259/300 Write block 2918 len 16384 data: 3 3 3 2
260/300 Write block 2990 len 8192 data: 4 4
261/300 Read block 2088 len 4096
262/300 Read block 1620 len 28672
263/300 Read block 4704 len 12288
264/300 Write block 4800 len 28672 data: 7 7 6 8 7 10 6
265/300 Flush
266/300 Read block 3810 len 16384
267/300 Write block 3208 len 28672 data: 4 5 4 6 6 6 4
268/300 Read block 2518 len 40960
269/300 Read block 1969 len 24576
270/300 Read block 4311 len 32768
271/300 Read block 4109 len 32768
272/300 Write block 3720 len 36864 data: 5 6 5 5 6 7 7 7 6
273/300 Read block 4104 len 12288
274/300 Write block 4453 len 12288 data: 5 4 3
275/300 Read block 4262 len 28672
276/300 Read block 2436 len 28672
277/300 Read block 2873 len 12288
278/300 Write block 4271 len 36864 data: 9 6 6 5 8 7 7 7 7
279/300 Read block 2533 len 36864
280/300 Read block 821 len 36864
281/300 Read block 4092 len 16384
282/300 Write block 1842 len 24576 data: 2 3 3 4 4 5
283/300 Flush
284/300 Flush
285/300 Read block 4562 len 36864
286/300 Write block 2231 len 16384 data: 5 3 2 4
287/300 Write block 1825 len 12288 data: 3 3 3
288/300 Write block 368 len 20480 data: 5 4 3 5 6
289/300 Read block 2584 len 8192
290/300 Write block 759 len 32768 data: 3 3 3 3 3 3 4 5
291/300 Write block 3103 len 40960 data: 7 9 8 9 9 9 7 5 4 3
292/300 Read block 2307 len 16384
293/300 Write block 875 len 8192 data: 4 4
294/300 Write block 4593 len 32768 data: 5 6 5 5 5 3 4 4
295/300 Read block 3325 len 4096
296/300 Write block 1367 len 20480 data: 7 6 6 7 6
297/300 Write block 444 len 24576 data: 3 4 3 3 3 2
298/300 Read block 4409 len 16384
299/300 Write block 3002 len 40960 data: 7 7 7 6 6 6 5 5 4 5
300/300 Write block 3644 len 36864 data: 5 5 6 5 6 7 6 6 4
301/300 Write block 1476 len 24576 data: 5 4 5 7 8 8
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:41:46.606 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: d9043ca4-d777-4264-b9c7-856c6ed4042a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7640 Acked Read 7 Done Done Sent false
7641 Acked Read 3 Done Done Sent false
7642 Acked Read 3 Done Done Sent false
7643 Acked Read 7 Done Done Sent false
7644 Acked Read 3 Done Done Sent false
7645 Acked Write 6 Done Done Sent false
7646 Acked Read 3 Done Done Sent false
7647 Acked Write 3 Done Done Sent false
7648 Acked Write 6 Done Done Sent false
7649 Acked Write 7 Done Done Sent false
7650 Acked Read 5 Done Done Sent false
7651 Acked Read 9 Done Done Sent false
7652 Acked Write 8 Done Done Sent false
7653 Acked Write 2 Done Done Sent false
7654 Acked Read 10 Done Done Sent false
7655 Acked Write 4 Done Done Sent false
7656 Acked Read 4 Done Done Sent false
7657 Acked Read 8 Done Done Sent false
7658 Acked Write 6 Done Done Sent false
7659 Acked Read 1 Done Done Sent false
7660 Acked Read 7 Done Done Sent false
7661 Acked Read 1 Done Done Sent false
7662 Acked Write 4 Done Done Sent false
7663 Acked Read 6 Done Done Sent false
7664 Acked Write 1 Done Done Sent false
7665 Acked Read 10 Done Done Sent false
7666 Acked Flush 0 Done Done Sent false
7667 Acked Write 2 Done Done Sent false
7668 Acked Write 1 Done Done Sent false
7669 Acked Read 4 Done Done Sent false
7670 Acked Flush 0 Done Done Sent false
7671 Acked Read 5 Done Done Sent false
7672 Acked Read 10 Done Done Sent false
7673 Acked Read 6 Done Done Sent false
7674 Acked Write 5 Done Done Sent false
7675 Acked Write 1 Done Done Sent false
7676 Acked Read 9 Done Done Sent false
7677 Acked Write 8 Done Done Sent false
7678 Acked Read 1 Done Done Sent false
7679 Acked Read 8 Done Done Sent false
7680 Acked Read 2 Done Done Sent false
7681 Acked Flush 0 Done Done Sent false
7682 Acked Read 5 Done Done Sent false
7683 Acked Write 2 Done Done Sent false
7684 Acked Write 5 Done Done Sent false
7685 Acked Write 7 Done Done Sent false
7686 Acked Read 10 Done Done Sent false
7687 Acked Write 3 Done Done Sent false
7688 Acked Write 8 Done Done Sent false
7689 Acked Read 8 Done Done Sent false
7690 Acked Write 5 Done Done Sent false
7691 Acked Read 9 Done Done Sent false
7692 Acked Read 2 Done Done Sent false
7693 Acked Read 6 Done Done Sent false
7694 Acked Read 9 Done Done Sent false
7695 Acked Read 7 Done Done Sent false
7696 Acked Write 4 Done Done Sent false
7697 Acked Read 8 Done Done Sent false
7698 Acked Read 9 Done Done Sent false
7699 Acked Write 4 Done Done Sent false
7700 Acked Flush 0 Done Done Sent false
7701 Acked Flush 0 Done Done Sent false
7702 Acked Read 3 Done Done Sent false
7703 Acked Read 1 Done Done Sent false
7704 Acked Read 2 Done Done Sent false
7705 Acked Read 5 Done Done Sent false
7706 Acked Read 2 Done Done Sent false
7707 Acked Write 2 Done Done Sent false
7708 Acked Read 1 Done Done Sent false
7709 Acked Write 4 Done Done Sent false
7710 Acked Write 9 Done Done Sent false
7711 Acked Read 5 Done Done Sent false
7712 Acked Write 1 Done Done Sent false
7713 Acked Read 2 Done Done Sent false
7714 Acked Flush 0 Done Done Sent false
7715 Acked Write 5 Done Done Sent false
7716 Acked Write 4 Done Done Sent false
7717 Acked Write 3 Done Done Sent false
7718 Acked Read 2 Done Done Sent false
7719 Acked Read 3 Done Done Sent false
7720 Acked Write 8 Done Done Sent false
7721 Acked Write 8 Done Done Sent false
7722 Acked Write 1 Done Done Sent false
7723 Acked Read 10 Done Done Sent false
7724 Acked Read 8 Done Done Sent false
7725 Acked Read 1 Done Done Sent false
7726 Acked Read 1 Done Done Sent false
7727 Acked Write 2 Done Done Sent false
7728 Acked Read 2 Done Done Sent false
7729 Acked Write 8 Done Done Sent false
7730 Acked Flush 0 Done Done Sent false
7731 Acked Write 7 Done Done Sent false
7732 Acked Flush 0 Done Done Sent false
7733 Acked Read 3 Done Done Sent false
7734 Acked Write 5 Done Done Sent false
7735 Acked Flush 0 Done Done Sent false
7736 Acked Read 5 Done Done Sent false
7737 Acked Write 6 Done Done Sent false
7738 Acked Read 1 Done Done Sent false
7739 Acked Write 8 Done Done Sent false
7740 Acked Write 8 Done Done Sent false
7741 Acked Write 5 Done Done Sent false
7742 Acked Write 10 Done Done Sent false
7743 Acked Read 7 Done Done Sent false
7744 Acked Read 2 Done Done Sent false
7745 Acked Write 7 Done Done Sent false
7746 Acked Read 1 Done Done Sent false
7747 Acked Write 10 Done Done Sent false
7748 Acked Write 6 Done Done Sent false
7749 Acked Read 10 Done Done Sent false
7750 Acked Write 10 Done Done Sent false
7751 Acked Write 6 Done Done Sent false
7752 Acked Write 1 Done Done Sent false
7753 Acked Write 4 Done Done Sent false
7754 Acked Read 6 Done Done Sent false
7755 Acked Read 8 Done Done Sent false
7756 Acked Read 5 Done Done Sent false
7757 Acked Write 10 Done Done Sent false
7758 Acked Write 4 Done Done Sent false
7759 Acked Write 10 Done Done Sent false
7760 Acked Flush 0 Done Done Sent false
7761 Acked Write 8 Done Done Sent false
7762 Acked Write 3 Done Done Sent false
7763 Acked Read 8 Done Done Sent false
7764 Acked Write 9 Done Done Sent false
7765 Acked Read 7 Done Done Sent false
7766 Acked Write 2 Done Done Sent false
7767 Acked Read 6 Done Done Sent false
7768 Acked Write 7 Done Done Sent false
7769 Acked Read 6 Done Done Sent false
7770 Acked Write 1 Done Done Sent false
7771 Acked Write 5 Done Done Sent false
7772 Acked Write 6 Done Done Sent false
7773 Acked Read 4 Done Done Sent false
7774 Acked Write 9 Done Done Sent false
7775 Acked Flush 0 Done Done Sent false
7776 Acked Write 3 Done Done Sent false
7777 Acked Read 10 Done Done Sent false
7778 Acked Flush 0 Done Done Sent false
7779 Acked Read 3 Done Done Sent false
7780 Acked Read 10 Done Done Sent false
7781 Acked Flush 0 Done Done Sent false
7782 Acked Read 4 Done Done Sent false
7783 Acked Flush 0 Done Done Sent false
7784 Acked Write 1 Done Done Sent false
7785 Acked Read 5 Done Done Sent false
7786 Acked Read 2 Done Done Sent false
7787 Acked Write 3 Done Done Sent false
7788 Acked Read 5 Done Done Sent false
7789 Acked Write 3 Done Done Sent false
7790 Acked Write 6 Done Done Sent false
7791 Acked Write 3 Done Done Sent false
7792 Acked Flush 0 Done Done Sent false
7793 Acked Write 10 Done Done Sent false
7794 Acked Read 1 Done Done Sent false
7795 Acked Read 7 Done Done Sent false
7796 Acked Read 7 Done Done Sent false
7797 Acked Read 4 Done Done Sent false
7798 Acked Write 7 Done Done Sent false
7799 Acked Read 1 Done Done Sent false
7800 Acked Read 3 Done Done Sent false
7801 Acked Read 5 Done Done Sent false
7802 Acked Write 9 Done Done Sent false
7803 Acked Read 1 Done Done Sent false
7804 Acked Write 10 Done Done Sent false
7805 Acked Read 10 Done Done Sent false
7806 Acked Read 9 Done Done Sent false
7807 Acked Read 10 Done Done Sent false
7808 Acked Read 6 Done Done Sent false
7809 Acked Write 5 Done Done Sent false
7810 Acked Read 1 Done Done Sent false
7811 Acked Flush 0 Done Done Sent false
7812 Acked Write 6 Done Done Sent false
7813 Acked Write 6 Done Done Sent false
7814 Acked Read 7 Done Done Sent false
7815 Acked Flush 0 Done Done Sent false
7816 Acked Write 9 Done Done Sent false
7817 Acked Read 4 Done Done Sent false
7818 Acked Write 6 Done Done Sent false
7819 Acked Read 7 Done Done Sent false
7820 Acked Read 6 Done Done Sent false
7821 Acked Read 7 Done Done Sent false
7822 Acked Read 7 Done Done Sent false
7823 Acked Read 10 Done Done Sent false
7824 Acked Write 3 Done Done Sent false
7825 Acked Read 5 Done Done Sent false
7826 Acked Read 9 Done Done Sent false
7827 Acked Write 3 Done Done Sent false
7828 Acked Read 2 Done Done Sent false
7829 Acked Write 4 Done Done Sent false
7830 Acked Read 4 Done Done Sent false
7831 Acked Read 4 Done Done Sent false
7832 Acked Read 1 Done Done Sent false
7833 Acked Read 1 Done Done Sent false
7834 Acked Write 9 Done Done Sent false
7835 Acked Write 7 Done Done Sent false
7836 Acked Write 1 Done Done Sent false
7837 Acked Flush 0 Done Done Sent false
7838 Acked Write 6 Done Done Sent false
7839 Acked Read 4 Done Done Sent false
7840 Acked Write 2 Done Done Sent false
7841 Acked Write 3 Done Done Sent false
7842 Acked Flush 0 Done Done Sent false
7843 Acked Read 6 Done Done Sent false
7844 Acked Read 8 Done Done Sent false
7845 Acked Write 3 Done Done Sent false
7846 Acked Read 6 Done Done Sent false
7847 Acked Write 5 Done Done Sent false
7848 Acked Read 7 Done Done Sent false
7849 Acked Flush 0 Done Done Sent false
7850 Acked Write 5 Done Done Sent false
7851 Acked Read 3 Done Done Sent false
7852 Acked Read 7 Done Done Sent false
7853 Acked Read 3 Done Done Sent false
7854 Acked Flush 0 Done Done Sent false
7855 Acked Read 4 Done Done Sent false
7856 Acked Write 10 Done Done Sent false
7857 Acked Read 8 Done Done Sent false
7858 Acked Read 3 Done Done Sent false
7859 Acked Read 5 Done Done Sent false
7860 Acked Write 4 Done Done Sent false
7861 Acked Read 7 Done Done Sent false
7862 Acked Read 2 Done Done Sent false
7863 Acked Write 10 Done Done Sent false
7864 Acked Write 6 Done Done Sent false
7865 Acked Read 6 Done Done Sent false
7866 Acked Read 5 Done Done Sent false
7867 Acked Write 8 Done Done Sent false
7868 Acked Read 5 Done Done Sent false
7869 Acked Read 8 Done Done Sent false
7870 Acked Write 8 Done Done Sent false
7871 Acked Read 8 Done Done Sent false
7872 Acked Read 9 Done Done Sent false
7873 Acked Write 7 Done Done Sent false
7874 Acked Read 5 Done Done Sent false
7875 Acked Read 7 Done Done Sent false
7876 Acked Write 2 Done Done Sent false
7877 Acked Write 9 Done Done Sent false
7878 Acked Write 3 Done Done Sent false
7879 Acked Write 1 Done Done Sent false
7880 Acked Write 5 Done Done Sent false
7881 Acked Write 3 Done Done Sent false
7882 Acked Read 9 Done Done Sent false
7883 Acked Read 3 Done Done Sent false
7884 Acked Flush 0 Done Done Sent false
7885 Acked Read 2 Done Done Sent false
7886 Acked Read 3 Done Done Sent false
7887 Acked Read 2 Done Done Sent false
7888 Acked Read 7 Done Done Sent false
7889 Acked Read 7 Done Done Sent false
7890 Acked Write 5 Done Done Sent false
7891 Acked Flush 0 Done Done Sent false
7892 Acked Flush 0 Done Done Sent false
7893 Acked Read 4 Done Done Sent false
7894 Acked Write 3 Done Done Sent false
7895 Acked Read 8 Done Done Sent false
7896 Acked Read 2 Done Done Sent false
7897 Acked Read 1 Done Done Sent false
7898 Acked Write 4 Done Done Sent false
7899 Acked Write 2 Done Done Sent false
7900 Acked Read 1 Done Done Sent false
7901 Acked Read 7 Done Done Sent false
7902 Acked Read 3 Done Done Sent false
7903 Acked Write 7 Done Done Sent false
7904 Acked Flush 0 Done Done Sent false
7905 Acked Read 4 Done Done Sent false
7906 Acked Write 7 Done Done Sent false
7907 Acked Read 10 Done Done Sent false
7908 Acked Read 6 Done Done Sent false
7909 Acked Read 8 Done Done Sent false
7910 Acked Read 8 Done Done Sent false
7911 Acked Write 9 Done Done Sent false
7912 Acked Read 3 Done Done Sent false
7913 Acked Write 3 Done Done Sent false
7914 Acked Read 7 Done Done Sent false
7915 Acked Read 7 Done Done Sent false
7916 Acked Read 3 Done Done Sent false
7917 Acked Write 9 Done Done Sent false
7918 Acked Read 9 Done Done Sent false
7919 Acked Read 9 Done Done Sent false
7920 Acked Read 4 Done Done Sent false
7921 Acked Write 6 Done Done Sent false
7922 Acked Flush 0 Done Done Sent false
7923 Acked Flush 0 Done Done Sent false
7924 Acked Read 9 Done Done Sent false
7925 Acked Write 4 Done Done Sent false
7926 Acked Write 3 Done Done Sent false
7927 Acked Write 5 Done Done Sent false
7928 Acked Read 2 Done Done Sent false
7929 Acked Write 8 Done Done Sent false
7930 Acked Write 10 Done Done Sent false
7931 Acked Read 4 Done Done Sent false
7932 Acked Write 2 Done Done Sent false
7933 Acked Write 8 Done Done Sent false
7934 Acked Read 1 Done Done Sent false
7935 Acked Write 5 Done Done Sent false
7936 Acked Write 6 Done Done Sent false
7937 Acked Read 4 Done Done Sent false
7938 Acked Write 10 Done Done Sent false
7939 Acked Write 9 Done Done Sent false
7940 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 301 303
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7923 7923 7639
Downstairs last five completed: 7639 7638 7637 7636 7635
Upstairs last five completed: 7940 7939 7938 7937 7936
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7640 Acked Read 7 Done Done Sent false
7641 Acked Read 3 Done Done Sent false
7642 Acked Read 3 Done Done Sent false
7643 Acked Read 7 Done Done Sent false
7644 Acked Read 3 Done Done Sent false
7645 Acked Write 6 Done Done Sent false
7646 Acked Read 3 Done Done Sent false
7647 Acked Write 3 Done Done Sent false
7648 Acked Write 6 Done Done Sent false
7649 Acked Write 7 Done Done Sent false
7650 Acked Read 5 Done Done Sent false
7651 Acked Read 9 Done Done Sent false
7652 Acked Write 8 Done Done Sent false
7653 Acked Write 2 Done Done Sent false
7654 Acked Read 10 Done Done Sent false
7655 Acked Write 4 Done Done Sent false
7656 Acked Read 4 Done Done Sent false
7657 Acked Read 8 Done Done Sent false
7658 Acked Write 6 Done Done Sent false
7659 Acked Read 1 Done Done Sent false
7660 Acked Read 7 Done Done Sent false
7661 Acked Read 1 Done Done Sent false
7662 Acked Write 4 Done Done Sent false
7663 Acked Read 6 Done Done Sent false
7664 Acked Write 1 Done Done Sent false
7665 Acked Read 10 Done Done Sent false
7666 Acked Flush 0 Done Done Sent false
7667 Acked Write 2 Done Done Sent false
7668 Acked Write 1 Done Done Sent false
7669 Acked Read 4 Done Done Sent false
7670 Acked Flush 0 Done Done Sent false
7671 Acked Read 5 Done Done Sent false
7672 Acked Read 10 Done Done Sent false
7673 Acked Read 6 Done Done Sent false
7674 Acked Write 5 Done Done Sent false
7675 Acked Write 1 Done Done Sent false
7676 Acked Read 9 Done Done Sent false
7677 Acked Write 8 Done Done Sent false
7678 Acked Read 1 Done Done Sent false
7679 Acked Read 8 Done Done Sent false
7680 Acked Read 2 Done Done Sent false
7681 Acked Flush 0 Done Done Sent false
7682 Acked Read 5 Done Done Sent false
7683 Acked Write 2 Done Done Sent false
7684 Acked Write 5 Done Done Sent false
7685 Acked Write 7 Done Done Sent false
7686 Acked Read 10 Done Done Sent false
7687 Acked Write 3 Done Done Sent false
7688 Acked Write 8 Done Done Sent false
7689 Acked Read 8 Done Done Sent false
7690 Acked Write 5 Done Done Sent false
7691 Acked Read 9 Done Done Sent false
7692 Acked Read 2 Done Done Sent false
7693 Acked Read 6 Done Done Sent false
7694 Acked Read 9 Done Done Sent false
7695 Acked Read 7 Done Done Sent false
7696 Acked Write 4 Done Done Sent false
7697 Acked Read 8 Done Done Sent false
7698 Acked Read 9 Done Done Sent false
7699 Acked Write 4 Done Done Sent false
7700 Acked Flush 0 Done Done Sent false
7701 Acked Flush 0 Done Done Sent false
7702 Acked Read 3 Done Done Sent false
7703 Acked Read 1 Done Done Sent false
7704 Acked Read 2 Done Done Sent false
7705 Acked Read 5 Done Done Sent false
7706 Acked Read 2 Done Done Sent false
7707 Acked Write 2 Done Done Sent false
7708 Acked Read 1 Done Done Sent false
7709 Acked Write 4 Done Done Sent false
7710 Acked Write 9 Done Done Sent false
7711 Acked Read 5 Done Done Sent false
7712 Acked Write 1 Done Done Sent false
7713 Acked Read 2 Done Done Sent false
7714 Acked Flush 0 Done Done Sent false
7715 Acked Write 5 Done Done Sent false
7716 Acked Write 4 Done Done Sent false
7717 Acked Write 3 Done Done Sent false
7718 Acked Read 2 Done Done Sent false
7719 Acked Read 3 Done Done Sent false
7720 Acked Write 8 Done Done Sent false
7721 Acked Write 8 Done Done Sent false
7722 Acked Write 1 Done Done Sent false
7723 Acked Read 10 Done Done Sent false
7724 Acked Read 8 Done Done Sent false
7725 Acked Read 1 Done Done Sent false
7726 Acked Read 1 Done Done Sent false
7727 Acked Write 2 Done Done Sent false
7728 Acked Read 2 Done Done Sent false
7729 Acked Write 8 Done Done Sent false
7730 Acked Flush 0 Done Done Sent false
7731 Acked Write 7 Done Done Sent false
7732 Acked Flush 0 Done Done Sent false
7733 Acked Read 3 Done Done Sent false
7734 Acked Write 5 Done Done Sent false
7735 Acked Flush 0 Done Done Sent false
7736 Acked Read 5 Done Done Sent false
7737 Acked Write 6 Done Done Sent false
7738 Acked Read 1 Done Done Sent false
7739 Acked Write 8 Done Done Sent false
7740 Acked Write 8 Done Done Sent false
7741 Acked Write 5 Done Done Sent false
7742 Acked Write 10 Done Done Sent false
7743 Acked Read 7 Done Done Sent false
7744 Acked Read 2 Done Done Sent false
7745 Acked Write 7 Done Done Sent false
7746 Acked Read 1 Done Done Sent false
7747 Acked Write 10 Done Done Sent false
7748 Acked Write 6 Done Done Sent false
7749 Acked Read 10 Done Done Sent false
7750 Acked Write 10 Done Done Sent false
7751 Acked Write 6 Done Done Sent false
7752 Acked Write 1 Done Done Sent false
7753 Acked Write 4 Done Done Sent false
7754 Acked Read 6 Done Done Sent false
7755 Acked Read 8 Done Done Sent false
7756 Acked Read 5 Done Done Sent false
7757 Acked Write 10 Done Done Sent false
7758 Acked Write 4 Done Done Sent false
7759 Acked Write 10 Done Done Sent false
7760 Acked Flush 0 Done Done Sent false
7761 Acked Write 8 Done Done Sent false
7762 Acked Write 3 Done Done Sent false
7763 Acked Read 8 Done Done Sent false
7764 Acked Write 9 Done Done Sent false
7765 Acked Read 7 Done Done Sent false
7766 Acked Write 2 Done Done Sent false
7767 Acked Read 6 Done Done Sent false
7768 Acked Write 7 Done Done Sent false
7769 Acked Read 6 Done Done Sent false
7770 Acked Write 1 Done Done Sent false
7771 Acked Write 5 Done Done Sent false
7772 Acked Write 6 Done Done Sent false
7773 Acked Read 4 Done Done Sent false
7774 Acked Write 9 Done Done Sent false
7775 Acked Flush 0 Done Done Sent false
7776 Acked Write 3 Done Done Sent false
7777 Acked Read 10 Done Done Sent false
7778 Acked Flush 0 Done Done Sent false
7779 Acked Read 3 Done Done Sent false
7780 Acked Read 10 Done Done Sent false
7781 Acked Flush 0 Done Done Sent false
7782 Acked Read 4 Done Done Sent false
7783 Acked Flush 0 Done Done Sent false
7784 Acked Write 1 Done Done Sent false
7785 Acked Read 5 Done Done Sent false
7786 Acked Read 2 Done Done Sent false
7787 Acked Write 3 Done Done Sent false
7788 Acked Read 5 Done Done Sent false
7789 Acked Write 3 Done Done Sent false
7790 Acked Write 6 Done Done Sent false
7791 Acked Write 3 Done Done Sent false
7792 Acked Flush 0 Done Done Sent false
7793 Acked Write 10 Done Done Sent false
7794 Acked Read 1 Done Done Sent false
7795 Acked Read 7 Done Done Sent false
7796 Acked Read 7 Done Done Sent false
7797 Acked Read 4 Done Done Sent false
7798 Acked Write 7 Done Done Sent false
7799 Acked Read 1 Done Done Sent false
7800 Acked Read 3 Done Done Sent false
7801 Acked Read 5 Done Done Sent false
7802 Acked Write 9 Done Done Sent false
7803 Acked Read 1 Done Done Sent false
7804 Acked Write 10 Done Done Sent false
7805 Acked Read 10 Done Done Sent false
7806 Acked Read 9 Done Done Sent false
7807 Acked Read 10 Done Done Sent false
7808 Acked Read 6 Done Done Sent false
7809 Acked Write 5 Done Done Sent false
7810 Acked Read 1 Done Done Sent false
7811 Acked Flush 0 Done Done Sent false
7812 Acked Write 6 Done Done Sent false
7813 Acked Write 6 Done Done Sent false
7814 Acked Read 7 Done Done Sent false
7815 Acked Flush 0 Done Done Sent false
7816 Acked Write 9 Done Done Sent false
7817 Acked Read 4 Done Done Sent false
7818 Acked Write 6 Done Done Sent false
7819 Acked Read 7 Done Done Sent false
7820 Acked Read 6 Done Done Sent false
7821 Acked Read 7 Done Done Sent false
7822 Acked Read 7 Done Done Sent false
7823 Acked Read 10 Done Done Sent false
7824 Acked Write 3 Done Done Sent false
7825 Acked Read 5 Done Done Sent false
7826 Acked Read 9 Done Done Sent false
7827 Acked Write 3 Done Done Sent false
7828 Acked Read 2 Done Done Sent false
7829 Acked Write 4 Done Done Sent false
7830 Acked Read 4 Done Done Sent false
7831 Acked Read 4 Done Done Sent false
7832 Acked Read 1 Done Done Sent false
7833 Acked Read 1 Done Done Sent false
7834 Acked Write 9 Done Done Sent false
7835 Acked Write 7 Done Done Sent false
7836 Acked Write 1 Done Done Sent false
7837 Acked Flush 0 Done Done Sent false
7838 Acked Write 6 Done Done Sent false
7839 Acked Read 4 Done Done Sent false
7840 Acked Write 2 Done Done Sent false
7841 Acked Write 3 Done Done Sent false
7842 Acked Flush 0 Done Done Sent false
7843 Acked Read 6 Done Done Sent false
7844 Acked Read 8 Done Done Sent false
7845 Acked Write 3 Done Done Sent false
7846 Acked Read 6 Done Done Sent false
7847 Acked Write 5 Done Done Sent false
7848 Acked Read 7 Done Done Sent false
7849 Acked Flush 0 Done Done Sent false
7850 Acked Write 5 Done Done Sent false
7851 Acked Read 3 Done Done Sent false
7852 Acked Read 7 Done Done Sent false
7853 Acked Read 3 Done Done Sent false
7854 Acked Flush 0 Done Done Sent false
7855 Acked Read 4 Done Done Sent false
7856 Acked Write 10 Done Done Sent false
7857 Acked Read 8 Done Done Sent false
7858 Acked Read 3 Done Done Sent false
7859 Acked Read 5 Done Done Sent false
7860 Acked Write 4 Done Done Sent false
7861 Acked Read 7 Done Done Sent false
7862 Acked Read 2 Done Done Sent false
7863 Acked Write 10 Done Done Sent false
7864 Acked Write 6 Done Done Sent false
7865 Acked Read 6 Done Done Sent false
7866 Acked Read 5 Done Done Sent false
7867 Acked Write 8 Done Done Sent false
7868 Acked Read 5 Done Done Sent false
7869 Acked Read 8 Done Done Sent false
7870 Acked Write 8 Done Done Sent false
7871 Acked Read 8 Done Done Sent false
7872 Acked Read 9 Done Done Sent false
7873 Acked Write 7 Done Done Sent false
7874 Acked Read 5 Done Done Sent false
7875 Acked Read 7 Done Done Sent false
7876 Acked Write 2 Done Done Sent false
7877 Acked Write 9 Done Done Sent false
7878 Acked Write 3 Done Done Sent false
7879 Acked Write 1 Done Done Sent false
7880 Acked Write 5 Done Done Sent false
7881 Acked Write 3 Done Done Sent false
7882 Acked Read 9 Done Done Sent false
7883 Acked Read 3 Done Done Sent false
7884 Acked Flush 0 Done Done Sent false
7885 Acked Read 2 Done Done Sent false
7886 Acked Read 3 Done Done Sent false
7887 Acked Read 2 Done Done Sent false
7888 Acked Read 7 Done Done Sent false
7889 Acked Read 7 Done Done Sent false
7890 Acked Write 5 Done Done Sent false
7891 Acked Flush 0 Done Done Sent false
7892 Acked Flush 0 Done Done Sent false
7893 Acked Read 4 Done Done Sent false
7894 Acked Write 3 Done Done Sent false
7895 Acked Read 8 Done Done Sent false
7896 Acked Read 2 Done Done Sent false
7897 Acked Read 1 Done Done Sent false
7898 Acked Write 4 Done Done Sent false
7899 Acked Write 2 Done Done Sent false
7900 Acked Read 1 Done Done Sent false
7901 Acked Read 7 Done Done Sent false
7902 Acked Read 3 Done Done Sent false
7903 Acked Write 7 Done Done Sent false
7904 Acked Flush 0 Done Done Sent false
7905 Acked Read 4 Done Done Sent false
7906 Acked Write 7 Done Done Sent false
7907 Acked Read 10 Done Done Sent false
7908 Acked Read 6 Done Done Sent false
7909 Acked Read 8 Done Done Sent false
7910 Acked Read 8 Done Done Sent false
7911 Acked Write 9 Done Done Sent false
7912 Acked Read 3 Done Done Sent false
7913 Acked Write 3 Done Done Sent false
7914 Acked Read 7 Done Done Sent false
7915 Acked Read 7 Done Done Sent false
7916 Acked Read 3 Done Done Sent false
7917 Acked Write 9 Done Done Sent false
7918 Acked Read 9 Done Done Sent false
7919 Acked Read 9 Done Done Sent false
7920 Acked Read 4 Done Done Sent false
7921 Acked Write 6 Done Done Sent false
7922 Acked Flush 0 Done Done Sent false
7923 Acked Flush 0 Done Done Sent false
7924 Acked Read 9 Done Done Sent false
7925 Acked Write 4 Done Done Sent false
7926 Acked Write 3 Done Done Sent false
7927 Acked Write 5 Done Done Sent false
7928 Acked Read 2 Done Done Sent false
7929 Acked Write 8 Done Done Sent false
7930 Acked Write 10 Done Done Sent false
7931 Acked Read 4 Done Done Sent false
7932 Acked Write 2 Done Done Sent false
7933 Acked Write 8 Done Done Sent false
7934 Acked Read 1 Done Done Sent false
7935 Acked Write 5 Done Done Sent false
7936 Acked Write 6 Done Done Sent false
7937 Acked Read 4 Done Done Sent false
7938 Acked Write 10 Done Done Sent false
7939 Acked Write 9 Done Done Sent false
7940 Acked Write 6 Done Done Sent false
7941 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7941 7941 7639
Downstairs last five completed: 7639 7638 7637 7636 7635
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.371202952Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58585)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.371310309Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.372347818Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.372942859Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.372966386Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 7639","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.372980404Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 7639","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.373246023Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 7639","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.373268541Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 7639","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:52.373488175Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7941 7940 7939 7938 7937
Upstairs last five completed: 7941 7940 7939 7938 7937
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Jan 27 19:41:54.612 [8810] Got stop action so:false kr:false
INFO request completed, latency_us: 551, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 77e30fea-9383-40fc-a9c1-86a3ffbd5e36, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:54.613658347Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:54.613709011Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:54.613724799Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:41:54.613 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1c945dd5-008e-45ef-a253-751e96d179a6{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:54.613820438Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:41:58.616 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: a8cde49d-cc86-446b-b859-f9879c2d016a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Read block 4199 len 32768
003/300 Write block 1242 len 8192 data: 4 2
004/300 Read block 3262 len 24576
005/300 Read block 2485 len 32768
006/300 Write block 2195 len 24576 data: 5 5 5 6 6 5
007/300 Read block 2719 len 16384
008/300 Write block 4447 len 12288 data: 5 5 5
009/300 Read block 2424 len 8192
010/300 Write block 4956 len 28672 data: 4 3 4 5 6 6 7
011/300 Read block 4591 len 16384
012/300 Read block 1596 len 24576
013/300 Read block 3684 len 16384
014/300 Write block 2926 len 4096 data: 4
015/300 Read block 3951 len 20480
016/300 Read block 3880 len 40960
017/300 Read block 1679 len 36864
018/300 Write block 4268 len 4096 data: 8
019/300 Write block 4231 len 16384 data: 5 4 3 3
020/300 Flush
021/300 Write block 1003 len 8192 data: 5 6
022/300 Flush
023/300 Write block 1934 len 8192 data: 5 5
024/300 Read block 749 len 36864
025/300 Write block 2745 len 4096 data: 3
026/300 Write block 4115 len 12288 data: 5 6 6
027/300 Write block 3998 len 20480 data: 4 4 3 4 4
028/300 Write block 2453 len 20480 data: 4 4 3 3 4
029/300 Flush
030/300 Write block 2648 len 4096 data: 3
031/300 Read block 2816 len 28672
032/300 Write block 3336 len 36864 data: 6 7 6 6 6 3 2 2 2
033/300 Write block 2944 len 28672 data: 4 4 4 3 3 4 5
034/300 Read block 1752 len 12288
035/300 Write block 1183 len 12288 data: 3 4 5
036/300 Write block 4719 len 36864 data: 5 3 3 3 3 3 3 3 2
037/300 Read block 1133 len 40960
038/300 Write block 1186 len 28672 data: 4 4 3 4 5 5 4
039/300 Write block 1897 len 16384 data: 2 2 3 3
040/300 Write block 2578 len 40960 data: 4 5 5 4 3 3 3 4 6 7
041/300 Read block 3684 len 8192
042/300 Read block 4556 len 32768
043/300 Read block 416 len 36864
044/300 Write block 4154 len 4096 data: 4
045/300 Read block 4859 len 16384
046/300 Write block 4643 len 24576 data: 5 5 4 5 5 4
047/300 Read block 2674 len 8192
048/300 Read block 761 len 40960
049/300 Write block 945 len 40960 data: 3 5 5 4 4 4 3 4 4 4
050/300 Write block 499 len 28672 data: 11 9 10 10 7 6 8
051/300 Flush
052/300 Read block 3010 len 12288
053/300 Flush
054/300 Read block 2711 len 20480
055/300 Write block 1483 len 36864 data: 5 4 2 2 2 2 2 2 3
056/300 Read block 2600 len 4096
057/300 Read block 417 len 36864
058/300 Read block 3496 len 16384
059/300 Read block 3242 len 8192
060/300 Read block 811 len 28672
061/300 Read block 1985 len 28672
062/300 Write block 2028 len 16384 data: 6 6 6 5
063/300 Write block 4531 len 12288 data: 8 8 7
064/300 Write block 2029 len 4096 data: 7
065/300 Flush
066/300 Read block 4198 len 20480
067/300 Write block 1670 len 8192 data: 4 4
068/300 Read block 2521 len 36864
069/300 Read block 2867 len 4096
070/300 Read block 4228 len 40960
071/300 Read block 957 len 32768
072/300 Read block 1214 len 4096
073/300 Write block 3881 len 8192 data: 7 5
074/300 Read block 1827 len 12288
075/300 Flush
076/300 Write block 3716 len 20480 data: 7 7 6 4 6
077/300 Flush
078/300 Read block 2129 len 28672
079/300 Read block 2550 len 20480
080/300 Flush
081/300 Write block 836 len 24576 data: 4 3 3 3 2 2
082/300 Read block 958 len 4096
083/300 Write block 265 len 4096 data: 3
084/300 Write block 2395 len 4096 data: 5
085/300 Read block 2088 len 4096
086/300 Read block 1327 len 12288
087/300 Read block 182 len 28672
088/300 Read block 103 len 12288
089/300 Read block 3588 len 4096
090/300 Read block 988 len 36864
091/300 Flush
092/300 Read block 308 len 36864
093/300 Write block 1616 len 40960 data: 5 5 5 6 6 6 6 5 6 6
094/300 Write block 1937 len 4096 data: 4
095/300 Write block 2596 len 12288 data: 5 4 5
096/300 Write block 3118 len 40960 data: 4 4 4 4 6 8 8 7 6 6
097/300 Read block 1209 len 20480
098/300 Read block 2515 len 28672
099/300 Flush
100/300 Write block 717 len 16384 data: 7 6 7 8
101/300 Read block 4165 len 36864
102/300 Read block 1458 len 4096
103/300 Read block 1368 len 16384
104/300 Flush
105/300 Write block 2804 len 24576 data: 4 3 4 4 4 4
106/300 Write block 1001 len 28672 data: 5 4 6 7 5 4 4
107/300 Read block 1928 len 12288
108/300 Write block 1689 len 12288 data: 3 4 4
109/300 Write block 490 len 4096 data: 3
110/300 Write block 1995 len 24576 data: 4 5 5 6 6 4
111/300 Write block 2692 len 36864 data: 4 4 2 2 2 2 2 2 3
112/300 Flush
113/300 Read block 1928 len 24576
114/300 Read block 1411 len 24576
115/300 Read block 3134 len 20480
116/300 Write block 1274 len 24576 data: 2 3 4 4 3 3
117/300 Write block 3430 len 16384 data: 10 9 8 9
118/300 Read block 250 len 24576
119/300 Flush
120/300 Read block 829 len 20480
121/300 Write block 964 len 32768 data: 6 5 3 4 5 4 4 4
122/300 Flush
123/300 Read block 3138 len 40960
124/300 Write block 1568 len 36864 data: 3 4 4 5 6 7 6 6 6
125/300 Read block 3538 len 12288
126/300 Read block 4125 len 8192
127/300 Read block 3552 len 8192
128/300 Write block 3081 len 12288 data: 5 4 3
129/300 Write block 2750 len 36864 data: 5 7 6 6 5 5 5 3 4
130/300 Read block 3217 len 20480
131/300 Read block 1223 len 20480
132/300 Write block 4340 len 20480 data: 4 4 2 2 3
133/300 Write block 3836 len 28672 data: 5 5 6 6 7 8 9
134/300 Read block 4504 len 16384
135/300 Flush
136/300 Write block 4833 len 28672 data: 5 4 7 7 7 8 7
137/300 Read block 485 len 12288
138/300 Read block 540 len 8192
139/300 Read block 2463 len 8192
140/300 Read block 16 len 4096
141/300 Read block 3474 len 40960
142/300 Write block 4901 len 4096 data: 4
143/300 Read block 3668 len 36864
144/300 Write block 4907 len 32768 data: 5 6 6 6 6 6 6 5
145/300 Write block 1441 len 24576 data: 4 4 3 4 4 5
146/300 Write block 4882 len 24576 data: 5 5 5 4 5 6
147/300 Read block 4722 len 4096
148/300 Read block 3631 len 16384
149/300 Write block 1485 len 12288 data: 3 3 3
150/300 Write block 1365 len 32768 data: 7 7 8 7 7 8 7 5
151/300 Write block 1483 len 24576 data: 6 5 4 4 4 3
152/300 Write block 4131 len 16384 data: 4 4 5 6
153/300 Flush
154/300 Read block 393 len 20480
155/300 Read block 2955 len 16384
156/300 Write block 1038 len 20480 data: 4 4 4 4 5
157/300 Write block 1415 len 40960 data: 6 6 7 5 4 4 4 5 5 5
158/300 Flush
159/300 Write block 3026 len 4096 data: 5
160/300 Read block 4977 len 28672
161/300 Write block 3461 len 16384 data: 4 5 4 5
162/300 Write block 691 len 12288 data: 5 4 4
163/300 Write block 1815 len 4096 data: 10
164/300 Read block 955 len 4096
165/300 Read block 1028 len 24576
166/300 Read block 838 len 32768
167/300 Write block 2103 len 32768 data: 3 4 5 5 5 5 5 4
168/300 Read block 1533 len 16384
169/300 Read block 4679 len 24576
170/300 Flush
171/300 Read block 1267 len 8192
172/300 Write block 3087 len 36864 data: 3 4 3 4 5 5 5 5 5
173/300 Read block 1971 len 20480
174/300 Write block 91 len 20480 data: 5 5 4 5 5
175/300 Read block 1325 len 12288
176/300 Read block 3667 len 28672
177/300 Write block 4647 len 32768 data: 6 5 5 3 4 4 4 4
178/300 Read block 1888 len 36864
179/300 Write block 4186 len 40960 data: 3 3 4 6 9 9 9 8 8 8
180/300 Write block 1593 len 16384 data: 7 7 7 9
181/300 Read block 4540 len 32768
182/300 Read block 837 len 20480
183/300 Read block 4455 len 4096
184/300 Write block 4751 len 36864 data: 4 4 5 5 6 5 2 2 3
185/300 Read block 226 len 28672
186/300 Read block 946 len 28672
187/300 Read block 79 len 4096
188/300 Read block 2730 len 36864
189/300 Write block 1097 len 8192 data: 6 7
190/300 Flush
191/300 Write block 2471 len 40960 data: 3 4 5 5 6 6 5 3 4 4
192/300 Write block 4560 len 4096 data: 5
193/300 Write block 4957 len 32768 data: 4 5 6 7 7 8 5 5
194/300 Read block 3540 len 36864
195/300 Read block 2025 len 16384
196/300 Read block 1330 len 24576
197/300 Read block 806 len 32768
198/300 Write block 2880 len 12288 data: 2 2 2
199/300 Write block 49 len 28672 data: 8 8 8 7 5 7 5
200/300 Write block 372 len 4096 data: 7
201/300 Read block 2556 len 32768
202/300 Read block 1596 len 24576
203/300 Read block 2504 len 28672
204/300 Write block 3978 len 28672 data: 3 3 3 3 3 3 3
205/300 Flush
206/300 Write block 4220 len 4096 data: 4
207/300 Read block 3304 len 32768
208/300 Read block 516 len 28672
209/300 Read block 4969 len 24576
210/300 Write block 2037 len 20480 data: 5 5 4 4 5
211/300 Flush
212/300 Read block 4409 len 8192
213/300 Flush
214/300 Read block 1601 len 28672
215/300 Write block 4455 len 12288 data: 4 2 3
216/300 Flush
217/300 Read block 14 len 8192
218/300 Read block 830 len 12288
219/300 Read block 4905 len 4096
220/300 Read block 3107 len 4096
221/300 Write block 2321 len 28672 data: 3 3 3 3 2 2 2
222/300 Read block 1363 len 40960
223/300 Write block 1329 len 40960 data: 6 5 4 5 6 5 5 5 5 6
224/300 Read block 1693 len 16384
225/300 Read block 4388 len 8192
226/300 Write block 288 len 24576 data: 7 6 4 4 4 5
227/300 Read block 3246 len 40960
228/300 Read block 83 len 40960
229/300 Read block 4730 len 8192
230/300 Flush
231/300 Write block 3056 len 20480 data: 6 5 5 5 6
232/300 Read block 1088 len 24576
233/300 Read block 3423 len 24576
234/300 Read block 1976 len 36864
235/300 Write block 3845 len 32768 data: 6 6 5 5 5 4 4 3
236/300 Read block 3111 len 36864
237/300 Flush
238/300 Write block 3000 len 28672 data: 6 6 8 8 8 7 7
239/300 Read block 2506 len 20480
240/300 Read block 762 len 8192
241/300 Read block 4347 len 24576
242/300 Read block 1131 len 16384
243/300 Read block 4039 len 28672
244/300 Write block 569 len 4096 data: 4
245/300 Write block 279 len 40960 data: 9 11 12 10 8 7 7 8 7 8
246/300 Write block 1154 len 24576 data: 2 2 2 2 3 3
247/300 Read block 1621 len 24576
248/300 Write block 1636 len 4096 data: 5
249/300 Read block 924 len 20480
250/300 Read block 1277 len 32768
251/300 Write block 3692 len 28672 data: 7 6 5 4 5 5 3
252/300 Read block 4691 len 36864
253/300 Write block 721 len 4096 data: 7
254/300 Write block 2700 len 40960 data: 4 4 5 5 4 3 3 3 3 5
255/300 Flush
256/300 Write block 2688 len 36864 data: 7 4 3 3 5 5 3 3 3
257/300 Read block 3175 len 36864
258/300 Read block 3729 len 32768
259/300 Write block 2920 len 28672 data: 4 3 2 2 2 4 5
260/300 Write block 567 len 40960 data: 3 4 5 4 4 3 4 4 5 5
261/300 Write block 2430 len 32768 data: 5 4 3 3 2 2 4 7
262/300 Read block 2915 len 20480
263/300 Read block 1117 len 40960
264/300 Read block 1513 len 8192
265/300 Read block 2013 len 36864
266/300 Write block 2816 len 8192 data: 4 2
267/300 Read block 4544 len 32768
268/300 Read block 3380 len 36864
269/300 Read block 2444 len 16384
270/300 Read block 1116 len 4096
271/300 Write block 4018 len 24576 data: 4 4 4 4 3 4
272/300 Write block 4958 len 4096 data: 6
273/300 Write block 4404 len 28672 data: 4 5 5 6 6 5 5
274/300 Read block 4294 len 28672
275/300 Write block 3184 len 28672 data: 5 2 3 4 4 4 5
276/300 Flush
277/300 Write block 786 len 8192 data: 4 4
278/300 Write block 726 len 36864 data: 4 3 3 3 3 2 2 2 2
279/300 Read block 2161 len 8192
280/300 Read block 448 len 20480
281/300 Write block 2075 len 36864 data: 4 6 6 6 6 5 4 4 4
282/300 Write block 1614 len 28672 data: 5 5 6 6 6 7 7
283/300 Read block 3150 len 12288
284/300 Read block 4408 len 32768
285/300 Write block 3394 len 16384 data: 3 3 4 7
286/300 Read block 246 len 4096
287/300 Read block 559 len 24576
288/300 Read block 1994 len 36864
289/300 Read block 4104 len 40960
290/300 Write block 2509 len 36864 data: 6 4 5 6 5 6 6 6 5
291/300 Read block 2258 len 28672
292/300 Write block 3360 len 8192 data: 6 7
293/300 Read block 4107 len 28672
294/300 Write block 4989 len 20480 data: 7 6 6 6 6
295/300 Read block 2879 len 20480
296/300 Flush
297/300 Read block 1338 len 32768
298/300 Flush
299/300 Write block 4388 len 40960 data: 4 3 3 4 4 4 4 3 3 3
300/300 Read block 4840 len 20480
301/300 Read block 2713 len 36864
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:41:58.855 INFO request completed, latency_us: 258, response_code: 204, uri: /start/cid/0, method: POST, req_id: 080a6df0-0671-4c32-ab95-6d70d89e594b, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7942 Acked Flush 0 Sent Done Done false
7943 Acked Read 8 Sent Done Done false
7944 Acked Write 2 Sent Done Done false
7945 Acked Read 6 Sent Done Done false
7946 Acked Read 8 Sent Done Done false
7947 Acked Write 6 Sent Done Done false
7948 Acked Read 4 Sent Done Done false
7949 Acked Write 3 Sent Done Done false
7950 Acked Read 2 Sent Done Done false
7951 Acked Write 7 Sent Done Done false
7952 Acked Read 4 Sent Done Done false
7953 Acked Read 6 Sent Done Done false
7954 Acked Read 4 Sent Done Done false
7955 Acked Write 1 Sent Done Done false
7956 Acked Read 5 Sent Done Done false
7957 Acked Read 10 Sent Done Done false
7958 Acked Read 9 Sent Done Done false
7959 Acked Write 1 Sent Done Done false
7960 Acked Write 4 Sent Done Done false
7961 Acked Flush 0 Sent Done Done false
7962 Acked Write 2 Sent Done Done false
7963 Acked Flush 0 Sent Done Done false
7964 Acked Write 2 Sent Done Done false
7965 Acked Read 9 Sent Done Done false
7966 Acked Write 1 Sent Done Done false
7967 Acked Write 3 Sent Done Done false
7968 Acked Write 5 Sent Done Done false
7969 Acked Write 5 Sent Done Done false
7970 Acked Flush 0 Sent Done Done false
7971 Acked Write 1 Sent Done Done false
7972 Acked Read 7 Sent Done Done false
7973 Acked Write 9 Sent Done Done false
7974 Acked Write 7 Sent Done Done false
7975 Acked Read 3 Sent Done Done false
7976 Acked Write 3 Sent Done Done false
7977 Acked Write 9 Sent Done Done false
7978 Acked Read 10 Sent Done Done false
7979 Acked Write 7 Sent Done Done false
7980 Acked Write 4 Sent Done Done false
7981 Acked Write 10 Sent Done Done false
7982 Acked Read 2 Sent Done Done false
7983 Acked Read 8 Sent Done Done false
7984 Acked Read 9 Sent Done Done false
7985 Acked Write 1 Sent Done Done false
7986 Acked Read 4 Sent Done Done false
7987 Acked Write 6 Sent Done Done false
7988 Acked Read 2 Sent Done Done false
7989 Acked Read 10 Sent Done Done false
7990 Acked Write 10 Sent Done Done false
7991 Acked Write 7 Sent Done Done false
7992 Acked Flush 0 Sent Done Done false
7993 Acked Read 3 Sent Done Done false
7994 Acked Flush 0 Sent Done Done false
7995 Acked Read 5 Sent Done Done false
7996 Acked Write 9 Sent Done Done false
7997 Acked Read 1 Sent Done Done false
7998 Acked Read 9 Sent Done Done false
7999 Acked Read 4 Sent Done Done false
8000 Acked Read 2 Sent Done Done false
8001 Acked Read 7 Sent Done Done false
8002 Acked Read 7 Sent Done Done false
8003 Acked Write 4 Sent Done Done false
8004 Acked Write 3 Sent Done Done false
8005 Acked Write 1 Sent Done Done false
8006 Acked Flush 0 Sent Done Done false
8007 Acked Read 5 Sent Done Done false
8008 Acked Write 2 Sent Done Done false
8009 Acked Read 9 Sent Done Done false
8010 Acked Read 1 Sent Done Done false
8011 Acked Read 10 Sent Done Done false
8012 Acked Read 8 Sent Done Done false
8013 Acked Read 1 Sent Done Done false
8014 Acked Write 2 Sent Done Done false
8015 Acked Read 3 Sent Done Done false
8016 Acked Flush 0 Sent Done Done false
8017 Acked Write 5 Sent Done Done false
8018 Acked Flush 0 Sent Done Done false
8019 Acked Read 7 Sent Done Done false
8020 Acked Read 5 Sent Done Done false
8021 Acked Flush 0 Sent Done Done false
8022 Acked Write 6 Sent Done Done false
8023 Acked Read 1 Sent Done Done false
8024 Acked Write 1 Sent Done Done false
8025 Acked Write 1 Sent Done Done false
8026 Acked Read 1 Sent Done Done false
8027 Acked Read 3 Sent Done Done false
8028 Acked Read 7 Sent Done Done false
8029 Acked Read 3 Sent Done Done false
8030 Acked Read 1 Sent Done Done false
8031 Acked Read 9 Sent Done Done false
8032 Acked Flush 0 Sent Done Done false
8033 Acked Read 9 Sent Done Done false
8034 Acked Write 10 Sent Done Done false
8035 Acked Write 1 Sent Done Done false
8036 Acked Write 3 Sent Done Done false
8037 Acked Write 10 Sent Done Done false
8038 Acked Read 5 Sent Done Done false
8039 Acked Read 7 Sent Done Done false
8040 Acked Flush 0 Sent Done Done false
8041 Acked Write 4 Sent Done Done false
8042 Acked Read 9 Sent Done Done false
8043 Acked Read 1 Sent Done Done false
8044 Acked Read 4 Sent Done Done false
8045 Acked Flush 0 Sent Done Done false
8046 Acked Write 6 Sent Done Done false
8047 Acked Write 7 Sent Done Done false
8048 Acked Read 3 Sent Done Done false
8049 Acked Write 3 Sent Done Done false
8050 Acked Write 1 Sent Done Done false
8051 Acked Write 6 Sent Done Done false
8052 Acked Write 9 Sent Done Done false
8053 Acked Flush 0 Sent Done Done false
8054 Acked Read 6 Sent Done Done false
8055 Acked Read 6 Sent Done Done false
8056 Acked Read 5 Sent Done Done false
8057 Acked Write 6 Sent Done Done false
8058 Acked Write 4 Sent Done Done false
8059 Acked Read 6 Sent Done Done false
8060 Acked Flush 0 Sent Done Done false
8061 Acked Read 5 Sent Done Done false
8062 Acked Write 8 Sent Done Done false
8063 Acked Flush 0 Sent Done Done false
8064 Acked Read 10 Sent Done Done false
8065 Acked Write 9 Sent Done Done false
8066 Acked Read 3 Sent Done Done false
8067 Acked Read 2 Sent Done Done false
8068 Acked Read 2 Sent Done Done false
8069 Acked Write 3 Sent Done Done false
8070 Acked Write 9 Sent Done Done false
8071 Acked Read 5 Sent Done Done false
8072 Acked Read 5 Sent Done Done false
8073 Acked Write 5 Sent Done Done false
8074 Acked Write 7 Sent Done Done false
8075 Acked Read 4 Sent Done Done false
8076 Acked Flush 0 Sent Done Done false
8077 Acked Write 7 Sent Done Done false
8078 Acked Read 3 Sent Done Done false
8079 Acked Read 2 Sent Done Done false
8080 Acked Read 2 Sent Done Done false
8081 Acked Read 1 Sent Done Done false
8082 Acked Read 10 Sent Done Done false
8083 Acked Write 1 Sent Done Done false
8084 Acked Read 9 Sent Done Done false
8085 Acked Write 8 Sent Done Done false
8086 Acked Write 6 Sent Done Done false
8087 Acked Write 6 Sent Done Done false
8088 Acked Read 1 Sent Done Done false
8089 Acked Read 4 Sent Done Done false
8090 Acked Write 3 Sent Done Done false
8091 Acked Write 8 Sent Done Done false
8092 Acked Write 6 Sent Done Done false
8093 Acked Write 4 Sent Done Done false
8094 Acked Flush 0 Sent Done Done false
8095 Acked Read 5 Sent Done Done false
8096 Acked Read 4 Sent Done Done false
8097 Acked Write 5 Sent Done Done false
8098 Acked Write 10 Sent Done Done false
8099 Acked Flush 0 Sent Done Done false
8100 Acked Write 1 Sent Done Done false
8101 Acked Read 7 Sent Done Done false
8102 Acked Write 4 Sent Done Done false
8103 Acked Write 3 Sent Done Done false
8104 Acked Write 1 Sent Done Done false
8105 Acked Read 1 Sent Done Done false
8106 Acked Read 6 Sent Done Done false
8107 Acked Read 8 Sent Done Done false
8108 Acked Write 8 Sent Done Done false
8109 Acked Read 4 Sent Done Done false
8110 Acked Read 6 Sent Done Done false
8111 Acked Flush 0 Sent Done Done false
8112 Acked Read 2 Sent Done Done false
8113 Acked Write 9 Sent Done Done false
8114 Acked Read 5 Sent Done Done false
8115 Acked Write 5 Sent Done Done false
8116 Acked Read 3 Sent Done Done false
8117 Acked Read 7 Sent Done Done false
8118 Acked Write 8 Sent Done Done false
8119 Acked Read 9 Sent Done Done false
8120 Acked Write 10 Sent Done Done false
8121 Acked Write 4 Sent Done Done false
8122 Acked Read 8 Sent Done Done false
8123 Acked Read 5 Sent Done Done false
8124 Acked Read 1 Sent Done Done false
8125 Acked Write 9 Sent Done Done false
8126 Acked Read 7 Sent Done Done false
8127 Acked Read 7 Sent Done Done false
8128 Acked Read 1 Sent Done Done false
8129 Acked Read 9 Sent Done Done false
8130 Acked Write 2 Sent Done Done false
8131 Acked Flush 0 Sent Done Done false
8132 Acked Write 10 Sent Done Done false
8133 Acked Write 1 Sent Done Done false
8134 Acked Write 8 Sent Done Done false
8135 Acked Read 9 Sent Done Done false
8136 Acked Read 4 Sent Done Done false
8137 Acked Read 6 Sent Done Done false
8138 Acked Read 8 Sent Done Done false
8139 Acked Write 3 Sent Done Done false
8140 Acked Write 7 Sent Done Done false
8141 Acked Write 1 Sent Done Done false
8142 Acked Read 8 Sent Done Done false
8143 Acked Read 6 Sent Done Done false
8144 Acked Read 7 Sent Done Done false
8145 Acked Write 7 Sent Done Done false
8146 Acked Flush 0 Sent Done Done false
8147 Acked Write 1 Sent Done Done false
8148 Acked Read 8 Sent Done Done false
8149 Acked Read 7 Sent Done Done false
8150 Acked Read 6 Sent Done Done false
8151 Acked Write 5 Sent Done Done false
8152 Acked Flush 0 Sent Done Done false
8153 Acked Read 2 Sent Done Done false
8154 Acked Flush 0 Sent Done Done false
8155 Acked Read 7 Sent Done Done false
8156 Acked Write 3 Sent Done Done false
8157 Acked Flush 0 Sent Done Done false
8158 Acked Read 2 Sent Done Done false
8159 Acked Read 3 Sent Done Done false
8160 Acked Read 1 Sent Done Done false
8161 Acked Read 1 Sent Done Done false
8162 Acked Write 7 Sent Done Done false
8163 Acked Read 10 Sent Done Done false
8164 Acked Write 10 Sent Done Done false
8165 Acked Read 4 Sent Done Done false
8166 Acked Read 2 Sent Done Done false
8167 Acked Write 6 Sent Done Done false
8168 Acked Read 10 Sent Done Done false
8169 Acked Read 10 Sent Done Done false
8170 Acked Read 2 Sent Done Done false
8171 Acked Flush 0 Sent Done Done false
8172 Acked Write 5 Sent Done Done false
8173 Acked Read 6 Sent Done Done false
8174 Acked Read 6 Sent Done Done false
8175 Acked Read 9 Sent Done Done false
8176 Acked Write 8 Sent Done Done false
8177 Acked Read 9 Sent Done Done false
8178 Acked Flush 0 Sent Done Done false
8179 Acked Write 7 Sent Done Done false
8180 Acked Read 5 Sent Done Done false
8181 Acked Read 2 Sent Done Done false
8182 Acked Read 6 Sent Done Done false
8183 Acked Read 4 Sent Done Done false
8184 Acked Read 7 Sent Done Done false
8185 Acked Write 1 Sent Done Done false
8186 Acked Write 10 Sent Done Done false
8187 Acked Write 6 Sent Done Done false
8188 Acked Read 6 Sent Done Done false
8189 Acked Write 1 Sent Done Done false
8190 Acked Read 5 Sent Done Done false
8191 Acked Read 8 Sent Done Done false
8192 Acked Write 7 Sent Done Done false
8193 Acked Read 9 Sent Done Done false
8194 Acked Write 1 Sent Done Done false
8195 Acked Write 10 Sent Done Done false
8196 Acked Flush 0 Sent Done Done false
8197 Acked Write 9 Sent Done Done false
8198 Acked Read 9 Sent Done Done false
8199 Acked Read 8 Sent Done Done false
8200 Acked Write 7 Sent Done Done false
8201 Acked Write 10 Sent Done Done false
8202 Acked Write 8 Sent Done Done false
8203 Acked Read 5 Sent Done Done false
8204 Acked Read 10 Sent Done Done false
8205 Acked Read 2 Sent Done Done false
8206 Acked Read 9 Sent Done Done false
8207 Acked Write 2 Sent Done Done false
8208 Acked Read 8 Sent Done Done false
8209 Acked Read 9 Sent Done Done false
8210 Acked Read 4 Sent Done Done false
8211 Acked Read 1 Sent Done Done false
8212 Acked Write 6 Sent Done Done false
8213 Acked Write 1 Sent Done Done false
8214 Acked Write 7 Sent Done Done false
8215 Acked Read 7 Sent Done Done false
8216 Acked Write 7 Sent Done Done false
8217 Acked Flush 0 Sent Done Done false
8218 Acked Write 2 Sent Done Done false
8219 Acked Write 9 Sent Done Done false
8220 Acked Read 2 Sent Done Done false
8221 Acked Read 5 Sent Done Done false
8222 Acked Write 9 Sent Done Done false
8223 Acked Write 7 Sent Done Done false
8224 Acked Read 3 Sent Done Done false
8225 Acked Read 8 Sent Done Done false
8226 Acked Write 4 Sent Done Done false
8227 Acked Read 1 Sent Done Done false
8228 Acked Read 6 Sent Done Done false
8229 Acked Read 9 Sent Done Done false
8230 Acked Read 10 Sent Done Done false
8231 Acked Write 9 Sent Done Done false
8232 Acked Read 7 Sent Done Done false
8233 Acked Write 2 Sent Done Done false
8234 Acked Read 7 Sent Done Done false
8235 Acked Write 5 Sent Done Done false
8236 Acked Read 5 Sent Done Done false
8237 Acked Flush 0 Sent Done Done false
8238 Acked Read 8 Sent Done Done false
8239 Acked Flush 0 Sent Done Done false
8240 Acked Write 10 Sent Done Done false
8241 Acked Read 5 Sent Done Done false
8242 Acked Read 9 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7941 8239 8239
Downstairs last five completed: 7941 7940 7939 7938 7937
Upstairs last five completed: 8242 8241 8240 8239 8238
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7942 Acked Flush 0 Sent Done Done false
7943 Acked Read 8 Sent Done Done false
7944 Acked Write 2 Sent Done Done false
7945 Acked Read 6 Sent Done Done false
7946 Acked Read 8 Sent Done Done false
7947 Acked Write 6 Sent Done Done false
7948 Acked Read 4 Sent Done Done false
7949 Acked Write 3 Sent Done Done false
7950 Acked Read 2 Sent Done Done false
7951 Acked Write 7 Sent Done Done false
7952 Acked Read 4 Sent Done Done false
7953 Acked Read 6 Sent Done Done false
7954 Acked Read 4 Sent Done Done false
7955 Acked Write 1 Sent Done Done false
7956 Acked Read 5 Sent Done Done false
7957 Acked Read 10 Sent Done Done false
7958 Acked Read 9 Sent Done Done false
7959 Acked Write 1 Sent Done Done false
7960 Acked Write 4 Sent Done Done false
7961 Acked Flush 0 Sent Done Done false
7962 Acked Write 2 Sent Done Done false
7963 Acked Flush 0 Sent Done Done false
7964 Acked Write 2 Sent Done Done false
7965 Acked Read 9 Sent Done Done false
7966 Acked Write 1 Sent Done Done false
7967 Acked Write 3 Sent Done Done false
7968 Acked Write 5 Sent Done Done false
7969 Acked Write 5 Sent Done Done false
7970 Acked Flush 0 Sent Done Done false
7971 Acked Write 1 Sent Done Done false
7972 Acked Read 7 Sent Done Done false
7973 Acked Write 9 Sent Done Done false
7974 Acked Write 7 Sent Done Done false
7975 Acked Read 3 Sent Done Done false
7976 Acked Write 3 Sent Done Done false
7977 Acked Write 9 Sent Done Done false
7978 Acked Read 10 Sent Done Done false
7979 Acked Write 7 Sent Done Done false
7980 Acked Write 4 Sent Done Done false
7981 Acked Write 10 Sent Done Done false
7982 Acked Read 2 Sent Done Done false
7983 Acked Read 8 Sent Done Done false
7984 Acked Read 9 Sent Done Done false
7985 Acked Write 1 Sent Done Done false
7986 Acked Read 4 Sent Done Done false
7987 Acked Write 6 Sent Done Done false
7988 Acked Read 2 Sent Done Done false
7989 Acked Read 10 Sent Done Done false
7990 Acked Write 10 Sent Done Done false
7991 Acked Write 7 Sent Done Done false
7992 Acked Flush 0 Sent Done Done false
7993 Acked Read 3 Sent Done Done false
7994 Acked Flush 0 Sent Done Done false
7995 Acked Read 5 Sent Done Done false
7996 Acked Write 9 Sent Done Done false
7997 Acked Read 1 Sent Done Done false
7998 Acked Read 9 Sent Done Done false
7999 Acked Read 4 Sent Done Done false
8000 Acked Read 2 Sent Done Done false
8001 Acked Read 7 Sent Done Done false
8002 Acked Read 7 Sent Done Done false
8003 Acked Write 4 Sent Done Done false
8004 Acked Write 3 Sent Done Done false
8005 Acked Write 1 Sent Done Done false
8006 Acked Flush 0 Sent Done Done false
8007 Acked Read 5 Sent Done Done false
8008 Acked Write 2 Sent Done Done false
8009 Acked Read 9 Sent Done Done false
8010 Acked Read 1 Sent Done Done false
8011 Acked Read 10 Sent Done Done false
8012 Acked Read 8 Sent Done Done false
8013 Acked Read 1 Sent Done Done false
8014 Acked Write 2 Sent Done Done false
8015 Acked Read 3 Sent Done Done false
8016 Acked Flush 0 Sent Done Done false
8017 Acked Write 5 Sent Done Done false
8018 Acked Flush 0 Sent Done Done false
8019 Acked Read 7 Sent Done Done false
8020 Acked Read 5 Sent Done Done false
8021 Acked Flush 0 Sent Done Done false
8022 Acked Write 6 Sent Done Done false
8023 Acked Read 1 Sent Done Done false
8024 Acked Write 1 Sent Done Done false
8025 Acked Write 1 Sent Done Done false
8026 Acked Read 1 Sent Done Done false
8027 Acked Read 3 Sent Done Done false
8028 Acked Read 7 Sent Done Done false
8029 Acked Read 3 Sent Done Done false
8030 Acked Read 1 Sent Done Done false
8031 Acked Read 9 Sent Done Done false
8032 Acked Flush 0 Sent Done Done false
8033 Acked Read 9 Sent Done Done false
8034 Acked Write 10 Sent Done Done false
8035 Acked Write 1 Sent Done Done false
8036 Acked Write 3 Sent Done Done false
8037 Acked Write 10 Sent Done Done false
8038 Acked Read 5 Sent Done Done false
8039 Acked Read 7 Sent Done Done false
8040 Acked Flush 0 Sent Done Done false
8041 Acked Write 4 Sent Done Done false
8042 Acked Read 9 Sent Done Done false
8043 Acked Read 1 Sent Done Done false
8044 Acked Read 4 Sent Done Done false
8045 Acked Flush 0 Sent Done Done false
8046 Acked Write 6 Sent Done Done false
8047 Acked Write 7 Sent Done Done false
8048 Acked Read 3 Sent Done Done false
8049 Acked Write 3 Sent Done Done false
8050 Acked Write 1 Sent Done Done false
8051 Acked Write 6 Sent Done Done false
8052 Acked Write 9 Sent Done Done false
8053 Acked Flush 0 Sent Done Done false
8054 Acked Read 6 Sent Done Done false
8055 Acked Read 6 Sent Done Done false
8056 Acked Read 5 Sent Done Done false
8057 Acked Write 6 Sent Done Done false
8058 Acked Write 4 Sent Done Done false
8059 Acked Read 6 Sent Done Done false
8060 Acked Flush 0 Sent Done Done false
8061 Acked Read 5 Sent Done Done false
8062 Acked Write 8 Sent Done Done false
8063 Acked Flush 0 Sent Done Done false
8064 Acked Read 10 Sent Done Done false
8065 Acked Write 9 Sent Done Done false
8066 Acked Read 3 Sent Done Done false
8067 Acked Read 2 Sent Done Done false
8068 Acked Read 2 Sent Done Done false
8069 Acked Write 3 Sent Done Done false
8070 Acked Write 9 Sent Done Done false
8071 Acked Read 5 Sent Done Done false
8072 Acked Read 5 Sent Done Done false
8073 Acked Write 5 Sent Done Done false
8074 Acked Write 7 Sent Done Done false
8075 Acked Read 4 Sent Done Done false
8076 Acked Flush 0 Sent Done Done false
8077 Acked Write 7 Sent Done Done false
8078 Acked Read 3 Sent Done Done false
8079 Acked Read 2 Sent Done Done false
8080 Acked Read 2 Sent Done Done false
8081 Acked Read 1 Sent Done Done false
8082 Acked Read 10 Sent Done Done false
8083 Acked Write 1 Sent Done Done false
8084 Acked Read 9 Sent Done Done false
8085 Acked Write 8 Sent Done Done false
8086 Acked Write 6 Sent Done Done false
8087 Acked Write 6 Sent Done Done false
8088 Acked Read 1 Sent Done Done false
8089 Acked Read 4 Sent Done Done false
8090 Acked Write 3 Sent Done Done false
8091 Acked Write 8 Sent Done Done false
8092 Acked Write 6 Sent Done Done false
8093 Acked Write 4 Sent Done Done false
8094 Acked Flush 0 Sent Done Done false
8095 Acked Read 5 Sent Done Done false
8096 Acked Read 4 Sent Done Done false
8097 Acked Write 5 Sent Done Done false
8098 Acked Write 10 Sent Done Done false
8099 Acked Flush 0 Sent Done Done false
8100 Acked Write 1 Sent Done Done false
8101 Acked Read 7 Sent Done Done false
8102 Acked Write 4 Sent Done Done false
8103 Acked Write 3 Sent Done Done false
8104 Acked Write 1 Sent Done Done false
8105 Acked Read 1 Sent Done Done false
8106 Acked Read 6 Sent Done Done false
8107 Acked Read 8 Sent Done Done false
8108 Acked Write 8 Sent Done Done false
8109 Acked Read 4 Sent Done Done false
8110 Acked Read 6 Sent Done Done false
8111 Acked Flush 0 Sent Done Done false
8112 Acked Read 2 Sent Done Done false
8113 Acked Write 9 Sent Done Done false
8114 Acked Read 5 Sent Done Done false
8115 Acked Write 5 Sent Done Done false
8116 Acked Read 3 Sent Done Done false
8117 Acked Read 7 Sent Done Done false
8118 Acked Write 8 Sent Done Done false
8119 Acked Read 9 Sent Done Done false
8120 Acked Write 10 Sent Done Done false
8121 Acked Write 4 Sent Done Done false
8122 Acked Read 8 Sent Done Done false
8123 Acked Read 5 Sent Done Done false
8124 Acked Read 1 Sent Done Done false
8125 Acked Write 9 Sent Done Done false
8126 Acked Read 7 Sent Done Done false
8127 Acked Read 7 Sent Done Done false
8128 Acked Read 1 Sent Done Done false
8129 Acked Read 9 Sent Done Done false
8130 Acked Write 2 Sent Done Done false
8131 Acked Flush 0 Sent Done Done false
8132 Acked Write 10 Sent Done Done false
8133 Acked Write 1 Sent Done Done false
8134 Acked Write 8 Sent Done Done false
8135 Acked Read 9 Sent Done Done false
8136 Acked Read 4 Sent Done Done false
8137 Acked Read 6 Sent Done Done false
8138 Acked Read 8 Sent Done Done false
8139 Acked Write 3 Sent Done Done false
8140 Acked Write 7 Sent Done Done false
8141 Acked Write 1 Sent Done Done false
8142 Acked Read 8 Sent Done Done false
8143 Acked Read 6 Sent Done Done false
8144 Acked Read 7 Sent Done Done false
8145 Acked Write 7 Sent Done Done false
8146 Acked Flush 0 Sent Done Done false
8147 Acked Write 1 Sent Done Done false
8148 Acked Read 8 Sent Done Done false
8149 Acked Read 7 Sent Done Done false
8150 Acked Read 6 Sent Done Done false
8151 Acked Write 5 Sent Done Done false
8152 Acked Flush 0 Sent Done Done false
8153 Acked Read 2 Sent Done Done false
8154 Acked Flush 0 Sent Done Done false
8155 Acked Read 7 Sent Done Done false
8156 Acked Write 3 Sent Done Done false
8157 Acked Flush 0 Sent Done Done false
8158 Acked Read 2 Sent Done Done false
8159 Acked Read 3 Sent Done Done false
8160 Acked Read 1 Sent Done Done false
8161 Acked Read 1 Sent Done Done false
8162 Acked Write 7 Sent Done Done false
8163 Acked Read 10 Sent Done Done false
8164 Acked Write 10 Sent Done Done false
8165 Acked Read 4 Sent Done Done false
8166 Acked Read 2 Sent Done Done false
8167 Acked Write 6 Sent Done Done false
8168 Acked Read 10 Sent Done Done false
8169 Acked Read 10 Sent Done Done false
8170 Acked Read 2 Sent Done Done false
8171 Acked Flush 0 Sent Done Done false
8172 Acked Write 5 Sent Done Done false
8173 Acked Read 6 Sent Done Done false
8174 Acked Read 6 Sent Done Done false
8175 Acked Read 9 Sent Done Done false
8176 Acked Write 8 Sent Done Done false
8177 Acked Read 9 Sent Done Done false
8178 Acked Flush 0 Sent Done Done false
8179 Acked Write 7 Sent Done Done false
8180 Acked Read 5 Sent Done Done false
8181 Acked Read 2 Sent Done Done false
8182 Acked Read 6 Sent Done Done false
8183 Acked Read 4 Sent Done Done false
8184 Acked Read 7 Sent Done Done false
8185 Acked Write 1 Sent Done Done false
8186 Acked Write 10 Sent Done Done false
8187 Acked Write 6 Sent Done Done false
8188 Acked Read 6 Sent Done Done false
8189 Acked Write 1 Sent Done Done false
8190 Acked Read 5 Sent Done Done false
8191 Acked Read 8 Sent Done Done false
8192 Acked Write 7 Sent Done Done false
8193 Acked Read 9 Sent Done Done false
8194 Acked Write 1 Sent Done Done false
8195 Acked Write 10 Sent Done Done false
8196 Acked Flush 0 Sent Done Done false
8197 Acked Write 9 Sent Done Done false
8198 Acked Read 9 Sent Done Done false
8199 Acked Read 8 Sent Done Done false
8200 Acked Write 7 Sent Done Done false
8201 Acked Write 10 Sent Done Done false
8202 Acked Write 8 Sent Done Done false
8203 Acked Read 5 Sent Done Done false
8204 Acked Read 10 Sent Done Done false
8205 Acked Read 2 Sent Done Done false
8206 Acked Read 9 Sent Done Done false
8207 Acked Write 2 Sent Done Done false
8208 Acked Read 8 Sent Done Done false
8209 Acked Read 9 Sent Done Done false
8210 Acked Read 4 Sent Done Done false
8211 Acked Read 1 Sent Done Done false
8212 Acked Write 6 Sent Done Done false
8213 Acked Write 1 Sent Done Done false
8214 Acked Write 7 Sent Done Done false
8215 Acked Read 7 Sent Done Done false
8216 Acked Write 7 Sent Done Done false
8217 Acked Flush 0 Sent Done Done false
8218 Acked Write 2 Sent Done Done false
8219 Acked Write 9 Sent Done Done false
8220 Acked Read 2 Sent Done Done false
8221 Acked Read 5 Sent Done Done false
8222 Acked Write 9 Sent Done Done false
8223 Acked Write 7 Sent Done Done false
8224 Acked Read 3 Sent Done Done false
8225 Acked Read 8 Sent Done Done false
8226 Acked Write 4 Sent Done Done false
8227 Acked Read 1 Sent Done Done false
8228 Acked Read 6 Sent Done Done false
8229 Acked Read 9 Sent Done Done false
8230 Acked Read 10 Sent Done Done false
8231 Acked Write 9 Sent Done Done false
8232 Acked Read 7 Sent Done Done false
8233 Acked Write 2 Sent Done Done false
8234 Acked Read 7 Sent Done Done false
8235 Acked Write 5 Sent Done Done false
8236 Acked Read 5 Sent Done Done false
8237 Acked Flush 0 Sent Done Done false
8238 Acked Read 8 Sent Done Done false
8239 Acked Flush 0 Sent Done Done false
8240 Acked Write 10 Sent Done Done false
8241 Acked Read 5 Sent Done Done false
8242 Acked Read 9 Sent Done Done false
8243 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7941 8243 8243
Downstairs last five completed: 7941 7940 7939 7938 7937
Upstairs last five completed: 8243 8242 8241 8240 8239
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.614761233Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57799)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.614978267Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.615992619Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.616644433Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.616693358Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 7941","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.616729783Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 7941","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.616979204Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 7941","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.617020369Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 7941","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.617261911Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8243 8242 8241 8240 8239
Upstairs last five completed: 8243 8242 8241 8240 8239
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Jan 27 19:42:06.860 INFO request completed, latency_us: 355, response_code: 204, uri:[8820] Got stop action so:false kr:false
/stop/cid/1, method: POST, req_id: c0d1a9da-a30b-4c0c-b513-e30c4e201253, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:06.861544673Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:06.861609015Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:06.861629693Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:42:06.861 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/1, method: GET, req_id: 17f1db11-25e3-4515-84c1-d3bc0db27315, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:06.861728212Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:42:10.863 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3763f3c0-4108-4c4a-882d-bc2596a448d3, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 1490 len 20480 data: 3 4 3 3 3
002/300 Read block 4265 len 32768
003/300 Write block 98 len 36864 data: 4 4 4 4 5 4 4 3 3
004/300 Read block 1960 len 24576
005/300 Write block 892 len 20480 data: 7 5 4 4 3
006/300 Write block 1860 len 28672 data: 9 7 7 6 5 6 7
007/300 Read block 68 len 16384
008/300 Write block 378 len 32768 data: 4 4 4 3 3 3 3 3
009/300 Write block 246 len 16384 data: 4 3 3 4
010/300 Read block 2249 len 36864
011/300 Read block 3160 len 28672
012/300 Write block 2123 len 8192 data: 2 4
013/300 Write block 3688 len 8192 data: 7 7
014/300 Read block 1173 len 28672
015/300 Read block 4543 len 4096
016/300 Write block 831 len 24576 data: 7 8 7 7 7 5
017/300 Read block 479 len 20480
018/300 Read block 3917 len 24576
019/300 Read block 1432 len 32768
020/300 Write block 62 len 4096 data: 7
021/300 Write block 473 len 28672 data: 4 4 5 5 4 6 6
022/300 Read block 3636 len 16384
023/300 Read block 3309 len 40960
024/300 Read block 4885 len 40960
025/300 Write block 3613 len 24576 data: 4 5 3 4 5 7
026/300 Write block 3883 len 20480 data: 4 4 4 4 4
027/300 Read block 325 len 8192
028/300 Read block 2425 len 32768
029/300 Write block 858 len 20480 data: 6 7 8 9 8
030/300 Write block 545 len 20480 data: 6 7 6 6 6
031/300 Write block 1493 len 20480 data: 4 4 3 3 3
032/300 Write block 4928 len 24576 data: 9 7 6 6 4 5
033/300 Read block 4033 len 12288
034/300 Write block 657 len 4096 data: 5
035/300 Read block 3902 len 16384
036/300 Read block 540 len 16384
037/300 Read block 4353 len 16384
038/300 Write block 4289 len 16384 data: 3 3 3 4
039/300 Write block 3462 len 12288 data: 6 5 6
040/300 Read block 4449 len 16384
041/300 Flush
042/300 Write block 2992 len 32768 data: 4 3 3 4 4 4 4 4
043/300 Read block 3349 len 12288
044/300 Write block 3682 len 36864 data: 2 2 3 4 5 6 8 8 7
045/300 Write block 2917 len 16384 data: 2 4 4 5
046/300 Write block 2187 len 12288 data: 6 5 3
047/300 Write block 2593 len 32768 data: 5 4 4 6 5 6 6 7
048/300 Read block 3497 len 16384
049/300 Read block 495 len 20480
050/300 Write block 4499 len 24576 data: 7 7 5 5 7 6
051/300 Write block 2573 len 36864 data: 2 4 3 4 4 5 6 6 5
052/300 Write block 1058 len 40960 data: 5 6 6 5 4 4 4 4 5 4
053/300 Flush
054/300 Read block 4473 len 28672
055/300 Write block 658 len 16384 data: 8 7 6 6
056/300 Write block 4248 len 40960 data: 8 8 9 6 5 4 4 5 6 5
057/300 Flush
058/300 Write block 214 len 16384 data: 4 4 3 4
059/300 Read block 4637 len 40960
060/300 Read block 3706 len 28672
061/300 Write block 2008 len 8192 data: 5 5
062/300 Read block 4888 len 4096
063/300 Write block 1351 len 40960 data: 4 4 7 6 8 7 8 11 11 8
064/300 Read block 2289 len 12288
065/300 Read block 719 len 36864
066/300 Write block 4245 len 32768 data: 6 7 8 9 9 10 7 6
067/300 Write block 154 len 12288 data: 4 4 4
068/300 Read block 2264 len 32768
069/300 Write block 2258 len 4096 data: 8
070/300 Write block 4688 len 20480 data: 3 3 4 4 5
071/300 Read block 3818 len 36864
072/300 Write block 3141 len 36864 data: 8 9 10 7 7 5 4 5 5
073/300 Write block 4755 len 20480 data: 7 6 3 3 4
074/300 Read block 3232 len 8192
075/300 Read block 1766 len 40960
076/300 Write block 2250 len 28672 data: 6 5 6 6 7 10 11
077/300 Write block 1177 len 12288 data: 3 3 2
078/300 Flush
079/300 Write block 349 len 32768 data: 4 4 7 8 8 8 7 8
080/300 Write block 527 len 4096 data: 3
081/300 Write block 3131 len 16384 data: 7 7 7 6
082/300 Read block 1937 len 40960
083/300 Write block 2144 len 4096 data: 8
084/300 Read block 3435 len 4096
085/300 Write block 4253 len 24576 data: 5 5 6 7 6 5
086/300 Write block 735 len 24576 data: 3 3 3 6 6 6
087/300 Read block 2080 len 12288
088/300 Flush
089/300 Write block 729 len 20480 data: 4 4 3 3 3
090/300 Read block 706 len 24576
091/300 Read block 2476 len 20480
092/300 Read block 4231 len 8192
093/300 Write block 3703 len 28672 data: 4 3 4 4 6 8 8
094/300 Read block 4042 len 36864
095/300 Read block 2764 len 8192
096/300 Read block 3760 len 24576
097/300 Read block 3350 len 16384
098/300 Flush
099/300 Read block 681 len 12288
100/300 Read block 2166 len 36864
101/300 Write block 1657 len 40960 data: 4 5 5 5 5 6 6 5 4 4
102/300 Write block 3822 len 12288 data: 2 2 2
103/300 Read block 3104 len 16384
104/300 Read block 2232 len 4096
105/300 Flush
106/300 Read block 3036 len 24576
107/300 Read block 3197 len 8192
108/300 Read block 4306 len 36864
109/300 Read block 1046 len 8192
110/300 Flush
111/300 Read block 2330 len 12288
112/300 Flush
113/300 Write block 1648 len 28672 data: 2 2 2 3 4 4 5
114/300 Write block 2314 len 28672 data: 4 4 3 3 3 2 3
115/300 Read block 1199 len 40960
116/300 Write block 2712 len 24576 data: 6 7 6 7 6 5
117/300 Flush
118/300 Write block 3479 len 40960 data: 6 6 5 5 4 3 3 3 2 3
119/300 Read block 607 len 24576
120/300 Write block 3488 len 36864 data: 4 3 3 4 5 5 4 4 3
121/300 Flush
122/300 Write block 2585 len 24576 data: 5 7 8 6 5 6
123/300 Read block 1981 len 36864
124/300 Read block 2402 len 16384
125/300 Write block 4153 len 28672 data: 4 5 4 4 3 5 6
126/300 Read block 611 len 8192
127/300 Read block 2641 len 8192
128/300 Write block 3159 len 8192 data: 3 3
129/300 Read block 913 len 20480
130/300 Write block 1666 len 24576 data: 5 3 2 4 5 5
131/300 Flush
132/300 Read block 1391 len 16384
133/300 Read block 1408 len 4096
134/300 Read block 4223 len 20480
135/300 Write block 1412 len 24576 data: 6 6 6 7 7 8
136/300 Write block 3437 len 8192 data: 7 7
137/300 Read block 1916 len 20480
138/300 Write block 3599 len 16384 data: 5 6 4 4
139/300 Write block 3872 len 40960 data: 4 6 6 5 6 6 7 6 9 8
140/300 Read block 1053 len 24576
141/300 Write block 4074 len 4096 data: 3
142/300 Read block 2103 len 8192
143/300 Read block 3562 len 8192
144/300 Read block 4751 len 8192
145/300 Flush
146/300 Read block 2425 len 40960
147/300 Write block 580 len 4096 data: 7
148/300 Read block 856 len 16384
149/300 Read block 4063 len 28672
150/300 Write block 4372 len 24576 data: 3 3 3 4 4 4
151/300 Read block 3779 len 28672
152/300 Flush
153/300 Write block 1851 len 12288 data: 7 7 7
154/300 Write block 680 len 12288 data: 5 6 6
155/300 Write block 2708 len 32768 data: 4 6 6 6 7 8 7 8
156/300 Write block 1749 len 8192 data: 2 2
157/300 Read block 4718 len 36864
158/300 Read block 1802 len 28672
159/300 Flush
160/300 Read block 4232 len 8192
161/300 Read block 4181 len 20480
162/300 Write block 382 len 4096 data: 4
163/300 Write block 2824 len 36864 data: 4 5 6 5 7 7 8 8 9
164/300 Read block 568 len 12288
165/300 Read block 970 len 12288
166/300 Read block 2128 len 12288
167/300 Read block 3077 len 24576
168/300 Write block 2683 len 8192 data: 10 10
169/300 Read block 784 len 20480
170/300 Read block 2752 len 32768
171/300 Read block 103 len 28672
172/300 Read block 2853 len 32768
173/300 Write block 3099 len 24576 data: 6 5 6 6 8 10
174/300 Write block 4502 len 24576 data: 6 8 7 6 5 5
175/300 Read block 3999 len 24576
176/300 Read block 4956 len 40960
177/300 Write block 1292 len 12288 data: 3 3 6
178/300 Read block 1056 len 28672
179/300 Flush
180/300 Read block 3921 len 8192
181/300 Write block 3781 len 32768 data: 5 4 4 4 3 4 4 4
182/300 Write block 2369 len 4096 data: 6
183/300 Flush
184/300 Write block 4771 len 12288 data: 7 5 6
185/300 Read block 4540 len 16384
186/300 Read block 4659 len 28672
187/300 Read block 1577 len 12288
188/300 Read block 1065 len 32768
189/300 Write block 4375 len 28672 data: 5 5 5 4 3 4 4
190/300 Read block 3770 len 28672
191/300 Read block 4333 len 24576
192/300 Read block 5 len 28672
193/300 Write block 1569 len 24576 data: 5 5 6 7 8 7
194/300 Write block 1589 len 4096 data: 2
195/300 Read block 884 len 28672
196/300 Read block 4434 len 4096
197/300 Read block 994 len 36864
198/300 Write block 2071 len 36864 data: 3 5 5 5 5 7 7 7 7
199/300 Read block 2669 len 12288
200/300 Read block 789 len 28672
201/300 Flush
202/300 Read block 2805 len 16384
203/300 Write block 3478 len 12288 data: 4 7 7
204/300 Flush
205/300 Read block 2572 len 20480
206/300 Flush
207/300 Write block 4306 len 20480 data: 3 3 3 3 3
208/300 Read block 677 len 24576
209/300 Write block 345 len 28672 data: 3 4 4 4 5 5 8
210/300 Write block 2590 len 32768 data: 7 6 6 6 5 5 7 6
211/300 Flush
212/300 Write block 1464 len 20480 data: 6 6 6 6 5
213/300 Read block 4189 len 36864
214/300 Read block 1410 len 24576
215/300 Read block 2601 len 32768
216/300 Read block 1646 len 20480
217/300 Write block 1243 len 36864 data: 3 3 3 3 3 3 4 3 5
218/300 Write block 2250 len 12288 data: 7 6 7
219/300 Write block 4026 len 28672 data: 3 4 3 3 3 3 3
220/300 Write block 4481 len 28672 data: 8 7 6 4 5 4 4
221/300 Flush
222/300 Write block 4072 len 12288 data: 4 4 4
223/300 Write block 1925 len 36864 data: 7 6 4 4 4 4 2 2 3
224/300 Read block 4875 len 36864
225/300 Write block 339 len 12288 data: 6 5 5
226/300 Write block 3670 len 20480 data: 4 5 5 5 5
227/300 Read block 3126 len 40960
228/300 Write block 4409 len 12288 data: 6 6 7
229/300 Read block 4042 len 4096
230/300 Read block 3903 len 28672
231/300 Write block 514 len 36864 data: 6 6 5 5 5 7 7 6 5
232/300 Write block 3456 len 20480 data: 8 8 5 4 3
233/300 Write block 1619 len 12288 data: 8 8 7
234/300 Read block 338 len 28672
235/300 Read block 1604 len 28672
236/300 Flush
237/300 Read block 3242 len 24576
238/300 Read block 397 len 40960
239/300 Read block 4007 len 4096
240/300 Write block 1575 len 36864 data: 7 7 6 6 5 7 7 7 5
241/300 Read block 2531 len 8192
242/300 Flush
243/300 Write block 2117 len 24576 data: 2 2 2 3 2 2
244/300 Write block 3726 len 12288 data: 8 8 7
245/300 Read block 346 len 40960
246/300 Write block 1726 len 32768 data: 7 7 6 7 6 5 4 4
247/300 Write block 3854 len 36864 data: 6 6 5 5 3 3 4 5 3
248/300 Read block 2657 len 20480
249/300 Read block 2032 len 16384
250/300 Write block 4323 len 12288 data: 4 4 5
251/300 Flush
252/300 Flush
253/300 Write block 204 len 12288 data: 4 7 7
254/300 Write block 4806 len 40960 data: 7 8 8 6 8 7 5 3 3 3
255/300 Write block 2358 len 20480 data: 3 3 3 3 2
256/300 Read block 269 len 28672
257/300 Write block 2172 len 20480 data: 6 7 6 7 8
258/300 Read block 2444 len 4096
259/300 Write block 4678 len 20480 data: 3 3 7 8 6
260/300 Write block 4597 len 4096 data: 6
261/300 Read block 683 len 32768
262/300 Write block 3279 len 40960 data: 6 5 5 5 4 3 3 3 4 3
263/300 Read block 1521 len 24576
264/300 Read block 3355 len 24576
265/300 Read block 4264 len 36864
266/300 Read block 4196 len 36864
267/300 Flush
268/300 Write block 1722 len 12288 data: 5 5 4
269/300 Write block 720 len 28672 data: 9 8 7 5 6 4 5
270/300 Flush
271/300 Read block 2720 len 12288
272/300 Read block 3042 len 20480
273/300 Write block 489 len 36864 data: 4 4 3 4 4 6 7 8 10
274/300 Write block 2749 len 16384 data: 4 6 8 7
275/300 Write block 2818 len 32768 data: 2 2 3 3 3 3 5 6
276/300 Read block 746 len 36864
277/300 Read block 267 len 40960
278/300 Write block 319 len 36864 data: 4 6 7 5 5 5 6 5 6
279/300 Read block 4440 len 16384
280/300 Read block 392 len 20480
281/300 Read block 1694 len 40960
282/300 Read block 1556 len 12288
283/300 Write block 3098 len 4096 data: 6
284/300 Read block 2673 len 32768
285/300 Read block 4105 len 12288
286/300 Write block 4687 len 40960 data: 5 4 4 5 5 6 6 6 6 5
287/300 Write block 2743 len 32768 data: 2 3 4 3 3 4 5 7
288/300 Write block 3730 len 36864 data: 5 4 4 4 3 5 5 4 3
289/300 Read block 1910 len 24576
290/300 Read block 234 len 24576
291/300 Flush
292/300 Read block 3405 len 16384
293/300 Write block 4220 len 8192 data: 5 4
294/300 Write block 839 len 32768 data: 4 3 3 3 4 4 4 5
295/300 Write block 12 len 4096 data: 3
296/300 Write block 2445 len 12288 data: 6 6 3
297/300 Write block 610 len 8192 data: 5 4
298/300 Read block 2406 len 12288
299/300 Write block 887 len 20480 data: 4 3 3 4 6
300/300 Write block 2927 len 28672 data: 5 6 8 7 7 7 8
301/300 Write block 235 len 4096 data: 6
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:42:11.097 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/1, method: POST, req_id: 168749a6-92f1-4205-9539-169d38059a2c, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8244 Acked Write 5 Done Sent Done false
8245 Acked Read 8 Done Sent Done false
8246 Acked Write 9 Done Sent Done false
8247 Acked Read 6 Done Sent Done false
8248 Acked Write 5 Done Sent Done false
8249 Acked Write 7 Done Sent Done false
8250 Acked Read 4 Done Sent Done false
8251 Acked Write 8 Done Sent Done false
8252 Acked Write 4 Done Sent Done false
8253 Acked Read 9 Done Sent Done false
8254 Acked Read 7 Done Sent Done false
8255 Acked Write 2 Done Sent Done false
8256 Acked Write 2 Done Sent Done false
8257 Acked Read 7 Done Sent Done false
8258 Acked Read 1 Done Sent Done false
8259 Acked Write 6 Done Sent Done false
8260 Acked Read 5 Done Sent Done false
8261 Acked Read 6 Done Sent Done false
8262 Acked Read 8 Done Sent Done false
8263 Acked Write 1 Done Sent Done false
8264 Acked Write 7 Done Sent Done false
8265 Acked Read 4 Done Sent Done false
8266 Acked Read 10 Done Sent Done false
8267 Acked Read 10 Done Sent Done false
8268 Acked Write 6 Done Sent Done false
8269 Acked Write 5 Done Sent Done false
8270 Acked Read 2 Done Sent Done false
8271 Acked Read 8 Done Sent Done false
8272 Acked Write 5 Done Sent Done false
8273 Acked Write 5 Done Sent Done false
8274 Acked Write 5 Done Sent Done false
8275 Acked Write 6 Done Sent Done false
8276 Acked Read 3 Done Sent Done false
8277 Acked Write 1 Done Sent Done false
8278 Acked Read 4 Done Sent Done false
8279 Acked Read 4 Done Sent Done false
8280 Acked Read 4 Done Sent Done false
8281 Acked Write 4 Done Sent Done false
8282 Acked Write 3 Done Sent Done false
8283 Acked Read 4 Done Sent Done false
8284 Acked Flush 0 Done Sent Done false
8285 Acked Write 8 Done Sent Done false
8286 Acked Read 3 Done Sent Done false
8287 Acked Write 9 Done Sent Done false
8288 Acked Write 4 Done Sent Done false
8289 Acked Write 3 Done Sent Done false
8290 Acked Write 8 Done Sent Done false
8291 Acked Read 4 Done Sent Done false
8292 Acked Read 5 Done Sent Done false
8293 Acked Write 6 Done Sent Done false
8294 Acked Write 9 Done Sent Done false
8295 Acked Write 10 Done Sent Done false
8296 Acked Flush 0 Done Sent Done false
8297 Acked Read 7 Done Sent Done false
8298 Acked Write 4 Done Sent Done false
8299 Acked Write 10 Done Sent Done false
8300 Acked Flush 0 Done Sent Done false
8301 Acked Write 4 Done Sent Done false
8302 Acked Read 10 Done Sent Done false
8303 Acked Read 7 Done Sent Done false
8304 Acked Write 2 Done Sent Done false
8305 Acked Read 1 Done Sent Done false
8306 Acked Write 10 Done Sent Done false
8307 Acked Read 3 Done Sent Done false
8308 Acked Read 9 Done Sent Done false
8309 Acked Write 8 Done Sent Done false
8310 Acked Write 3 Done Sent Done false
8311 Acked Read 8 Done Sent Done false
8312 Acked Write 1 Done Sent Done false
8313 Acked Write 5 Done Sent Done false
8314 Acked Read 9 Done Sent Done false
8315 Acked Write 9 Done Sent Done false
8316 Acked Write 5 Done Sent Done false
8317 Acked Read 2 Done Sent Done false
8318 Acked Read 10 Done Sent Done false
8319 Acked Write 7 Done Sent Done false
8320 Acked Write 3 Done Sent Done false
8321 Acked Flush 0 Done Sent Done false
8322 Acked Write 8 Done Sent Done false
8323 Acked Write 1 Done Sent Done false
8324 Acked Write 4 Done Sent Done false
8325 Acked Read 10 Done Sent Done false
8326 Acked Write 1 Done Sent Done false
8327 Acked Read 1 Done Sent Done false
8328 Acked Write 6 Done Sent Done false
8329 Acked Write 6 Done Sent Done false
8330 Acked Read 3 Done Sent Done false
8331 Acked Flush 0 Done Sent Done false
8332 Acked Write 5 Done Sent Done false
8333 Acked Read 6 Done Sent Done false
8334 Acked Read 5 Done Sent Done false
8335 Acked Read 2 Done Sent Done false
8336 Acked Write 7 Done Sent Done false
8337 Acked Read 9 Done Sent Done false
8338 Acked Read 2 Done Sent Done false
8339 Acked Read 6 Done Sent Done false
8340 Acked Read 4 Done Sent Done false
8341 Acked Flush 0 Done Sent Done false
8342 Acked Read 3 Done Sent Done false
8343 Acked Read 9 Done Sent Done false
8344 Acked Write 10 Done Sent Done false
8345 Acked Write 3 Done Sent Done false
8346 Acked Read 4 Done Sent Done false
8347 Acked Read 1 Done Sent Done false
8348 Acked Flush 0 Done Sent Done false
8349 Acked Read 6 Done Sent Done false
8350 Acked Read 2 Done Sent Done false
8351 Acked Read 9 Done Sent Done false
8352 Acked Read 2 Done Sent Done false
8353 Acked Flush 0 Done Sent Done false
8354 Acked Read 3 Done Sent Done false
8355 Acked Flush 0 Done Sent Done false
8356 Acked Write 7 Done Sent Done false
8357 Acked Write 7 Done Sent Done false
8358 Acked Read 10 Done Sent Done false
8359 Acked Write 6 Done Sent Done false
8360 Acked Flush 0 Done Sent Done false
8361 Acked Write 10 Done Sent Done false
8362 Acked Read 6 Done Sent Done false
8363 Acked Write 9 Done Sent Done false
8364 Acked Flush 0 Done Sent Done false
8365 Acked Write 6 Done Sent Done false
8366 Acked Read 9 Done Sent Done false
8367 Acked Read 4 Done Sent Done false
8368 Acked Write 7 Done Sent Done false
8369 Acked Read 2 Done Sent Done false
8370 Acked Read 2 Done Sent Done false
8371 Acked Write 2 Done Sent Done false
8372 Acked Read 5 Done Sent Done false
8373 Acked Write 6 Done Sent Done false
8374 Acked Flush 0 Done Sent Done false
8375 Acked Read 4 Done Sent Done false
8376 Acked Read 1 Done Sent Done false
8377 Acked Read 5 Done Sent Done false
8378 Acked Write 6 Done Sent Done false
8379 Acked Write 2 Done Sent Done false
8380 Acked Read 5 Done Sent Done false
8381 Acked Write 4 Done Sent Done false
8382 Acked Write 10 Done Sent Done false
8383 Acked Read 6 Done Sent Done false
8384 Acked Write 1 Done Sent Done false
8385 Acked Read 2 Done Sent Done false
8386 Acked Read 2 Done Sent Done false
8387 Acked Read 2 Done Sent Done false
8388 Acked Flush 0 Done Sent Done false
8389 Acked Read 10 Done Sent Done false
8390 Acked Write 1 Done Sent Done false
8391 Acked Read 4 Done Sent Done false
8392 Acked Read 7 Done Sent Done false
8393 Acked Write 6 Done Sent Done false
8394 Acked Read 7 Done Sent Done false
8395 Acked Flush 0 Done Sent Done false
8396 Acked Write 3 Done Sent Done false
8397 Acked Write 3 Done Sent Done false
8398 Acked Write 8 Done Sent Done false
8399 Acked Write 2 Done Sent Done false
8400 Acked Read 9 Done Sent Done false
8401 Acked Read 7 Done Sent Done false
8402 Acked Flush 0 Done Sent Done false
8403 Acked Read 2 Done Sent Done false
8404 Acked Read 5 Done Sent Done false
8405 Acked Write 1 Done Sent Done false
8406 Acked Write 9 Done Sent Done false
8407 Acked Read 3 Done Sent Done false
8408 Acked Read 3 Done Sent Done false
8409 Acked Read 3 Done Sent Done false
8410 Acked Read 6 Done Sent Done false
8411 Acked Write 2 Done Sent Done false
8412 Acked Read 5 Done Sent Done false
8413 Acked Read 8 Done Sent Done false
8414 Acked Read 7 Done Sent Done false
8415 Acked Read 8 Done Sent Done false
8416 Acked Write 6 Done Sent Done false
8417 Acked Write 6 Done Sent Done false
8418 Acked Read 6 Done Sent Done false
8419 Acked Read 10 Done Sent Done false
8420 Acked Write 3 Done Sent Done false
8421 Acked Read 7 Done Sent Done false
8422 Acked Flush 0 Done Sent Done false
8423 Acked Read 2 Done Sent Done false
8424 Acked Write 8 Done Sent Done false
8425 Acked Write 1 Done Sent Done false
8426 Acked Flush 0 Done Sent Done false
8427 Acked Write 3 Done Sent Done false
8428 Acked Read 4 Done Sent Done false
8429 Acked Read 7 Done Sent Done false
8430 Acked Read 3 Done Sent Done false
8431 Acked Read 8 Done Sent Done false
8432 Acked Write 7 Done Sent Done false
8433 Acked Read 7 Done Sent Done false
8434 Acked Read 6 Done Sent Done false
8435 Acked Read 7 Done Sent Done false
8436 Acked Write 6 Done Sent Done false
8437 Acked Write 1 Done Sent Done false
8438 Acked Read 7 Done Sent Done false
8439 Acked Read 1 Done Sent Done false
8440 Acked Read 9 Done Sent Done false
8441 Acked Write 9 Done Sent Done false
8442 Acked Read 3 Done Sent Done false
8443 Acked Read 7 Done Sent Done false
8444 Acked Flush 0 Done Sent Done false
8445 Acked Read 4 Done Sent Done false
8446 Acked Write 3 Done Sent Done false
8447 Acked Flush 0 Done Sent Done false
8448 Acked Read 5 Done Sent Done false
8449 Acked Flush 0 Done Sent Done false
8450 Acked Write 5 Done Sent Done false
8451 Acked Read 6 Done Sent Done false
8452 Acked Write 7 Done Sent Done false
8453 Acked Write 8 Done Sent Done false
8454 Acked Flush 0 Done Sent Done false
8455 Acked Write 5 Done Sent Done false
8456 Acked Read 9 Done Sent Done false
8457 Acked Read 6 Done Sent Done false
8458 Acked Read 8 Done Sent Done false
8459 Acked Read 5 Done Sent Done false
8460 Acked Write 9 Done Sent Done false
8461 Acked Write 3 Done Sent Done false
8462 Acked Write 7 Done Sent Done false
8463 Acked Write 7 Done Sent Done false
8464 Acked Flush 0 Done Sent Done false
8465 Acked Write 3 Done Sent Done false
8466 Acked Write 9 Done Sent Done false
8467 Acked Read 9 Done Sent Done false
8468 Acked Write 3 Done Sent Done false
8469 Acked Write 5 Done Sent Done false
8470 Acked Read 10 Done Sent Done false
8471 Acked Write 3 Done Sent Done false
8472 Acked Read 1 Done Sent Done false
8473 Acked Read 7 Done Sent Done false
8474 Acked Write 9 Done Sent Done false
8475 Acked Write 5 Done Sent Done false
8476 Acked Write 3 Done Sent Done false
8477 Acked Read 7 Done Sent Done false
8478 Acked Read 7 Done Sent Done false
8479 Acked Flush 0 Done Sent Done false
8480 Acked Read 6 Done Sent Done false
8481 Acked Read 10 Done Sent Done false
8482 Acked Read 1 Done Sent Done false
8483 Acked Write 9 Done Sent Done false
8484 Acked Read 2 Done Sent Done false
8485 Acked Flush 0 Done Sent Done false
8486 Acked Write 6 Done Sent Done false
8487 Acked Write 3 Done Sent Done false
8488 Acked Read 10 Done Sent Done false
8489 Acked Write 8 Done Sent Done false
8490 Acked Write 9 Done Sent Done false
8491 Acked Read 5 Done Sent Done false
8492 Acked Read 4 Done Sent Done false
8493 Acked Write 3 Done Sent Done false
8494 Acked Flush 0 Done Sent Done false
8495 Acked Flush 0 Done Sent Done false
8496 Acked Write 3 Done Sent Done false
8497 Acked Write 10 Done Sent Done false
8498 Acked Write 5 Done Sent Done false
8499 Acked Read 7 Done Sent Done false
8500 Acked Write 5 Done Sent Done false
8501 Acked Read 1 Done Sent Done false
8502 Acked Write 5 Done Sent Done false
8503 Acked Write 1 Done Sent Done false
8504 Acked Read 8 Done Sent Done false
8505 Acked Write 10 Done Sent Done false
8506 Acked Read 6 Done Sent Done false
8507 Acked Read 6 Done Sent Done false
8508 Acked Read 9 Done Sent Done false
8509 Acked Read 9 Done Sent Done false
8510 Acked Flush 0 Done Sent Done false
8511 Acked Write 3 Done Sent Done false
8512 Acked Write 7 Done Sent Done false
8513 Acked Flush 0 Done Sent Done false
8514 Acked Read 3 Done Sent Done false
8515 Acked Read 5 Done Sent Done false
8516 Acked Write 9 Done Sent Done false
8517 Acked Write 4 Done Sent Done false
8518 Acked Write 8 Done Sent Done false
8519 Acked Read 9 Done Sent Done false
8520 Acked Read 10 Done Sent Done false
8521 Acked Write 9 Done Sent Done false
8522 Acked Read 4 Done Sent Done false
8523 Acked Read 5 Done Sent Done false
8524 Acked Read 10 Done Sent Done false
8525 Acked Read 3 Done Sent Done false
8526 Acked Write 1 Done Sent Done false
8527 Acked Read 8 Done Sent Done false
8528 Acked Read 3 Done Sent Done false
8529 Acked Write 10 Done Sent Done false
8530 Acked Write 8 Done Sent Done false
8531 Acked Write 9 Done Sent Done false
8532 Acked Read 6 Done Sent Done false
8533 Acked Read 6 Done Sent Done false
8534 Acked Flush 0 Done Sent Done false
8535 Acked Read 4 Done Sent Done false
8536 Acked Write 2 Done Sent Done false
8537 Acked Write 8 Done Sent Done false
8538 Acked Write 1 Done Sent Done false
8539 Acked Write 3 Done Sent Done false
8540 Acked Write 2 Done Sent Done false
8541 Acked Read 3 Done Sent Done false
8542 Acked Write 5 Done Sent Done false
8543 Acked Write 7 Done Sent Done false
8544 Acked Write 1 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8534 8243 8534
Downstairs last five completed: 8243 8242 8241 8240 8239
Upstairs last five completed: 8544 8543 8542 8541 8540
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8244 Acked Write 5 Done Sent Done false
8245 Acked Read 8 Done Sent Done false
8246 Acked Write 9 Done Sent Done false
8247 Acked Read 6 Done Sent Done false
8248 Acked Write 5 Done Sent Done false
8249 Acked Write 7 Done Sent Done false
8250 Acked Read 4 Done Sent Done false
8251 Acked Write 8 Done Sent Done false
8252 Acked Write 4 Done Sent Done false
8253 Acked Read 9 Done Sent Done false
8254 Acked Read 7 Done Sent Done false
8255 Acked Write 2 Done Sent Done false
8256 Acked Write 2 Done Sent Done false
8257 Acked Read 7 Done Sent Done false
8258 Acked Read 1 Done Sent Done false
8259 Acked Write 6 Done Sent Done false
8260 Acked Read 5 Done Sent Done false
8261 Acked Read 6 Done Sent Done false
8262 Acked Read 8 Done Sent Done false
8263 Acked Write 1 Done Sent Done false
8264 Acked Write 7 Done Sent Done false
8265 Acked Read 4 Done Sent Done false
8266 Acked Read 10 Done Sent Done false
8267 Acked Read 10 Done Sent Done false
8268 Acked Write 6 Done Sent Done false
8269 Acked Write 5 Done Sent Done false
8270 Acked Read 2 Done Sent Done false
8271 Acked Read 8 Done Sent Done false
8272 Acked Write 5 Done Sent Done false
8273 Acked Write 5 Done Sent Done false
8274 Acked Write 5 Done Sent Done false
8275 Acked Write 6 Done Sent Done false
8276 Acked Read 3 Done Sent Done false
8277 Acked Write 1 Done Sent Done false
8278 Acked Read 4 Done Sent Done false
8279 Acked Read 4 Done Sent Done false
8280 Acked Read 4 Done Sent Done false
8281 Acked Write 4 Done Sent Done false
8282 Acked Write 3 Done Sent Done false
8283 Acked Read 4 Done Sent Done false
8284 Acked Flush 0 Done Sent Done false
8285 Acked Write 8 Done Sent Done false
8286 Acked Read 3 Done Sent Done false
8287 Acked Write 9 Done Sent Done false
8288 Acked Write 4 Done Sent Done false
8289 Acked Write 3 Done Sent Done false
8290 Acked Write 8 Done Sent Done false
8291 Acked Read 4 Done Sent Done false
8292 Acked Read 5 Done Sent Done false
8293 Acked Write 6 Done Sent Done false
8294 Acked Write 9 Done Sent Done false
8295 Acked Write 10 Done Sent Done false
8296 Acked Flush 0 Done Sent Done false
8297 Acked Read 7 Done Sent Done false
8298 Acked Write 4 Done Sent Done false
8299 Acked Write 10 Done Sent Done false
8300 Acked Flush 0 Done Sent Done false
8301 Acked Write 4 Done Sent Done false
8302 Acked Read 10 Done Sent Done false
8303 Acked Read 7 Done Sent Done false
8304 Acked Write 2 Done Sent Done false
8305 Acked Read 1 Done Sent Done false
8306 Acked Write 10 Done Sent Done false
8307 Acked Read 3 Done Sent Done false
8308 Acked Read 9 Done Sent Done false
8309 Acked Write 8 Done Sent Done false
8310 Acked Write 3 Done Sent Done false
8311 Acked Read 8 Done Sent Done false
8312 Acked Write 1 Done Sent Done false
8313 Acked Write 5 Done Sent Done false
8314 Acked Read 9 Done Sent Done false
8315 Acked Write 9 Done Sent Done false
8316 Acked Write 5 Done Sent Done false
8317 Acked Read 2 Done Sent Done false
8318 Acked Read 10 Done Sent Done false
8319 Acked Write 7 Done Sent Done false
8320 Acked Write 3 Done Sent Done false
8321 Acked Flush 0 Done Sent Done false
8322 Acked Write 8 Done Sent Done false
8323 Acked Write 1 Done Sent Done false
8324 Acked Write 4 Done Sent Done false
8325 Acked Read 10 Done Sent Done false
8326 Acked Write 1 Done Sent Done false
8327 Acked Read 1 Done Sent Done false
8328 Acked Write 6 Done Sent Done false
8329 Acked Write 6 Done Sent Done false
8330 Acked Read 3 Done Sent Done false
8331 Acked Flush 0 Done Sent Done false
8332 Acked Write 5 Done Sent Done false
8333 Acked Read 6 Done Sent Done false
8334 Acked Read 5 Done Sent Done false
8335 Acked Read 2 Done Sent Done false
8336 Acked Write 7 Done Sent Done false
8337 Acked Read 9 Done Sent Done false
8338 Acked Read 2 Done Sent Done false
8339 Acked Read 6 Done Sent Done false
8340 Acked Read 4 Done Sent Done false
8341 Acked Flush 0 Done Sent Done false
8342 Acked Read 3 Done Sent Done false
8343 Acked Read 9 Done Sent Done false
8344 Acked Write 10 Done Sent Done false
8345 Acked Write 3 Done Sent Done false
8346 Acked Read 4 Done Sent Done false
8347 Acked Read 1 Done Sent Done false
8348 Acked Flush 0 Done Sent Done false
8349 Acked Read 6 Done Sent Done false
8350 Acked Read 2 Done Sent Done false
8351 Acked Read 9 Done Sent Done false
8352 Acked Read 2 Done Sent Done false
8353 Acked Flush 0 Done Sent Done false
8354 Acked Read 3 Done Sent Done false
8355 Acked Flush 0 Done Sent Done false
8356 Acked Write 7 Done Sent Done false
8357 Acked Write 7 Done Sent Done false
8358 Acked Read 10 Done Sent Done false
8359 Acked Write 6 Done Sent Done false
8360 Acked Flush 0 Done Sent Done false
8361 Acked Write 10 Done Sent Done false
8362 Acked Read 6 Done Sent Done false
8363 Acked Write 9 Done Sent Done false
8364 Acked Flush 0 Done Sent Done false
8365 Acked Write 6 Done Sent Done false
8366 Acked Read 9 Done Sent Done false
8367 Acked Read 4 Done Sent Done false
8368 Acked Write 7 Done Sent Done false
8369 Acked Read 2 Done Sent Done false
8370 Acked Read 2 Done Sent Done false
8371 Acked Write 2 Done Sent Done false
8372 Acked Read 5 Done Sent Done false
8373 Acked Write 6 Done Sent Done false
8374 Acked Flush 0 Done Sent Done false
8375 Acked Read 4 Done Sent Done false
8376 Acked Read 1 Done Sent Done false
8377 Acked Read 5 Done Sent Done false
8378 Acked Write 6 Done Sent Done false
8379 Acked Write 2 Done Sent Done false
8380 Acked Read 5 Done Sent Done false
8381 Acked Write 4 Done Sent Done false
8382 Acked Write 10 Done Sent Done false
8383 Acked Read 6 Done Sent Done false
8384 Acked Write 1 Done Sent Done false
8385 Acked Read 2 Done Sent Done false
8386 Acked Read 2 Done Sent Done false
8387 Acked Read 2 Done Sent Done false
8388 Acked Flush 0 Done Sent Done false
8389 Acked Read 10 Done Sent Done false
8390 Acked Write 1 Done Sent Done false
8391 Acked Read 4 Done Sent Done false
8392 Acked Read 7 Done Sent Done false
8393 Acked Write 6 Done Sent Done false
8394 Acked Read 7 Done Sent Done false
8395 Acked Flush 0 Done Sent Done false
8396 Acked Write 3 Done Sent Done false
8397 Acked Write 3 Done Sent Done false
8398 Acked Write 8 Done Sent Done false
8399 Acked Write 2 Done Sent Done false
8400 Acked Read 9 Done Sent Done false
8401 Acked Read 7 Done Sent Done false
8402 Acked Flush 0 Done Sent Done false
8403 Acked Read 2 Done Sent Done false
8404 Acked Read 5 Done Sent Done false
8405 Acked Write 1 Done Sent Done false
8406 Acked Write 9 Done Sent Done false
8407 Acked Read 3 Done Sent Done false
8408 Acked Read 3 Done Sent Done false
8409 Acked Read 3 Done Sent Done false
8410 Acked Read 6 Done Sent Done false
8411 Acked Write 2 Done Sent Done false
8412 Acked Read 5 Done Sent Done false
8413 Acked Read 8 Done Sent Done false
8414 Acked Read 7 Done Sent Done false
8415 Acked Read 8 Done Sent Done false
8416 Acked Write 6 Done Sent Done false
8417 Acked Write 6 Done Sent Done false
8418 Acked Read 6 Done Sent Done false
8419 Acked Read 10 Done Sent Done false
8420 Acked Write 3 Done Sent Done false
8421 Acked Read 7 Done Sent Done false
8422 Acked Flush 0 Done Sent Done false
8423 Acked Read 2 Done Sent Done false
8424 Acked Write 8 Done Sent Done false
8425 Acked Write 1 Done Sent Done false
8426 Acked Flush 0 Done Sent Done false
8427 Acked Write 3 Done Sent Done false
8428 Acked Read 4 Done Sent Done false
8429 Acked Read 7 Done Sent Done false
8430 Acked Read 3 Done Sent Done false
8431 Acked Read 8 Done Sent Done false
8432 Acked Write 7 Done Sent Done false
8433 Acked Read 7 Done Sent Done false
8434 Acked Read 6 Done Sent Done false
8435 Acked Read 7 Done Sent Done false
8436 Acked Write 6 Done Sent Done false
8437 Acked Write 1 Done Sent Done false
8438 Acked Read 7 Done Sent Done false
8439 Acked Read 1 Done Sent Done false
8440 Acked Read 9 Done Sent Done false
8441 Acked Write 9 Done Sent Done false
8442 Acked Read 3 Done Sent Done false
8443 Acked Read 7 Done Sent Done false
8444 Acked Flush 0 Done Sent Done false
8445 Acked Read 4 Done Sent Done false
8446 Acked Write 3 Done Sent Done false
8447 Acked Flush 0 Done Sent Done false
8448 Acked Read 5 Done Sent Done false
8449 Acked Flush 0 Done Sent Done false
8450 Acked Write 5 Done Sent Done false
8451 Acked Read 6 Done Sent Done false
8452 Acked Write 7 Done Sent Done false
8453 Acked Write 8 Done Sent Done false
8454 Acked Flush 0 Done Sent Done false
8455 Acked Write 5 Done Sent Done false
8456 Acked Read 9 Done Sent Done false
8457 Acked Read 6 Done Sent Done false
8458 Acked Read 8 Done Sent Done false
8459 Acked Read 5 Done Sent Done false
8460 Acked Write 9 Done Sent Done false
8461 Acked Write 3 Done Sent Done false
8462 Acked Write 7 Done Sent Done false
8463 Acked Write 7 Done Sent Done false
8464 Acked Flush 0 Done Sent Done false
8465 Acked Write 3 Done Sent Done false
8466 Acked Write 9 Done Sent Done false
8467 Acked Read 9 Done Sent Done false
8468 Acked Write 3 Done Sent Done false
8469 Acked Write 5 Done Sent Done false
8470 Acked Read 10 Done Sent Done false
8471 Acked Write 3 Done Sent Done false
8472 Acked Read 1 Done Sent Done false
8473 Acked Read 7 Done Sent Done false
8474 Acked Write 9 Done Sent Done false
8475 Acked Write 5 Done Sent Done false
8476 Acked Write 3 Done Sent Done false
8477 Acked Read 7 Done Sent Done false
8478 Acked Read 7 Done Sent Done false
8479 Acked Flush 0 Done Sent Done false
8480 Acked Read 6 Done Sent Done false
8481 Acked Read 10 Done Sent Done false
8482 Acked Read 1 Done Sent Done false
8483 Acked Write 9 Done Sent Done false
8484 Acked Read 2 Done Sent Done false
8485 Acked Flush 0 Done Sent Done false
8486 Acked Write 6 Done Sent Done false
8487 Acked Write 3 Done Sent Done false
8488 Acked Read 10 Done Sent Done false
8489 Acked Write 8 Done Sent Done false
8490 Acked Write 9 Done Sent Done false
8491 Acked Read 5 Done Sent Done false
8492 Acked Read 4 Done Sent Done false
8493 Acked Write 3 Done Sent Done false
8494 Acked Flush 0 Done Sent Done false
8495 Acked Flush 0 Done Sent Done false
8496 Acked Write 3 Done Sent Done false
8497 Acked Write 10 Done Sent Done false
8498 Acked Write 5 Done Sent Done false
8499 Acked Read 7 Done Sent Done false
8500 Acked Write 5 Done Sent Done false
8501 Acked Read 1 Done Sent Done false
8502 Acked Write 5 Done Sent Done false
8503 Acked Write 1 Done Sent Done false
8504 Acked Read 8 Done Sent Done false
8505 Acked Write 10 Done Sent Done false
8506 Acked Read 6 Done Sent Done false
8507 Acked Read 6 Done Sent Done false
8508 Acked Read 9 Done Sent Done false
8509 Acked Read 9 Done Sent Done false
8510 Acked Flush 0 Done Sent Done false
8511 Acked Write 3 Done Sent Done false
8512 Acked Write 7 Done Sent Done false
8513 Acked Flush 0 Done Sent Done false
8514 Acked Read 3 Done Sent Done false
8515 Acked Read 5 Done Sent Done false
8516 Acked Write 9 Done Sent Done false
8517 Acked Write 4 Done Sent Done false
8518 Acked Write 8 Done Sent Done false
8519 Acked Read 9 Done Sent Done false
8520 Acked Read 10 Done Sent Done false
8521 Acked Write 9 Done Sent Done false
8522 Acked Read 4 Done Sent Done false
8523 Acked Read 5 Done Sent Done false
8524 Acked Read 10 Done Sent Done false
8525 Acked Read 3 Done Sent Done false
8526 Acked Write 1 Done Sent Done false
8527 Acked Read 8 Done Sent Done false
8528 Acked Read 3 Done Sent Done false
8529 Acked Write 10 Done Sent Done false
8530 Acked Write 8 Done Sent Done false
8531 Acked Write 9 Done Sent Done false
8532 Acked Read 6 Done Sent Done false
8533 Acked Read 6 Done Sent Done false
8534 Acked Flush 0 Done Sent Done false
8535 Acked Read 4 Done Sent Done false
8536 Acked Write 2 Done Sent Done false
8537 Acked Write 8 Done Sent Done false
8538 Acked Write 1 Done Sent Done false
8539 Acked Write 3 Done Sent Done false
8540 Acked Write 2 Done Sent Done false
8541 Acked Read 3 Done Sent Done false
8542 Acked Write 5 Done Sent Done false
8543 Acked Write 7 Done Sent Done false
8544 Acked Write 1 Done Sent Done false
8545 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8545 8243 8545
Downstairs last five completed: 8243 8242 8241 8240 8239
Upstairs last five completed: 8545 8544 8543 8542 8541
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.862337157Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59311)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.862518226Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.863632906Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.864251914Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.86427954Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 8243","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.864295018Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 8243","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.8645403Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 8243","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.864559428Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 8243","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:16.864783821Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8545 8544 8543 8542 8541
Upstairs last five completed: 8545 8544 8543 8542 8541
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Jan 27 19:42:19.102 INFO request completed, latency_us: 326, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4c6004c8-3205-43a8-b8b2-f7bf5a792cd4, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:19.103691701Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:42:19.103 INFO request completed, latency_us: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:19.103762573Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
115, response_code: 200, uri: /state/cid/2, method: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:19.103826955Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
GET, req_id: f2b1f0ad-594f-4eda-900c-78ee667c86c5, remote_addr: 127.0.0.1:55047, local_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:19.103857632Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:42:23.106 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2559416e-2fa3-4fba-99f2-2dd3132525ba, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 48 len 20480 data: 8 9 9 9 8
002/300 Read block 182 len 28672
003/300 Read block 362 len 40960
004/300 Read block 4930 len 32768
005/300 Read block 479 len 16384
006/300 Read block 4806 len 20480
007/300 Flush
008/300 Flush
009/300 Write block 4008 len 32768 data: 7 8 8 6 5 6 4 4
010/300 Write block 3315 len 8192 data: 5 5
011/300 Write block 916 len 28672 data: 4 4 6 8 9 9 7
012/300 Read block 2879 len 40960
013/300 Read block 2600 len 4096
014/300 Read block 1518 len 32768
015/300 Write block 1991 len 36864 data: 4 4 4 4 5 6 6 7 7
016/300 Read block 4013 len 20480
017/300 Write block 760 len 40960 data: 4 4 4 4 4 5 6 5 5 5
018/300 Read block 869 len 28672
019/300 Read block 1410 len 36864
020/300 Read block 4248 len 8192
021/300 Read block 2209 len 40960
022/300 Write block 2378 len 28672 data: 11 6 6 5 4 4 4
023/300 Read block 1791 len 24576
024/300 Write block 2555 len 24576 data: 6 6 7 7 7 7
025/300 Read block 2560 len 40960
026/300 Write block 1400 len 40960 data: 8 7 6 7 6 9 9 8 8 7
027/300 Flush
028/300 Write block 2182 len 12288 data: 8 7 8
029/300 Read block 323 len 8192
030/300 Write block 4534 len 32768 data: 7 6 5 4 3 2 3 3
031/300 Read block 4281 len 28672
032/300 Write block 4785 len 20480 data: 5 6 5 5 6
033/300 Write block 3090 len 16384 data: 5 6 6 6
034/300 Read block 4721 len 28672
035/300 Read block 1148 len 16384
036/300 Read block 1133 len 24576
037/300 Read block 4018 len 28672
038/300 Read block 660 len 4096
039/300 Read block 3934 len 16384
040/300 Read block 3656 len 32768
041/300 Write block 2303 len 20480 data: 7 7 7 5 6
042/300 Read block 3677 len 32768
043/300 Write block 3817 len 20480 data: 4 3 3 2 2
044/300 Write block 861 len 36864 data: 10 9 7 8 7 9 9 9 6
045/300 Flush
046/300 Read block 1527 len 20480
047/300 Read block 2835 len 32768
048/300 Read block 2106 len 16384
049/300 Read block 1300 len 20480
050/300 Write block 4682 len 8192 data: 7 6
051/300 Flush
052/300 Flush
053/300 Read block 4500 len 16384
054/300 Write block 425 len 32768 data: 6 6 4 4 5 4 4 3
055/300 Read block 554 len 16384
056/300 Read block 4436 len 8192
057/300 Flush
058/300 Flush
059/300 Write block 3617 len 28672 data: 6 8 7 6 8 7 8
060/300 Write block 2346 len 8192 data: 5 6
061/300 Write block 3046 len 40960 data: 6 6 7 7 8 9 9 8 8 7
062/300 Read block 2703 len 36864
063/300 Read block 752 len 4096
064/300 Write block 4820 len 16384 data: 3 4 5 5
065/300 Read block 4663 len 12288
066/300 Write block 4938 len 40960 data: 6 6 5 5 5 6 6 4 4 4
067/300 Read block 1029 len 8192
068/300 Flush
069/300 Read block 2117 len 4096
070/300 Read block 4074 len 16384
071/300 Flush
072/300 Write block 3660 len 12288 data: 4 4 4
073/300 Read block 440 len 32768
074/300 Write block 4589 len 16384 data: 4 4 5 5
075/300 Read block 4882 len 8192
076/300 Read block 3285 len 24576
077/300 Flush
078/300 Flush
079/300 Read block 3452 len 20480
080/300 Flush
081/300 Read block 2536 len 12288
082/300 Write block 1395 len 28672 data: 6 6 7 8 9 9 8
083/300 Write block 2501 len 20480 data: 3 3 2 2 2
084/300 Read block 185 len 16384
085/300 Read block 1801 len 24576
086/300 Write block 1411 len 12288 data: 5 7 7
087/300 Write block 4164 len 40960 data: 9 9 8 8 8 7 6 5 5 7
088/300 Write block 2447 len 36864 data: 4 4 5 6 5 4 5 5 4
089/300 Write block 4866 len 36864 data: 6 9 8 9 8 6 5 2 2
090/300 Write block 2359 len 4096 data: 4
091/300 Flush
092/300 Read block 3133 len 32768
093/300 Write block 3804 len 36864 data: 3 4 4 4 4 6 7 6 6
094/300 Read block 4805 len 36864
095/300 Write block 2329 len 32768 data: 2 2 4 4 4 3 4 4
096/300 Write block 4764 len 8192 data: 7 7
097/300 Flush
098/300 Read block 2911 len 36864
099/300 Write block 4829 len 24576 data: 6 7 6 5 6 5
100/300 Read block 1365 len 24576
101/300 Write block 126 len 4096 data: 5
102/300 Read block 3473 len 28672
103/300 Flush
104/300 Flush
105/300 Read block 3371 len 8192
106/300 Write block 2139 len 36864 data: 3 4 5 5 5 9 7 7 5
107/300 Read block 3054 len 24576
108/300 Flush
109/300 Write block 2546 len 8192 data: 2 2
110/300 Read block 2040 len 20480
111/300 Write block 1821 len 24576 data: 7 6 3 3 4 4
112/300 Flush
113/300 Read block 1846 len 20480
114/300 Read block 4277 len 8192
115/300 Read block 1789 len 36864
116/300 Read block 4158 len 40960
117/300 Read block 1193 len 40960
118/300 Read block 1844 len 36864
119/300 Read block 2623 len 16384
120/300 Read block 4528 len 20480
121/300 Write block 2754 len 24576 data: 6 6 6 4 5 4
122/300 Read block 1518 len 4096
123/300 Write block 1290 len 32768 data: 3 3 4 4 7 6 6 8
124/300 Flush
125/300 Read block 3523 len 32768
126/300 Read block 536 len 32768
127/300 Read block 2933 len 40960
128/300 Flush
129/300 Write block 835 len 36864 data: 8 6 4 4 5 4 4 4 5
130/300 Read block 4888 len 4096
131/300 Write block 4790 len 16384 data: 8 8 7 6
132/300 Read block 3870 len 8192
133/300 Write block 4564 len 36864 data: 9 7 6 6 7 5 3 3 3
134/300 Write block 1488 len 32768 data: 4 3 4 5 4 5 5 4
135/300 Write block 4131 len 36864 data: 5 5 6 7 6 6 6 4 4
136/300 Read block 3614 len 8192
137/300 Write block 873 len 36864 data: 7 5 5 5 4 5 5 4 4
138/300 Read block 1569 len 40960
139/300 Write block 4237 len 20480 data: 4 5 4 5 5
140/300 Read block 1364 len 24576
141/300 Write block 1739 len 4096 data: 7
142/300 Read block 1379 len 20480
143/300 Read block 864 len 40960
144/300 Write block 2942 len 16384 data: 6 4 5 5
145/300 Write block 3716 len 40960 data: 8 8 7 5 7 7 6 6 7 8
146/300 Read block 2974 len 4096
147/300 Read block 612 len 40960
148/300 Read block 2846 len 12288
149/300 Read block 2356 len 4096
150/300 Read block 190 len 8192
151/300 Write block 2614 len 28672 data: 6 5 5 6 6 8 8
152/300 Write block 2372 len 8192 data: 8 8
153/300 Read block 255 len 28672
154/300 Write block 223 len 20480 data: 7 7 7 9 7
155/300 Flush
156/300 Read block 2025 len 40960
157/300 Write block 726 len 32768 data: 6 4 4 5 5 4 4 4
158/300 Read block 3453 len 12288
159/300 Read block 3948 len 28672
160/300 Read block 3933 len 20480
161/300 Flush
162/300 Read block 3937 len 20480
163/300 Write block 3691 len 12288 data: 7 8 7
164/300 Write block 1819 len 20480 data: 6 8 8 7 4
165/300 Write block 705 len 8192 data: 6 6
166/300 Write block 1815 len 32768 data: 11 9 8 6 7 9 9 8
167/300 Flush
168/300 Write block 3560 len 4096 data: 6
169/300 Write block 1056 len 20480 data: 2 4 6 7 7
170/300 Write block 3673 len 28672 data: 6 6 4 4 4 4 4
171/300 Write block 3688 len 16384 data: 9 9 8 8
172/300 Write block 852 len 12288 data: 3 3 3
173/300 Write block 4592 len 8192 data: 6 6
174/300 Write block 1744 len 8192 data: 6 5
175/300 Write block 3653 len 36864 data: 4 4 4 2 2 4 4 5 5
176/300 Write block 2018 len 20480 data: 6 5 5 5 5
177/300 Read block 1265 len 4096
178/300 Read block 1022 len 40960
179/300 Read block 2304 len 16384
180/300 Read block 2806 len 32768
181/300 Read block 3420 len 8192
182/300 Read block 2566 len 8192
183/300 Flush
184/300 Write block 1438 len 40960 data: 4 4 4 5 5 4 5 5 6 5
185/300 Flush
186/300 Read block 1779 len 40960
187/300 Read block 2267 len 8192
188/300 Write block 401 len 4096 data: 7
189/300 Read block 2655 len 16384
190/300 Write block 325 len 24576 data: 7 6 7 6 6 5
191/300 Write block 556 len 40960 data: 4 5 5 6 6 5 5 5 6 4
192/300 Write block 434 len 36864 data: 2 3 3 4 4 6 7 4 4
193/300 Flush
194/300 Read block 2584 len 12288
195/300 Write block 3699 len 36864 data: 3 3 3 3 5 4 5 5 7
196/300 Read block 3105 len 16384
197/300 Write block 4160 len 24576 data: 7 9 9 9 10 10
198/300 Flush
199/300 Write block 2920 len 12288 data: 6 4 3
200/300 Flush
201/300 Flush
202/300 Write block 790 len 8192 data: 5 5
203/300 Write block 1997 len 40960 data: 7 8 8 5 4 5 4 4 4 4
204/300 Read block 1560 len 40960
205/300 Read block 1348 len 8192
206/300 Write block 1227 len 36864 data: 5 5 3 2 4 3 5 9 10
207/300 Read block 1040 len 20480
208/300 Read block 2659 len 24576
209/300 Write block 1387 len 4096 data: 5
210/300 Read block 1893 len 12288
211/300 Read block 3077 len 16384
212/300 Write block 2612 len 12288 data: 6 6 7
213/300 Read block 2409 len 36864
214/300 Write block 377 len 12288 data: 5 5 5
215/300 Write block 4255 len 32768 data: 7 8 7 6 5 7 6 5
216/300 Read block 4069 len 4096
217/300 Read block 3822 len 4096
218/300 Read block 2456 len 20480
219/300 Write block 758 len 8192 data: 4 4
220/300 Write block 3187 len 24576 data: 5 5 5 6 6 6
221/300 Flush
222/300 Read block 2264 len 40960
223/300 Read block 1647 len 8192
224/300 Read block 258 len 12288
225/300 Read block 3957 len 36864
226/300 Read block 2216 len 40960
227/300 Write block 1108 len 8192 data: 3 3
228/300 Read block 1102 len 40960
229/300 Read block 4776 len 16384
230/300 Read block 1308 len 28672
231/300 Read block 884 len 8192
232/300 Write block 495 len 36864 data: 8 9 11 11 12 10 11 11 8
233/300 Write block 3404 len 16384 data: 4 5 4 4
234/300 Read block 4159 len 16384
235/300 Read block 4538 len 32768
236/300 Write block 2337 len 16384 data: 4 4 6 7
237/300 Read block 562 len 4096
238/300 Read block 1952 len 20480
239/300 Write block 3757 len 4096 data: 6
240/300 Write block 918 len 36864 data: 7 9 10 10 8 5 6 6 5
241/300 Read block 489 len 36864
242/300 Flush
243/300 Read block 970 len 36864
244/300 Write block 4244 len 32768 data: 5 7 8 9 10 10 11 8
245/300 Write block 4061 len 4096 data: 4
246/300 Read block 4323 len 36864
247/300 Write block 602 len 28672 data: 9 9 8 7 4 4 6
248/300 Write block 1829 len 12288 data: 2 2 2
249/300 Read block 2121 len 4096
250/300 Read block 103 len 40960
251/300 Write block 4181 len 24576 data: 3 3 2 2 3 4
252/300 Write block 2242 len 28672 data: 6 6 6 7 8 9 8
253/300 Flush
254/300 Read block 821 len 16384
255/300 Read block 4209 len 32768
256/300 Read block 4722 len 16384
257/300 Read block 2700 len 28672
258/300 Write block 2665 len 16384 data: 6 5 3 2
259/300 Read block 2053 len 8192
260/300 Write block 3703 len 16384 data: 6 5 6 6
261/300 Read block 4656 len 20480
262/300 Write block 37 len 4096 data: 5
263/300 Read block 1684 len 36864
264/300 Write block 1957 len 4096 data: 6
265/300 Read block 3290 len 12288
266/300 Read block 2446 len 4096
267/300 Read block 4684 len 40960
268/300 Write block 4193 len 20480 data: 9 9 9 9 9
269/300 Read block 2070 len 24576
270/300 Write block 2909 len 8192 data: 4 4
271/300 Read block 2139 len 28672
272/300 Flush
273/300 Read block 2432 len 40960
274/300 Write block 2615 len 32768 data: 6 6 7 7 9 9 9 9
275/300 Read block 4867 len 20480
276/300 Read block 3329 len 4096
277/300 Write block 61 len 8192 data: 6 8
278/300 Read block 2857 len 4096
279/300 Flush
280/300 Write block 3894 len 4096 data: 6
281/300 Write block 2106 len 4096 data: 6
282/300 Read block 1800 len 36864
283/300 Read block 3444 len 24576
284/300 Read block 2973 len 32768
285/300 Read block 4957 len 32768
286/300 Read block 1227 len 16384
287/300 Write block 4892 len 28672 data: 4 4 4 3 4 5 5
288/300 Write block 2912 len 20480 data: 4 4 3 2 2
289/300 Write block 785 len 36864 data: 3 5 5 4 5 6 6 4 4
290/300 Read block 791 len 36864
291/300 Read block 4458 len 24576
292/300 Write block 2282 len 4096 data: 7
293/300 Write block 572 len 12288 data: 4 5 5
294/300 Write block 2524 len 16384 data: 5 4 3 3
295/300 Flush
296/300 Read block 4028 len 28672
297/300 Read block 3799 len 28672
298/300 Read block 2740 len 12288
299/300 Flush
300/300 Read block 4046 len 8192
301/300 Write block 3442 len 12288 data: 5 5 5
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:42:23.344 INFO request completed, latency_us: 164, response_code: 204, uri: /start/cid/2, method: POST, req_id: f67f2bc2-32ae-4db5-b772-65ea82b7b2fc, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8546 Acked Write 5 Done Done Sent false
8547 Acked Read 7 Done Done Sent false
8548 Acked Read 10 Done Done Sent false
8549 Acked Read 8 Done Done Sent false
8550 Acked Read 4 Done Done Sent false
8551 Acked Read 5 Done Done Sent false
8552 Acked Flush 0 Done Done Sent false
8553 Acked Flush 0 Done Done Sent false
8554 Acked Write 8 Done Done Sent false
8555 Acked Write 2 Done Done Sent false
8556 Acked Write 7 Done Done Sent false
8557 Acked Read 10 Done Done Sent false
8558 Acked Read 1 Done Done Sent false
8559 Acked Read 8 Done Done Sent false
8560 Acked Write 9 Done Done Sent false
8561 Acked Read 5 Done Done Sent false
8562 Acked Write 10 Done Done Sent false
8563 Acked Read 7 Done Done Sent false
8564 Acked Read 9 Done Done Sent false
8565 Acked Read 2 Done Done Sent false
8566 Acked Read 10 Done Done Sent false
8567 Acked Write 7 Done Done Sent false
8568 Acked Read 6 Done Done Sent false
8569 Acked Write 6 Done Done Sent false
8570 Acked Read 10 Done Done Sent false
8571 Acked Write 10 Done Done Sent false
8572 Acked Flush 0 Done Done Sent false
8573 Acked Write 3 Done Done Sent false
8574 Acked Read 2 Done Done Sent false
8575 Acked Write 8 Done Done Sent false
8576 Acked Read 7 Done Done Sent false
8577 Acked Write 5 Done Done Sent false
8578 Acked Write 4 Done Done Sent false
8579 Acked Read 7 Done Done Sent false
8580 Acked Read 4 Done Done Sent false
8581 Acked Read 6 Done Done Sent false
8582 Acked Read 7 Done Done Sent false
8583 Acked Read 1 Done Done Sent false
8584 Acked Read 4 Done Done Sent false
8585 Acked Read 8 Done Done Sent false
8586 Acked Write 5 Done Done Sent false
8587 Acked Read 8 Done Done Sent false
8588 Acked Write 5 Done Done Sent false
8589 Acked Write 9 Done Done Sent false
8590 Acked Flush 0 Done Done Sent false
8591 Acked Read 5 Done Done Sent false
8592 Acked Read 8 Done Done Sent false
8593 Acked Read 4 Done Done Sent false
8594 Acked Read 5 Done Done Sent false
8595 Acked Write 2 Done Done Sent false
8596 Acked Flush 0 Done Done Sent false
8597 Acked Flush 0 Done Done Sent false
8598 Acked Read 4 Done Done Sent false
8599 Acked Write 8 Done Done Sent false
8600 Acked Read 4 Done Done Sent false
8601 Acked Read 2 Done Done Sent false
8602 Acked Flush 0 Done Done Sent false
8603 Acked Flush 0 Done Done Sent false
8604 Acked Write 7 Done Done Sent false
8605 Acked Write 2 Done Done Sent false
8606 Acked Write 10 Done Done Sent false
8607 Acked Read 9 Done Done Sent false
8608 Acked Read 1 Done Done Sent false
8609 Acked Write 4 Done Done Sent false
8610 Acked Read 3 Done Done Sent false
8611 Acked Write 10 Done Done Sent false
8612 Acked Read 2 Done Done Sent false
8613 Acked Flush 0 Done Done Sent false
8614 Acked Read 1 Done Done Sent false
8615 Acked Read 4 Done Done Sent false
8616 Acked Flush 0 Done Done Sent false
8617 Acked Write 3 Done Done Sent false
8618 Acked Read 8 Done Done Sent false
8619 Acked Write 4 Done Done Sent false
8620 Acked Read 2 Done Done Sent false
8621 Acked Read 6 Done Done Sent false
8622 Acked Flush 0 Done Done Sent false
8623 Acked Flush 0 Done Done Sent false
8624 Acked Read 5 Done Done Sent false
8625 Acked Flush 0 Done Done Sent false
8626 Acked Read 3 Done Done Sent false
8627 Acked Write 7 Done Done Sent false
8628 Acked Write 5 Done Done Sent false
8629 Acked Read 4 Done Done Sent false
8630 Acked Read 6 Done Done Sent false
8631 Acked Write 3 Done Done Sent false
8632 Acked Write 10 Done Done Sent false
8633 Acked Write 9 Done Done Sent false
8634 Acked Write 9 Done Done Sent false
8635 Acked Write 1 Done Done Sent false
8636 Acked Flush 0 Done Done Sent false
8637 Acked Read 8 Done Done Sent false
8638 Acked Write 9 Done Done Sent false
8639 Acked Read 9 Done Done Sent false
8640 Acked Write 8 Done Done Sent false
8641 Acked Write 2 Done Done Sent false
8642 Acked Flush 0 Done Done Sent false
8643 Acked Read 9 Done Done Sent false
8644 Acked Write 6 Done Done Sent false
8645 Acked Read 6 Done Done Sent false
8646 Acked Write 1 Done Done Sent false
8647 Acked Read 7 Done Done Sent false
8648 Acked Flush 0 Done Done Sent false
8649 Acked Flush 0 Done Done Sent false
8650 Acked Read 2 Done Done Sent false
8651 Acked Write 9 Done Done Sent false
8652 Acked Read 6 Done Done Sent false
8653 Acked Flush 0 Done Done Sent false
8654 Acked Write 2 Done Done Sent false
8655 Acked Read 5 Done Done Sent false
8656 Acked Write 6 Done Done Sent false
8657 Acked Flush 0 Done Done Sent false
8658 Acked Read 5 Done Done Sent false
8659 Acked Read 2 Done Done Sent false
8660 Acked Read 9 Done Done Sent false
8661 Acked Read 10 Done Done Sent false
8662 Acked Read 10 Done Done Sent false
8663 Acked Read 9 Done Done Sent false
8664 Acked Read 4 Done Done Sent false
8665 Acked Read 5 Done Done Sent false
8666 Acked Write 6 Done Done Sent false
8667 Acked Read 1 Done Done Sent false
8668 Acked Write 8 Done Done Sent false
8669 Acked Flush 0 Done Done Sent false
8670 Acked Read 8 Done Done Sent false
8671 Acked Read 8 Done Done Sent false
8672 Acked Read 10 Done Done Sent false
8673 Acked Flush 0 Done Done Sent false
8674 Acked Write 9 Done Done Sent false
8675 Acked Read 1 Done Done Sent false
8676 Acked Write 4 Done Done Sent false
8677 Acked Read 2 Done Done Sent false
8678 Acked Write 9 Done Done Sent false
8679 Acked Write 8 Done Done Sent false
8680 Acked Write 9 Done Done Sent false
8681 Acked Read 2 Done Done Sent false
8682 Acked Write 9 Done Done Sent false
8683 Acked Read 10 Done Done Sent false
8684 Acked Write 5 Done Done Sent false
8685 Acked Read 6 Done Done Sent false
8686 Acked Write 1 Done Done Sent false
8687 Acked Read 5 Done Done Sent false
8688 Acked Read 10 Done Done Sent false
8689 Acked Write 4 Done Done Sent false
8690 Acked Write 10 Done Done Sent false
8691 Acked Read 1 Done Done Sent false
8692 Acked Read 10 Done Done Sent false
8693 Acked Read 3 Done Done Sent false
8694 Acked Read 1 Done Done Sent false
8695 Acked Read 2 Done Done Sent false
8696 Acked Write 7 Done Done Sent false
8697 Acked Write 2 Done Done Sent false
8698 Acked Read 7 Done Done Sent false
8699 Acked Write 5 Done Done Sent false
8700 Acked Flush 0 Done Done Sent false
8701 Acked Read 10 Done Done Sent false
8702 Acked Write 8 Done Done Sent false
8703 Acked Read 3 Done Done Sent false
8704 Acked Read 7 Done Done Sent false
8705 Acked Read 5 Done Done Sent false
8706 Acked Flush 0 Done Done Sent false
8707 Acked Read 5 Done Done Sent false
8708 Acked Write 3 Done Done Sent false
8709 Acked Write 5 Done Done Sent false
8710 Acked Write 2 Done Done Sent false
8711 Acked Write 8 Done Done Sent false
8712 Acked Flush 0 Done Done Sent false
8713 Acked Write 1 Done Done Sent false
8714 Acked Write 5 Done Done Sent false
8715 Acked Write 7 Done Done Sent false
8716 Acked Write 4 Done Done Sent false
8717 Acked Write 3 Done Done Sent false
8718 Acked Write 2 Done Done Sent false
8719 Acked Write 2 Done Done Sent false
8720 Acked Write 9 Done Done Sent false
8721 Acked Write 5 Done Done Sent false
8722 Acked Read 1 Done Done Sent false
8723 Acked Read 10 Done Done Sent false
8724 Acked Read 4 Done Done Sent false
8725 Acked Read 8 Done Done Sent false
8726 Acked Read 2 Done Done Sent false
8727 Acked Read 2 Done Done Sent false
8728 Acked Flush 0 Done Done Sent false
8729 Acked Write 10 Done Done Sent false
8730 Acked Flush 0 Done Done Sent false
8731 Acked Read 10 Done Done Sent false
8732 Acked Read 2 Done Done Sent false
8733 Acked Write 1 Done Done Sent false
8734 Acked Read 4 Done Done Sent false
8735 Acked Write 6 Done Done Sent false
8736 Acked Write 10 Done Done Sent false
8737 Acked Write 9 Done Done Sent false
8738 Acked Flush 0 Done Done Sent false
8739 Acked Read 3 Done Done Sent false
8740 Acked Write 9 Done Done Sent false
8741 Acked Read 4 Done Done Sent false
8742 Acked Write 6 Done Done Sent false
8743 Acked Flush 0 Done Done Sent false
8744 Acked Write 3 Done Done Sent false
8745 Acked Flush 0 Done Done Sent false
8746 Acked Flush 0 Done Done Sent false
8747 Acked Write 2 Done Done Sent false
8748 Acked Write 10 Done Done Sent false
8749 Acked Read 10 Done Done Sent false
8750 Acked Read 2 Done Done Sent false
8751 Acked Write 9 Done Done Sent false
8752 Acked Read 5 Done Done Sent false
8753 Acked Read 6 Done Done Sent false
8754 Acked Write 1 Done Done Sent false
8755 Acked Read 3 Done Done Sent false
8756 Acked Read 4 Done Done Sent false
8757 Acked Write 3 Done Done Sent false
8758 Acked Read 9 Done Done Sent false
8759 Acked Write 3 Done Done Sent false
8760 Acked Write 8 Done Done Sent false
8761 Acked Read 1 Done Done Sent false
8762 Acked Read 1 Done Done Sent false
8763 Acked Read 5 Done Done Sent false
8764 Acked Write 2 Done Done Sent false
8765 Acked Write 6 Done Done Sent false
8766 Acked Flush 0 Done Done Sent false
8767 Acked Read 10 Done Done Sent false
8768 Acked Read 2 Done Done Sent false
8769 Acked Read 3 Done Done Sent false
8770 Acked Read 9 Done Done Sent false
8771 Acked Read 10 Done Done Sent false
8772 Acked Write 2 Done Done Sent false
8773 Acked Read 10 Done Done Sent false
8774 Acked Read 4 Done Done Sent false
8775 Acked Read 7 Done Done Sent false
8776 Acked Read 2 Done Done Sent false
8777 Acked Write 9 Done Done Sent false
8778 Acked Write 4 Done Done Sent false
8779 Acked Read 4 Done Done Sent false
8780 Acked Read 8 Done Done Sent false
8781 Acked Write 4 Done Done Sent false
8782 Acked Read 1 Done Done Sent false
8783 Acked Read 5 Done Done Sent false
8784 Acked Write 1 Done Done Sent false
8785 Acked Write 9 Done Done Sent false
8786 Acked Read 9 Done Done Sent false
8787 Acked Flush 0 Done Done Sent false
8788 Acked Read 9 Done Done Sent false
8789 Acked Write 8 Done Done Sent false
8790 Acked Write 1 Done Done Sent false
8791 Acked Read 9 Done Done Sent false
8792 Acked Write 7 Done Done Sent false
8793 Acked Write 3 Done Done Sent false
8794 Acked Read 1 Done Done Sent false
8795 Acked Read 10 Done Done Sent false
8796 Acked Write 6 Done Done Sent false
8797 Acked Write 7 Done Done Sent false
8798 Acked Flush 0 Done Done Sent false
8799 Acked Read 4 Done Done Sent false
8800 Acked Read 8 Done Done Sent false
8801 Acked Read 4 Done Done Sent false
8802 Acked Read 7 Done Done Sent false
8803 Acked Write 4 Done Done Sent false
8804 Acked Read 2 Done Done Sent false
8805 Acked Write 4 Done Done Sent false
8806 Acked Read 5 Done Done Sent false
8807 Acked Write 1 Done Done Sent false
8808 Acked Read 9 Done Done Sent false
8809 Acked Write 1 Done Done Sent false
8810 Acked Read 3 Done Done Sent false
8811 Acked Read 1 Done Done Sent false
8812 Acked Read 10 Done Done Sent false
8813 Acked Write 5 Done Done Sent false
8814 Acked Read 6 Done Done Sent false
8815 Acked Write 2 Done Done Sent false
8816 Acked Read 7 Done Done Sent false
8817 Acked Flush 0 Done Done Sent false
8818 Acked Read 10 Done Done Sent false
8819 Acked Write 8 Done Done Sent false
8820 Acked Read 5 Done Done Sent false
8821 Acked Read 1 Done Done Sent false
8822 Acked Write 2 Done Done Sent false
8823 Acked Read 1 Done Done Sent false
8824 Acked Flush 0 Done Done Sent false
8825 Acked Write 1 Done Done Sent false
8826 Acked Write 1 Done Done Sent false
8827 Acked Read 9 Done Done Sent false
8828 Acked Read 6 Done Done Sent false
8829 Acked Read 8 Done Done Sent false
8830 Acked Read 8 Done Done Sent false
8831 Acked Read 4 Done Done Sent false
8832 Acked Write 7 Done Done Sent false
8833 Acked Write 5 Done Done Sent false
8834 Acked Write 9 Done Done Sent false
8835 Acked Read 9 Done Done Sent false
8836 Acked Read 6 Done Done Sent false
8837 Acked Write 1 Done Done Sent false
8838 Acked Write 3 Done Done Sent false
8839 Acked Write 4 Done Done Sent false
8840 Acked Flush 0 Done Done Sent false
8841 Acked Read 7 Done Done Sent false
8842 Acked Read 7 Done Done Sent false
8843 Acked Read 3 Done Done Sent false
8844 Acked Flush 0 Done Done Sent false
8845 Acked Read 2 Done Done Sent false
8846 Acked Write 3 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8844 8844 8545
Downstairs last five completed: 8545 8544 8543 8542 8541
Upstairs last five completed: 8846 8845 8844 8843 8842
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8546 Acked Write 5 Done Done Sent false
8547 Acked Read 7 Done Done Sent false
8548 Acked Read 10 Done Done Sent false
8549 Acked Read 8 Done Done Sent false
8550 Acked Read 4 Done Done Sent false
8551 Acked Read 5 Done Done Sent false
8552 Acked Flush 0 Done Done Sent false
8553 Acked Flush 0 Done Done Sent false
8554 Acked Write 8 Done Done Sent false
8555 Acked Write 2 Done Done Sent false
8556 Acked Write 7 Done Done Sent false
8557 Acked Read 10 Done Done Sent false
8558 Acked Read 1 Done Done Sent false
8559 Acked Read 8 Done Done Sent false
8560 Acked Write 9 Done Done Sent false
8561 Acked Read 5 Done Done Sent false
8562 Acked Write 10 Done Done Sent false
8563 Acked Read 7 Done Done Sent false
8564 Acked Read 9 Done Done Sent false
8565 Acked Read 2 Done Done Sent false
8566 Acked Read 10 Done Done Sent false
8567 Acked Write 7 Done Done Sent false
8568 Acked Read 6 Done Done Sent false
8569 Acked Write 6 Done Done Sent false
8570 Acked Read 10 Done Done Sent false
8571 Acked Write 10 Done Done Sent false
8572 Acked Flush 0 Done Done Sent false
8573 Acked Write 3 Done Done Sent false
8574 Acked Read 2 Done Done Sent false
8575 Acked Write 8 Done Done Sent false
8576 Acked Read 7 Done Done Sent false
8577 Acked Write 5 Done Done Sent false
8578 Acked Write 4 Done Done Sent false
8579 Acked Read 7 Done Done Sent false
8580 Acked Read 4 Done Done Sent false
8581 Acked Read 6 Done Done Sent false
8582 Acked Read 7 Done Done Sent false
8583 Acked Read 1 Done Done Sent false
8584 Acked Read 4 Done Done Sent false
8585 Acked Read 8 Done Done Sent false
8586 Acked Write 5 Done Done Sent false
8587 Acked Read 8 Done Done Sent false
8588 Acked Write 5 Done Done Sent false
8589 Acked Write 9 Done Done Sent false
8590 Acked Flush 0 Done Done Sent false
8591 Acked Read 5 Done Done Sent false
8592 Acked Read 8 Done Done Sent false
8593 Acked Read 4 Done Done Sent false
8594 Acked Read 5 Done Done Sent false
8595 Acked Write 2 Done Done Sent false
8596 Acked Flush 0 Done Done Sent false
8597 Acked Flush 0 Done Done Sent false
8598 Acked Read 4 Done Done Sent false
8599 Acked Write 8 Done Done Sent false
8600 Acked Read 4 Done Done Sent false
8601 Acked Read 2 Done Done Sent false
8602 Acked Flush 0 Done Done Sent false
8603 Acked Flush 0 Done Done Sent false
8604 Acked Write 7 Done Done Sent false
8605 Acked Write 2 Done Done Sent false
8606 Acked Write 10 Done Done Sent false
8607 Acked Read 9 Done Done Sent false
8608 Acked Read 1 Done Done Sent false
8609 Acked Write 4 Done Done Sent false
8610 Acked Read 3 Done Done Sent false
8611 Acked Write 10 Done Done Sent false
8612 Acked Read 2 Done Done Sent false
8613 Acked Flush 0 Done Done Sent false
8614 Acked Read 1 Done Done Sent false
8615 Acked Read 4 Done Done Sent false
8616 Acked Flush 0 Done Done Sent false
8617 Acked Write 3 Done Done Sent false
8618 Acked Read 8 Done Done Sent false
8619 Acked Write 4 Done Done Sent false
8620 Acked Read 2 Done Done Sent false
8621 Acked Read 6 Done Done Sent false
8622 Acked Flush 0 Done Done Sent false
8623 Acked Flush 0 Done Done Sent false
8624 Acked Read 5 Done Done Sent false
8625 Acked Flush 0 Done Done Sent false
8626 Acked Read 3 Done Done Sent false
8627 Acked Write 7 Done Done Sent false
8628 Acked Write 5 Done Done Sent false
8629 Acked Read 4 Done Done Sent false
8630 Acked Read 6 Done Done Sent false
8631 Acked Write 3 Done Done Sent false
8632 Acked Write 10 Done Done Sent false
8633 Acked Write 9 Done Done Sent false
8634 Acked Write 9 Done Done Sent false
8635 Acked Write 1 Done Done Sent false
8636 Acked Flush 0 Done Done Sent false
8637 Acked Read 8 Done Done Sent false
8638 Acked Write 9 Done Done Sent false
8639 Acked Read 9 Done Done Sent false
8640 Acked Write 8 Done Done Sent false
8641 Acked Write 2 Done Done Sent false
8642 Acked Flush 0 Done Done Sent false
8643 Acked Read 9 Done Done Sent false
8644 Acked Write 6 Done Done Sent false
8645 Acked Read 6 Done Done Sent false
8646 Acked Write 1 Done Done Sent false
8647 Acked Read 7 Done Done Sent false
8648 Acked Flush 0 Done Done Sent false
8649 Acked Flush 0 Done Done Sent false
8650 Acked Read 2 Done Done Sent false
8651 Acked Write 9 Done Done Sent false
8652 Acked Read 6 Done Done Sent false
8653 Acked Flush 0 Done Done Sent false
8654 Acked Write 2 Done Done Sent false
8655 Acked Read 5 Done Done Sent false
8656 Acked Write 6 Done Done Sent false
8657 Acked Flush 0 Done Done Sent false
8658 Acked Read 5 Done Done Sent false
8659 Acked Read 2 Done Done Sent false
8660 Acked Read 9 Done Done Sent false
8661 Acked Read 10 Done Done Sent false
8662 Acked Read 10 Done Done Sent false
8663 Acked Read 9 Done Done Sent false
8664 Acked Read 4 Done Done Sent false
8665 Acked Read 5 Done Done Sent false
8666 Acked Write 6 Done Done Sent false
8667 Acked Read 1 Done Done Sent false
8668 Acked Write 8 Done Done Sent false
8669 Acked Flush 0 Done Done Sent false
8670 Acked Read 8 Done Done Sent false
8671 Acked Read 8 Done Done Sent false
8672 Acked Read 10 Done Done Sent false
8673 Acked Flush 0 Done Done Sent false
8674 Acked Write 9 Done Done Sent false
8675 Acked Read 1 Done Done Sent false
8676 Acked Write 4 Done Done Sent false
8677 Acked Read 2 Done Done Sent false
8678 Acked Write 9 Done Done Sent false
8679 Acked Write 8 Done Done Sent false
8680 Acked Write 9 Done Done Sent false
8681 Acked Read 2 Done Done Sent false
8682 Acked Write 9 Done Done Sent false
8683 Acked Read 10 Done Done Sent false
8684 Acked Write 5 Done Done Sent false
8685 Acked Read 6 Done Done Sent false
8686 Acked Write 1 Done Done Sent false
8687 Acked Read 5 Done Done Sent false
8688 Acked Read 10 Done Done Sent false
8689 Acked Write 4 Done Done Sent false
8690 Acked Write 10 Done Done Sent false
8691 Acked Read 1 Done Done Sent false
8692 Acked Read 10 Done Done Sent false
8693 Acked Read 3 Done Done Sent false
8694 Acked Read 1 Done Done Sent false
8695 Acked Read 2 Done Done Sent false
8696 Acked Write 7 Done Done Sent false
8697 Acked Write 2 Done Done Sent false
8698 Acked Read 7 Done Done Sent false
8699 Acked Write 5 Done Done Sent false
8700 Acked Flush 0 Done Done Sent false
8701 Acked Read 10 Done Done Sent false
8702 Acked Write 8 Done Done Sent false
8703 Acked Read 3 Done Done Sent false
8704 Acked Read 7 Done Done Sent false
8705 Acked Read 5 Done Done Sent false
8706 Acked Flush 0 Done Done Sent false
8707 Acked Read 5 Done Done Sent false
8708 Acked Write 3 Done Done Sent false
8709 Acked Write 5 Done Done Sent false
8710 Acked Write 2 Done Done Sent false
8711 Acked Write 8 Done Done Sent false
8712 Acked Flush 0 Done Done Sent false
8713 Acked Write 1 Done Done Sent false
8714 Acked Write 5 Done Done Sent false
8715 Acked Write 7 Done Done Sent false
8716 Acked Write 4 Done Done Sent false
8717 Acked Write 3 Done Done Sent false
8718 Acked Write 2 Done Done Sent false
8719 Acked Write 2 Done Done Sent false
8720 Acked Write 9 Done Done Sent false
8721 Acked Write 5 Done Done Sent false
8722 Acked Read 1 Done Done Sent false
8723 Acked Read 10 Done Done Sent false
8724 Acked Read 4 Done Done Sent false
8725 Acked Read 8 Done Done Sent false
8726 Acked Read 2 Done Done Sent false
8727 Acked Read 2 Done Done Sent false
8728 Acked Flush 0 Done Done Sent false
8729 Acked Write 10 Done Done Sent false
8730 Acked Flush 0 Done Done Sent false
8731 Acked Read 10 Done Done Sent false
8732 Acked Read 2 Done Done Sent false
8733 Acked Write 1 Done Done Sent false
8734 Acked Read 4 Done Done Sent false
8735 Acked Write 6 Done Done Sent false
8736 Acked Write 10 Done Done Sent false
8737 Acked Write 9 Done Done Sent false
8738 Acked Flush 0 Done Done Sent false
8739 Acked Read 3 Done Done Sent false
8740 Acked Write 9 Done Done Sent false
8741 Acked Read 4 Done Done Sent false
8742 Acked Write 6 Done Done Sent false
8743 Acked Flush 0 Done Done Sent false
8744 Acked Write 3 Done Done Sent false
8745 Acked Flush 0 Done Done Sent false
8746 Acked Flush 0 Done Done Sent false
8747 Acked Write 2 Done Done Sent false
8748 Acked Write 10 Done Done Sent false
8749 Acked Read 10 Done Done Sent false
8750 Acked Read 2 Done Done Sent false
8751 Acked Write 9 Done Done Sent false
8752 Acked Read 5 Done Done Sent false
8753 Acked Read 6 Done Done Sent false
8754 Acked Write 1 Done Done Sent false
8755 Acked Read 3 Done Done Sent false
8756 Acked Read 4 Done Done Sent false
8757 Acked Write 3 Done Done Sent false
8758 Acked Read 9 Done Done Sent false
8759 Acked Write 3 Done Done Sent false
8760 Acked Write 8 Done Done Sent false
8761 Acked Read 1 Done Done Sent false
8762 Acked Read 1 Done Done Sent false
8763 Acked Read 5 Done Done Sent false
8764 Acked Write 2 Done Done Sent false
8765 Acked Write 6 Done Done Sent false
8766 Acked Flush 0 Done Done Sent false
8767 Acked Read 10 Done Done Sent false
8768 Acked Read 2 Done Done Sent false
8769 Acked Read 3 Done Done Sent false
8770 Acked Read 9 Done Done Sent false
8771 Acked Read 10 Done Done Sent false
8772 Acked Write 2 Done Done Sent false
8773 Acked Read 10 Done Done Sent false
8774 Acked Read 4 Done Done Sent false
8775 Acked Read 7 Done Done Sent false
8776 Acked Read 2 Done Done Sent false
8777 Acked Write 9 Done Done Sent false
8778 Acked Write 4 Done Done Sent false
8779 Acked Read 4 Done Done Sent false
8780 Acked Read 8 Done Done Sent false
8781 Acked Write 4 Done Done Sent false
8782 Acked Read 1 Done Done Sent false
8783 Acked Read 5 Done Done Sent false
8784 Acked Write 1 Done Done Sent false
8785 Acked Write 9 Done Done Sent false
8786 Acked Read 9 Done Done Sent false
8787 Acked Flush 0 Done Done Sent false
8788 Acked Read 9 Done Done Sent false
8789 Acked Write 8 Done Done Sent false
8790 Acked Write 1 Done Done Sent false
8791 Acked Read 9 Done Done Sent false
8792 Acked Write 7 Done Done Sent false
8793 Acked Write 3 Done Done Sent false
8794 Acked Read 1 Done Done Sent false
8795 Acked Read 10 Done Done Sent false
8796 Acked Write 6 Done Done Sent false
8797 Acked Write 7 Done Done Sent false
8798 Acked Flush 0 Done Done Sent false
8799 Acked Read 4 Done Done Sent false
8800 Acked Read 8 Done Done Sent false
8801 Acked Read 4 Done Done Sent false
8802 Acked Read 7 Done Done Sent false
8803 Acked Write 4 Done Done Sent false
8804 Acked Read 2 Done Done Sent false
8805 Acked Write 4 Done Done Sent false
8806 Acked Read 5 Done Done Sent false
8807 Acked Write 1 Done Done Sent false
8808 Acked Read 9 Done Done Sent false
8809 Acked Write 1 Done Done Sent false
8810 Acked Read 3 Done Done Sent false
8811 Acked Read 1 Done Done Sent false
8812 Acked Read 10 Done Done Sent false
8813 Acked Write 5 Done Done Sent false
8814 Acked Read 6 Done Done Sent false
8815 Acked Write 2 Done Done Sent false
8816 Acked Read 7 Done Done Sent false
8817 Acked Flush 0 Done Done Sent false
8818 Acked Read 10 Done Done Sent false
8819 Acked Write 8 Done Done Sent false
8820 Acked Read 5 Done Done Sent false
8821 Acked Read 1 Done Done Sent false
8822 Acked Write 2 Done Done Sent false
8823 Acked Read 1 Done Done Sent false
8824 Acked Flush 0 Done Done Sent false
8825 Acked Write 1 Done Done Sent false
8826 Acked Write 1 Done Done Sent false
8827 Acked Read 9 Done Done Sent false
8828 Acked Read 6 Done Done Sent false
8829 Acked Read 8 Done Done Sent false
8830 Acked Read 8 Done Done Sent false
8831 Acked Read 4 Done Done Sent false
8832 Acked Write 7 Done Done Sent false
8833 Acked Write 5 Done Done Sent false
8834 Acked Write 9 Done Done Sent false
8835 Acked Read 9 Done Done Sent false
8836 Acked Read 6 Done Done Sent false
8837 Acked Write 1 Done Done Sent false
8838 Acked Write 3 Done Done Sent false
8839 Acked Write 4 Done Done Sent false
8840 Acked Flush 0 Done Done Sent false
8841 Acked Read 7 Done Done Sent false
8842 Acked Read 7 Done Done Sent false
8843 Acked Read 3 Done Done Sent false
8844 Acked Flush 0 Done Done Sent false
8845 Acked Read 2 Done Done Sent false
8846 Acked Write 3 Done Done Sent false
8847 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8847 8847 8545
Downstairs last five completed: 8545 8544 8543 8542 8541
Upstairs last five completed: 8847 8846 8845 8844 8843
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.10495684Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41890)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.105159636Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.106295554Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.106885235Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.106949398Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 8545","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.106985333Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 8545","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.107260791Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 8545","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.107292898Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 8545","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:29.107512592Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8847 8846 8845 8844 8843
Upstairs last five completed: 8847 8846 8845 8844 8843
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:42:31.349 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
, req_id: ff72eda5-63dc-4484-b46b-b159014a8a84, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:31.350584245Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:31.35063361Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:31.350649918Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:31.350724399Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:42:31.350 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/2, method: GET, req_id: 76737829-05e0-4df0-b1ca-f79d502e691f, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:42:35.354 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/2, method: GET, req_id: 45cd4c02-4d92-4801-9cc1-f98dd2e5b2f0, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 3055 len 36864
002/300 Write block 1467 len 24576 data: 7 6 4 6 6 4
003/300 Read block 1295 len 40960
004/300 Read block 2859 len 16384
005/300 Write block 3120 len 4096 data: 5
006/300 Write block 3984 len 20480 data: 4 3 3 4 4
007/300 Write block 75 len 28672 data: 3 4 4 5 6 5 4
008/300 Read block 4727 len 20480
009/300 Read block 2175 len 20480
010/300 Read block 349 len 24576
011/300 Read block 3951 len 24576
012/300 Read block 2995 len 24576
013/300 Write block 4136 len 36864 data: 7 7 5 5 4 5 5 5 5
014/300 Write block 3945 len 28672 data: 2 3 3 3 2 3 3
015/300 Read block 2820 len 40960
016/300 Read block 582 len 32768
017/300 Flush
018/300 Write block 3556 len 28672 data: 7 8 7 7 7 7 8
019/300 Write block 4048 len 36864 data: 4 3 3 2 2 3 3 4 6
020/300 Write block 3096 len 24576 data: 4 4 7 7 6 7
021/300 Read block 2775 len 28672
022/300 Write block 1198 len 20480 data: 8 6 5 5 5
023/300 Write block 2506 len 36864 data: 4 5 7 7 5 6 7 6 7
024/300 Write block 2170 len 36864 data: 4 6 7 8 7 8 9 7 7
025/300 Read block 2258 len 12288
026/300 Write block 1597 len 12288 data: 11 12 8
027/300 Read block 1482 len 24576
028/300 Read block 1841 len 24576
029/300 Read block 3914 len 32768
030/300 Write block 2008 len 32768 data: 6 6 6 9 6 7 6 6
031/300 Write block 3216 len 8192 data: 7 5
032/300 Write block 1480 len 36864 data: 9 9 8 7 6 5 5 5 5
033/300 Write block 4639 len 28672 data: 6 5 5 4 6 6 5
034/300 Read block 370 len 20480
035/300 Read block 652 len 12288
036/300 Write block 2590 len 8192 data: 8 7
037/300 Read block 1184 len 24576
038/300 Read block 2691 len 32768
039/300 Read block 3277 len 36864
040/300 Read block 446 len 4096
041/300 Flush
042/300 Flush
043/300 Read block 3878 len 8192
044/300 Write block 650 len 20480 data: 4 3 3 6 6
045/300 Write block 4739 len 20480 data: 7 7 7 6 5
046/300 Write block 4447 len 36864 data: 6 6 6 5 5 4 6 5 5
047/300 Write block 1721 len 8192 data: 5 6
048/300 Read block 4399 len 32768
049/300 Read block 965 len 20480
050/300 Read block 202 len 8192
051/300 Read block 175 len 40960
052/300 Write block 1136 len 12288 data: 5 6 5
053/300 Flush
054/300 Write block 4062 len 20480 data: 5 5 4 4 4
055/300 Write block 4536 len 12288 data: 6 5 4
056/300 Read block 1085 len 32768
057/300 Write block 1827 len 4096 data: 4
058/300 Read block 413 len 4096
059/300 Read block 4938 len 28672
060/300 Read block 3651 len 32768
061/300 Read block 480 len 20480
062/300 Write block 4603 len 12288 data: 7 5 5
063/300 Write block 4646 len 40960 data: 6 7 6 6 4 5 5 5 5 5
064/300 Read block 3045 len 20480
065/300 Read block 309 len 40960
066/300 Read block 4625 len 40960
067/300 Read block 4535 len 36864
068/300 Read block 709 len 4096
069/300 Flush
070/300 Write block 2627 len 36864 data: 5 6 6 5 5 6 8 6 6
071/300 Flush
072/300 Read block 1498 len 20480
073/300 Read block 1020 len 24576
074/300 Write block 4292 len 36864 data: 5 4 5 6 4 6 6 6 4
075/300 Read block 4217 len 40960
076/300 Read block 585 len 28672
077/300 Read block 4697 len 8192
078/300 Read block 1182 len 20480
079/300 Read block 566 len 12288
080/300 Read block 3112 len 32768
081/300 Read block 2485 len 16384
082/300 Read block 325 len 28672
083/300 Write block 3629 len 12288 data: 4 4 4
084/300 Write block 1854 len 4096 data: 8
085/300 Read block 3440 len 36864
086/300 Read block 445 len 28672
087/300 Read block 1077 len 28672
088/300 Write block 4619 len 4096 data: 7
089/300 Read block 2314 len 36864
090/300 Read block 235 len 20480
091/300 Write block 2722 len 16384 data: 6 4 4 4
092/300 Write block 358 len 36864 data: 6 4 4 5 5 5 7 6 6
093/300 Read block 1547 len 16384
094/300 Write block 2179 len 4096 data: 6
095/300 Read block 3087 len 32768
096/300 Write block 3485 len 12288 data: 4 4 3
097/300 Flush
098/300 Read block 1207 len 20480
099/300 Read block 3216 len 20480
100/300 Read block 2249 len 24576
101/300 Read block 4113 len 24576
102/300 Read block 73 len 24576
103/300 Flush
104/300 Read block 1119 len 12288
105/300 Read block 2181 len 28672
106/300 Read block 4978 len 16384
107/300 Read block 2453 len 8192
108/300 Read block 697 len 40960
109/300 Read block 4614 len 32768
110/300 Write block 3864 len 20480 data: 4 4 5 5 5
111/300 Write block 1073 len 28672 data: 7 6 6 5 5 4 6
112/300 Write block 2014 len 4096 data: 7
113/300 Read block 1435 len 28672
114/300 Read block 2992 len 4096
115/300 Read block 4168 len 4096
116/300 Read block 2255 len 32768
117/300 Read block 1651 len 16384
118/300 Read block 1811 len 12288
119/300 Read block 1775 len 24576
120/300 Read block 1281 len 12288
121/300 Read block 3673 len 4096
122/300 Write block 4440 len 4096 data: 5
123/300 Write block 517 len 24576 data: 6 6 8 8 7 6
124/300 Write block 3082 len 40960 data: 5 4 4 3 3 4 5 4 6 7
125/300 Flush
126/300 Read block 2932 len 8192
127/300 Read block 3100 len 20480
128/300 Write block 783 len 32768 data: 3 4 4 6 6 5 6 7
129/300 Read block 979 len 32768
130/300 Read block 1081 len 24576
131/300 Write block 3292 len 16384 data: 5 6 6 6
132/300 Read block 4556 len 12288
133/300 Read block 1273 len 32768
134/300 Write block 3302 len 28672 data: 4 4 4 4 4 3 3
135/300 Write block 4366 len 28672 data: 4 3 4 5 4 4 4
136/300 Write block 2952 len 4096 data: 7
137/300 Flush
138/300 Read block 2683 len 12288
139/300 Write block 703 len 28672 data: 4 5 7 7 7 8 7
140/300 Flush
141/300 Write block 4885 len 36864 data: 5 6 7 5 4 4 3 5 5
142/300 Read block 1957 len 40960
143/300 Write block 2169 len 32768 data: 5 5 7 8 9 8 9 10
144/300 Read block 2414 len 40960
145/300 Write block 4160 len 12288 data: 8 10 10
146/300 Flush
147/300 Write block 2498 len 24576 data: 4 4 3 4 4 3
148/300 Read block 2444 len 8192
149/300 Write block 1368 len 32768 data: 8 8 9 8 6 7 8 9
150/300 Read block 1273 len 36864
151/300 Write block 1461 len 16384 data: 4 2 4 7
152/300 Read block 2266 len 36864
153/300 Read block 3847 len 12288
154/300 Read block 4234 len 40960
155/300 Write block 1453 len 20480 data: 6 6 5 7 8
156/300 Read block 4079 len 16384
157/300 Read block 2363 len 24576
158/300 Read block 164 len 28672
159/300 Read block 2520 len 20480
160/300 Write block 298 len 28672 data: 6 6 7 5 6 6 5
161/300 Write block 127 len 20480 data: 5 5 6 7 9
162/300 Flush
163/300 Flush
164/300 Write block 126 len 40960 data: 6 6 6 7 8 10 7 6 6 6
165/300 Write block 406 len 12288 data: 7 5 5
166/300 Write block 4575 len 8192 data: 4 5
167/300 Read block 2931 len 4096
168/300 Write block 185 len 8192 data: 8 8
169/300 Write block 555 len 8192 data: 5 5
170/300 Flush
171/300 Write block 1146 len 24576 data: 5 6 7 7 5 2
172/300 Read block 3512 len 24576
173/300 Read block 1309 len 12288
174/300 Read block 1265 len 28672
175/300 Write block 1670 len 8192 data: 6 6
176/300 Write block 2494 len 40960 data: 5 4 3 3 5 5 4 5 5 4
177/300 Write block 818 len 36864 data: 5 7 6 5 4 4 4 4 4
178/300 Write block 4355 len 12288 data: 7 4 4
179/300 Read block 1931 len 20480
180/300 Flush
181/300 Read block 2565 len 16384
182/300 Read block 581 len 16384
183/300 Read block 1734 len 36864
184/300 Write block 219 len 20480 data: 5 4 5 7 8
185/300 Read block 2608 len 16384
186/300 Read block 1113 len 12288
187/300 Write block 3888 len 4096 data: 6
188/300 Read block 4983 len 36864
189/300 Write block 1407 len 4096 data: 9
190/300 Read block 861 len 32768
191/300 Read block 3024 len 40960
192/300 Write block 317 len 20480 data: 5 5 5 7 8
193/300 Read block 3140 len 8192
194/300 Write block 102 len 12288 data: 6 5 5
195/300 Read block 4221 len 28672
196/300 Flush
197/300 Write block 768 len 24576 data: 6 6 5 5 5 5
198/300 Write block 1298 len 40960 data: 9 8 7 9 8 10 8 8 7 8
199/300 Read block 258 len 8192
200/300 Write block 562 len 8192 data: 6 6
201/300 Read block 1 len 12288
202/300 Read block 2175 len 28672
203/300 Flush
204/300 Write block 1400 len 4096 data: 10
205/300 Write block 2136 len 8192 data: 4 4
206/300 Write block 2818 len 28672 data: 3 3 4 4 4 4 6
207/300 Write block 3303 len 40960 data: 5 5 5 5 4 4 4 3 3 3
208/300 Read block 111 len 24576
209/300 Read block 3164 len 12288
210/300 Read block 1252 len 40960
211/300 Read block 4341 len 40960
212/300 Write block 3578 len 16384 data: 3 3 4 5
213/300 Read block 1294 len 24576
214/300 Read block 332 len 20480
215/300 Read block 4558 len 16384
216/300 Write block 1361 len 4096 data: 8
217/300 Write block 300 len 24576 data: 8 6 7 7 6 6
218/300 Write block 4698 len 4096 data: 5
219/300 Write block 1262 len 28672 data: 4 5 4 4 4 3 3
220/300 Read block 3667 len 36864
221/300 Read block 4565 len 8192
222/300 Write block 96 len 16384 data: 4 4 5 5
223/300 Read block 1309 len 40960
224/300 Write block 1160 len 28672 data: 3 2 2 3 4 4 4
225/300 Flush
226/300 Write block 4401 len 4096 data: 4
227/300 Write block 2365 len 24576 data: 4 4 4 5 7 8
228/300 Read block 296 len 4096
229/300 Read block 2707 len 16384
230/300 Write block 1780 len 8192 data: 3 4
231/300 Write block 178 len 4096 data: 5
232/300 Flush
233/300 Write block 207 len 12288 data: 7 7 7
234/300 Read block 177 len 8192
235/300 Write block 732 len 4096 data: 5
236/300 Read block 3517 len 16384
237/300 Read block 726 len 36864
238/300 Read block 1669 len 40960
239/300 Read block 2921 len 32768
240/300 Read block 3100 len 28672
241/300 Write block 2215 len 12288 data: 2 3 3
242/300 Read block 9 len 16384
243/300 Read block 2333 len 24576
244/300 Write block 1 len 24576 data: 3 3 2 3 3 3
245/300 Write block 3810 len 32768 data: 8 7 7 4 4 3 3 5
246/300 Write block 2301 len 20480 data: 4 6 8 8 8
247/300 Read block 4019 len 40960
248/300 Write block 642 len 4096 data: 4
249/300 Write block 1185 len 40960 data: 6 5 5 4 5 6 6 5 5 4
250/300 Write block 3038 len 28672 data: 5 5 4 4 3 3 3
251/300 Read block 4103 len 12288
252/300 Read block 4389 len 24576
253/300 Read block 862 len 40960
254/300 Read block 3823 len 20480
255/300 Write block 3388 len 28672 data: 6 7 6 6 5 4 4
256/300 Write block 554 len 8192 data: 8 6
257/300 Write block 4232 len 24576 data: 5 4 4 3 4 5
258/300 Write block 772 len 8192 data: 6 6
259/300 Read block 4232 len 4096
260/300 Write block 1635 len 4096 data: 6
261/300 Read block 1403 len 20480
262/300 Write block 4652 len 16384 data: 6 6 6 6
263/300 Flush
264/300 Flush
265/300 Read block 2253 len 20480
266/300 Read block 3431 len 28672
267/300 Write block 2030 len 16384 data: 7 6 6 4
268/300 Write block 2079 len 32768 data: 8 6 5 5 5 4 3 2
269/300 Write block 3534 len 36864 data: 2 3 3 4 4 5 5 5 6
270/300 Read block 1523 len 8192
271/300 Write block 2624 len 32768 data: 9 6 4 6 7 7 6 6
272/300 Read block 585 len 36864
273/300 Read block 1445 len 32768
274/300 Write block 4233 len 24576 data: 5 5 4 5 6 6
275/300 Read block 3292 len 20480
276/300 Read block 4022 len 16384
277/300 Read block 3127 len 36864
278/300 Read block 337 len 40960
279/300 Write block 4221 len 8192 data: 5 4
280/300 Flush
281/300 Read block 1893 len 16384
282/300 Write block 2844 len 24576 data: 6 7 8 6 7 7
283/300 Flush
284/300 Read block 4045 len 32768
285/300 Flush
286/300 Write block 2826 len 36864 data: 7 6 8 8 9 9 10 9 6
287/300 Write block 3694 len 16384 data: 6 5 6 6
288/300 Read block 485 len 12288
289/300 Write block 1149 len 40960 data: 8 6 3 2 2 3 3 3 3 4
290/300 Write block 2122 len 20480 data: 3 3 5 5 4
291/300 Read block 4043 len 20480
292/300 Read block 4081 len 4096
293/300 Write block 3378 len 40960 data: 4 4 4 3 3 4 4 4 4 5
294/300 Read block 4363 len 20480
295/300 Read block 1274 len 28672
296/300 Write block 126 len 4096 data: 7
297/300 Write block 1093 len 16384 data: 5 5 6 6
298/300 Read block 3070 len 32768
299/300 Write block 1435 len 36864 data: 3 3 3 5 5 5 6 6 5
300/300 Flush
301/300 Write block 4689 len 4096 data: 5
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:42:35.593 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/2, method: POST, req_id: a45320f6-6c30-4a61-a1a3-5e57a492f63b, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8848 Acked Read 9 Done Done Sent false
8849 Acked Write 6 Done Done Sent false
8850 Acked Read 10 Done Done Sent false
8851 Acked Read 4 Done Done Sent false
8852 Acked Write 1 Done Done Sent false
8853 Acked Write 5 Done Done Sent false
8854 Acked Write 7 Done Done Sent false
8855 Acked Read 5 Done Done Sent false
8856 Acked Read 5 Done Done Sent false
8857 Acked Read 6 Done Done Sent false
8858 Acked Read 6 Done Done Sent false
8859 Acked Read 6 Done Done Sent false
8860 Acked Write 9 Done Done Sent false
8861 Acked Write 7 Done Done Sent false
8862 Acked Read 10 Done Done Sent false
8863 Acked Read 8 Done Done Sent false
8864 Acked Flush 0 Done Done Sent false
8865 Acked Write 7 Done Done Sent false
8866 Acked Write 9 Done Done Sent false
8867 Acked Write 6 Done Done Sent false
8868 Acked Read 7 Done Done Sent false
8869 Acked Write 5 Done Done Sent false
8870 Acked Write 9 Done Done Sent false
8871 Acked Write 9 Done Done Sent false
8872 Acked Read 3 Done Done Sent false
8873 Acked Write 3 Done Done Sent false
8874 Acked Read 6 Done Done Sent false
8875 Acked Read 6 Done Done Sent false
8876 Acked Read 8 Done Done Sent false
8877 Acked Write 8 Done Done Sent false
8878 Acked Write 2 Done Done Sent false
8879 Acked Write 9 Done Done Sent false
8880 Acked Write 7 Done Done Sent false
8881 Acked Read 5 Done Done Sent false
8882 Acked Read 3 Done Done Sent false
8883 Acked Write 2 Done Done Sent false
8884 Acked Read 6 Done Done Sent false
8885 Acked Read 8 Done Done Sent false
8886 Acked Read 9 Done Done Sent false
8887 Acked Read 1 Done Done Sent false
8888 Acked Flush 0 Done Done Sent false
8889 Acked Flush 0 Done Done Sent false
8890 Acked Read 2 Done Done Sent false
8891 Acked Write 5 Done Done Sent false
8892 Acked Write 5 Done Done Sent false
8893 Acked Write 9 Done Done Sent false
8894 Acked Write 2 Done Done Sent false
8895 Acked Read 8 Done Done Sent false
8896 Acked Read 5 Done Done Sent false
8897 Acked Read 2 Done Done Sent false
8898 Acked Read 10 Done Done Sent false
8899 Acked Write 3 Done Done Sent false
8900 Acked Flush 0 Done Done Sent false
8901 Acked Write 5 Done Done Sent false
8902 Acked Write 3 Done Done Sent false
8903 Acked Read 8 Done Done Sent false
8904 Acked Write 1 Done Done Sent false
8905 Acked Read 1 Done Done Sent false
8906 Acked Read 7 Done Done Sent false
8907 Acked Read 8 Done Done Sent false
8908 Acked Read 5 Done Done Sent false
8909 Acked Write 3 Done Done Sent false
8910 Acked Write 10 Done Done Sent false
8911 Acked Read 5 Done Done Sent false
8912 Acked Read 10 Done Done Sent false
8913 Acked Read 10 Done Done Sent false
8914 Acked Read 9 Done Done Sent false
8915 Acked Read 1 Done Done Sent false
8916 Acked Flush 0 Done Done Sent false
8917 Acked Write 9 Done Done Sent false
8918 Acked Flush 0 Done Done Sent false
8919 Acked Read 5 Done Done Sent false
8920 Acked Read 6 Done Done Sent false
8921 Acked Write 9 Done Done Sent false
8922 Acked Read 10 Done Done Sent false
8923 Acked Read 7 Done Done Sent false
8924 Acked Read 2 Done Done Sent false
8925 Acked Read 5 Done Done Sent false
8926 Acked Read 3 Done Done Sent false
8927 Acked Read 8 Done Done Sent false
8928 Acked Read 4 Done Done Sent false
8929 Acked Read 7 Done Done Sent false
8930 Acked Write 3 Done Done Sent false
8931 Acked Write 1 Done Done Sent false
8932 Acked Read 9 Done Done Sent false
8933 Acked Read 7 Done Done Sent false
8934 Acked Read 7 Done Done Sent false
8935 Acked Write 1 Done Done Sent false
8936 Acked Read 9 Done Done Sent false
8937 Acked Read 5 Done Done Sent false
8938 Acked Write 4 Done Done Sent false
8939 Acked Write 9 Done Done Sent false
8940 Acked Read 4 Done Done Sent false
8941 Acked Write 1 Done Done Sent false
8942 Acked Read 8 Done Done Sent false
8943 Acked Write 3 Done Done Sent false
8944 Acked Flush 0 Done Done Sent false
8945 Acked Read 5 Done Done Sent false
8946 Acked Read 5 Done Done Sent false
8947 Acked Read 6 Done Done Sent false
8948 Acked Read 6 Done Done Sent false
8949 Acked Read 6 Done Done Sent false
8950 Acked Flush 0 Done Done Sent false
8951 Acked Read 3 Done Done Sent false
8952 Acked Read 7 Done Done Sent false
8953 Acked Read 4 Done Done Sent false
8954 Acked Read 2 Done Done Sent false
8955 Acked Read 10 Done Done Sent false
8956 Acked Read 8 Done Done Sent false
8957 Acked Write 5 Done Done Sent false
8958 Acked Write 7 Done Done Sent false
8959 Acked Write 1 Done Done Sent false
8960 Acked Read 7 Done Done Sent false
8961 Acked Read 1 Done Done Sent false
8962 Acked Read 1 Done Done Sent false
8963 Acked Read 8 Done Done Sent false
8964 Acked Read 4 Done Done Sent false
8965 Acked Read 3 Done Done Sent false
8966 Acked Read 6 Done Done Sent false
8967 Acked Read 3 Done Done Sent false
8968 Acked Read 1 Done Done Sent false
8969 Acked Write 1 Done Done Sent false
8970 Acked Write 6 Done Done Sent false
8971 Acked Write 10 Done Done Sent false
8972 Acked Flush 0 Done Done Sent false
8973 Acked Read 2 Done Done Sent false
8974 Acked Read 5 Done Done Sent false
8975 Acked Write 8 Done Done Sent false
8976 Acked Read 8 Done Done Sent false
8977 Acked Read 6 Done Done Sent false
8978 Acked Write 4 Done Done Sent false
8979 Acked Read 3 Done Done Sent false
8980 Acked Read 8 Done Done Sent false
8981 Acked Write 7 Done Done Sent false
8982 Acked Write 7 Done Done Sent false
8983 Acked Write 1 Done Done Sent false
8984 Acked Flush 0 Done Done Sent false
8985 Acked Read 3 Done Done Sent false
8986 Acked Write 7 Done Done Sent false
8987 Acked Flush 0 Done Done Sent false
8988 Acked Write 9 Done Done Sent false
8989 Acked Read 10 Done Done Sent false
8990 Acked Write 8 Done Done Sent false
8991 Acked Read 10 Done Done Sent false
8992 Acked Write 3 Done Done Sent false
8993 Acked Flush 0 Done Done Sent false
8994 Acked Write 6 Done Done Sent false
8995 Acked Read 2 Done Done Sent false
8996 Acked Write 8 Done Done Sent false
8997 Acked Read 9 Done Done Sent false
8998 Acked Write 4 Done Done Sent false
8999 Acked Read 9 Done Done Sent false
9000 Acked Read 3 Done Done Sent false
9001 Acked Read 10 Done Done Sent false
9002 Acked Write 5 Done Done Sent false
9003 Acked Read 4 Done Done Sent false
9004 Acked Read 6 Done Done Sent false
9005 Acked Read 7 Done Done Sent false
9006 Acked Read 5 Done Done Sent false
9007 Acked Write 7 Done Done Sent false
9008 Acked Write 5 Done Done Sent false
9009 Acked Flush 0 Done Done Sent false
9010 Acked Flush 0 Done Done Sent false
9011 Acked Write 10 Done Done Sent false
9012 Acked Write 3 Done Done Sent false
9013 Acked Write 2 Done Done Sent false
9014 Acked Read 1 Done Done Sent false
9015 Acked Write 2 Done Done Sent false
9016 Acked Write 2 Done Done Sent false
9017 Acked Flush 0 Done Done Sent false
9018 Acked Write 6 Done Done Sent false
9019 Acked Read 6 Done Done Sent false
9020 Acked Read 3 Done Done Sent false
9021 Acked Read 7 Done Done Sent false
9022 Acked Write 2 Done Done Sent false
9023 Acked Write 10 Done Done Sent false
9024 Acked Write 9 Done Done Sent false
9025 Acked Write 3 Done Done Sent false
9026 Acked Read 5 Done Done Sent false
9027 Acked Flush 0 Done Done Sent false
9028 Acked Read 4 Done Done Sent false
9029 Acked Read 4 Done Done Sent false
9030 Acked Read 9 Done Done Sent false
9031 Acked Write 5 Done Done Sent false
9032 Acked Read 4 Done Done Sent false
9033 Acked Read 3 Done Done Sent false
9034 Acked Write 1 Done Done Sent false
9035 Acked Read 9 Done Done Sent false
9036 Acked Write 1 Done Done Sent false
9037 Acked Read 8 Done Done Sent false
9038 Acked Read 10 Done Done Sent false
9039 Acked Write 5 Done Done Sent false
9040 Acked Read 2 Done Done Sent false
9041 Acked Write 3 Done Done Sent false
9042 Acked Read 7 Done Done Sent false
9043 Acked Flush 0 Done Done Sent false
9044 Acked Write 6 Done Done Sent false
9045 Acked Write 10 Done Done Sent false
9046 Acked Read 2 Done Done Sent false
9047 Acked Write 2 Done Done Sent false
9048 Acked Read 3 Done Done Sent false
9049 Acked Read 7 Done Done Sent false
9050 Acked Flush 0 Done Done Sent false
9051 Acked Write 1 Done Done Sent false
9052 Acked Write 2 Done Done Sent false
9053 Acked Write 7 Done Done Sent false
9054 Acked Write 10 Done Done Sent false
9055 Acked Read 6 Done Done Sent false
9056 Acked Read 3 Done Done Sent false
9057 Acked Read 10 Done Done Sent false
9058 Acked Read 10 Done Done Sent false
9059 Acked Write 4 Done Done Sent false
9060 Acked Read 6 Done Done Sent false
9061 Acked Read 5 Done Done Sent false
9062 Acked Read 4 Done Done Sent false
9063 Acked Write 1 Done Done Sent false
9064 Acked Write 6 Done Done Sent false
9065 Acked Write 1 Done Done Sent false
9066 Acked Write 7 Done Done Sent false
9067 Acked Read 9 Done Done Sent false
9068 Acked Read 2 Done Done Sent false
9069 Acked Write 4 Done Done Sent false
9070 Acked Read 10 Done Done Sent false
9071 Acked Write 7 Done Done Sent false
9072 Acked Flush 0 Done Done Sent false
9073 Acked Write 1 Done Done Sent false
9074 Acked Write 6 Done Done Sent false
9075 Acked Read 1 Done Done Sent false
9076 Acked Read 4 Done Done Sent false
9077 Acked Write 2 Done Done Sent false
9078 Acked Write 1 Done Done Sent false
9079 Acked Flush 0 Done Done Sent false
9080 Acked Write 3 Done Done Sent false
9081 Acked Read 2 Done Done Sent false
9082 Acked Write 1 Done Done Sent false
9083 Acked Read 4 Done Done Sent false
9084 Acked Read 9 Done Done Sent false
9085 Acked Read 10 Done Done Sent false
9086 Acked Read 8 Done Done Sent false
9087 Acked Read 7 Done Done Sent false
9088 Acked Write 3 Done Done Sent false
9089 Acked Read 4 Done Done Sent false
9090 Acked Read 6 Done Done Sent false
9091 Acked Write 6 Done Done Sent false
9092 Acked Write 8 Done Done Sent false
9093 Acked Write 5 Done Done Sent false
9094 Acked Read 10 Done Done Sent false
9095 Acked Write 1 Done Done Sent false
9096 Acked Write 10 Done Done Sent false
9097 Acked Write 7 Done Done Sent false
9098 Acked Read 3 Done Done Sent false
9099 Acked Read 6 Done Done Sent false
9100 Acked Read 10 Done Done Sent false
9101 Acked Read 5 Done Done Sent false
9102 Acked Write 7 Done Done Sent false
9103 Acked Write 2 Done Done Sent false
9104 Acked Write 6 Done Done Sent false
9105 Acked Write 2 Done Done Sent false
9106 Acked Read 1 Done Done Sent false
9107 Acked Write 1 Done Done Sent false
9108 Acked Read 5 Done Done Sent false
9109 Acked Write 4 Done Done Sent false
9110 Acked Flush 0 Done Done Sent false
9111 Acked Flush 0 Done Done Sent false
9112 Acked Read 5 Done Done Sent false
9113 Acked Read 7 Done Done Sent false
9114 Acked Write 4 Done Done Sent false
9115 Acked Write 8 Done Done Sent false
9116 Acked Write 9 Done Done Sent false
9117 Acked Read 2 Done Done Sent false
9118 Acked Write 8 Done Done Sent false
9119 Acked Read 9 Done Done Sent false
9120 Acked Read 8 Done Done Sent false
9121 Acked Write 6 Done Done Sent false
9122 Acked Read 5 Done Done Sent false
9123 Acked Read 4 Done Done Sent false
9124 Acked Read 9 Done Done Sent false
9125 Acked Read 10 Done Done Sent false
9126 Acked Write 2 Done Done Sent false
9127 Acked Flush 0 Done Done Sent false
9128 Acked Read 4 Done Done Sent false
9129 Acked Write 6 Done Done Sent false
9130 Acked Flush 0 Done Done Sent false
9131 Acked Read 8 Done Done Sent false
9132 Acked Flush 0 Done Done Sent false
9133 Acked Write 9 Done Done Sent false
9134 Acked Write 4 Done Done Sent false
9135 Acked Read 3 Done Done Sent false
9136 Acked Write 10 Done Done Sent false
9137 Acked Write 5 Done Done Sent false
9138 Acked Read 5 Done Done Sent false
9139 Acked Read 1 Done Done Sent false
9140 Acked Write 10 Done Done Sent false
9141 Acked Read 5 Done Done Sent false
9142 Acked Read 7 Done Done Sent false
9143 Acked Write 1 Done Done Sent false
9144 Acked Write 4 Done Done Sent false
9145 Acked Read 8 Done Done Sent false
9146 Acked Write 9 Done Done Sent false
9147 Acked Flush 0 Done Done Sent false
9148 Acked Write 1 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9147 9147 8847
Downstairs last five completed: 8847 8846 8845 8844 8843
Upstairs last five completed: 9148 9147 9146 9145 9144
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1227)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8848 Acked Read 9 Done Done Sent false
8849 Acked Write 6 Done Done Sent false
8850 Acked Read 10 Done Done Sent false
8851 Acked Read 4 Done Done Sent false
8852 Acked Write 1 Done Done Sent false
8853 Acked Write 5 Done Done Sent false
8854 Acked Write 7 Done Done Sent false
8855 Acked Read 5 Done Done Sent false
8856 Acked Read 5 Done Done Sent false
8857 Acked Read 6 Done Done Sent false
8858 Acked Read 6 Done Done Sent false
8859 Acked Read 6 Done Done Sent false
8860 Acked Write 9 Done Done Sent false
8861 Acked Write 7 Done Done Sent false
8862 Acked Read 10 Done Done Sent false
8863 Acked Read 8 Done Done Sent false
8864 Acked Flush 0 Done Done Sent false
8865 Acked Write 7 Done Done Sent false
8866 Acked Write 9 Done Done Sent false
8867 Acked Write 6 Done Done Sent false
8868 Acked Read 7 Done Done Sent false
8869 Acked Write 5 Done Done Sent false
8870 Acked Write 9 Done Done Sent false
8871 Acked Write 9 Done Done Sent false
8872 Acked Read 3 Done Done Sent false
8873 Acked Write 3 Done Done Sent false
8874 Acked Read 6 Done Done Sent false
8875 Acked Read 6 Done Done Sent false
8876 Acked Read 8 Done Done Sent false
8877 Acked Write 8 Done Done Sent false
8878 Acked Write 2 Done Done Sent false
8879 Acked Write 9 Done Done Sent false
8880 Acked Write 7 Done Done Sent false
8881 Acked Read 5 Done Done Sent false
8882 Acked Read 3 Done Done Sent false
8883 Acked Write 2 Done Done Sent false
8884 Acked Read 6 Done Done Sent false
8885 Acked Read 8 Done Done Sent false
8886 Acked Read 9 Done Done Sent false
8887 Acked Read 1 Done Done Sent false
8888 Acked Flush 0 Done Done Sent false
8889 Acked Flush 0 Done Done Sent false
8890 Acked Read 2 Done Done Sent false
8891 Acked Write 5 Done Done Sent false
8892 Acked Write 5 Done Done Sent false
8893 Acked Write 9 Done Done Sent false
8894 Acked Write 2 Done Done Sent false
8895 Acked Read 8 Done Done Sent false
8896 Acked Read 5 Done Done Sent false
8897 Acked Read 2 Done Done Sent false
8898 Acked Read 10 Done Done Sent false
8899 Acked Write 3 Done Done Sent false
8900 Acked Flush 0 Done Done Sent false
8901 Acked Write 5 Done Done Sent false
8902 Acked Write 3 Done Done Sent false
8903 Acked Read 8 Done Done Sent false
8904 Acked Write 1 Done Done Sent false
8905 Acked Read 1 Done Done Sent false
8906 Acked Read 7 Done Done Sent false
8907 Acked Read 8 Done Done Sent false
8908 Acked Read 5 Done Done Sent false
8909 Acked Write 3 Done Done Sent false
8910 Acked Write 10 Done Done Sent false
8911 Acked Read 5 Done Done Sent false
8912 Acked Read 10 Done Done Sent false
8913 Acked Read 10 Done Done Sent false
8914 Acked Read 9 Done Done Sent false
8915 Acked Read 1 Done Done Sent false
8916 Acked Flush 0 Done Done Sent false
8917 Acked Write 9 Done Done Sent false
8918 Acked Flush 0 Done Done Sent false
8919 Acked Read 5 Done Done Sent false
8920 Acked Read 6 Done Done Sent false
8921 Acked Write 9 Done Done Sent false
8922 Acked Read 10 Done Done Sent false
8923 Acked Read 7 Done Done Sent false
8924 Acked Read 2 Done Done Sent false
8925 Acked Read 5 Done Done Sent false
8926 Acked Read 3 Done Done Sent false
8927 Acked Read 8 Done Done Sent false
8928 Acked Read 4 Done Done Sent false
8929 Acked Read 7 Done Done Sent false
8930 Acked Write 3 Done Done Sent false
8931 Acked Write 1 Done Done Sent false
8932 Acked Read 9 Done Done Sent false
8933 Acked Read 7 Done Done Sent false
8934 Acked Read 7 Done Done Sent false
8935 Acked Write 1 Done Done Sent false
8936 Acked Read 9 Done Done Sent false
8937 Acked Read 5 Done Done Sent false
8938 Acked Write 4 Done Done Sent false
8939 Acked Write 9 Done Done Sent false
8940 Acked Read 4 Done Done Sent false
8941 Acked Write 1 Done Done Sent false
8942 Acked Read 8 Done Done Sent false
8943 Acked Write 3 Done Done Sent false
8944 Acked Flush 0 Done Done Sent false
8945 Acked Read 5 Done Done Sent false
8946 Acked Read 5 Done Done Sent false
8947 Acked Read 6 Done Done Sent false
8948 Acked Read 6 Done Done Sent false
8949 Acked Read 6 Done Done Sent false
8950 Acked Flush 0 Done Done Sent false
8951 Acked Read 3 Done Done Sent false
8952 Acked Read 7 Done Done Sent false
8953 Acked Read 4 Done Done Sent false
8954 Acked Read 2 Done Done Sent false
8955 Acked Read 10 Done Done Sent false
8956 Acked Read 8 Done Done Sent false
8957 Acked Write 5 Done Done Sent false
8958 Acked Write 7 Done Done Sent false
8959 Acked Write 1 Done Done Sent false
8960 Acked Read 7 Done Done Sent false
8961 Acked Read 1 Done Done Sent false
8962 Acked Read 1 Done Done Sent false
8963 Acked Read 8 Done Done Sent false
8964 Acked Read 4 Done Done Sent false
8965 Acked Read 3 Done Done Sent false
8966 Acked Read 6 Done Done Sent false
8967 Acked Read 3 Done Done Sent false
8968 Acked Read 1 Done Done Sent false
8969 Acked Write 1 Done Done Sent false
8970 Acked Write 6 Done Done Sent false
8971 Acked Write 10 Done Done Sent false
8972 Acked Flush 0 Done Done Sent false
8973 Acked Read 2 Done Done Sent false
8974 Acked Read 5 Done Done Sent false
8975 Acked Write 8 Done Done Sent false
8976 Acked Read 8 Done Done Sent false
8977 Acked Read 6 Done Done Sent false
8978 Acked Write 4 Done Done Sent false
8979 Acked Read 3 Done Done Sent false
8980 Acked Read 8 Done Done Sent false
8981 Acked Write 7 Done Done Sent false
8982 Acked Write 7 Done Done Sent false
8983 Acked Write 1 Done Done Sent false
8984 Acked Flush 0 Done Done Sent false
8985 Acked Read 3 Done Done Sent false
8986 Acked Write 7 Done Done Sent false
8987 Acked Flush 0 Done Done Sent false
8988 Acked Write 9 Done Done Sent false
8989 Acked Read 10 Done Done Sent false
8990 Acked Write 8 Done Done Sent false
8991 Acked Read 10 Done Done Sent false
8992 Acked Write 3 Done Done Sent false
8993 Acked Flush 0 Done Done Sent false
8994 Acked Write 6 Done Done Sent false
8995 Acked Read 2 Done Done Sent false
8996 Acked Write 8 Done Done Sent false
8997 Acked Read 9 Done Done Sent false
8998 Acked Write 4 Done Done Sent false
8999 Acked Read 9 Done Done Sent false
9000 Acked Read 3 Done Done Sent false
9001 Acked Read 10 Done Done Sent false
9002 Acked Write 5 Done Done Sent false
9003 Acked Read 4 Done Done Sent false
9004 Acked Read 6 Done Done Sent false
9005 Acked Read 7 Done Done Sent false
9006 Acked Read 5 Done Done Sent false
9007 Acked Write 7 Done Done Sent false
9008 Acked Write 5 Done Done Sent false
9009 Acked Flush 0 Done Done Sent false
9010 Acked Flush 0 Done Done Sent false
9011 Acked Write 10 Done Done Sent false
9012 Acked Write 3 Done Done Sent false
9013 Acked Write 2 Done Done Sent false
9014 Acked Read 1 Done Done Sent false
9015 Acked Write 2 Done Done Sent false
9016 Acked Write 2 Done Done Sent false
9017 Acked Flush 0 Done Done Sent false
9018 Acked Write 6 Done Done Sent false
9019 Acked Read 6 Done Done Sent false
9020 Acked Read 3 Done Done Sent false
9021 Acked Read 7 Done Done Sent false
9022 Acked Write 2 Done Done Sent false
9023 Acked Write 10 Done Done Sent false
9024 Acked Write 9 Done Done Sent false
9025 Acked Write 3 Done Done Sent false
9026 Acked Read 5 Done Done Sent false
9027 Acked Flush 0 Done Done Sent false
9028 Acked Read 4 Done Done Sent false
9029 Acked Read 4 Done Done Sent false
9030 Acked Read 9 Done Done Sent false
9031 Acked Write 5 Done Done Sent false
9032 Acked Read 4 Done Done Sent false
9033 Acked Read 3 Done Done Sent false
9034 Acked Write 1 Done Done Sent false
9035 Acked Read 9 Done Done Sent false
9036 Acked Write 1 Done Done Sent false
9037 Acked Read 8 Done Done Sent false
9038 Acked Read 10 Done Done Sent false
9039 Acked Write 5 Done Done Sent false
9040 Acked Read 2 Done Done Sent false
9041 Acked Write 3 Done Done Sent false
9042 Acked Read 7 Done Done Sent false
9043 Acked Flush 0 Done Done Sent false
9044 Acked Write 6 Done Done Sent false
9045 Acked Write 10 Done Done Sent false
9046 Acked Read 2 Done Done Sent false
9047 Acked Write 2 Done Done Sent false
9048 Acked Read 3 Done Done Sent false
9049 Acked Read 7 Done Done Sent false
9050 Acked Flush 0 Done Done Sent false
9051 Acked Write 1 Done Done Sent false
9052 Acked Write 2 Done Done Sent false
9053 Acked Write 7 Done Done Sent false
9054 Acked Write 10 Done Done Sent false
9055 Acked Read 6 Done Done Sent false
9056 Acked Read 3 Done Done Sent false
9057 Acked Read 10 Done Done Sent false
9058 Acked Read 10 Done Done Sent false
9059 Acked Write 4 Done Done Sent false
9060 Acked Read 6 Done Done Sent false
9061 Acked Read 5 Done Done Sent false
9062 Acked Read 4 Done Done Sent false
9063 Acked Write 1 Done Done Sent false
9064 Acked Write 6 Done Done Sent false
9065 Acked Write 1 Done Done Sent false
9066 Acked Write 7 Done Done Sent false
9067 Acked Read 9 Done Done Sent false
9068 Acked Read 2 Done Done Sent false
9069 Acked Write 4 Done Done Sent false
9070 Acked Read 10 Done Done Sent false
9071 Acked Write 7 Done Done Sent false
9072 Acked Flush 0 Done Done Sent false
9073 Acked Write 1 Done Done Sent false
9074 Acked Write 6 Done Done Sent false
9075 Acked Read 1 Done Done Sent false
9076 Acked Read 4 Done Done Sent false
9077 Acked Write 2 Done Done Sent false
9078 Acked Write 1 Done Done Sent false
9079 Acked Flush 0 Done Done Sent false
9080 Acked Write 3 Done Done Sent false
9081 Acked Read 2 Done Done Sent false
9082 Acked Write 1 Done Done Sent false
9083 Acked Read 4 Done Done Sent false
9084 Acked Read 9 Done Done Sent false
9085 Acked Read 10 Done Done Sent false
9086 Acked Read 8 Done Done Sent false
9087 Acked Read 7 Done Done Sent false
9088 Acked Write 3 Done Done Sent false
9089 Acked Read 4 Done Done Sent false
9090 Acked Read 6 Done Done Sent false
9091 Acked Write 6 Done Done Sent false
9092 Acked Write 8 Done Done Sent false
9093 Acked Write 5 Done Done Sent false
9094 Acked Read 10 Done Done Sent false
9095 Acked Write 1 Done Done Sent false
9096 Acked Write 10 Done Done Sent false
9097 Acked Write 7 Done Done Sent false
9098 Acked Read 3 Done Done Sent false
9099 Acked Read 6 Done Done Sent false
9100 Acked Read 10 Done Done Sent false
9101 Acked Read 5 Done Done Sent false
9102 Acked Write 7 Done Done Sent false
9103 Acked Write 2 Done Done Sent false
9104 Acked Write 6 Done Done Sent false
9105 Acked Write 2 Done Done Sent false
9106 Acked Read 1 Done Done Sent false
9107 Acked Write 1 Done Done Sent false
9108 Acked Read 5 Done Done Sent false
9109 Acked Write 4 Done Done Sent false
9110 Acked Flush 0 Done Done Sent false
9111 Acked Flush 0 Done Done Sent false
9112 Acked Read 5 Done Done Sent false
9113 Acked Read 7 Done Done Sent false
9114 Acked Write 4 Done Done Sent false
9115 Acked Write 8 Done Done Sent false
9116 Acked Write 9 Done Done Sent false
9117 Acked Read 2 Done Done Sent false
9118 Acked Write 8 Done Done Sent false
9119 Acked Read 9 Done Done Sent false
9120 Acked Read 8 Done Done Sent false
9121 Acked Write 6 Done Done Sent false
9122 Acked Read 5 Done Done Sent false
9123 Acked Read 4 Done Done Sent false
9124 Acked Read 9 Done Done Sent false
9125 Acked Read 10 Done Done Sent false
9126 Acked Write 2 Done Done Sent false
9127 Acked Flush 0 Done Done Sent false
9128 Acked Read 4 Done Done Sent false
9129 Acked Write 6 Done Done Sent false
9130 Acked Flush 0 Done Done Sent false
9131 Acked Read 8 Done Done Sent false
9132 Acked Flush 0 Done Done Sent false
9133 Acked Write 9 Done Done Sent false
9134 Acked Write 4 Done Done Sent false
9135 Acked Read 3 Done Done Sent false
9136 Acked Write 10 Done Done Sent false
9137 Acked Write 5 Done Done Sent false
9138 Acked Read 5 Done Done Sent false
9139 Acked Read 1 Done Done Sent false
9140 Acked Write 10 Done Done Sent false
9141 Acked Read 5 Done Done Sent false
9142 Acked Read 7 Done Done Sent false
9143 Acked Write 1 Done Done Sent false
9144 Acked Write 4 Done Done Sent false
9145 Acked Read 8 Done Done Sent false
9146 Acked Write 9 Done Done Sent false
9147 Acked Flush 0 Done Done Sent false
9148 Acked Write 1 Done Done Sent false
9149 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9149 9149 8847
Downstairs last five completed: 8847 8846 8845 8844 8843
Upstairs last five completed: 9149 9148 9147 9146 9145
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.351896258Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54580)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.352116962Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.353090278Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.353626066Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.353654642Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 8847","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.35367242Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 8847","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.353882096Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 8847","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.353914122Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 8847","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:41.354123578Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9149 9148 9147 9146 9145
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Jan 27 19:42:43.598[8820] Got stop action so:false kr:false
INFO request completed, latency_us: 381, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 54296742-2bfb-45d9-9390-1c1ae63b3c0b, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:43.599455209Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:43.599510092Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:43.59952991Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:43.599575745Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:42:43.599 INFO request completed, latency_us: 145, response_code: 200, uri: /state/cid/1, method: GET, req_id: e28d64ab-13a6-4daf-b2a7-061b37505070, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
Jan 27 19:42:47.601 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/1, method: GET, req_id: b31129bd-c4fd-48d8-b14b-f32b4ad69282, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 1327 len 8192
002/300 Read block 2046 len 28672
003/300 Read block 3968 len 4096
004/300 Read block 4955 len 16384
005/300 Flush
006/300 Write block 2737 len 4096 data: 5
007/300 Write block 2776 len 40960 data: 5 5 4 5 4 4 6 6 6 6
008/300 Read block 4330 len 20480
009/300 Write block 3907 len 8192 data: 5 6
010/300 Read block 492 len 20480
011/300 Read block 358 len 16384
012/300 Read block 2588 len 24576
013/300 Write block 1646 len 20480 data: 3 2 3 3 3
014/300 Read block 777 len 12288
015/300 Read block 4695 len 20480
016/300 Write block 2420 len 4096 data: 5
017/300 Read block 2067 len 20480
018/300 Write block 3229 len 8192 data: 5 6
019/300 Write block 153 len 32768 data: 3 5 5 5 5 5 5 4
020/300 Write block 3234 len 40960 data: 5 6 4 4 3 4 4 6 7 7
021/300 Write block 2981 len 16384 data: 6 6 7 6
022/300 Write block 2612 len 36864 data: 7 7 8 7 7 8 8 10 10
023/300 Write block 2984 len 20480 data: 7 6 6 6 5
024/300 Read block 2842 len 4096
025/300 Write block 1332 len 36864 data: 6 7 6 6 6 6 7 6 5
026/300 Write block 1078 len 24576 data: 5 7 6 4 4 3
027/300 Read block 2377 len 12288
028/300 Read block 2517 len 40960
029/300 Read block 3379 len 4096
030/300 Write block 4758 len 36864 data: 4 5 3 3 5 5 8 8 6
031/300 Read block 1981 len 16384
032/300 Write block 1227 len 4096 data: 6
033/300 Write block 4826 len 40960 data: 6 5 3 7 8 7 6 7 6 8
034/300 Read block 4741 len 40960
035/300 Read block 104 len 12288
036/300 Flush
037/300 Read block 1442 len 24576
038/300 Write block 811 len 40960 data: 6 6 4 4 5 5 5 6 8 7
039/300 Read block 4874 len 8192
040/300 Read block 52 len 16384
041/300 Write block 218 len 32768 data: 5 6 5 6 8 9 8 8
042/300 Write block 4722 len 20480 data: 4 4 4 4 4
043/300 Read block 3156 len 28672
044/300 Write block 3644 len 12288 data: 6 6 7
045/300 Write block 2622 len 28672 data: 10 8 10 7 5 7 8
046/300 Write block 4493 len 16384 data: 7 8 8 6
047/300 Write block 4308 len 36864 data: 4 4 4 3 3 3 3 3 3
048/300 Read block 3842 len 40960
049/300 Write block 1315 len 4096 data: 6
050/300 Read block 3118 len 4096
051/300 Read block 1817 len 40960
052/300 Read block 119 len 20480
053/300 Read block 4567 len 32768
054/300 Write block 4680 len 40960 data: 8 9 8 7 5 5 6 6 5 6
055/300 Read block 3654 len 28672
056/300 Read block 3167 len 36864
057/300 Write block 4589 len 28672 data: 5 5 6 7 7 7 6
058/300 Write block 3464 len 24576 data: 7 6 6 6 5 5
059/300 Write block 2924 len 36864 data: 3 5 6 6 7 9 8 8 8
060/300 Read block 4563 len 40960
061/300 Read block 1354 len 8192
062/300 Write block 2223 len 28672 data: 3 3 3 3 4 3 3
063/300 Write block 2766 len 12288 data: 8 9 9
064/300 Flush
065/300 Write block 3060 len 40960 data: 7 7 7 7 4 4 4 4 6 5
066/300 Write block 1740 len 40960 data: 8 8 6 6 7 6 4 3 3 3
067/300 Read block 2326 len 16384
068/300 Read block 303 len 20480
069/300 Read block 4990 len 12288
070/300 Write block 1079 len 28672 data: 8 7 5 5 4 3 2
071/300 Read block 960 len 40960
072/300 Read block 4825 len 36864
073/300 Write block 1114 len 28672 data: 6 6 6 7 7 5 7
074/300 Write block 2658 len 12288 data: 6 6 6
075/300 Flush
076/300 Write block 2408 len 16384 data: 9 8 9 8
077/300 Read block 4472 len 36864
078/300 Write block 3334 len 8192 data: 5 6
079/300 Flush
080/300 Read block 742 len 20480
081/300 Read block 507 len 12288
082/300 Write block 1533 len 16384 data: 5 5 5 5
083/300 Flush
084/300 Read block 4868 len 24576
085/300 Read block 4094 len 36864
086/300 Read block 190 len 8192
087/300 Write block 1905 len 28672 data: 6 6 5 4 4 3 3
088/300 Read block 1107 len 8192
089/300 Read block 1520 len 12288
090/300 Read block 3540 len 12288
091/300 Read block 672 len 28672
092/300 Write block 1423 len 20480 data: 6 6 7 6 6
093/300 Read block 1336 len 32768
094/300 Read block 1997 len 32768
095/300 Flush
096/300 Flush
097/300 Read block 1421 len 24576
098/300 Read block 4352 len 32768
099/300 Write block 2644 len 4096 data: 5
100/300 Flush
101/300 Write block 1902 len 20480 data: 3 4 6 7 7
102/300 Write block 3021 len 32768 data: 6 6 6 6 6 6 5 5
103/300 Write block 2303 len 28672 data: 9 9 9 6 7 7 6
104/300 Write block 2515 len 4096 data: 7
105/300 Read block 2663 len 20480
106/300 Read block 3810 len 20480
107/300 Write block 821 len 36864 data: 6 5 5 5 5 5 4 6 6
108/300 Flush
109/300 Write block 382 len 8192 data: 5 4
110/300 Read block 2801 len 36864
111/300 Read block 2308 len 12288
112/300 Read block 60 len 20480
113/300 Read block 2919 len 24576
114/300 Read block 2128 len 8192
115/300 Write block 3016 len 12288 data: 3 3 3
116/300 Write block 3782 len 8192 data: 5 5
117/300 Read block 2888 len 40960
118/300 Write block 4336 len 20480 data: 10 9 8 7 5
119/300 Read block 3982 len 32768
120/300 Read block 102 len 28672
121/300 Read block 2856 len 8192
122/300 Write block 2028 len 8192 data: 7 8
123/300 Write block 4386 len 8192 data: 5 5
124/300 Read block 1474 len 40960
125/300 Read block 3859 len 28672
126/300 Write block 2481 len 16384 data: 3 3 4 5
127/300 Read block 3415 len 40960
128/300 Write block 2132 len 40960 data: 4 5 5 4 5 5 4 4 5 6
129/300 Read block 4689 len 40960
130/300 Write block 3273 len 28672 data: 7 5 6 6 7 6 7
131/300 Write block 3858 len 20480 data: 4 4 5 6 4
132/300 Write block 4262 len 4096 data: 6
133/300 Read block 4677 len 12288
134/300 Write block 4256 len 4096 data: 9
135/300 Read block 4030 len 40960
136/300 Write block 791 len 4096 data: 7
137/300 Write block 2557 len 24576 data: 8 8 8 8 7 8
138/300 Write block 4363 len 32768 data: 5 5 4 5 4 5 6 5
139/300 Read block 2003 len 36864
140/300 Read block 4391 len 4096
141/300 Write block 2589 len 12288 data: 6 9 8
142/300 Read block 2471 len 4096
143/300 Read block 4653 len 4096
144/300 Read block 3908 len 24576
145/300 Write block 347 len 32768 data: 5 5 6 6 9 9 9 9
146/300 Read block 658 len 36864
147/300 Read block 747 len 40960
148/300 Write block 3804 len 20480 data: 4 5 5 5 5
149/300 Write block 2706 len 4096 data: 4
150/300 Read block 3802 len 36864
151/300 Write block 2835 len 12288 data: 4 5 6
152/300 Flush
153/300 Read block 4146 len 32768
154/300 Read block 477 len 36864
155/300 Read block 3275 len 8192
156/300 Read block 1796 len 40960
157/300 Write block 504 len 4096 data: 7
158/300 Write block 1112 len 8192 data: 5 5
159/300 Read block 2261 len 16384
160/300 Read block 1680 len 4096
161/300 Read block 2729 len 32768
162/300 Write block 3720 len 28672 data: 8 8 7 7 8 9 9
163/300 Write block 1654 len 28672 data: 6 5 5 5 6 6 6
164/300 Write block 4752 len 32768 data: 5 6 6 8 7 4 5 6
165/300 Write block 3550 len 16384 data: 5 5 4 6
166/300 Write block 3824 len 36864 data: 3 2 4 4 4 5 6 5 5
167/300 Read block 1259 len 12288
168/300 Flush
169/300 Read block 847 len 4096
170/300 Flush
171/300 Write block 3266 len 12288 data: 2 3 3
172/300 Read block 1802 len 16384
173/300 Flush
174/300 Write block 4268 len 28672 data: 9 7 7 10 7 7 6
175/300 Write block 1511 len 12288 data: 2 2 3
176/300 Write block 210 len 24576 data: 6 6 4 3 5 5
177/300 Read block 1638 len 12288
178/300 Write block 4662 len 40960 data: 2 2 2 2 3 4 3 3 3 4
179/300 Write block 3767 len 36864 data: 8 7 5 5 5 5 7 8 9
180/300 Read block 4762 len 4096
181/300 Write block 3220 len 12288 data: 6 7 8
182/300 Read block 1262 len 32768
183/300 Read block 4248 len 40960
184/300 Read block 4826 len 4096
185/300 Flush
186/300 Flush
187/300 Write block 81 len 36864 data: 5 4 5 3 3 4 5 6 5
188/300 Read block 1211 len 28672
189/300 Read block 778 len 12288
190/300 Write block 3190 len 12288 data: 7 7 7
191/300 Read block 165 len 32768
192/300 Read block 885 len 28672
193/300 Read block 242 len 4096
194/300 Write block 3110 len 16384 data: 6 5 4 3
195/300 Write block 2240 len 20480 data: 6 6 7 7 7
196/300 Write block 1682 len 36864 data: 3 4 4 4 4 4 3 4 5
197/300 Flush
198/300 Read block 1729 len 28672
199/300 Read block 275 len 4096
200/300 Write block 3682 len 20480 data: 3 3 4 5 6
201/300 Write block 3258 len 16384 data: 4 4 4 5
202/300 Write block 2599 len 36864 data: 7 8 6 5 5 6 6 7 7
203/300 Write block 3770 len 4096 data: 6
204/300 Read block 322 len 16384
205/300 Read block 4026 len 8192
206/300 Write block 4464 len 12288 data: 3 3 5
207/300 Write block 3024 len 24576 data: 7 7 7 6 6 5
208/300 Write block 4292 len 4096 data: 6
209/300 Write block 4787 len 20480 data: 6 6 7 9 9
210/300 Read block 4172 len 36864
211/300 Write block 4163 len 40960 data: 10 11 11 9 9 9 8 7 6 6
212/300 Write block 3861 len 12288 data: 7 5 5
213/300 Read block 2322 len 4096
214/300 Read block 2836 len 4096
215/300 Write block 233 len 8192 data: 4 5
216/300 Read block 3824 len 8192
217/300 Write block 312 len 36864 data: 4 4 5 5 5 6 6 6 8
218/300 Read block 3674 len 40960
219/300 Read block 4192 len 20480
220/300 Flush
221/300 Write block 2794 len 16384 data: 7 6 4 3
222/300 Read block 140 len 36864
223/300 Read block 4220 len 12288
224/300 Write block 4523 len 16384 data: 4 5 5 8
225/300 Read block 3572 len 32768
226/300 Write block 4355 len 16384 data: 8 5 5 4
227/300 Write block 3007 len 40960 data: 7 6 6 5 6 5 4 3 3 4
228/300 Read block 1108 len 12288
229/300 Read block 3276 len 8192
230/300 Read block 2337 len 40960
231/300 Read block 2322 len 16384
232/300 Write block 248 len 28672 data: 4 5 4 4 6 7 6
233/300 Read block 4144 len 40960
234/300 Read block 178 len 4096
235/300 Read block 1518 len 12288
236/300 Read block 807 len 8192
237/300 Read block 576 len 12288
238/300 Read block 4370 len 24576
239/300 Flush
240/300 Write block 3617 len 12288 data: 7 9 8
241/300 Write block 1497 len 32768 data: 4 3 3 3 3 3 3 4
242/300 Read block 2137 len 20480
243/300 Write block 746 len 12288 data: 7 7 7
244/300 Flush
245/300 Write block 2621 len 8192 data: 10 11
246/300 Read block 3083 len 8192
247/300 Write block 3192 len 40960 data: 8 5 5 5 5 4 4 3 4 4
248/300 Write block 339 len 16384 data: 7 6 6 5
249/300 Write block 3382 len 36864 data: 4 5 5 5 5 6 7 8 7
250/300 Read block 1122 len 4096
251/300 Write block 846 len 40960 data: 6 4 5 4 4 4 4 4 4 3
252/300 Write block 848 len 40960 data: 6 5 5 5 5 5 5 4 4 4
253/300 Write block 1699 len 20480 data: 4 4 4 4 4
254/300 Read block 2304 len 28672
255/300 Read block 1860 len 4096
256/300 Read block 2914 len 16384
257/300 Flush
258/300 Flush
259/300 Read block 2960 len 24576
260/300 Read block 4024 len 16384
261/300 Write block 4395 len 12288 data: 4 4 4
262/300 Read block 1156 len 24576
263/300 Read block 2636 len 28672
264/300 Read block 360 len 36864
265/300 Read block 1930 len 24576
266/300 Read block 4467 len 8192
267/300 Read block 1943 len 40960
268/300 Read block 2315 len 24576
269/300 Flush
270/300 Read block 923 len 12288
271/300 Write block 4730 len 4096 data: 5
272/300 Read block 3229 len 12288
273/300 Read block 3391 len 20480
274/300 Read block 4081 len 16384
275/300 Write block 1298 len 32768 data: 10 9 8 10 9 11 9 9
276/300 Flush
277/300 Write block 3194 len 8192 data: 6 6
278/300 Write block 474 len 36864 data: 5 6 6 5 7 7 6 6 6
279/300 Write block 1531 len 24576 data: 7 6 6 6 6 6
280/300 Read block 2432 len 24576
281/300 Write block 1886 len 24576 data: 8 8 9 9 8 6
282/300 Write block 4402 len 28672 data: 3 4 5 6 6 7 7
283/300 Read block 2933 len 40960
284/300 Read block 1844 len 32768
285/300 Read block 4938 len 36864
286/300 Write block 1718 len 12288 data: 4 6 5
287/300 Read block 2510 len 24576
288/300 Write block 4707 len 32768 data: 6 8 7 7 6 6 7 7
289/300 Read block 890 len 40960
290/300 Read block 2745 len 16384
291/300 Write block 2282 len 4096 data: 8
292/300 Read block 3681 len 16384
293/300 Read block 1412 len 4096
294/300 Flush
295/300 Read block 4729 len 12288
296/300 Flush
297/300 Read block 4383 len 4096
298/300 Write block 3669 len 40960 data: 4 5 6 6 7 7 5 5 5 5
299/300 Read block 3699 len 32768
300/300 Write block 1324 len 20480 data: 5 5 5 5 5
301/300 Read block 3006 len 16384
Main task has work to do, go find it
got dsc Start(1)
start 1
Jan 27 19:42:47.835 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5012df3b-d860-4d41-bbc5-b8ed10022a85, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9150 Acked Read 2 Done Sent Done false
9151 Acked Read 7 Done Sent Done false
9152 Acked Read 1 Done Sent Done false
9153 Acked Read 4 Done Sent Done false
9154 Acked Flush 0 Done Sent Done false
9155 Acked Write 1 Done Sent Done false
9156 Acked Write 10 Done Sent Done false
9157 Acked Read 5 Done Sent Done false
9158 Acked Write 2 Done Sent Done false
9159 Acked Read 5 Done Sent Done false
9160 Acked Read 4 Done Sent Done false
9161 Acked Read 6 Done Sent Done false
9162 Acked Write 5 Done Sent Done false
9163 Acked Read 3 Done Sent Done false
9164 Acked Read 5 Done Sent Done false
9165 Acked Write 1 Done Sent Done false
9166 Acked Read 5 Done Sent Done false
9167 Acked Write 2 Done Sent Done false
9168 Acked Write 8 Done Sent Done false
9169 Acked Write 10 Done Sent Done false
9170 Acked Write 4 Done Sent Done false
9171 Acked Write 9 Done Sent Done false
9172 Acked Write 5 Done Sent Done false
9173 Acked Read 1 Done Sent Done false
9174 Acked Write 9 Done Sent Done false
9175 Acked Write 6 Done Sent Done false
9176 Acked Read 3 Done Sent Done false
9177 Acked Read 10 Done Sent Done false
9178 Acked Read 1 Done Sent Done false
9179 Acked Write 9 Done Sent Done false
9180 Acked Read 4 Done Sent Done false
9181 Acked Write 1 Done Sent Done false
9182 Acked Write 10 Done Sent Done false
9183 Acked Read 10 Done Sent Done false
9184 Acked Read 3 Done Sent Done false
9185 Acked Flush 0 Done Sent Done false
9186 Acked Read 6 Done Sent Done false
9187 Acked Write 10 Done Sent Done false
9188 Acked Read 2 Done Sent Done false
9189 Acked Read 4 Done Sent Done false
9190 Acked Write 8 Done Sent Done false
9191 Acked Write 5 Done Sent Done false
9192 Acked Read 7 Done Sent Done false
9193 Acked Write 3 Done Sent Done false
9194 Acked Write 7 Done Sent Done false
9195 Acked Write 4 Done Sent Done false
9196 Acked Write 9 Done Sent Done false
9197 Acked Read 10 Done Sent Done false
9198 Acked Write 1 Done Sent Done false
9199 Acked Read 1 Done Sent Done false
9200 Acked Read 10 Done Sent Done false
9201 Acked Read 5 Done Sent Done false
9202 Acked Read 8 Done Sent Done false
9203 Acked Write 10 Done Sent Done false
9204 Acked Read 7 Done Sent Done false
9205 Acked Read 9 Done Sent Done false
9206 Acked Write 7 Done Sent Done false
9207 Acked Write 6 Done Sent Done false
9208 Acked Write 9 Done Sent Done false
9209 Acked Read 10 Done Sent Done false
9210 Acked Read 2 Done Sent Done false
9211 Acked Write 7 Done Sent Done false
9212 Acked Write 3 Done Sent Done false
9213 Acked Flush 0 Done Sent Done false
9214 Acked Write 10 Done Sent Done false
9215 Acked Write 10 Done Sent Done false
9216 Acked Read 4 Done Sent Done false
9217 Acked Read 5 Done Sent Done false
9218 Acked Read 3 Done Sent Done false
9219 Acked Write 7 Done Sent Done false
9220 Acked Read 10 Done Sent Done false
9221 Acked Read 9 Done Sent Done false
9222 Acked Write 7 Done Sent Done false
9223 Acked Write 3 Done Sent Done false
9224 Acked Flush 0 Done Sent Done false
9225 Acked Write 4 Done Sent Done false
9226 Acked Read 9 Done Sent Done false
9227 Acked Write 2 Done Sent Done false
9228 Acked Flush 0 Done Sent Done false
9229 Acked Read 5 Done Sent Done false
9230 Acked Read 3 Done Sent Done false
9231 Acked Write 4 Done Sent Done false
9232 Acked Flush 0 Done Sent Done false
9233 Acked Read 6 Done Sent Done false
9234 Acked Read 9 Done Sent Done false
9235 Acked Read 2 Done Sent Done false
9236 Acked Write 7 Done Sent Done false
9237 Acked Read 2 Done Sent Done false
9238 Acked Read 3 Done Sent Done false
9239 Acked Read 3 Done Sent Done false
9240 Acked Read 7 Done Sent Done false
9241 Acked Write 5 Done Sent Done false
9242 Acked Read 8 Done Sent Done false
9243 Acked Read 8 Done Sent Done false
9244 Acked Flush 0 Done Sent Done false
9245 Acked Flush 0 Done Sent Done false
9246 Acked Read 6 Done Sent Done false
9247 Acked Read 8 Done Sent Done false
9248 Acked Write 1 Done Sent Done false
9249 Acked Flush 0 Done Sent Done false
9250 Acked Write 5 Done Sent Done false
9251 Acked Write 8 Done Sent Done false
9252 Acked Write 7 Done Sent Done false
9253 Acked Write 1 Done Sent Done false
9254 Acked Read 5 Done Sent Done false
9255 Acked Read 5 Done Sent Done false
9256 Acked Write 9 Done Sent Done false
9257 Acked Flush 0 Done Sent Done false
9258 Acked Write 2 Done Sent Done false
9259 Acked Read 9 Done Sent Done false
9260 Acked Read 3 Done Sent Done false
9261 Acked Read 5 Done Sent Done false
9262 Acked Read 6 Done Sent Done false
9263 Acked Read 2 Done Sent Done false
9264 Acked Write 3 Done Sent Done false
9265 Acked Write 2 Done Sent Done false
9266 Acked Read 10 Done Sent Done false
9267 Acked Write 5 Done Sent Done false
9268 Acked Read 8 Done Sent Done false
9269 Acked Read 7 Done Sent Done false
9270 Acked Read 2 Done Sent Done false
9271 Acked Write 2 Done Sent Done false
9272 Acked Write 2 Done Sent Done false
9273 Acked Read 10 Done Sent Done false
9274 Acked Read 7 Done Sent Done false
9275 Acked Write 4 Done Sent Done false
9276 Acked Read 10 Done Sent Done false
9277 Acked Write 10 Done Sent Done false
9278 Acked Read 10 Done Sent Done false
9279 Acked Write 7 Done Sent Done false
9280 Acked Write 5 Done Sent Done false
9281 Acked Write 1 Done Sent Done false
9282 Acked Read 3 Done Sent Done false
9283 Acked Write 1 Done Sent Done false
9284 Acked Read 10 Done Sent Done false
9285 Acked Write 1 Done Sent Done false
9286 Acked Write 6 Done Sent Done false
9287 Acked Write 8 Done Sent Done false
9288 Acked Read 9 Done Sent Done false
9289 Acked Read 1 Done Sent Done false
9290 Acked Write 3 Done Sent Done false
9291 Acked Read 1 Done Sent Done false
9292 Acked Read 1 Done Sent Done false
9293 Acked Read 6 Done Sent Done false
9294 Acked Write 8 Done Sent Done false
9295 Acked Read 9 Done Sent Done false
9296 Acked Read 10 Done Sent Done false
9297 Acked Write 5 Done Sent Done false
9298 Acked Write 1 Done Sent Done false
9299 Acked Read 9 Done Sent Done false
9300 Acked Write 3 Done Sent Done false
9301 Acked Flush 0 Done Sent Done false
9302 Acked Read 8 Done Sent Done false
9303 Acked Read 9 Done Sent Done false
9304 Acked Read 2 Done Sent Done false
9305 Acked Read 10 Done Sent Done false
9306 Acked Write 1 Done Sent Done false
9307 Acked Write 2 Done Sent Done false
9308 Acked Read 4 Done Sent Done false
9309 Acked Read 1 Done Sent Done false
9310 Acked Read 8 Done Sent Done false
9311 Acked Write 7 Done Sent Done false
9312 Acked Write 7 Done Sent Done false
9313 Acked Write 8 Done Sent Done false
9314 Acked Write 4 Done Sent Done false
9315 Acked Write 9 Done Sent Done false
9316 Acked Read 3 Done Sent Done false
9317 Acked Flush 0 Done Sent Done false
9318 Acked Read 1 Done Sent Done false
9319 Acked Flush 0 Done Sent Done false
9320 Acked Write 3 Done Sent Done false
9321 Acked Read 4 Done Sent Done false
9322 Acked Flush 0 Done Sent Done false
9323 Acked Write 7 Done Sent Done false
9324 Acked Write 3 Done Sent Done false
9325 Acked Write 6 Done Sent Done false
9326 Acked Read 3 Done Sent Done false
9327 Acked Write 10 Done Sent Done false
9328 Acked Write 9 Done Sent Done false
9329 Acked Read 1 Done Sent Done false
9330 Acked Write 3 Done Sent Done false
9331 Acked Read 8 Done Sent Done false
9332 Acked Read 10 Done Sent Done false
9333 Acked Read 1 Done Sent Done false
9334 Acked Flush 0 Done Sent Done false
9335 Acked Flush 0 Done Sent Done false
9336 Acked Write 9 Done Sent Done false
9337 Acked Read 7 Done Sent Done false
9338 Acked Read 3 Done Sent Done false
9339 Acked Write 3 Done Sent Done false
9340 Acked Read 8 Done Sent Done false
9341 Acked Read 7 Done Sent Done false
9342 Acked Read 1 Done Sent Done false
9343 Acked Write 4 Done Sent Done false
9344 Acked Write 5 Done Sent Done false
9345 Acked Write 9 Done Sent Done false
9346 Acked Flush 0 Done Sent Done false
9347 Acked Read 7 Done Sent Done false
9348 Acked Read 1 Done Sent Done false
9349 Acked Write 5 Done Sent Done false
9350 Acked Write 4 Done Sent Done false
9351 Acked Write 9 Done Sent Done false
9352 Acked Write 1 Done Sent Done false
9353 Acked Read 4 Done Sent Done false
9354 Acked Read 2 Done Sent Done false
9355 Acked Write 3 Done Sent Done false
9356 Acked Write 6 Done Sent Done false
9357 Acked Write 1 Done Sent Done false
9358 Acked Write 5 Done Sent Done false
9359 Acked Read 9 Done Sent Done false
9360 Acked Write 10 Done Sent Done false
9361 Acked Write 3 Done Sent Done false
9362 Acked Read 1 Done Sent Done false
9363 Acked Read 1 Done Sent Done false
9364 Acked Write 2 Done Sent Done false
9365 Acked Read 2 Done Sent Done false
9366 Acked Write 9 Done Sent Done false
9367 Acked Read 10 Done Sent Done false
9368 Acked Read 5 Done Sent Done false
9369 Acked Flush 0 Done Sent Done false
9370 Acked Write 4 Done Sent Done false
9371 Acked Read 9 Done Sent Done false
9372 Acked Read 3 Done Sent Done false
9373 Acked Write 4 Done Sent Done false
9374 Acked Read 8 Done Sent Done false
9375 Acked Write 4 Done Sent Done false
9376 Acked Write 10 Done Sent Done false
9377 Acked Read 3 Done Sent Done false
9378 Acked Read 2 Done Sent Done false
9379 Acked Read 10 Done Sent Done false
9380 Acked Read 4 Done Sent Done false
9381 Acked Write 7 Done Sent Done false
9382 Acked Read 10 Done Sent Done false
9383 Acked Read 1 Done Sent Done false
9384 Acked Read 3 Done Sent Done false
9385 Acked Read 2 Done Sent Done false
9386 Acked Read 3 Done Sent Done false
9387 Acked Read 6 Done Sent Done false
9388 Acked Flush 0 Done Sent Done false
9389 Acked Write 3 Done Sent Done false
9390 Acked Write 8 Done Sent Done false
9391 Acked Read 5 Done Sent Done false
9392 Acked Write 3 Done Sent Done false
9393 Acked Flush 0 Done Sent Done false
9394 Acked Write 2 Done Sent Done false
9395 Acked Read 2 Done Sent Done false
9396 Acked Write 10 Done Sent Done false
9397 Acked Write 4 Done Sent Done false
9398 Acked Write 9 Done Sent Done false
9399 Acked Read 1 Done Sent Done false
9400 Acked Write 10 Done Sent Done false
9401 Acked Write 10 Done Sent Done false
9402 Acked Write 5 Done Sent Done false
9403 Acked Read 7 Done Sent Done false
9404 Acked Read 1 Done Sent Done false
9405 Acked Read 4 Done Sent Done false
9406 Acked Flush 0 Done Sent Done false
9407 Acked Flush 0 Done Sent Done false
9408 Acked Read 6 Done Sent Done false
9409 Acked Read 4 Done Sent Done false
9410 Acked Write 3 Done Sent Done false
9411 Acked Read 6 Done Sent Done false
9412 Acked Read 7 Done Sent Done false
9413 Acked Read 9 Done Sent Done false
9414 Acked Read 6 Done Sent Done false
9415 Acked Read 2 Done Sent Done false
9416 Acked Read 10 Done Sent Done false
9417 Acked Read 6 Done Sent Done false
9418 Acked Flush 0 Done Sent Done false
9419 Acked Read 3 Done Sent Done false
9420 Acked Write 1 Done Sent Done false
9421 Acked Read 3 Done Sent Done false
9422 Acked Read 5 Done Sent Done false
9423 Acked Read 4 Done Sent Done false
9424 Acked Write 8 Done Sent Done false
9425 Acked Flush 0 Done Sent Done false
9426 Acked Write 2 Done Sent Done false
9427 Acked Write 9 Done Sent Done false
9428 Acked Write 6 Done Sent Done false
9429 Acked Read 6 Done Sent Done false
9430 Acked Write 6 Done Sent Done false
9431 Acked Write 7 Done Sent Done false
9432 Acked Read 10 Done Sent Done false
9433 Acked Read 8 Done Sent Done false
9434 Acked Read 9 Done Sent Done false
9435 Acked Write 3 Done Sent Done false
9436 Acked Read 6 Done Sent Done false
9437 Acked Write 8 Done Sent Done false
9438 Acked Read 10 Done Sent Done false
9439 Acked Read 4 Done Sent Done false
9440 Acked Write 1 Done Sent Done false
9441 Acked Read 4 Done Sent Done false
9442 Acked Read 1 Done Sent Done false
9443 Acked Flush 0 Done Sent Done false
9444 Acked Read 3 Done Sent Done false
9445 Acked Flush 0 Done Sent Done false
9446 Acked Read 1 Done Sent Done false
9447 Acked Write 10 Done Sent Done false
9448 Acked Read 8 Done Sent Done false
9449 Acked Write 5 Done Sent Done false
9450 Acked Read 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 301 0 301
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9445 9149 9445
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9450 9449 9448 9447 9446
CLIENT: Up:0 ds:301 act:2
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1228)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9150 Acked Read 2 Done Sent Done false
9151 Acked Read 7 Done Sent Done false
9152 Acked Read 1 Done Sent Done false
9153 Acked Read 4 Done Sent Done false
9154 Acked Flush 0 Done Sent Done false
9155 Acked Write 1 Done Sent Done false
9156 Acked Write 10 Done Sent Done false
9157 Acked Read 5 Done Sent Done false
9158 Acked Write 2 Done Sent Done false
9159 Acked Read 5 Done Sent Done false
9160 Acked Read 4 Done Sent Done false
9161 Acked Read 6 Done Sent Done false
9162 Acked Write 5 Done Sent Done false
9163 Acked Read 3 Done Sent Done false
9164 Acked Read 5 Done Sent Done false
9165 Acked Write 1 Done Sent Done false
9166 Acked Read 5 Done Sent Done false
9167 Acked Write 2 Done Sent Done false
9168 Acked Write 8 Done Sent Done false
9169 Acked Write 10 Done Sent Done false
9170 Acked Write 4 Done Sent Done false
9171 Acked Write 9 Done Sent Done false
9172 Acked Write 5 Done Sent Done false
9173 Acked Read 1 Done Sent Done false
9174 Acked Write 9 Done Sent Done false
9175 Acked Write 6 Done Sent Done false
9176 Acked Read 3 Done Sent Done false
9177 Acked Read 10 Done Sent Done false
9178 Acked Read 1 Done Sent Done false
9179 Acked Write 9 Done Sent Done false
9180 Acked Read 4 Done Sent Done false
9181 Acked Write 1 Done Sent Done false
9182 Acked Write 10 Done Sent Done false
9183 Acked Read 10 Done Sent Done false
9184 Acked Read 3 Done Sent Done false
9185 Acked Flush 0 Done Sent Done false
9186 Acked Read 6 Done Sent Done false
9187 Acked Write 10 Done Sent Done false
9188 Acked Read 2 Done Sent Done false
9189 Acked Read 4 Done Sent Done false
9190 Acked Write 8 Done Sent Done false
9191 Acked Write 5 Done Sent Done false
9192 Acked Read 7 Done Sent Done false
9193 Acked Write 3 Done Sent Done false
9194 Acked Write 7 Done Sent Done false
9195 Acked Write 4 Done Sent Done false
9196 Acked Write 9 Done Sent Done false
9197 Acked Read 10 Done Sent Done false
9198 Acked Write 1 Done Sent Done false
9199 Acked Read 1 Done Sent Done false
9200 Acked Read 10 Done Sent Done false
9201 Acked Read 5 Done Sent Done false
9202 Acked Read 8 Done Sent Done false
9203 Acked Write 10 Done Sent Done false
9204 Acked Read 7 Done Sent Done false
9205 Acked Read 9 Done Sent Done false
9206 Acked Write 7 Done Sent Done false
9207 Acked Write 6 Done Sent Done false
9208 Acked Write 9 Done Sent Done false
9209 Acked Read 10 Done Sent Done false
9210 Acked Read 2 Done Sent Done false
9211 Acked Write 7 Done Sent Done false
9212 Acked Write 3 Done Sent Done false
9213 Acked Flush 0 Done Sent Done false
9214 Acked Write 10 Done Sent Done false
9215 Acked Write 10 Done Sent Done false
9216 Acked Read 4 Done Sent Done false
9217 Acked Read 5 Done Sent Done false
9218 Acked Read 3 Done Sent Done false
9219 Acked Write 7 Done Sent Done false
9220 Acked Read 10 Done Sent Done false
9221 Acked Read 9 Done Sent Done false
9222 Acked Write 7 Done Sent Done false
9223 Acked Write 3 Done Sent Done false
9224 Acked Flush 0 Done Sent Done false
9225 Acked Write 4 Done Sent Done false
9226 Acked Read 9 Done Sent Done false
9227 Acked Write 2 Done Sent Done false
9228 Acked Flush 0 Done Sent Done false
9229 Acked Read 5 Done Sent Done false
9230 Acked Read 3 Done Sent Done false
9231 Acked Write 4 Done Sent Done false
9232 Acked Flush 0 Done Sent Done false
9233 Acked Read 6 Done Sent Done false
9234 Acked Read 9 Done Sent Done false
9235 Acked Read 2 Done Sent Done false
9236 Acked Write 7 Done Sent Done false
9237 Acked Read 2 Done Sent Done false
9238 Acked Read 3 Done Sent Done false
9239 Acked Read 3 Done Sent Done false
9240 Acked Read 7 Done Sent Done false
9241 Acked Write 5 Done Sent Done false
9242 Acked Read 8 Done Sent Done false
9243 Acked Read 8 Done Sent Done false
9244 Acked Flush 0 Done Sent Done false
9245 Acked Flush 0 Done Sent Done false
9246 Acked Read 6 Done Sent Done false
9247 Acked Read 8 Done Sent Done false
9248 Acked Write 1 Done Sent Done false
9249 Acked Flush 0 Done Sent Done false
9250 Acked Write 5 Done Sent Done false
9251 Acked Write 8 Done Sent Done false
9252 Acked Write 7 Done Sent Done false
9253 Acked Write 1 Done Sent Done false
9254 Acked Read 5 Done Sent Done false
9255 Acked Read 5 Done Sent Done false
9256 Acked Write 9 Done Sent Done false
9257 Acked Flush 0 Done Sent Done false
9258 Acked Write 2 Done Sent Done false
9259 Acked Read 9 Done Sent Done false
9260 Acked Read 3 Done Sent Done false
9261 Acked Read 5 Done Sent Done false
9262 Acked Read 6 Done Sent Done false
9263 Acked Read 2 Done Sent Done false
9264 Acked Write 3 Done Sent Done false
9265 Acked Write 2 Done Sent Done false
9266 Acked Read 10 Done Sent Done false
9267 Acked Write 5 Done Sent Done false
9268 Acked Read 8 Done Sent Done false
9269 Acked Read 7 Done Sent Done false
9270 Acked Read 2 Done Sent Done false
9271 Acked Write 2 Done Sent Done false
9272 Acked Write 2 Done Sent Done false
9273 Acked Read 10 Done Sent Done false
9274 Acked Read 7 Done Sent Done false
9275 Acked Write 4 Done Sent Done false
9276 Acked Read 10 Done Sent Done false
9277 Acked Write 10 Done Sent Done false
9278 Acked Read 10 Done Sent Done false
9279 Acked Write 7 Done Sent Done false
9280 Acked Write 5 Done Sent Done false
9281 Acked Write 1 Done Sent Done false
9282 Acked Read 3 Done Sent Done false
9283 Acked Write 1 Done Sent Done false
9284 Acked Read 10 Done Sent Done false
9285 Acked Write 1 Done Sent Done false
9286 Acked Write 6 Done Sent Done false
9287 Acked Write 8 Done Sent Done false
9288 Acked Read 9 Done Sent Done false
9289 Acked Read 1 Done Sent Done false
9290 Acked Write 3 Done Sent Done false
9291 Acked Read 1 Done Sent Done false
9292 Acked Read 1 Done Sent Done false
9293 Acked Read 6 Done Sent Done false
9294 Acked Write 8 Done Sent Done false
9295 Acked Read 9 Done Sent Done false
9296 Acked Read 10 Done Sent Done false
9297 Acked Write 5 Done Sent Done false
9298 Acked Write 1 Done Sent Done false
9299 Acked Read 9 Done Sent Done false
9300 Acked Write 3 Done Sent Done false
9301 Acked Flush 0 Done Sent Done false
9302 Acked Read 8 Done Sent Done false
9303 Acked Read 9 Done Sent Done false
9304 Acked Read 2 Done Sent Done false
9305 Acked Read 10 Done Sent Done false
9306 Acked Write 1 Done Sent Done false
9307 Acked Write 2 Done Sent Done false
9308 Acked Read 4 Done Sent Done false
9309 Acked Read 1 Done Sent Done false
9310 Acked Read 8 Done Sent Done false
9311 Acked Write 7 Done Sent Done false
9312 Acked Write 7 Done Sent Done false
9313 Acked Write 8 Done Sent Done false
9314 Acked Write 4 Done Sent Done false
9315 Acked Write 9 Done Sent Done false
9316 Acked Read 3 Done Sent Done false
9317 Acked Flush 0 Done Sent Done false
9318 Acked Read 1 Done Sent Done false
9319 Acked Flush 0 Done Sent Done false
9320 Acked Write 3 Done Sent Done false
9321 Acked Read 4 Done Sent Done false
9322 Acked Flush 0 Done Sent Done false
9323 Acked Write 7 Done Sent Done false
9324 Acked Write 3 Done Sent Done false
9325 Acked Write 6 Done Sent Done false
9326 Acked Read 3 Done Sent Done false
9327 Acked Write 10 Done Sent Done false
9328 Acked Write 9 Done Sent Done false
9329 Acked Read 1 Done Sent Done false
9330 Acked Write 3 Done Sent Done false
9331 Acked Read 8 Done Sent Done false
9332 Acked Read 10 Done Sent Done false
9333 Acked Read 1 Done Sent Done false
9334 Acked Flush 0 Done Sent Done false
9335 Acked Flush 0 Done Sent Done false
9336 Acked Write 9 Done Sent Done false
9337 Acked Read 7 Done Sent Done false
9338 Acked Read 3 Done Sent Done false
9339 Acked Write 3 Done Sent Done false
9340 Acked Read 8 Done Sent Done false
9341 Acked Read 7 Done Sent Done false
9342 Acked Read 1 Done Sent Done false
9343 Acked Write 4 Done Sent Done false
9344 Acked Write 5 Done Sent Done false
9345 Acked Write 9 Done Sent Done false
9346 Acked Flush 0 Done Sent Done false
9347 Acked Read 7 Done Sent Done false
9348 Acked Read 1 Done Sent Done false
9349 Acked Write 5 Done Sent Done false
9350 Acked Write 4 Done Sent Done false
9351 Acked Write 9 Done Sent Done false
9352 Acked Write 1 Done Sent Done false
9353 Acked Read 4 Done Sent Done false
9354 Acked Read 2 Done Sent Done false
9355 Acked Write 3 Done Sent Done false
9356 Acked Write 6 Done Sent Done false
9357 Acked Write 1 Done Sent Done false
9358 Acked Write 5 Done Sent Done false
9359 Acked Read 9 Done Sent Done false
9360 Acked Write 10 Done Sent Done false
9361 Acked Write 3 Done Sent Done false
9362 Acked Read 1 Done Sent Done false
9363 Acked Read 1 Done Sent Done false
9364 Acked Write 2 Done Sent Done false
9365 Acked Read 2 Done Sent Done false
9366 Acked Write 9 Done Sent Done false
9367 Acked Read 10 Done Sent Done false
9368 Acked Read 5 Done Sent Done false
9369 Acked Flush 0 Done Sent Done false
9370 Acked Write 4 Done Sent Done false
9371 Acked Read 9 Done Sent Done false
9372 Acked Read 3 Done Sent Done false
9373 Acked Write 4 Done Sent Done false
9374 Acked Read 8 Done Sent Done false
9375 Acked Write 4 Done Sent Done false
9376 Acked Write 10 Done Sent Done false
9377 Acked Read 3 Done Sent Done false
9378 Acked Read 2 Done Sent Done false
9379 Acked Read 10 Done Sent Done false
9380 Acked Read 4 Done Sent Done false
9381 Acked Write 7 Done Sent Done false
9382 Acked Read 10 Done Sent Done false
9383 Acked Read 1 Done Sent Done false
9384 Acked Read 3 Done Sent Done false
9385 Acked Read 2 Done Sent Done false
9386 Acked Read 3 Done Sent Done false
9387 Acked Read 6 Done Sent Done false
9388 Acked Flush 0 Done Sent Done false
9389 Acked Write 3 Done Sent Done false
9390 Acked Write 8 Done Sent Done false
9391 Acked Read 5 Done Sent Done false
9392 Acked Write 3 Done Sent Done false
9393 Acked Flush 0 Done Sent Done false
9394 Acked Write 2 Done Sent Done false
9395 Acked Read 2 Done Sent Done false
9396 Acked Write 10 Done Sent Done false
9397 Acked Write 4 Done Sent Done false
9398 Acked Write 9 Done Sent Done false
9399 Acked Read 1 Done Sent Done false
9400 Acked Write 10 Done Sent Done false
9401 Acked Write 10 Done Sent Done false
9402 Acked Write 5 Done Sent Done false
9403 Acked Read 7 Done Sent Done false
9404 Acked Read 1 Done Sent Done false
9405 Acked Read 4 Done Sent Done false
9406 Acked Flush 0 Done Sent Done false
9407 Acked Flush 0 Done Sent Done false
9408 Acked Read 6 Done Sent Done false
9409 Acked Read 4 Done Sent Done false
9410 Acked Write 3 Done Sent Done false
9411 Acked Read 6 Done Sent Done false
9412 Acked Read 7 Done Sent Done false
9413 Acked Read 9 Done Sent Done false
9414 Acked Read 6 Done Sent Done false
9415 Acked Read 2 Done Sent Done false
9416 Acked Read 10 Done Sent Done false
9417 Acked Read 6 Done Sent Done false
9418 Acked Flush 0 Done Sent Done false
9419 Acked Read 3 Done Sent Done false
9420 Acked Write 1 Done Sent Done false
9421 Acked Read 3 Done Sent Done false
9422 Acked Read 5 Done Sent Done false
9423 Acked Read 4 Done Sent Done false
9424 Acked Write 8 Done Sent Done false
9425 Acked Flush 0 Done Sent Done false
9426 Acked Write 2 Done Sent Done false
9427 Acked Write 9 Done Sent Done false
9428 Acked Write 6 Done Sent Done false
9429 Acked Read 6 Done Sent Done false
9430 Acked Write 6 Done Sent Done false
9431 Acked Write 7 Done Sent Done false
9432 Acked Read 10 Done Sent Done false
9433 Acked Read 8 Done Sent Done false
9434 Acked Read 9 Done Sent Done false
9435 Acked Write 3 Done Sent Done false
9436 Acked Read 6 Done Sent Done false
9437 Acked Write 8 Done Sent Done false
9438 Acked Read 10 Done Sent Done false
9439 Acked Read 4 Done Sent Done false
9440 Acked Write 1 Done Sent Done false
9441 Acked Read 4 Done Sent Done false
9442 Acked Read 1 Done Sent Done false
9443 Acked Flush 0 Done Sent Done false
9444 Acked Read 3 Done Sent Done false
9445 Acked Flush 0 Done Sent Done false
9446 Acked Read 1 Done Sent Done false
9447 Acked Write 10 Done Sent Done false
9448 Acked Read 8 Done Sent Done false
9449 Acked Write 5 Done Sent Done false
9450 Acked Read 4 Done Sent Done false
9451 Acked Flush 0 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 302 0 302
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9451 9149 9451
Downstairs last five completed: 9149 9148 9147 9146 9145
Upstairs last five completed: 9451 9450 9449 9448 9447
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.600354309Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47001)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.600452627Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.601610172Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.60214147Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.602166767Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 9149","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.602181636Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 9149","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.60240107Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"1","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] client re-new 302 jobs since flush 9149","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.602420868Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[1] Replayed 302 jobs since flush: 9149","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:53.602648951Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9451 9450 9449 9448 9447
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:42:55.841 INFO request completed[8830] Got stop action so:false kr:false
, latency_us: 412, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a9e5d28d-3f73-4f90-9f57-d0bfc936378a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:55.8421619Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:55.84224127Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:42:55.842 INFO request completed, latency_us: 127{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:55.842279476Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, response_code: 200, uri: /state/cid/2, method: GET, req_id: faa5f01c-b08a-49ad-80f1-51741722a80d, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:55.84241668Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:42:59.845 INFO request completed, latency_us: 330, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8df8dcda-bef0-468e-88c0-a4d6ccdd3003, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 2243 len 4096
002/300 Read block 900 len 4096
003/300 Read block 4414 len 28672
004/300 Read block 632 len 20480
005/300 Write block 3269 len 36864 data: 3 4 6 7 8 6 7 7 8
006/300 Write block 3966 len 20480 data: 7 7 7 5 4
007/300 Write block 4545 len 40960 data: 3 3 3 2 3 3 4 4 4 4
008/300 Read block 2540 len 24576
009/300 Write block 2564 len 8192 data: 8 8
010/300 Write block 4281 len 40960 data: 6 5 5 6 5 4 3 4 4 4
011/300 Write block 2274 len 36864 data: 3 4 4 4 5 5 5 6 9
012/300 Read block 4941 len 40960
013/300 Read block 4465 len 36864
014/300 Write block 1445 len 16384 data: 6 7 6 6
015/300 Write block 1413 len 16384 data: 8 7 8 8
016/300 Read block 1218 len 8192
017/300 Write block 2970 len 36864 data: 4 5 6 7 7 5 4 4 5
018/300 Read block 1090 len 32768
019/300 Read block 569 len 8192
020/300 Write block 3230 len 8192 data: 7 6
021/300 Write block 4811 len 8192 data: 8 6
022/300 Write block 28 len 40960 data: 5 4 4 3 3 2 3 4 4 6
023/300 Read block 3089 len 20480
024/300 Flush
025/300 Read block 1957 len 36864
026/300 Write block 246 len 40960 data: 5 4 5 6 5 5 7 8 7 6
027/300 Read block 1287 len 12288
028/300 Write block 1682 len 12288 data: 4 5 5
029/300 Read block 3292 len 20480
030/300 Write block 4092 len 16384 data: 4 4 4 4
031/300 Write block 3186 len 16384 data: 4 6 6 6
032/300 Read block 4218 len 20480
033/300 Write block 1716 len 12288 data: 5 5 5
034/300 Read block 4534 len 12288
035/300 Read block 2239 len 16384
036/300 Write block 947 len 36864 data: 6 5 5 5 4 5 5 5 5
037/300 Read block 2322 len 16384
038/300 Write block 1451 len 16384 data: 6 6 7 7
039/300 Read block 1863 len 16384
040/300 Read block 2488 len 36864
041/300 Write block 1690 len 16384 data: 6 5 4 4
042/300 Flush
043/300 Read block 3183 len 4096
044/300 Flush
045/300 Read block 3459 len 4096
046/300 Flush
047/300 Write block 1855 len 8192 data: 9 8
048/300 Read block 4580 len 8192
049/300 Read block 4657 len 4096
050/300 Write block 3382 len 4096 data: 5
051/300 Read block 4731 len 4096
052/300 Read block 4376 len 20480
053/300 Read block 1223 len 12288
054/300 Read block 4099 len 8192
055/300 Flush
056/300 Flush
057/300 Write block 3097 len 28672 data: 5 8 8 7 8 7 9
058/300 Read block 1262 len 8192
059/300 Read block 642 len 24576
060/300 Read block 3961 len 12288
061/300 Read block 3210 len 40960
062/300 Write block 347 len 16384 data: 6 6 7 7
063/300 Flush
064/300 Read block 4269 len 8192
065/300 Write block 4833 len 40960 data: 8 7 9 8 8 9 8 6 7 8
066/300 Write block 1674 len 36864 data: 8 7 8 5 5 5 4 4 5
067/300 Read block 3595 len 24576
068/300 Read block 1508 len 8192
069/300 Write block 4317 len 32768 data: 3 3 3 4 5 4 5 5
070/300 Write block 3576 len 20480 data: 4 3 4 4 5
071/300 Read block 2196 len 40960
072/300 Read block 1704 len 16384
073/300 Read block 3288 len 24576
074/300 Write block 2911 len 12288 data: 4 5 5
075/300 Write block 1723 len 12288 data: 6 5 5
076/300 Read block 1535 len 40960
077/300 Read block 3941 len 32768
078/300 Write block 2271 len 24576 data: 4 3 3 4 5 5
079/300 Write block 2602 len 40960 data: 6 6 7 7 8 8 7 6 6 8
080/300 Read block 3622 len 20480
081/300 Flush
082/300 Read block 3637 len 12288
083/300 Read block 4297 len 24576
084/300 Read block 1842 len 36864
085/300 Read block 74 len 4096
086/300 Read block 1426 len 8192
087/300 Write block 151 len 8192 data: 2 2
088/300 Flush
089/300 Read block 3506 len 4096
090/300 Read block 3727 len 40960
091/300 Write block 4959 len 16384 data: 7 8 8 9
092/300 Read block 493 len 4096
093/300 Read block 4478 len 32768
094/300 Write block 3494 len 24576 data: 5 5 4 3 3 4
095/300 Write block 3317 len 8192 data: 5 4
096/300 Write block 2095 len 40960 data: 3 4 3 3 4 3 3 4 4 5
097/300 Write block 1413 len 12288 data: 9 8 9
098/300 Write block 3397 len 16384 data: 8 7 5 5
099/300 Read block 1352 len 16384
100/300 Write block 484 len 40960 data: 2 3 3 3 4 5 5 4 5 5
101/300 Read block 3273 len 20480
102/300 Write block 4981 len 24576 data: 4 5 4 3 5 5
103/300 Read block 322 len 32768
104/300 Flush
105/300 Read block 4334 len 36864
106/300 Read block 4259 len 36864
107/300 Read block 3004 len 40960
108/300 Write block 2778 len 40960 data: 5 6 5 5 7 7 7 7 6 5
109/300 Flush
110/300 Write block 3047 len 4096 data: 7
111/300 Flush
112/300 Read block 2883 len 40960
113/300 Flush
114/300 Write block 3342 len 4096 data: 3
115/300 Read block 3370 len 32768
116/300 Read block 3178 len 32768
117/300 Flush
118/300 Write block 3014 len 16384 data: 4 4 5 4
119/300 Read block 3060 len 20480
120/300 Write block 595 len 16384 data: 6 6 7 7
121/300 Read block 4994 len 20480
122/300 Read block 1656 len 40960
123/300 Write block 4652 len 8192 data: 7 7
124/300 Write block 1180 len 32768 data: 2 2 2 4 5 7 6 6
125/300 Flush
126/300 Read block 1400 len 36864
127/300 Read block 4085 len 8192
128/300 Write block 3495 len 12288 data: 6 5 4
129/300 Read block 4311 len 40960
130/300 Read block 4238 len 12288
131/300 Read block 1822 len 40960
132/300 Write block 819 len 20480 data: 9 8 7 6 6
133/300 Write block 4019 len 12288 data: 5 5 5
134/300 Read block 4138 len 24576
135/300 Write block 2038 len 8192 data: 6 5
136/300 Write block 4967 len 4096 data: 6
137/300 Write block 2331 len 12288 data: 5 5 5
138/300 Read block 3945 len 36864
139/300 Read block 3565 len 12288
140/300 Write block 1916 len 12288 data: 3 3 4
141/300 Write block 4981 len 36864 data: 5 6 5 4 6 6 6 6 8
142/300 Read block 2925 len 8192
143/300 Write block 3364 len 36864 data: 7 7 7 7 6 6 7 7 9
144/300 Write block 4486 len 24576 data: 5 5 4 4 5 5
145/300 Read block 3659 len 36864
146/300 Write block 4173 len 28672 data: 8 5 5 5 6 6 3
147/300 Write block 1427 len 8192 data: 7 5
148/300 Read block 4138 len 40960
149/300 Write block 4138 len 8192 data: 6 6
150/300 Write block 4513 len 12288 data: 6 7 5
151/300 Read block 2825 len 8192
152/300 Write block 1997 len 28672 data: 8 9 9 6 5 6 5
153/300 Write block 1756 len 28672 data: 5 5 4 3 3 4 4
154/300 Write block 6 len 32768 data: 4 2 2 3 3 3 4 4
155/300 Read block 1970 len 36864
156/300 Write block 3647 len 40960 data: 6 7 8 7 7 5 5 5 5 3
157/300 Read block 1325 len 20480
158/300 Read block 3491 len 24576
159/300 Write block 3285 len 16384 data: 4 4 5 4
160/300 Write block 326 len 16384 data: 7 8 7 7
161/300 Read block 537 len 16384
162/300 Write block 662 len 8192 data: 5 4
163/300 Write block 1284 len 20480 data: 4 3 3 3 3
164/300 Flush
165/300 Read block 1772 len 40960
166/300 Read block 2821 len 36864
167/300 Read block 1201 len 28672
168/300 Write block 878 len 20480 data: 6 6 5 5 4
169/300 Flush
170/300 Read block 3919 len 16384
171/300 Read block 3135 len 36864
172/300 Read block 589 len 40960
173/300 Read block 3968 len 16384
174/300 Write block 2343 len 36864 data: 6 6 5 6 7 6 6 7 7
175/300 Flush
176/300 Read block 221 len 20480
177/300 Write block 3298 len 40960 data: 4 3 2 4 5 6 6 6 6 5
178/300 Flush
179/300 Write block 4292 len 8192 data: 7 5
180/300 Read block 177 len 32768
181/300 Read block 4033 len 40960
182/300 Write block 1274 len 32768 data: 3 4 5 5 4 4 3 3
183/300 Read block 263 len 28672
184/300 Write block 1688 len 24576 data: 4 5 7 6 5 5
185/300 Write block 4957 len 36864 data: 5 7 8 9 9 10 6 6 3
186/300 Write block 2213 len 4096 data: 3
187/300 Flush
188/300 Read block 3801 len 20480
189/300 Read block 4667 len 4096
190/300 Write block 2162 len 4096 data: 7
191/300 Read block 1268 len 12288
192/300 Read block 2058 len 28672
193/300 Read block 3028 len 12288
194/300 Read block 1606 len 32768
195/300 Write block 209 len 36864 data: 8 7 7 5 4 6 6 4 5
196/300 Flush
197/300 Read block 3281 len 36864
198/300 Write block 4267 len 28672 data: 7 10 8 8 11 8 8
199/300 Write block 2095 len 16384 data: 4 5 4 4
200/300 Read block 4848 len 4096
201/300 Flush
202/300 Write block 3408 len 4096 data: 5
203/300 Write block 4072 len 8192 data: 5 5
204/300 Read block 294 len 24576
205/300 Read block 4478 len 20480
206/300 Read block 1746 len 36864
207/300 Write block 330 len 36864 data: 6 5 5 3 3 3 4 5 6
208/300 Read block 3331 len 8192
209/300 Write block 2858 len 28672 data: 5 4 5 5 6 6 6
210/300 Read block 2771 len 12288
211/300 Read block 1922 len 32768
212/300 Flush
213/300 Write block 611 len 40960 data: 5 4 5 5 4 4 3 3 4 3
214/300 Read block 303 len 20480
215/300 Write block 1330 len 16384 data: 6 5 7 8
216/300 Read block 2599 len 12288
217/300 Write block 1947 len 16384 data: 5 5 4 4
218/300 Read block 3871 len 36864
219/300 Write block 4052 len 36864 data: 3 4 4 5 7 6 6 6 4
220/300 Read block 2929 len 4096
221/300 Read block 1980 len 12288
222/300 Write block 4939 len 28672 data: 7 6 6 6 7 7 5
223/300 Read block 3215 len 12288
224/300 Write block 146 len 32768 data: 3 3 4 3 3 3 3 4
225/300 Read block 1671 len 32768
226/300 Write block 4069 len 28672 data: 4 4 4 6 6 5 4
227/300 Read block 204 len 32768
228/300 Read block 219 len 8192
229/300 Read block 3364 len 8192
230/300 Read block 104 len 12288
231/300 Read block 109 len 36864
232/300 Read block 1338 len 32768
233/300 Read block 1264 len 4096
234/300 Write block 1327 len 32768 data: 6 6 7 7 6 8 9 7
235/300 Write block 2393 len 8192 data: 6 5
236/300 Write block 4234 len 20480 data: 6 5 6 7 7
237/300 Read block 2439 len 40960
238/300 Read block 3859 len 40960
239/300 Write block 4661 len 8192 data: 3 3
240/300 Flush
241/300 Write block 2454 len 32768 data: 6 5 4 5 3 4 4 3
242/300 Read block 1010 len 24576
243/300 Write block 1825 len 8192 data: 5 5
244/300 Write block 3102 len 32768 data: 8 10 11 9 10 10 10 8
245/300 Write block 2490 len 28672 data: 6 6 6 6 6 5 4
246/300 Read block 1367 len 40960
247/300 Read block 3433 len 24576
248/300 Read block 1170 len 16384
249/300 Read block 3156 len 16384
250/300 Read block 4705 len 16384
251/300 Write block 2549 len 36864 data: 3 5 5 5 5 5 7 7 9
252/300 Read block 1112 len 36864
253/300 Read block 4713 len 4096
254/300 Read block 447 len 8192
255/300 Write block 3169 len 12288 data: 5 5 5
256/300 Read block 1399 len 40960
257/300 Write block 1542 len 32768 data: 6 8 8 6 3 3 4 4
258/300 Write block 1254 len 40960 data: 5 4 5 4 5 4 4 4 5 6
259/300 Read block 2279 len 12288
260/300 Read block 3098 len 24576
261/300 Write block 2341 len 16384 data: 7 6 7 7
262/300 Read block 3993 len 20480
263/300 Flush
264/300 Read block 4838 len 4096
265/300 Read block 851 len 20480
266/300 Read block 332 len 24576
267/300 Write block 292 len 32768 data: 5 6 6 6 6 6 7 7
268/300 Read block 429 len 36864
269/300 Write block 824 len 20480 data: 6 6 6 5 7
270/300 Write block 578 len 36864 data: 6 6 8 7 7 9 6 5 4
271/300 Read block 2770 len 28672
272/300 Read block 3384 len 4096
273/300 Read block 3004 len 36864
274/300 Read block 3926 len 8192
275/300 Read block 2266 len 16384
276/300 Write block 2155 len 36864 data: 8 10 11 10 8 7 8 8 6
277/300 Read block 3559 len 32768
278/300 Read block 692 len 20480
279/300 Read block 443 len 8192
280/300 Read block 3682 len 12288
281/300 Read block 4871 len 40960
282/300 Read block 4346 len 4096
283/300 Write block 1651 len 12288 data: 4 5 5
284/300 Read block 4848 len 20480
285/300 Read block 3911 len 8192
286/300 Write block 3706 len 36864 data: 7 8 9 9 8 9 8 8 8
287/300 Read block 2243 len 4096
288/300 Flush
289/300 Write block 2803 len 8192 data: 5 5
290/300 Write block 235 len 24576 data: 7 5 5 5 5 5
291/300 Write block 1721 len 8192 data: 6 7
292/300 Read block 422 len 12288
293/300 Read block 4589 len 24576
294/300 Read block 2466 len 28672
295/300 Read block 1646 len 16384
296/300 Write block 353 len 28672 data: 10 10 8 9 7 7 5
297/300 Read block 2315 len 12288
298/300 Read block 838 len 16384
299/300 Flush
300/300 Read block 3805 len 12288
301/300 Read block 4505 len 40960
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:43:00.086 INFO request completed, latency_us: 164, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7faf77d4-2acf-4e55-9cd3-4d22394ffc48, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Read 1 Done Done Sent false
9453 Acked Read 1 Done Done Sent false
9454 Acked Read 7 Done Done Sent false
9455 Acked Read 5 Done Done Sent false
9456 Acked Write 9 Done Done Sent false
9457 Acked Write 5 Done Done Sent false
9458 Acked Write 10 Done Done Sent false
9459 Acked Read 6 Done Done Sent false
9460 Acked Write 2 Done Done Sent false
9461 Acked Write 10 Done Done Sent false
9462 Acked Write 9 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 9 Done Done Sent false
9465 Acked Write 4 Done Done Sent false
9466 Acked Write 4 Done Done Sent false
9467 Acked Read 2 Done Done Sent false
9468 Acked Write 9 Done Done Sent false
9469 Acked Read 8 Done Done Sent false
9470 Acked Read 2 Done Done Sent false
9471 Acked Write 2 Done Done Sent false
9472 Acked Write 2 Done Done Sent false
9473 Acked Write 10 Done Done Sent false
9474 Acked Read 5 Done Done Sent false
9475 Acked Flush 0 Done Done Sent false
9476 Acked Read 9 Done Done Sent false
9477 Acked Write 10 Done Done Sent false
9478 Acked Read 3 Done Done Sent false
9479 Acked Write 3 Done Done Sent false
9480 Acked Read 5 Done Done Sent false
9481 Acked Write 4 Done Done Sent false
9482 Acked Write 4 Done Done Sent false
9483 Acked Read 5 Done Done Sent false
9484 Acked Write 3 Done Done Sent false
9485 Acked Read 3 Done Done Sent false
9486 Acked Read 4 Done Done Sent false
9487 Acked Write 9 Done Done Sent false
9488 Acked Read 4 Done Done Sent false
9489 Acked Write 4 Done Done Sent false
9490 Acked Read 4 Done Done Sent false
9491 Acked Read 9 Done Done Sent false
9492 Acked Write 4 Done Done Sent false
9493 Acked Flush 0 Done Done Sent false
9494 Acked Read 1 Done Done Sent false
9495 Acked Flush 0 Done Done Sent false
9496 Acked Read 1 Done Done Sent false
9497 Acked Flush 0 Done Done Sent false
9498 Acked Write 2 Done Done Sent false
9499 Acked Read 2 Done Done Sent false
9500 Acked Read 1 Done Done Sent false
9501 Acked Write 1 Done Done Sent false
9502 Acked Read 1 Done Done Sent false
9503 Acked Read 5 Done Done Sent false
9504 Acked Read 3 Done Done Sent false
9505 Acked Read 2 Done Done Sent false
9506 Acked Flush 0 Done Done Sent false
9507 Acked Flush 0 Done Done Sent false
9508 Acked Write 7 Done Done Sent false
9509 Acked Read 2 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Read 3 Done Done Sent false
9512 Acked Read 10 Done Done Sent false
9513 Acked Write 4 Done Done Sent false
9514 Acked Flush 0 Done Done Sent false
9515 Acked Read 2 Done Done Sent false
9516 Acked Write 10 Done Done Sent false
9517 Acked Write 9 Done Done Sent false
9518 Acked Read 6 Done Done Sent false
9519 Acked Read 2 Done Done Sent false
9520 Acked Write 8 Done Done Sent false
9521 Acked Write 5 Done Done Sent false
9522 Acked Read 10 Done Done Sent false
9523 Acked Read 4 Done Done Sent false
9524 Acked Read 6 Done Done Sent false
9525 Acked Write 3 Done Done Sent false
9526 Acked Write 3 Done Done Sent false
9527 Acked Read 10 Done Done Sent false
9528 Acked Read 8 Done Done Sent false
9529 Acked Write 6 Done Done Sent false
9530 Acked Write 10 Done Done Sent false
9531 Acked Read 5 Done Done Sent false
9532 Acked Flush 0 Done Done Sent false
9533 Acked Read 3 Done Done Sent false
9534 Acked Read 6 Done Done Sent false
9535 Acked Read 9 Done Done Sent false
9536 Acked Read 1 Done Done Sent false
9537 Acked Read 2 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Flush 0 Done Done Sent false
9540 Acked Read 1 Done Done Sent false
9541 Acked Read 10 Done Done Sent false
9542 Acked Write 4 Done Done Sent false
9543 Acked Read 1 Done Done Sent false
9544 Acked Read 8 Done Done Sent false
9545 Acked Write 6 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 10 Done Done Sent false
9548 Acked Write 3 Done Done Sent false
9549 Acked Write 4 Done Done Sent false
9550 Acked Read 4 Done Done Sent false
9551 Acked Write 10 Done Done Sent false
9552 Acked Read 5 Done Done Sent false
9553 Acked Write 6 Done Done Sent false
9554 Acked Read 8 Done Done Sent false
9555 Acked Flush 0 Done Done Sent false
9556 Acked Read 9 Done Done Sent false
9557 Acked Read 9 Done Done Sent false
9558 Acked Read 10 Done Done Sent false
9559 Acked Write 10 Done Done Sent false
9560 Acked Flush 0 Done Done Sent false
9561 Acked Write 1 Done Done Sent false
9562 Acked Flush 0 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Flush 0 Done Done Sent false
9565 Acked Write 1 Done Done Sent false
9566 Acked Read 8 Done Done Sent false
9567 Acked Read 8 Done Done Sent false
9568 Acked Flush 0 Done Done Sent false
9569 Acked Write 4 Done Done Sent false
9570 Acked Read 5 Done Done Sent false
9571 Acked Write 4 Done Done Sent false
9572 Acked Read 5 Done Done Sent false
9573 Acked Read 10 Done Done Sent false
9574 Acked Write 2 Done Done Sent false
9575 Acked Write 8 Done Done Sent false
9576 Acked Flush 0 Done Done Sent false
9577 Acked Read 9 Done Done Sent false
9578 Acked Read 2 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 10 Done Done Sent false
9581 Acked Read 3 Done Done Sent false
9582 Acked Read 10 Done Done Sent false
9583 Acked Write 5 Done Done Sent false
9584 Acked Write 3 Done Done Sent false
9585 Acked Read 6 Done Done Sent false
9586 Acked Write 2 Done Done Sent false
9587 Acked Write 1 Done Done Sent false
9588 Acked Write 3 Done Done Sent false
9589 Acked Read 9 Done Done Sent false
9590 Acked Read 3 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 9 Done Done Sent false
9593 Acked Read 2 Done Done Sent false
9594 Acked Write 9 Done Done Sent false
9595 Acked Write 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Write 7 Done Done Sent false
9598 Acked Write 2 Done Done Sent false
9599 Acked Read 10 Done Done Sent false
9600 Acked Write 2 Done Done Sent false
9601 Acked Write 3 Done Done Sent false
9602 Acked Read 2 Done Done Sent false
9603 Acked Write 7 Done Done Sent false
9604 Acked Write 7 Done Done Sent false
9605 Acked Write 8 Done Done Sent false
9606 Acked Read 9 Done Done Sent false
9607 Acked Write 10 Done Done Sent false
9608 Acked Read 5 Done Done Sent false
9609 Acked Read 6 Done Done Sent false
9610 Acked Write 4 Done Done Sent false
9611 Acked Write 4 Done Done Sent false
9612 Acked Read 4 Done Done Sent false
9613 Acked Write 2 Done Done Sent false
9614 Acked Write 5 Done Done Sent false
9615 Acked Flush 0 Done Done Sent false
9616 Acked Read 10 Done Done Sent false
9617 Acked Read 9 Done Done Sent false
9618 Acked Read 7 Done Done Sent false
9619 Acked Write 5 Done Done Sent false
9620 Acked Flush 0 Done Done Sent false
9621 Acked Read 4 Done Done Sent false
9622 Acked Read 9 Done Done Sent false
9623 Acked Read 10 Done Done Sent false
9624 Acked Read 4 Done Done Sent false
9625 Acked Write 9 Done Done Sent false
9626 Acked Flush 0 Done Done Sent false
9627 Acked Read 5 Done Done Sent false
9628 Acked Write 10 Done Done Sent false
9629 Acked Flush 0 Done Done Sent false
9630 Acked Write 2 Done Done Sent false
9631 Acked Read 8 Done Done Sent false
9632 Acked Read 10 Done Done Sent false
9633 Acked Write 8 Done Done Sent false
9634 Acked Read 7 Done Done Sent false
9635 Acked Write 6 Done Done Sent false
9636 Acked Write 9 Done Done Sent false
9637 Acked Write 1 Done Done Sent false
9638 Acked Flush 0 Done Done Sent false
9639 Acked Read 5 Done Done Sent false
9640 Acked Read 1 Done Done Sent false
9641 Acked Write 1 Done Done Sent false
9642 Acked Read 3 Done Done Sent false
9643 Acked Read 7 Done Done Sent false
9644 Acked Read 3 Done Done Sent false
9645 Acked Read 8 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 9 Done Done Sent false
9649 Acked Write 7 Done Done Sent false
9650 Acked Write 4 Done Done Sent false
9651 Acked Read 1 Done Done Sent false
9652 Acked Flush 0 Done Done Sent false
9653 Acked Write 1 Done Done Sent false
9654 Acked Write 2 Done Done Sent false
9655 Acked Read 6 Done Done Sent false
9656 Acked Read 5 Done Done Sent false
9657 Acked Read 9 Done Done Sent false
9658 Acked Write 9 Done Done Sent false
9659 Acked Read 2 Done Done Sent false
9660 Acked Write 7 Done Done Sent false
9661 Acked Read 3 Done Done Sent false
9662 Acked Read 8 Done Done Sent false
9663 Acked Flush 0 Done Done Sent false
9664 Acked Write 10 Done Done Sent false
9665 Acked Read 5 Done Done Sent false
9666 Acked Write 4 Done Done Sent false
9667 Acked Read 3 Done Done Sent false
9668 Acked Write 4 Done Done Sent false
9669 Acked Read 9 Done Done Sent false
9670 Acked Write 9 Done Done Sent false
9671 Acked Read 1 Done Done Sent false
9672 Acked Read 3 Done Done Sent false
9673 Acked Write 7 Done Done Sent false
9674 Acked Read 3 Done Done Sent false
9675 Acked Write 8 Done Done Sent false
9676 Acked Read 8 Done Done Sent false
9677 Acked Write 7 Done Done Sent false
9678 Acked Read 8 Done Done Sent false
9679 Acked Read 2 Done Done Sent false
9680 Acked Read 2 Done Done Sent false
9681 Acked Read 3 Done Done Sent false
9682 Acked Read 9 Done Done Sent false
9683 Acked Read 8 Done Done Sent false
9684 Acked Read 1 Done Done Sent false
9685 Acked Write 8 Done Done Sent false
9686 Acked Write 2 Done Done Sent false
9687 Acked Write 5 Done Done Sent false
9688 Acked Read 10 Done Done Sent false
9689 Acked Read 10 Done Done Sent false
9690 Acked Write 2 Done Done Sent false
9691 Acked Flush 0 Done Done Sent false
9692 Acked Write 8 Done Done Sent false
9693 Acked Read 6 Done Done Sent false
9694 Acked Write 2 Done Done Sent false
9695 Acked Write 8 Done Done Sent false
9696 Acked Write 7 Done Done Sent false
9697 Acked Read 10 Done Done Sent false
9698 Acked Read 6 Done Done Sent false
9699 Acked Read 4 Done Done Sent false
9700 Acked Read 4 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 9 Done Done Sent false
9703 Acked Read 9 Done Done Sent false
9704 Acked Read 1 Done Done Sent false
9705 Acked Read 2 Done Done Sent false
9706 Acked Write 3 Done Done Sent false
9707 Acked Read 10 Done Done Sent false
9708 Acked Write 8 Done Done Sent false
9709 Acked Write 10 Done Done Sent false
9710 Acked Read 3 Done Done Sent false
9711 Acked Read 6 Done Done Sent false
9712 Acked Write 4 Done Done Sent false
9713 Acked Read 5 Done Done Sent false
9714 Acked Flush 0 Done Done Sent false
9715 Acked Read 1 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Read 6 Done Done Sent false
9718 Acked Write 8 Done Done Sent false
9719 Acked Read 9 Done Done Sent false
9720 Acked Write 5 Done Done Sent false
9721 Acked Write 9 Done Done Sent false
9722 Acked Read 7 Done Done Sent false
9723 Acked Read 1 Done Done Sent false
9724 Acked Read 9 Done Done Sent false
9725 Acked Read 2 Done Done Sent false
9726 Acked Read 4 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Read 8 Done Done Sent false
9729 Acked Read 5 Done Done Sent false
9730 Acked Read 2 Done Done Sent false
9731 Acked Read 3 Done Done Sent false
9732 Acked Read 10 Done Done Sent false
9733 Acked Read 1 Done Done Sent false
9734 Acked Write 3 Done Done Sent false
9735 Acked Read 5 Done Done Sent false
9736 Acked Read 2 Done Done Sent false
9737 Acked Write 9 Done Done Sent false
9738 Acked Read 1 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Write 2 Done Done Sent false
9741 Acked Write 6 Done Done Sent false
9742 Acked Write 2 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 7 Done Done Sent false
9746 Acked Read 4 Done Done Sent false
9747 Acked Write 7 Done Done Sent false
9748 Acked Read 3 Done Done Sent false
9749 Acked Read 4 Done Done Sent false
9750 Acked Flush 0 Done Done Sent false
9751 Acked Read 3 Done Done Sent false
9752 Acked Read 10 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9750 9750 9451
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9752 9751 9750 9749 9748
CLIENT: Up:0 ds:301 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Read 1 Done Done Sent false
9453 Acked Read 1 Done Done Sent false
9454 Acked Read 7 Done Done Sent false
9455 Acked Read 5 Done Done Sent false
9456 Acked Write 9 Done Done Sent false
9457 Acked Write 5 Done Done Sent false
9458 Acked Write 10 Done Done Sent false
9459 Acked Read 6 Done Done Sent false
9460 Acked Write 2 Done Done Sent false
9461 Acked Write 10 Done Done Sent false
9462 Acked Write 9 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 9 Done Done Sent false
9465 Acked Write 4 Done Done Sent false
9466 Acked Write 4 Done Done Sent false
9467 Acked Read 2 Done Done Sent false
9468 Acked Write 9 Done Done Sent false
9469 Acked Read 8 Done Done Sent false
9470 Acked Read 2 Done Done Sent false
9471 Acked Write 2 Done Done Sent false
9472 Acked Write 2 Done Done Sent false
9473 Acked Write 10 Done Done Sent false
9474 Acked Read 5 Done Done Sent false
9475 Acked Flush 0 Done Done Sent false
9476 Acked Read 9 Done Done Sent false
9477 Acked Write 10 Done Done Sent false
9478 Acked Read 3 Done Done Sent false
9479 Acked Write 3 Done Done Sent false
9480 Acked Read 5 Done Done Sent false
9481 Acked Write 4 Done Done Sent false
9482 Acked Write 4 Done Done Sent false
9483 Acked Read 5 Done Done Sent false
9484 Acked Write 3 Done Done Sent false
9485 Acked Read 3 Done Done Sent false
9486 Acked Read 4 Done Done Sent false
9487 Acked Write 9 Done Done Sent false
9488 Acked Read 4 Done Done Sent false
9489 Acked Write 4 Done Done Sent false
9490 Acked Read 4 Done Done Sent false
9491 Acked Read 9 Done Done Sent false
9492 Acked Write 4 Done Done Sent false
9493 Acked Flush 0 Done Done Sent false
9494 Acked Read 1 Done Done Sent false
9495 Acked Flush 0 Done Done Sent false
9496 Acked Read 1 Done Done Sent false
9497 Acked Flush 0 Done Done Sent false
9498 Acked Write 2 Done Done Sent false
9499 Acked Read 2 Done Done Sent false
9500 Acked Read 1 Done Done Sent false
9501 Acked Write 1 Done Done Sent false
9502 Acked Read 1 Done Done Sent false
9503 Acked Read 5 Done Done Sent false
9504 Acked Read 3 Done Done Sent false
9505 Acked Read 2 Done Done Sent false
9506 Acked Flush 0 Done Done Sent false
9507 Acked Flush 0 Done Done Sent false
9508 Acked Write 7 Done Done Sent false
9509 Acked Read 2 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Read 3 Done Done Sent false
9512 Acked Read 10 Done Done Sent false
9513 Acked Write 4 Done Done Sent false
9514 Acked Flush 0 Done Done Sent false
9515 Acked Read 2 Done Done Sent false
9516 Acked Write 10 Done Done Sent false
9517 Acked Write 9 Done Done Sent false
9518 Acked Read 6 Done Done Sent false
9519 Acked Read 2 Done Done Sent false
9520 Acked Write 8 Done Done Sent false
9521 Acked Write 5 Done Done Sent false
9522 Acked Read 10 Done Done Sent false
9523 Acked Read 4 Done Done Sent false
9524 Acked Read 6 Done Done Sent false
9525 Acked Write 3 Done Done Sent false
9526 Acked Write 3 Done Done Sent false
9527 Acked Read 10 Done Done Sent false
9528 Acked Read 8 Done Done Sent false
9529 Acked Write 6 Done Done Sent false
9530 Acked Write 10 Done Done Sent false
9531 Acked Read 5 Done Done Sent false
9532 Acked Flush 0 Done Done Sent false
9533 Acked Read 3 Done Done Sent false
9534 Acked Read 6 Done Done Sent false
9535 Acked Read 9 Done Done Sent false
9536 Acked Read 1 Done Done Sent false
9537 Acked Read 2 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Flush 0 Done Done Sent false
9540 Acked Read 1 Done Done Sent false
9541 Acked Read 10 Done Done Sent false
9542 Acked Write 4 Done Done Sent false
9543 Acked Read 1 Done Done Sent false
9544 Acked Read 8 Done Done Sent false
9545 Acked Write 6 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 10 Done Done Sent false
9548 Acked Write 3 Done Done Sent false
9549 Acked Write 4 Done Done Sent false
9550 Acked Read 4 Done Done Sent false
9551 Acked Write 10 Done Done Sent false
9552 Acked Read 5 Done Done Sent false
9553 Acked Write 6 Done Done Sent false
9554 Acked Read 8 Done Done Sent false
9555 Acked Flush 0 Done Done Sent false
9556 Acked Read 9 Done Done Sent false
9557 Acked Read 9 Done Done Sent false
9558 Acked Read 10 Done Done Sent false
9559 Acked Write 10 Done Done Sent false
9560 Acked Flush 0 Done Done Sent false
9561 Acked Write 1 Done Done Sent false
9562 Acked Flush 0 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Flush 0 Done Done Sent false
9565 Acked Write 1 Done Done Sent false
9566 Acked Read 8 Done Done Sent false
9567 Acked Read 8 Done Done Sent false
9568 Acked Flush 0 Done Done Sent false
9569 Acked Write 4 Done Done Sent false
9570 Acked Read 5 Done Done Sent false
9571 Acked Write 4 Done Done Sent false
9572 Acked Read 5 Done Done Sent false
9573 Acked Read 10 Done Done Sent false
9574 Acked Write 2 Done Done Sent false
9575 Acked Write 8 Done Done Sent false
9576 Acked Flush 0 Done Done Sent false
9577 Acked Read 9 Done Done Sent false
9578 Acked Read 2 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 10 Done Done Sent false
9581 Acked Read 3 Done Done Sent false
9582 Acked Read 10 Done Done Sent false
9583 Acked Write 5 Done Done Sent false
9584 Acked Write 3 Done Done Sent false
9585 Acked Read 6 Done Done Sent false
9586 Acked Write 2 Done Done Sent false
9587 Acked Write 1 Done Done Sent false
9588 Acked Write 3 Done Done Sent false
9589 Acked Read 9 Done Done Sent false
9590 Acked Read 3 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 9 Done Done Sent false
9593 Acked Read 2 Done Done Sent false
9594 Acked Write 9 Done Done Sent false
9595 Acked Write 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Write 7 Done Done Sent false
9598 Acked Write 2 Done Done Sent false
9599 Acked Read 10 Done Done Sent false
9600 Acked Write 2 Done Done Sent false
9601 Acked Write 3 Done Done Sent false
9602 Acked Read 2 Done Done Sent false
9603 Acked Write 7 Done Done Sent false
9604 Acked Write 7 Done Done Sent false
9605 Acked Write 8 Done Done Sent false
9606 Acked Read 9 Done Done Sent false
9607 Acked Write 10 Done Done Sent false
9608 Acked Read 5 Done Done Sent false
9609 Acked Read 6 Done Done Sent false
9610 Acked Write 4 Done Done Sent false
9611 Acked Write 4 Done Done Sent false
9612 Acked Read 4 Done Done Sent false
9613 Acked Write 2 Done Done Sent false
9614 Acked Write 5 Done Done Sent false
9615 Acked Flush 0 Done Done Sent false
9616 Acked Read 10 Done Done Sent false
9617 Acked Read 9 Done Done Sent false
9618 Acked Read 7 Done Done Sent false
9619 Acked Write 5 Done Done Sent false
9620 Acked Flush 0 Done Done Sent false
9621 Acked Read 4 Done Done Sent false
9622 Acked Read 9 Done Done Sent false
9623 Acked Read 10 Done Done Sent false
9624 Acked Read 4 Done Done Sent false
9625 Acked Write 9 Done Done Sent false
9626 Acked Flush 0 Done Done Sent false
9627 Acked Read 5 Done Done Sent false
9628 Acked Write 10 Done Done Sent false
9629 Acked Flush 0 Done Done Sent false
9630 Acked Write 2 Done Done Sent false
9631 Acked Read 8 Done Done Sent false
9632 Acked Read 10 Done Done Sent false
9633 Acked Write 8 Done Done Sent false
9634 Acked Read 7 Done Done Sent false
9635 Acked Write 6 Done Done Sent false
9636 Acked Write 9 Done Done Sent false
9637 Acked Write 1 Done Done Sent false
9638 Acked Flush 0 Done Done Sent false
9639 Acked Read 5 Done Done Sent false
9640 Acked Read 1 Done Done Sent false
9641 Acked Write 1 Done Done Sent false
9642 Acked Read 3 Done Done Sent false
9643 Acked Read 7 Done Done Sent false
9644 Acked Read 3 Done Done Sent false
9645 Acked Read 8 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 9 Done Done Sent false
9649 Acked Write 7 Done Done Sent false
9650 Acked Write 4 Done Done Sent false
9651 Acked Read 1 Done Done Sent false
9652 Acked Flush 0 Done Done Sent false
9653 Acked Write 1 Done Done Sent false
9654 Acked Write 2 Done Done Sent false
9655 Acked Read 6 Done Done Sent false
9656 Acked Read 5 Done Done Sent false
9657 Acked Read 9 Done Done Sent false
9658 Acked Write 9 Done Done Sent false
9659 Acked Read 2 Done Done Sent false
9660 Acked Write 7 Done Done Sent false
9661 Acked Read 3 Done Done Sent false
9662 Acked Read 8 Done Done Sent false
9663 Acked Flush 0 Done Done Sent false
9664 Acked Write 10 Done Done Sent false
9665 Acked Read 5 Done Done Sent false
9666 Acked Write 4 Done Done Sent false
9667 Acked Read 3 Done Done Sent false
9668 Acked Write 4 Done Done Sent false
9669 Acked Read 9 Done Done Sent false
9670 Acked Write 9 Done Done Sent false
9671 Acked Read 1 Done Done Sent false
9672 Acked Read 3 Done Done Sent false
9673 Acked Write 7 Done Done Sent false
9674 Acked Read 3 Done Done Sent false
9675 Acked Write 8 Done Done Sent false
9676 Acked Read 8 Done Done Sent false
9677 Acked Write 7 Done Done Sent false
9678 Acked Read 8 Done Done Sent false
9679 Acked Read 2 Done Done Sent false
9680 Acked Read 2 Done Done Sent false
9681 Acked Read 3 Done Done Sent false
9682 Acked Read 9 Done Done Sent false
9683 Acked Read 8 Done Done Sent false
9684 Acked Read 1 Done Done Sent false
9685 Acked Write 8 Done Done Sent false
9686 Acked Write 2 Done Done Sent false
9687 Acked Write 5 Done Done Sent false
9688 Acked Read 10 Done Done Sent false
9689 Acked Read 10 Done Done Sent false
9690 Acked Write 2 Done Done Sent false
9691 Acked Flush 0 Done Done Sent false
9692 Acked Write 8 Done Done Sent false
9693 Acked Read 6 Done Done Sent false
9694 Acked Write 2 Done Done Sent false
9695 Acked Write 8 Done Done Sent false
9696 Acked Write 7 Done Done Sent false
9697 Acked Read 10 Done Done Sent false
9698 Acked Read 6 Done Done Sent false
9699 Acked Read 4 Done Done Sent false
9700 Acked Read 4 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 9 Done Done Sent false
9703 Acked Read 9 Done Done Sent false
9704 Acked Read 1 Done Done Sent false
9705 Acked Read 2 Done Done Sent false
9706 Acked Write 3 Done Done Sent false
9707 Acked Read 10 Done Done Sent false
9708 Acked Write 8 Done Done Sent false
9709 Acked Write 10 Done Done Sent false
9710 Acked Read 3 Done Done Sent false
9711 Acked Read 6 Done Done Sent false
9712 Acked Write 4 Done Done Sent false
9713 Acked Read 5 Done Done Sent false
9714 Acked Flush 0 Done Done Sent false
9715 Acked Read 1 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Read 6 Done Done Sent false
9718 Acked Write 8 Done Done Sent false
9719 Acked Read 9 Done Done Sent false
9720 Acked Write 5 Done Done Sent false
9721 Acked Write 9 Done Done Sent false
9722 Acked Read 7 Done Done Sent false
9723 Acked Read 1 Done Done Sent false
9724 Acked Read 9 Done Done Sent false
9725 Acked Read 2 Done Done Sent false
9726 Acked Read 4 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Read 8 Done Done Sent false
9729 Acked Read 5 Done Done Sent false
9730 Acked Read 2 Done Done Sent false
9731 Acked Read 3 Done Done Sent false
9732 Acked Read 10 Done Done Sent false
9733 Acked Read 1 Done Done Sent false
9734 Acked Write 3 Done Done Sent false
9735 Acked Read 5 Done Done Sent false
9736 Acked Read 2 Done Done Sent false
9737 Acked Write 9 Done Done Sent false
9738 Acked Read 1 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Write 2 Done Done Sent false
9741 Acked Write 6 Done Done Sent false
9742 Acked Write 2 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 7 Done Done Sent false
9746 Acked Read 4 Done Done Sent false
9747 Acked Write 7 Done Done Sent false
9748 Acked Read 3 Done Done Sent false
9749 Acked Read 4 Done Done Sent false
9750 Acked Flush 0 Done Done Sent false
9751 Acked Read 3 Done Done Sent false
9752 Acked Read 10 Done Done Sent false
9753 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9753 9753 9451
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9753 9752 9751 9750 9749
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.844118307Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:05.844214466Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:05.844256901Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:05.844305615Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.84435442Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Read 1 Done Done Sent false
9453 Acked Read 1 Done Done Sent false
9454 Acked Read 7 Done Done Sent false
9455 Acked Read 5 Done Done Sent false
9456 Acked Write 9 Done Done Sent false
9457 Acked Write 5 Done Done Sent false
9458 Acked Write 10 Done Done Sent false
9459 Acked Read 6 Done Done Sent false
9460 Acked Write 2 Done Done Sent false
9461 Acked Write 10 Done Done Sent false
9462 Acked Write 9 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 9 Done Done Sent false
9465 Acked Write 4 Done Done Sent false
9466 Acked Write 4 Done Done Sent false
9467 Acked Read 2 Done Done Sent false
9468 Acked Write 9 Done Done Sent false
9469 Acked Read 8 Done Done Sent false
9470 Acked Read 2 Done Done Sent false
9471 Acked Write 2 Done Done Sent false
9472 Acked Write 2 Done Done Sent false
9473 Acked Write 10 Done Done Sent false
9474 Acked Read 5 Done Done Sent false
9475 Acked Flush 0 Done Done Sent false
9476 Acked Read 9 Done Done Sent false
9477 Acked Write 10 Done Done Sent false
9478 Acked Read 3 Done Done Sent false
9479 Acked Write 3 Done Done Sent false
9480 Acked Read 5 Done Done Sent false
9481 Acked Write 4 Done Done Sent false
9482 Acked Write 4 Done Done Sent false
9483 Acked Read 5 Done Done Sent false
9484 Acked Write 3 Done Done Sent false
9485 Acked Read 3 Done Done Sent false
9486 Acked Read 4 Done Done Sent false
9487 Acked Write 9 Done Done Sent false
9488 Acked Read 4 Done Done Sent false
9489 Acked Write 4 Done Done Sent false
9490 Acked Read 4 Done Done Sent false
9491 Acked Read 9 Done Done Sent false
9492 Acked Write 4 Done Done Sent false
9493 Acked Flush 0 Done Done Sent false
9494 Acked Read 1 Done Done Sent false
9495 Acked Flush 0 Done Done Sent false
9496 Acked Read 1 Done Done Sent false
9497 Acked Flush 0 Done Done Sent false
9498 Acked Write 2 Done Done Sent false
9499 Acked Read 2 Done Done Sent false
9500 Acked Read 1 Done Done Sent false
9501 Acked Write 1 Done Done Sent false
9502 Acked Read 1 Done Done Sent false
9503 Acked Read 5 Done Done Sent false
9504 Acked Read 3 Done Done Sent false
9505 Acked Read 2 Done Done Sent false
9506 Acked Flush 0 Done Done Sent false
9507 Acked Flush 0 Done Done Sent false
9508 Acked Write 7 Done Done Sent false
9509 Acked Read 2 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Read 3 Done Done Sent false
9512 Acked Read 10 Done Done Sent false
9513 Acked Write 4 Done Done Sent false
9514 Acked Flush 0 Done Done Sent false
9515 Acked Read 2 Done Done Sent false
9516 Acked Write 10 Done Done Sent false
9517 Acked Write 9 Done Done Sent false
9518 Acked Read 6 Done Done Sent false
9519 Acked Read 2 Done Done Sent false
9520 Acked Write 8 Done Done Sent false
9521 Acked Write 5 Done Done Sent false
9522 Acked Read 10 Done Done Sent false
9523 Acked Read 4 Done Done Sent false
9524 Acked Read 6 Done Done Sent false
9525 Acked Write 3 Done Done Sent false
9526 Acked Write 3 Done Done Sent false
9527 Acked Read 10 Done Done Sent false
9528 Acked Read 8 Done Done Sent false
9529 Acked Write 6 Done Done Sent false
9530 Acked Write 10 Done Done Sent false
9531 Acked Read 5 Done Done Sent false
9532 Acked Flush 0 Done Done Sent false
9533 Acked Read 3 Done Done Sent false
9534 Acked Read 6 Done Done Sent false
9535 Acked Read 9 Done Done Sent false
9536 Acked Read 1 Done Done Sent false
9537 Acked Read 2 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Flush 0 Done Done Sent false
9540 Acked Read 1 Done Done Sent false
9541 Acked Read 10 Done Done Sent false
9542 Acked Write 4 Done Done Sent false
9543 Acked Read 1 Done Done Sent false
9544 Acked Read 8 Done Done Sent false
9545 Acked Write 6 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 10 Done Done Sent false
9548 Acked Write 3 Done Done Sent false
9549 Acked Write 4 Done Done Sent false
9550 Acked Read 4 Done Done Sent false
9551 Acked Write 10 Done Done Sent false
9552 Acked Read 5 Done Done Sent false
9553 Acked Write 6 Done Done Sent false
9554 Acked Read 8 Done Done Sent false
9555 Acked Flush 0 Done Done Sent false
9556 Acked Read 9 Done Done Sent false
9557 Acked Read 9 Done Done Sent false
9558 Acked Read 10 Done Done Sent false
9559 Acked Write 10 Done Done Sent false
9560 Acked Flush 0 Done Done Sent false
9561 Acked Write 1 Done Done Sent false
9562 Acked Flush 0 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Flush 0 Done Done Sent false
9565 Acked Write 1 Done Done Sent false
9566 Acked Read 8 Done Done Sent false
9567 Acked Read 8 Done Done Sent false
9568 Acked Flush 0 Done Done Sent false
9569 Acked Write 4 Done Done Sent false
9570 Acked Read 5 Done Done Sent false
9571 Acked Write 4 Done Done Sent false
9572 Acked Read 5 Done Done Sent false
9573 Acked Read 10 Done Done Sent false
9574 Acked Write 2 Done Done Sent false
9575 Acked Write 8 Done Done Sent false
9576 Acked Flush 0 Done Done Sent false
9577 Acked Read 9 Done Done Sent false
9578 Acked Read 2 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 10 Done Done Sent false
9581 Acked Read 3 Done Done Sent false
9582 Acked Read 10 Done Done Sent false
9583 Acked Write 5 Done Done Sent false
9584 Acked Write 3 Done Done Sent false
9585 Acked Read 6 Done Done Sent false
9586 Acked Write 2 Done Done Sent false
9587 Acked Write 1 Done Done Sent false
9588 Acked Write 3 Done Done Sent false
9589 Acked Read 9 Done Done Sent false
9590 Acked Read 3 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 9 Done Done Sent false
9593 Acked Read 2 Done Done Sent false
9594 Acked Write 9 Done Done Sent false
9595 Acked Write 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Write 7 Done Done Sent false
9598 Acked Write 2 Done Done Sent false
9599 Acked Read 10 Done Done Sent false
9600 Acked Write 2 Done Done Sent false
9601 Acked Write 3 Done Done Sent false
9602 Acked Read 2 Done Done Sent false
9603 Acked Write 7 Done Done Sent false
9604 Acked Write 7 Done Done Sent false
9605 Acked Write 8 Done Done Sent false
9606 Acked Read 9 Done Done Sent false
9607 Acked Write 10 Done Done Sent false
9608 Acked Read 5 Done Done Sent false
9609 Acked Read 6 Done Done Sent false
9610 Acked Write 4 Done Done Sent false
9611 Acked Write 4 Done Done Sent false
9612 Acked Read 4 Done Done Sent false
9613 Acked Write 2 Done Done Sent false
9614 Acked Write 5 Done Done Sent false
9615 Acked Flush 0 Done Done Sent false
9616 Acked Read 10 Done Done Sent false
9617 Acked Read 9 Done Done Sent false
9618 Acked Read 7 Done Done Sent false
9619 Acked Write 5 Done Done Sent false
9620 Acked Flush 0 Done Done Sent false
9621 Acked Read 4 Done Done Sent false
9622 Acked Read 9 Done Done Sent false
9623 Acked Read 10 Done Done Sent false
9624 Acked Read 4 Done Done Sent false
9625 Acked Write 9 Done Done Sent false
9626 Acked Flush 0 Done Done Sent false
9627 Acked Read 5 Done Done Sent false
9628 Acked Write 10 Done Done Sent false
9629 Acked Flush 0 Done Done Sent false
9630 Acked Write 2 Done Done Sent false
9631 Acked Read 8 Done Done Sent false
9632 Acked Read 10 Done Done Sent false
9633 Acked Write 8 Done Done Sent false
9634 Acked Read 7 Done Done Sent false
9635 Acked Write 6 Done Done Sent false
9636 Acked Write 9 Done Done Sent false
9637 Acked Write 1 Done Done Sent false
9638 Acked Flush 0 Done Done Sent false
9639 Acked Read 5 Done Done Sent false
9640 Acked Read 1 Done Done Sent false
9641 Acked Write 1 Done Done Sent false
9642 Acked Read 3 Done Done Sent false
9643 Acked Read 7 Done Done Sent false
9644 Acked Read 3 Done Done Sent false
9645 Acked Read 8 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 9 Done Done Sent false
9649 Acked Write 7 Done Done Sent false
9650 Acked Write 4 Done Done Sent false
9651 Acked Read 1 Done Done Sent false
9652 Acked Flush 0 Done Done Sent false
9653 Acked Write 1 Done Done Sent false
9654 Acked Write 2 Done Done Sent false
9655 Acked Read 6 Done Done Sent false
9656 Acked Read 5 Done Done Sent false
9657 Acked Read 9 Done Done Sent false
9658 Acked Write 9 Done Done Sent false
9659 Acked Read 2 Done Done Sent false
9660 Acked Write 7 Done Done Sent false
9661 Acked Read 3 Done Done Sent false
9662 Acked Read 8 Done Done Sent false
9663 Acked Flush 0 Done Done Sent false
9664 Acked Write 10 Done Done Sent false
9665 Acked Read 5 Done Done Sent false
9666 Acked Write 4 Done Done Sent false
9667 Acked Read 3 Done Done Sent false
9668 Acked Write 4 Done Done Sent false
9669 Acked Read 9 Done Done Sent false
9670 Acked Write 9 Done Done Sent false
9671 Acked Read 1 Done Done Sent false
9672 Acked Read 3 Done Done Sent false
9673 Acked Write 7 Done Done Sent false
9674 Acked Read 3 Done Done Sent false
9675 Acked Write 8 Done Done Sent false
9676 Acked Read 8 Done Done Sent false
9677 Acked Write 7 Done Done Sent false
9678 Acked Read 8 Done Done Sent false
9679 Acked Read 2 Done Done Sent false
9680 Acked Read 2 Done Done Sent false
9681 Acked Read 3 Done Done Sent false
9682 Acked Read 9 Done Done Sent false
9683 Acked Read 8 Done Done Sent false
9684 Acked Read 1 Done Done Sent false
9685 Acked Write 8 Done Done Sent false
9686 Acked Write 2 Done Done Sent false
9687 Acked Write 5 Done Done Sent false
9688 Acked Read 10 Done Done Sent false
9689 Acked Read 10 Done Done Sent false
9690 Acked Write 2 Done Done Sent false
9691 Acked Flush 0 Done Done Sent false
9692 Acked Write 8 Done Done Sent false
9693 Acked Read 6 Done Done Sent false
9694 Acked Write 2 Done Done Sent false
9695 Acked Write 8 Done Done Sent false
9696 Acked Write 7 Done Done Sent false
9697 Acked Read 10 Done Done Sent false
9698 Acked Read 6 Done Done Sent false
9699 Acked Read 4 Done Done Sent false
9700 Acked Read 4 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 9 Done Done Sent false
9703 Acked Read 9 Done Done Sent false
9704 Acked Read 1 Done Done Sent false
9705 Acked Read 2 Done Done Sent false
9706 Acked Write 3 Done Done Sent false
9707 Acked Read 10 Done Done Sent false
9708 Acked Write 8 Done Done Sent false
9709 Acked Write 10 Done Done Sent false
9710 Acked Read 3 Done Done Sent false
9711 Acked Read 6 Done Done Sent false
9712 Acked Write 4 Done Done Sent false
9713 Acked Read 5 Done Done Sent false
9714 Acked Flush 0 Done Done Sent false
9715 Acked Read 1 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Read 6 Done Done Sent false
9718 Acked Write 8 Done Done Sent false
9719 Acked Read 9 Done Done Sent false
9720 Acked Write 5 Done Done Sent false
9721 Acked Write 9 Done Done Sent false
9722 Acked Read 7 Done Done Sent false
9723 Acked Read 1 Done Done Sent false
9724 Acked Read 9 Done Done Sent false
9725 Acked Read 2 Done Done Sent false
9726 Acked Read 4 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Read 8 Done Done Sent false
9729 Acked Read 5 Done Done Sent false
9730 Acked Read 2 Done Done Sent false
9731 Acked Read 3 Done Done Sent false
9732 Acked Read 10 Done Done Sent false
9733 Acked Read 1 Done Done Sent false
9734 Acked Write 3 Done Done Sent false
9735 Acked Read 5 Done Done Sent false
9736 Acked Read 2 Done Done Sent false
9737 Acked Write 9 Done Done Sent false
9738 Acked Read 1 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Write 2 Done Done Sent false
9741 Acked Write 6 Done Done Sent false
9742 Acked Write 2 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 7 Done Done Sent false
9746 Acked Read 4 Done Done Sent false
9747 Acked Write 7 Done Done Sent false
9748 Acked Read 3 Done Done Sent false
9749 Acked Read 4 Done Done Sent false
9750 Acked Flush 0 Done Done Sent false
9751 Acked Read 3 Done Done Sent false
9752 Acked Read 10 Done Done Sent false
9753 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9753 9753 9451
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9753 9752 9751 9750 9749
CLIENT: Up:0 ds:302 act:2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9452 Acked Read 1 Done Done Sent false
9453 Acked Read 1 Done Done Sent false
9454 Acked Read 7 Done Done Sent false
9455 Acked Read 5 Done Done Sent false
9456 Acked Write 9 Done Done Sent false
9457 Acked Write 5 Done Done Sent false
9458 Acked Write 10 Done Done Sent false
9459 Acked Read 6 Done Done Sent false
9460 Acked Write 2 Done Done Sent false
9461 Acked Write 10 Done Done Sent false
9462 Acked Write 9 Done Done Sent false
9463 Acked Read 10 Done Done Sent false
9464 Acked Read 9 Done Done Sent false
9465 Acked Write 4 Done Done Sent false
9466 Acked Write 4 Done Done Sent false
9467 Acked Read 2 Done Done Sent false
9468 Acked Write 9 Done Done Sent false
9469 Acked Read 8 Done Done Sent false
9470 Acked Read 2 Done Done Sent false
9471 Acked Write 2 Done Done Sent false
9472 Acked Write 2 Done Done Sent false
9473 Acked Write 10 Done Done Sent false
9474 Acked Read 5 Done Done Sent false
9475 Acked Flush 0 Done Done Sent false
9476 Acked Read 9 Done Done Sent false
9477 Acked Write 10 Done Done Sent false
9478 Acked Read 3 Done Done Sent false
9479 Acked Write 3 Done Done Sent false
9480 Acked Read 5 Done Done Sent false
9481 Acked Write 4 Done Done Sent false
9482 Acked Write 4 Done Done Sent false
9483 Acked Read 5 Done Done Sent false
9484 Acked Write 3 Done Done Sent false
9485 Acked Read 3 Done Done Sent false
9486 Acked Read 4 Done Done Sent false
9487 Acked Write 9 Done Done Sent false
9488 Acked Read 4 Done Done Sent false
9489 Acked Write 4 Done Done Sent false
9490 Acked Read 4 Done Done Sent false
9491 Acked Read 9 Done Done Sent false
9492 Acked Write 4 Done Done Sent false
9493 Acked Flush 0 Done Done Sent false
9494 Acked Read 1 Done Done Sent false
9495 Acked Flush 0 Done Done Sent false
9496 Acked Read 1 Done Done Sent false
9497 Acked Flush 0 Done Done Sent false
9498 Acked Write 2 Done Done Sent false
9499 Acked Read 2 Done Done Sent false
9500 Acked Read 1 Done Done Sent false
9501 Acked Write 1 Done Done Sent false
9502 Acked Read 1 Done Done Sent false
9503 Acked Read 5 Done Done Sent false
9504 Acked Read 3 Done Done Sent false
9505 Acked Read 2 Done Done Sent false
9506 Acked Flush 0 Done Done Sent false
9507 Acked Flush 0 Done Done Sent false
9508 Acked Write 7 Done Done Sent false
9509 Acked Read 2 Done Done Sent false
9510 Acked Read 6 Done Done Sent false
9511 Acked Read 3 Done Done Sent false
9512 Acked Read 10 Done Done Sent false
9513 Acked Write 4 Done Done Sent false
9514 Acked Flush 0 Done Done Sent false
9515 Acked Read 2 Done Done Sent false
9516 Acked Write 10 Done Done Sent false
9517 Acked Write 9 Done Done Sent false
9518 Acked Read 6 Done Done Sent false
9519 Acked Read 2 Done Done Sent false
9520 Acked Write 8 Done Done Sent false
9521 Acked Write 5 Done Done Sent false
9522 Acked Read 10 Done Done Sent false
9523 Acked Read 4 Done Done Sent false
9524 Acked Read 6 Done Done Sent false
9525 Acked Write 3 Done Done Sent false
9526 Acked Write 3 Done Done Sent false
9527 Acked Read 10 Done Done Sent false
9528 Acked Read 8 Done Done Sent false
9529 Acked Write 6 Done Done Sent false
9530 Acked Write 10 Done Done Sent false
9531 Acked Read 5 Done Done Sent false
9532 Acked Flush 0 Done Done Sent false
9533 Acked Read 3 Done Done Sent false
9534 Acked Read 6 Done Done Sent false
9535 Acked Read 9 Done Done Sent false
9536 Acked Read 1 Done Done Sent false
9537 Acked Read 2 Done Done Sent false
9538 Acked Write 2 Done Done Sent false
9539 Acked Flush 0 Done Done Sent false
9540 Acked Read 1 Done Done Sent false
9541 Acked Read 10 Done Done Sent false
9542 Acked Write 4 Done Done Sent false
9543 Acked Read 1 Done Done Sent false
9544 Acked Read 8 Done Done Sent false
9545 Acked Write 6 Done Done Sent false
9546 Acked Write 2 Done Done Sent false
9547 Acked Write 10 Done Done Sent false
9548 Acked Write 3 Done Done Sent false
9549 Acked Write 4 Done Done Sent false
9550 Acked Read 4 Done Done Sent false
9551 Acked Write 10 Done Done Sent false
9552 Acked Read 5 Done Done Sent false
9553 Acked Write 6 Done Done Sent false
9554 Acked Read 8 Done Done Sent false
9555 Acked Flush 0 Done Done Sent false
9556 Acked Read 9 Done Done Sent false
9557 Acked Read 9 Done Done Sent false
9558 Acked Read 10 Done Done Sent false
9559 Acked Write 10 Done Done Sent false
9560 Acked Flush 0 Done Done Sent false
9561 Acked Write 1 Done Done Sent false
9562 Acked Flush 0 Done Done Sent false
9563 Acked Read 10 Done Done Sent false
9564 Acked Flush 0 Done Done Sent false
9565 Acked Write 1 Done Done Sent false
9566 Acked Read 8 Done Done Sent false
9567 Acked Read 8 Done Done Sent false
9568 Acked Flush 0 Done Done Sent false
9569 Acked Write 4 Done Done Sent false
9570 Acked Read 5 Done Done Sent false
9571 Acked Write 4 Done Done Sent false
9572 Acked Read 5 Done Done Sent false
9573 Acked Read 10 Done Done Sent false
9574 Acked Write 2 Done Done Sent false
9575 Acked Write 8 Done Done Sent false
9576 Acked Flush 0 Done Done Sent false
9577 Acked Read 9 Done Done Sent false
9578 Acked Read 2 Done Done Sent false
9579 Acked Write 3 Done Done Sent false
9580 Acked Read 10 Done Done Sent false
9581 Acked Read 3 Done Done Sent false
9582 Acked Read 10 Done Done Sent false
9583 Acked Write 5 Done Done Sent false
9584 Acked Write 3 Done Done Sent false
9585 Acked Read 6 Done Done Sent false
9586 Acked Write 2 Done Done Sent false
9587 Acked Write 1 Done Done Sent false
9588 Acked Write 3 Done Done Sent false
9589 Acked Read 9 Done Done Sent false
9590 Acked Read 3 Done Done Sent false
9591 Acked Write 3 Done Done Sent false
9592 Acked Write 9 Done Done Sent false
9593 Acked Read 2 Done Done Sent false
9594 Acked Write 9 Done Done Sent false
9595 Acked Write 6 Done Done Sent false
9596 Acked Read 9 Done Done Sent false
9597 Acked Write 7 Done Done Sent false
9598 Acked Write 2 Done Done Sent false
9599 Acked Read 10 Done Done Sent false
9600 Acked Write 2 Done Done Sent false
9601 Acked Write 3 Done Done Sent false
9602 Acked Read 2 Done Done Sent false
9603 Acked Write 7 Done Done Sent false
9604 Acked Write 7 Done Done Sent false
9605 Acked Write 8 Done Done Sent false
9606 Acked Read 9 Done Done Sent false
9607 Acked Write 10 Done Done Sent false
9608 Acked Read 5 Done Done Sent false
9609 Acked Read 6 Done Done Sent false
9610 Acked Write 4 Done Done Sent false
9611 Acked Write 4 Done Done Sent false
9612 Acked Read 4 Done Done Sent false
9613 Acked Write 2 Done Done Sent false
9614 Acked Write 5 Done Done Sent false
9615 Acked Flush 0 Done Done Sent false
9616 Acked Read 10 Done Done Sent false
9617 Acked Read 9 Done Done Sent false
9618 Acked Read 7 Done Done Sent false
9619 Acked Write 5 Done Done Sent false
9620 Acked Flush 0 Done Done Sent false
9621 Acked Read 4 Done Done Sent false
9622 Acked Read 9 Done Done Sent false
9623 Acked Read 10 Done Done Sent false
9624 Acked Read 4 Done Done Sent false
9625 Acked Write 9 Done Done Sent false
9626 Acked Flush 0 Done Done Sent false
9627 Acked Read 5 Done Done Sent false
9628 Acked Write 10 Done Done Sent false
9629 Acked Flush 0 Done Done Sent false
9630 Acked Write 2 Done Done Sent false
9631 Acked Read 8 Done Done Sent false
9632 Acked Read 10 Done Done Sent false
9633 Acked Write 8 Done Done Sent false
9634 Acked Read 7 Done Done Sent false
9635 Acked Write 6 Done Done Sent false
9636 Acked Write 9 Done Done Sent false
9637 Acked Write 1 Done Done Sent false
9638 Acked Flush 0 Done Done Sent false
9639 Acked Read 5 Done Done Sent false
9640 Acked Read 1 Done Done Sent false
9641 Acked Write 1 Done Done Sent false
9642 Acked Read 3 Done Done Sent false
9643 Acked Read 7 Done Done Sent false
9644 Acked Read 3 Done Done Sent false
9645 Acked Read 8 Done Done Sent false
9646 Acked Write 9 Done Done Sent false
9647 Acked Flush 0 Done Done Sent false
9648 Acked Read 9 Done Done Sent false
9649 Acked Write 7 Done Done Sent false
9650 Acked Write 4 Done Done Sent false
9651 Acked Read 1 Done Done Sent false
9652 Acked Flush 0 Done Done Sent false
9653 Acked Write 1 Done Done Sent false
9654 Acked Write 2 Done Done Sent false
9655 Acked Read 6 Done Done Sent false
9656 Acked Read 5 Done Done Sent false
9657 Acked Read 9 Done Done Sent false
9658 Acked Write 9 Done Done Sent false
9659 Acked Read 2 Done Done Sent false
9660 Acked Write 7 Done Done Sent false
9661 Acked Read 3 Done Done Sent false
9662 Acked Read 8 Done Done Sent false
9663 Acked Flush 0 Done Done Sent false
9664 Acked Write 10 Done Done Sent false
9665 Acked Read 5 Done Done Sent false
9666 Acked Write 4 Done Done Sent false
9667 Acked Read 3 Done Done Sent false
9668 Acked Write 4 Done Done Sent false
9669 Acked Read 9 Done Done Sent false
9670 Acked Write 9 Done Done Sent false
9671 Acked Read 1 Done Done Sent false
9672 Acked Read 3 Done Done Sent false
9673 Acked Write 7 Done Done Sent false
9674 Acked Read 3 Done Done Sent false
9675 Acked Write 8 Done Done Sent false
9676 Acked Read 8 Done Done Sent false
9677 Acked Write 7 Done Done Sent false
9678 Acked Read 8 Done Done Sent false
9679 Acked Read 2 Done Done Sent false
9680 Acked Read 2 Done Done Sent false
9681 Acked Read 3 Done Done Sent false
9682 Acked Read 9 Done Done Sent false
9683 Acked Read 8 Done Done Sent false
9684 Acked Read 1 Done Done Sent false
9685 Acked Write 8 Done Done Sent false
9686 Acked Write 2 Done Done Sent false
9687 Acked Write 5 Done Done Sent false
9688 Acked Read 10 Done Done Sent false
9689 Acked Read 10 Done Done Sent false
9690 Acked Write 2 Done Done Sent false
9691 Acked Flush 0 Done Done Sent false
9692 Acked Write 8 Done Done Sent false
9693 Acked Read 6 Done Done Sent false
9694 Acked Write 2 Done Done Sent false
9695 Acked Write 8 Done Done Sent false
9696 Acked Write 7 Done Done Sent false
9697 Acked Read 10 Done Done Sent false
9698 Acked Read 6 Done Done Sent false
9699 Acked Read 4 Done Done Sent false
9700 Acked Read 4 Done Done Sent false
9701 Acked Read 4 Done Done Sent false
9702 Acked Write 9 Done Done Sent false
9703 Acked Read 9 Done Done Sent false
9704 Acked Read 1 Done Done Sent false
9705 Acked Read 2 Done Done Sent false
9706 Acked Write 3 Done Done Sent false
9707 Acked Read 10 Done Done Sent false
9708 Acked Write 8 Done Done Sent false
9709 Acked Write 10 Done Done Sent false
9710 Acked Read 3 Done Done Sent false
9711 Acked Read 6 Done Done Sent false
9712 Acked Write 4 Done Done Sent false
9713 Acked Read 5 Done Done Sent false
9714 Acked Flush 0 Done Done Sent false
9715 Acked Read 1 Done Done Sent false
9716 Acked Read 5 Done Done Sent false
9717 Acked Read 6 Done Done Sent false
9718 Acked Write 8 Done Done Sent false
9719 Acked Read 9 Done Done Sent false
9720 Acked Write 5 Done Done Sent false
9721 Acked Write 9 Done Done Sent false
9722 Acked Read 7 Done Done Sent false
9723 Acked Read 1 Done Done Sent false
9724 Acked Read 9 Done Done Sent false
9725 Acked Read 2 Done Done Sent false
9726 Acked Read 4 Done Done Sent false
9727 Acked Write 9 Done Done Sent false
9728 Acked Read 8 Done Done Sent false
9729 Acked Read 5 Done Done Sent false
9730 Acked Read 2 Done Done Sent false
9731 Acked Read 3 Done Done Sent false
9732 Acked Read 10 Done Done Sent false
9733 Acked Read 1 Done Done Sent false
9734 Acked Write 3 Done Done Sent false
9735 Acked Read 5 Done Done Sent false
9736 Acked Read 2 Done Done Sent false
9737 Acked Write 9 Done Done Sent false
9738 Acked Read 1 Done Done Sent false
9739 Acked Flush 0 Done Done Sent false
9740 Acked Write 2 Done Done Sent false
9741 Acked Write 6 Done Done Sent false
9742 Acked Write 2 Done Done Sent false
9743 Acked Read 3 Done Done Sent false
9744 Acked Read 6 Done Done Sent false
9745 Acked Read 7 Done Done Sent false
9746 Acked Read 4 Done Done Sent false
9747 Acked Write 7 Done Done Sent false
9748 Acked Read 3 Done Done Sent false
9749 Acked Read 4 Done Done Sent false
9750 Acked Flush 0 Done Done Sent false
9751 Acked Read 3 Done Done Sent false
9752 Acked Read 10 Done Done Sent false
9753 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9753 9753 9451
Downstairs last five completed: 9451 9450 9449 9448 9447
Upstairs last five completed: 9753 9752 9751 9750 9749
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.846011101Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50756)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.846146246Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.847169456Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.847733361Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.847767197Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 9451","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.847785074Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 9451","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.848035185Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 9451","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.848063412Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 9451","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:15.848315193Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 9753 9752 9751 9750 9749
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Jan 27 19:43:16.096 INFO request completed, latency_us: 461, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 934d509f-61e7-4071-a169-9a2251016960, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Jan 27 19:43:16.097 INFO request completed, latency_us: 127{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:16.097615413Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
, response_code: 200, uri: /state/cid/0, method: GET, req_id: 95391008-11be-456e-b818-8a1426ed50a6, remote_addr: 127.0.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:16.097679595Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
1:55047, local_addr: 127.0.0.1:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:16.097717411Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:16.097763825Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
Jan 27 19:43:20.099 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/0, method: GET, req_id: ea210c26-c502-4b2c-b553-9e684603ba09, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Read block 2468 len 8192
002/300 Write block 3132 len 24576 data: 8 8 7 7 5 5
003/300 Read block 3367 len 28672
004/300 Read block 2128 len 28672
005/300 Write block 4691 len 28672 data: 6 7 7 7 7 6 5
006/300 Write block 1101 len 4096 data: 7
007/300 Write block 1129 len 20480 data: 9 8 7 7 6
008/300 Write block 2071 len 24576 data: 4 6 6 6 6 8
009/300 Read block 938 len 4096
010/300 Read block 850 len 20480
011/300 Write block 2136 len 12288 data: 6 6 5
012/300 Write block 4544 len 12288 data: 3 4 4
013/300 Write block 4080 len 16384 data: 4 4 4 4
014/300 Write block 4968 len 36864 data: 7 7 8 8 6 6 5 4 3
015/300 Read block 4928 len 4096
016/300 Write block 1390 len 28672 data: 7 8 6 5 7 7 7
017/300 Write block 614 len 20480 data: 6 5 5 4 4
018/300 Read block 297 len 28672
019/300 Write block 2318 len 12288 data: 4 3 4
020/300 Flush
021/300 Read block 2354 len 16384
022/300 Write block 686 len 8192 data: 8 7
023/300 Flush
024/300 Read block 1983 len 4096
025/300 Write block 3262 len 32768 data: 5 5 5 3 3 4 4 4
026/300 Read block 4900 len 28672
027/300 Read block 3322 len 12288
028/300 Read block 3476 len 12288
029/300 Read block 1411 len 16384
030/300 Read block 2756 len 24576
031/300 Read block 4365 len 28672
032/300 Flush
033/300 Read block 1665 len 12288
034/300 Read block 1750 len 8192
035/300 Write block 2800 len 20480 data: 5 6 6 6 6
036/300 Read block 2581 len 4096
037/300 Write block 2198 len 20480 data: 7 7 6 8 7
038/300 Read block 1763 len 36864
039/300 Read block 899 len 28672
040/300 Read block 1674 len 12288
041/300 Read block 802 len 24576
042/300 Read block 950 len 16384
043/300 Write block 3975 len 8192 data: 3 3
044/300 Write block 3285 len 32768 data: 5 5 6 5 2 3 3 6
045/300 Read block 2998 len 20480
046/300 Write block 2801 len 4096 data: 7
047/300 Write block 3059 len 8192 data: 6 8
048/300 Read block 4769 len 24576
049/300 Flush
050/300 Read block 2948 len 20480
051/300 Write block 3572 len 12288 data: 5 5 5
052/300 Write block 1076 len 16384 data: 6 6 6 9
053/300 Write block 3125 len 32768 data: 8 7 7 7 7 6 8 9
054/300 Write block 2973 len 28672 data: 8 8 6 5 5 6 7
055/300 Read block 4802 len 12288
056/300 Read block 2873 len 40960
057/300 Flush
058/300 Write block 2001 len 32768 data: 6 7 6 5 5 5 4 7
059/300 Flush
060/300 Read block 4375 len 8192
061/300 Read block 862 len 8192
062/300 Write block 855 len 36864 data: 5 5 5 7 8 9 11 10 8
063/300 Read block 2608 len 20480
064/300 Write block 2372 len 36864 data: 9 9 7 7 8 9 12 7 7
065/300 Read block 4675 len 40960
066/300 Flush
067/300 Read block 2142 len 4096
068/300 Write block 751 len 4096 data: 4
069/300 Read block 1602 len 12288
070/300 Flush
071/300 Write block 3023 len 4096 data: 7
072/300 Write block 2772 len 20480 data: 8 9 8 6 6
073/300 Write block 779 len 16384 data: 5 4 5 5
074/300 Flush
075/300 Read block 3179 len 36864
076/300 Write block 3780 len 4096 data: 6
077/300 Write block 3072 len 8192 data: 4 3
078/300 Read block 407 len 36864
079/300 Read block 4211 len 24576
080/300 Read block 2208 len 16384
081/300 Read block 1674 len 16384
082/300 Write block 14 len 8192 data: 4 4
083/300 Write block 1666 len 16384 data: 6 4 3 5
084/300 Read block 2936 len 12288
085/300 Read block 2005 len 36864
086/300 Read block 2977 len 20480
087/300 Write block 2920 len 40960 data: 7 5 4 3 4 6 7 7 8 10
088/300 Write block 422 len 24576 data: 7 6 6 7 7 5
089/300 Write block 314 len 28672 data: 6 6 6 7 7 7 9
090/300 Flush
091/300 Read block 3419 len 4096
092/300 Write block 1738 len 12288 data: 6 8 9
093/300 Read block 2927 len 32768
094/300 Read block 3485 len 32768
095/300 Flush
096/300 Write block 1746 len 20480 data: 5 4 4 4 3
097/300 Write block 4645 len 32768 data: 6 7 8 7 7 5 6 8
098/300 Read block 2278 len 36864
099/300 Write block 3617 len 32768 data: 8 10 9 7 9 8 9 8
100/300 Write block 3002 len 24576 data: 9 9 9 8 8 8
101/300 Write block 3144 len 24576 data: 8 8 6 5 6 6
102/300 Read block 580 len 24576
103/300 Flush
104/300 Write block 2720 len 12288 data: 6 5 7
105/300 Read block 2247 len 12288
106/300 Write block 4252 len 12288 data: 7 6 6
107/300 Write block 3547 len 24576 data: 3 3 4 6 6 5
108/300 Read block 2664 len 40960
109/300 Write block 2473 len 8192 data: 6 6
110/300 Read block 1293 len 40960
111/300 Write block 2986 len 36864 data: 7 7 6 4 5 5 5 4 4
112/300 Read block 4926 len 28672
113/300 Write block 1925 len 36864 data: 8 7 5 5 5 5 3 3 4
114/300 Write block 1052 len 36864 data: 3 3 3 2 3 5 7 8 8
115/300 Read block 1395 len 40960
116/300 Write block 4091 len 16384 data: 4 5 5 5
117/300 Read block 3902 len 24576
118/300 Read block 2154 len 40960
119/300 Read block 1676 len 28672
120/300 Read block 4404 len 36864
121/300 Read block 2211 len 40960
122/300 Read block 3661 len 24576
123/300 Write block 2939 len 24576 data: 5 5 6 7 5 6
124/300 Read block 4392 len 8192
125/300 Read block 4979 len 24576
126/300 Write block 3177 len 20480 data: 10 10 10 8 8
127/300 Write block 4207 len 28672 data: 4 4 4 4 4 3 3
128/300 Write block 4278 len 12288 data: 8 8 8
129/300 Read block 36 len 28672
130/300 Read block 793 len 40960
131/300 Read block 4995 len 4096
132/300 Flush
133/300 Write block 908 len 4096 data: 5
134/300 Read block 2419 len 20480
135/300 Read block 2525 len 28672
136/300 Read block 3810 len 24576
137/300 Flush
138/300 Read block 3401 len 20480
139/300 Read block 4052 len 8192
140/300 Read block 1901 len 16384
141/300 Write block 1355 len 40960 data: 9 8 9 12 12 9 9 7 7 8
142/300 Write block 2682 len 12288 data: 9 11 11
143/300 Flush
144/300 Read block 505 len 32768
145/300 Flush
146/300 Write block 3991 len 8192 data: 3 4
147/300 Read block 3005 len 36864
148/300 Write block 4710 len 40960 data: 8 7 7 8 8 6 6 6 8 6
149/300 Write block 1154 len 36864 data: 4 4 4 4 5 4 4 3 3
150/300 Flush
151/300 Read block 3465 len 28672
152/300 Read block 4189 len 24576
153/300 Read block 586 len 24576
154/300 Write block 2986 len 36864 data: 8 8 7 5 6 6 6 5 5
155/300 Write block 871 len 8192 data: 7 6
156/300 Write block 1493 len 32768 data: 6 6 5 4 5 4 4 4
157/300 Read block 715 len 12288
158/300 Write block 1333 len 40960 data: 10 8 7 7 7 8 7 6 4 6
159/300 Read block 4105 len 12288
160/300 Write block 3231 len 40960 data: 7 5 5 6 7 5 5 4 5 5
161/300 Flush
162/300 Read block 4775 len 40960
163/300 Write block 3269 len 4096 data: 5
164/300 Read block 1892 len 8192
165/300 Write block 3795 len 28672 data: 2 2 2 3 3 3 2
166/300 Write block 1483 len 4096 data: 8
167/300 Flush
168/300 Read block 3456 len 20480
169/300 Write block 1438 len 24576 data: 6 6 6 7 7 6
170/300 Read block 4440 len 24576
171/300 Read block 3823 len 36864
172/300 Read block 466 len 20480
173/300 Read block 1812 len 20480
174/300 Write block 485 len 20480 data: 4 4 4 5 6
175/300 Write block 3038 len 20480 data: 6 6 5 5 4
176/300 Write block 1725 len 12288 data: 6 8 8
177/300 Flush
178/300 Write block 3630 len 36864 data: 5 5 4 5 5 6 6 4 5
179/300 Read block 2911 len 36864
180/300 Read block 2518 len 24576
181/300 Write block 4073 len 8192 data: 7 6
182/300 Write block 774 len 12288 data: 3 3 5
183/300 Write block 2515 len 36864 data: 8 7 6 5 4 5 4 4 4
184/300 Read block 2246 len 32768
185/300 Read block 3311 len 32768
186/300 Write block 898 len 28672 data: 3 3 4 5 5 6 7
187/300 Write block 1371 len 32768 data: 9 7 8 9 10 10 10 10
188/300 Read block 4865 len 20480
189/300 Read block 341 len 4096
190/300 Write block 733 len 32768 data: 5 3 4 4 4 7 7 7
191/300 Read block 356 len 8192
192/300 Read block 3506 len 40960
193/300 Read block 2583 len 28672
194/300 Read block 3950 len 12288
195/300 Write block 4642 len 8192 data: 5 7
196/300 Read block 1623 len 36864
197/300 Read block 2876 len 4096
198/300 Write block 3035 len 36864 data: 3 4 4 7 7 6 6 5 4
199/300 Write block 4537 len 32768 data: 6 5 3 4 4 3 3 4
200/300 Read block 1125 len 36864
201/300 Read block 3045 len 32768
202/300 Write block 1725 len 36864 data: 7 9 9 7 8 7 6 5 5
203/300 Write block 3656 len 20480 data: 4 3 5 5 6
204/300 Read block 4379 len 36864
205/300 Read block 2847 len 36864
206/300 Write block 3432 len 8192 data: 9 10
207/300 Flush
208/300 Read block 3719 len 28672
209/300 Read block 2195 len 28672
210/300 Read block 4153 len 4096
211/300 Write block 4224 len 16384 data: 5 5 4 3
212/300 Write block 2319 len 12288 data: 4 5 4
213/300 Read block 4679 len 40960
214/300 Write block 1662 len 20480 data: 7 7 6 5 7
215/300 Read block 4657 len 28672
216/300 Write block 4416 len 16384 data: 5 5 4 3
217/300 Read block 3841 len 12288
218/300 Flush
219/300 Read block 1526 len 32768
220/300 Write block 4906 len 28672 data: 4 6 7 7 7 7 7
221/300 Write block 2813 len 32768 data: 7 6 5 5 3 4 4 5
222/300 Read block 4146 len 4096
223/300 Write block 4744 len 32768 data: 5 5 6 4 4 5 6 5
224/300 Read block 499 len 8192
225/300 Read block 3059 len 40960
226/300 Read block 3250 len 24576
227/300 Write block 432 len 40960 data: 4 2 3 4 4 5 5 7 8 5
228/300 Read block 3836 len 24576
229/300 Write block 4813 len 40960 data: 4 4 4 3 2 2 3 4 5 6
230/300 Read block 1553 len 8192
231/300 Read block 571 len 40960
232/300 Read block 1537 len 40960
233/300 Read block 4919 len 24576
234/300 Read block 4309 len 36864
235/300 Read block 3277 len 4096
236/300 Read block 3710 len 36864
237/300 Write block 4663 len 4096 data: 3
238/300 Read block 211 len 32768
239/300 Read block 1548 len 16384
240/300 Write block 69 len 4096 data: 4
241/300 Write block 2808 len 40960 data: 5 5 5 7 7 8 7 6 6 4
242/300 Read block 4227 len 40960
243/300 Read block 391 len 40960
244/300 Write block 4094 len 32768 data: 6 5 4 3 3 3 5 4
245/300 Read block 2924 len 24576
246/300 Flush
247/300 Read block 4186 len 24576
248/300 Read block 747 len 20480
249/300 Read block 3487 len 36864
250/300 Flush
251/300 Write block 1587 len 16384 data: 2 2 3 3
252/300 Write block 4425 len 16384 data: 3 3 5 5
253/300 Flush
254/300 Write block 4042 len 20480 data: 6 7 6 6 6
255/300 Read block 2278 len 16384
256/300 Write block 861 len 28672 data: 12 11 9 9 8 10 10
257/300 Read block 3262 len 12288
258/300 Write block 949 len 40960 data: 6 6 5 6 6 6 6 8 8 7
259/300 Write block 4077 len 32768 data: 4 4 3 5 5 5 5 2
260/300 Write block 574 len 4096 data: 6
261/300 Write block 2413 len 16384 data: 8 9 8 7
262/300 Read block 4520 len 8192
263/300 Read block 4412 len 16384
264/300 Flush
265/300 Read block 2931 len 4096
266/300 Read block 4946 len 12288
267/300 Write block 3034 len 24576 data: 4 4 5 5 8 8
268/300 Read block 3752 len 4096
269/300 Read block 4070 len 32768
270/300 Read block 548 len 8192
271/300 Read block 4360 len 8192
272/300 Read block 901 len 24576
273/300 Write block 3345 len 36864 data: 2 4 3 3 3 4 3 3 2
274/300 Flush
275/300 Read block 99 len 40960
276/300 Flush
277/300 Read block 1712 len 8192
278/300 Read block 2492 len 28672
279/300 Read block 350 len 24576
280/300 Write block 875 len 24576 data: 6 6 5 7 7 6
281/300 Read block 655 len 16384
282/300 Read block 1084 len 40960
283/300 Read block 1791 len 16384
284/300 Write block 3024 len 16384 data: 8 8 8 7
285/300 Read block 4760 len 12288
286/300 Flush
287/300 Write block 770 len 16384 data: 6 6 7 7
288/300 Read block 2847 len 4096
289/300 Read block 4395 len 24576
290/300 Write block 2133 len 12288 data: 6 6 5
291/300 Read block 1993 len 20480
292/300 Write block 1542 len 12288 data: 7 9 9
293/300 Write block 4809 len 4096 data: 7
294/300 Write block 3889 len 24576 data: 6 4 3 3 5 7
295/300 Read block 4549 len 8192
296/300 Write block 2097 len 32768 data: 5 5 5 4 4 5 5 6
297/300 Read block 1531 len 32768
298/300 Read block 3692 len 8192
299/300 Read block 4636 len 4096
300/300 Read block 101 len 16384
301/300 Read block 4601 len 8192
Main task has work to do, go find it
got dsc Start(0)
start 0
Jan 27 19:43:20.336 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/0, method: POST, req_id: 12ac5137-617f-41e3-a8b1-dbd33bdfe205, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9754 Acked Read 2 Sent Done Done false
9755 Acked Write 6 Sent Done Done false
9756 Acked Read 7 Sent Done Done false
9757 Acked Read 7 Sent Done Done false
9758 Acked Write 7 Sent Done Done false
9759 Acked Write 1 Sent Done Done false
9760 Acked Write 5 Sent Done Done false
9761 Acked Write 6 Sent Done Done false
9762 Acked Read 1 Sent Done Done false
9763 Acked Read 5 Sent Done Done false
9764 Acked Write 3 Sent Done Done false
9765 Acked Write 3 Sent Done Done false
9766 Acked Write 4 Sent Done Done false
9767 Acked Write 9 Sent Done Done false
9768 Acked Read 1 Sent Done Done false
9769 Acked Write 7 Sent Done Done false
9770 Acked Write 5 Sent Done Done false
9771 Acked Read 7 Sent Done Done false
9772 Acked Write 3 Sent Done Done false
9773 Acked Flush 0 Sent Done Done false
9774 Acked Read 4 Sent Done Done false
9775 Acked Write 2 Sent Done Done false
9776 Acked Flush 0 Sent Done Done false
9777 Acked Read 1 Sent Done Done false
9778 Acked Write 8 Sent Done Done false
9779 Acked Read 7 Sent Done Done false
9780 Acked Read 3 Sent Done Done false
9781 Acked Read 3 Sent Done Done false
9782 Acked Read 4 Sent Done Done false
9783 Acked Read 6 Sent Done Done false
9784 Acked Read 7 Sent Done Done false
9785 Acked Flush 0 Sent Done Done false
9786 Acked Read 3 Sent Done Done false
9787 Acked Read 2 Sent Done Done false
9788 Acked Write 5 Sent Done Done false
9789 Acked Read 1 Sent Done Done false
9790 Acked Write 5 Sent Done Done false
9791 Acked Read 9 Sent Done Done false
9792 Acked Read 7 Sent Done Done false
9793 Acked Read 3 Sent Done Done false
9794 Acked Read 6 Sent Done Done false
9795 Acked Read 4 Sent Done Done false
9796 Acked Write 2 Sent Done Done false
9797 Acked Write 8 Sent Done Done false
9798 Acked Read 5 Sent Done Done false
9799 Acked Write 1 Sent Done Done false
9800 Acked Write 2 Sent Done Done false
9801 Acked Read 6 Sent Done Done false
9802 Acked Flush 0 Sent Done Done false
9803 Acked Read 5 Sent Done Done false
9804 Acked Write 3 Sent Done Done false
9805 Acked Write 4 Sent Done Done false
9806 Acked Write 8 Sent Done Done false
9807 Acked Write 7 Sent Done Done false
9808 Acked Read 3 Sent Done Done false
9809 Acked Read 10 Sent Done Done false
9810 Acked Flush 0 Sent Done Done false
9811 Acked Write 8 Sent Done Done false
9812 Acked Flush 0 Sent Done Done false
9813 Acked Read 2 Sent Done Done false
9814 Acked Read 2 Sent Done Done false
9815 Acked Write 9 Sent Done Done false
9816 Acked Read 5 Sent Done Done false
9817 Acked Write 9 Sent Done Done false
9818 Acked Read 10 Sent Done Done false
9819 Acked Flush 0 Sent Done Done false
9820 Acked Read 1 Sent Done Done false
9821 Acked Write 1 Sent Done Done false
9822 Acked Read 3 Sent Done Done false
9823 Acked Flush 0 Sent Done Done false
9824 Acked Write 1 Sent Done Done false
9825 Acked Write 5 Sent Done Done false
9826 Acked Write 4 Sent Done Done false
9827 Acked Flush 0 Sent Done Done false
9828 Acked Read 9 Sent Done Done false
9829 Acked Write 1 Sent Done Done false
9830 Acked Write 2 Sent Done Done false
9831 Acked Read 9 Sent Done Done false
9832 Acked Read 6 Sent Done Done false
9833 Acked Read 4 Sent Done Done false
9834 Acked Read 4 Sent Done Done false
9835 Acked Write 2 Sent Done Done false
9836 Acked Write 4 Sent Done Done false
9837 Acked Read 3 Sent Done Done false
9838 Acked Read 9 Sent Done Done false
9839 Acked Read 5 Sent Done Done false
9840 Acked Write 10 Sent Done Done false
9841 Acked Write 6 Sent Done Done false
9842 Acked Write 7 Sent Done Done false
9843 Acked Flush 0 Sent Done Done false
9844 Acked Read 1 Sent Done Done false
9845 Acked Write 3 Sent Done Done false
9846 Acked Read 8 Sent Done Done false
9847 Acked Read 8 Sent Done Done false
9848 Acked Flush 0 Sent Done Done false
9849 Acked Write 5 Sent Done Done false
9850 Acked Write 8 Sent Done Done false
9851 Acked Read 9 Sent Done Done false
9852 Acked Write 8 Sent Done Done false
9853 Acked Write 6 Sent Done Done false
9854 Acked Write 6 Sent Done Done false
9855 Acked Read 6 Sent Done Done false
9856 Acked Flush 0 Sent Done Done false
9857 Acked Write 3 Sent Done Done false
9858 Acked Read 3 Sent Done Done false
9859 Acked Write 3 Sent Done Done false
9860 Acked Write 6 Sent Done Done false
9861 Acked Read 10 Sent Done Done false
9862 Acked Write 2 Sent Done Done false
9863 Acked Read 10 Sent Done Done false
9864 Acked Write 9 Sent Done Done false
9865 Acked Read 7 Sent Done Done false
9866 Acked Write 9 Sent Done Done false
9867 Acked Write 9 Sent Done Done false
9868 Acked Read 10 Sent Done Done false
9869 Acked Write 4 Sent Done Done false
9870 Acked Read 6 Sent Done Done false
9871 Acked Read 10 Sent Done Done false
9872 Acked Read 7 Sent Done Done false
9873 Acked Read 9 Sent Done Done false
9874 Acked Read 10 Sent Done Done false
9875 Acked Read 6 Sent Done Done false
9876 Acked Write 6 Sent Done Done false
9877 Acked Read 2 Sent Done Done false
9878 Acked Read 6 Sent Done Done false
9879 Acked Write 5 Sent Done Done false
9880 Acked Write 7 Sent Done Done false
9881 Acked Write 3 Sent Done Done false
9882 Acked Read 7 Sent Done Done false
9883 Acked Read 10 Sent Done Done false
9884 Acked Read 1 Sent Done Done false
9885 Acked Flush 0 Sent Done Done false
9886 Acked Write 1 Sent Done Done false
9887 Acked Read 5 Sent Done Done false
9888 Acked Read 7 Sent Done Done false
9889 Acked Read 6 Sent Done Done false
9890 Acked Flush 0 Sent Done Done false
9891 Acked Read 5 Sent Done Done false
9892 Acked Read 2 Sent Done Done false
9893 Acked Read 4 Sent Done Done false
9894 Acked Write 10 Sent Done Done false
9895 Acked Write 3 Sent Done Done false
9896 Acked Flush 0 Sent Done Done false
9897 Acked Read 8 Sent Done Done false
9898 Acked Flush 0 Sent Done Done false
9899 Acked Write 2 Sent Done Done false
9900 Acked Read 9 Sent Done Done false
9901 Acked Write 10 Sent Done Done false
9902 Acked Write 9 Sent Done Done false
9903 Acked Flush 0 Sent Done Done false
9904 Acked Read 7 Sent Done Done false
9905 Acked Read 6 Sent Done Done false
9906 Acked Read 6 Sent Done Done false
9907 Acked Write 9 Sent Done Done false
9908 Acked Write 2 Sent Done Done false
9909 Acked Write 8 Sent Done Done false
9910 Acked Read 3 Sent Done Done false
9911 Acked Write 10 Sent Done Done false
9912 Acked Read 3 Sent Done Done false
9913 Acked Write 10 Sent Done Done false
9914 Acked Flush 0 Sent Done Done false
9915 Acked Read 10 Sent Done Done false
9916 Acked Write 1 Sent Done Done false
9917 Acked Read 2 Sent Done Done false
9918 Acked Write 7 Sent Done Done false
9919 Acked Write 1 Sent Done Done false
9920 Acked Flush 0 Sent Done Done false
9921 Acked Read 5 Sent Done Done false
9922 Acked Write 6 Sent Done Done false
9923 Acked Read 6 Sent Done Done false
9924 Acked Read 9 Sent Done Done false
9925 Acked Read 5 Sent Done Done false
9926 Acked Read 5 Sent Done Done false
9927 Acked Write 5 Sent Done Done false
9928 Acked Write 5 Sent Done Done false
9929 Acked Write 3 Sent Done Done false
9930 Acked Flush 0 Sent Done Done false
9931 Acked Write 9 Sent Done Done false
9932 Acked Read 9 Sent Done Done false
9933 Acked Read 6 Sent Done Done false
9934 Acked Write 2 Sent Done Done false
9935 Acked Write 3 Sent Done Done false
9936 Acked Write 9 Sent Done Done false
9937 Acked Read 8 Sent Done Done false
9938 Acked Read 8 Sent Done Done false
9939 Acked Write 7 Sent Done Done false
9940 Acked Write 8 Sent Done Done false
9941 Acked Read 5 Sent Done Done false
9942 Acked Read 1 Sent Done Done false
9943 Acked Write 8 Sent Done Done false
9944 Acked Read 2 Sent Done Done false
9945 Acked Read 10 Sent Done Done false
9946 Acked Read 7 Sent Done Done false
9947 Acked Read 3 Sent Done Done false
9948 Acked Write 2 Sent Done Done false
9949 Acked Read 9 Sent Done Done false
9950 Acked Read 1 Sent Done Done false
9951 Acked Write 9 Sent Done Done false
9952 Acked Write 8 Sent Done Done false
9953 Acked Read 9 Sent Done Done false
9954 Acked Read 8 Sent Done Done false
9955 Acked Write 9 Sent Done Done false
9956 Acked Write 5 Sent Done Done false
9957 Acked Read 9 Sent Done Done false
9958 Acked Read 9 Sent Done Done false
9959 Acked Write 2 Sent Done Done false
9960 Acked Flush 0 Sent Done Done false
9961 Acked Read 7 Sent Done Done false
9962 Acked Read 7 Sent Done Done false
9963 Acked Read 1 Sent Done Done false
9964 Acked Write 4 Sent Done Done false
9965 Acked Write 3 Sent Done Done false
9966 Acked Read 10 Sent Done Done false
9967 Acked Write 5 Sent Done Done false
9968 Acked Read 7 Sent Done Done false
9969 Acked Write 4 Sent Done Done false
9970 Acked Read 3 Sent Done Done false
9971 Acked Flush 0 Sent Done Done false
9972 Acked Read 8 Sent Done Done false
9973 Acked Write 7 Sent Done Done false
9974 Acked Write 8 Sent Done Done false
9975 Acked Read 1 Sent Done Done false
9976 Acked Write 8 Sent Done Done false
9977 Acked Read 2 Sent Done Done false
9978 Acked Read 10 Sent Done Done false
9979 Acked Read 6 Sent Done Done false
9980 Acked Write 10 Sent Done Done false
9981 Acked Read 6 Sent Done Done false
9982 Acked Write 10 Sent Done Done false
9983 Acked Read 2 Sent Done Done false
9984 Acked Read 10 Sent Done Done false
9985 Acked Read 10 Sent Done Done false
9986 Acked Read 6 Sent Done Done false
9987 Acked Read 9 Sent Done Done false
9988 Acked Read 1 Sent Done Done false
9989 Acked Read 9 Sent Done Done false
9990 Acked Write 1 Sent Done Done false
9991 Acked Read 8 Sent Done Done false
9992 Acked Read 4 Sent Done Done false
9993 Acked Write 1 Sent Done Done false
9994 Acked Write 10 Sent Done Done false
9995 Acked Read 10 Sent Done Done false
9996 Acked Read 10 Sent Done Done false
9997 Acked Write 8 Sent Done Done false
9998 Acked Read 6 Sent Done Done false
9999 Acked Flush 0 Sent Done Done false
10000 Acked Read 6 Sent Done Done false
10001 Acked Read 5 Sent Done Done false
10002 Acked Read 9 Sent Done Done false
10003 Acked Flush 0 Sent Done Done false
10004 Acked Write 4 Sent Done Done false
10005 Acked Write 4 Sent Done Done false
10006 Acked Flush 0 Sent Done Done false
10007 Acked Write 5 Sent Done Done false
10008 Acked Read 4 Sent Done Done false
10009 Acked Write 7 Sent Done Done false
10010 Acked Read 3 Sent Done Done false
10011 Acked Write 10 Sent Done Done false
10012 Acked Write 8 Sent Done Done false
10013 Acked Write 1 Sent Done Done false
10014 Acked Write 4 Sent Done Done false
10015 Acked Read 2 Sent Done Done false
10016 Acked Read 4 Sent Done Done false
10017 Acked Flush 0 Sent Done Done false
10018 Acked Read 1 Sent Done Done false
10019 Acked Read 3 Sent Done Done false
10020 Acked Write 6 Sent Done Done false
10021 Acked Read 1 Sent Done Done false
10022 Acked Read 8 Sent Done Done false
10023 Acked Read 2 Sent Done Done false
10024 Acked Read 2 Sent Done Done false
10025 Acked Read 6 Sent Done Done false
10026 Acked Write 9 Sent Done Done false
10027 Acked Flush 0 Sent Done Done false
10028 Acked Read 10 Sent Done Done false
10029 Acked Flush 0 Sent Done Done false
10030 Acked Read 2 Sent Done Done false
10031 Acked Read 7 Sent Done Done false
10032 Acked Read 6 Sent Done Done false
10033 Acked Write 6 Sent Done Done false
10034 Acked Read 4 Sent Done Done false
10035 Acked Read 10 Sent Done Done false
10036 Acked Read 4 Sent Done Done false
10037 Acked Write 4 Sent Done Done false
10038 Acked Read 3 Sent Done Done false
10039 Acked Flush 0 Sent Done Done false
10040 Acked Write 4 Sent Done Done false
10041 Acked Read 1 Sent Done Done false
10042 Acked Read 6 Sent Done Done false
10043 Acked Write 3 Sent Done Done false
10044 Acked Read 5 Sent Done Done false
10045 Acked Write 3 Sent Done Done false
10046 Acked Write 1 Sent Done Done false
10047 Acked Write 6 Sent Done Done false
10048 Acked Read 2 Sent Done Done false
10049 Acked Write 8 Sent Done Done false
10050 Acked Read 8 Sent Done Done false
10051 Acked Read 2 Sent Done Done false
10052 Acked Read 1 Sent Done Done false
10053 Acked Read 4 Sent Done Done false
10054 Acked Read 2 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 301 0 0 301
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9753 10039 10039
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 10054 10053 10052 10051 10050
CLIENT: Up:0 ds:301 act:2
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9754 Acked Read 2 Sent Done Done false
9755 Acked Write 6 Sent Done Done false
9756 Acked Read 7 Sent Done Done false
9757 Acked Read 7 Sent Done Done false
9758 Acked Write 7 Sent Done Done false
9759 Acked Write 1 Sent Done Done false
9760 Acked Write 5 Sent Done Done false
9761 Acked Write 6 Sent Done Done false
9762 Acked Read 1 Sent Done Done false
9763 Acked Read 5 Sent Done Done false
9764 Acked Write 3 Sent Done Done false
9765 Acked Write 3 Sent Done Done false
9766 Acked Write 4 Sent Done Done false
9767 Acked Write 9 Sent Done Done false
9768 Acked Read 1 Sent Done Done false
9769 Acked Write 7 Sent Done Done false
9770 Acked Write 5 Sent Done Done false
9771 Acked Read 7 Sent Done Done false
9772 Acked Write 3 Sent Done Done false
9773 Acked Flush 0 Sent Done Done false
9774 Acked Read 4 Sent Done Done false
9775 Acked Write 2 Sent Done Done false
9776 Acked Flush 0 Sent Done Done false
9777 Acked Read 1 Sent Done Done false
9778 Acked Write 8 Sent Done Done false
9779 Acked Read 7 Sent Done Done false
9780 Acked Read 3 Sent Done Done false
9781 Acked Read 3 Sent Done Done false
9782 Acked Read 4 Sent Done Done false
9783 Acked Read 6 Sent Done Done false
9784 Acked Read 7 Sent Done Done false
9785 Acked Flush 0 Sent Done Done false
9786 Acked Read 3 Sent Done Done false
9787 Acked Read 2 Sent Done Done false
9788 Acked Write 5 Sent Done Done false
9789 Acked Read 1 Sent Done Done false
9790 Acked Write 5 Sent Done Done false
9791 Acked Read 9 Sent Done Done false
9792 Acked Read 7 Sent Done Done false
9793 Acked Read 3 Sent Done Done false
9794 Acked Read 6 Sent Done Done false
9795 Acked Read 4 Sent Done Done false
9796 Acked Write 2 Sent Done Done false
9797 Acked Write 8 Sent Done Done false
9798 Acked Read 5 Sent Done Done false
9799 Acked Write 1 Sent Done Done false
9800 Acked Write 2 Sent Done Done false
9801 Acked Read 6 Sent Done Done false
9802 Acked Flush 0 Sent Done Done false
9803 Acked Read 5 Sent Done Done false
9804 Acked Write 3 Sent Done Done false
9805 Acked Write 4 Sent Done Done false
9806 Acked Write 8 Sent Done Done false
9807 Acked Write 7 Sent Done Done false
9808 Acked Read 3 Sent Done Done false
9809 Acked Read 10 Sent Done Done false
9810 Acked Flush 0 Sent Done Done false
9811 Acked Write 8 Sent Done Done false
9812 Acked Flush 0 Sent Done Done false
9813 Acked Read 2 Sent Done Done false
9814 Acked Read 2 Sent Done Done false
9815 Acked Write 9 Sent Done Done false
9816 Acked Read 5 Sent Done Done false
9817 Acked Write 9 Sent Done Done false
9818 Acked Read 10 Sent Done Done false
9819 Acked Flush 0 Sent Done Done false
9820 Acked Read 1 Sent Done Done false
9821 Acked Write 1 Sent Done Done false
9822 Acked Read 3 Sent Done Done false
9823 Acked Flush 0 Sent Done Done false
9824 Acked Write 1 Sent Done Done false
9825 Acked Write 5 Sent Done Done false
9826 Acked Write 4 Sent Done Done false
9827 Acked Flush 0 Sent Done Done false
9828 Acked Read 9 Sent Done Done false
9829 Acked Write 1 Sent Done Done false
9830 Acked Write 2 Sent Done Done false
9831 Acked Read 9 Sent Done Done false
9832 Acked Read 6 Sent Done Done false
9833 Acked Read 4 Sent Done Done false
9834 Acked Read 4 Sent Done Done false
9835 Acked Write 2 Sent Done Done false
9836 Acked Write 4 Sent Done Done false
9837 Acked Read 3 Sent Done Done false
9838 Acked Read 9 Sent Done Done false
9839 Acked Read 5 Sent Done Done false
9840 Acked Write 10 Sent Done Done false
9841 Acked Write 6 Sent Done Done false
9842 Acked Write 7 Sent Done Done false
9843 Acked Flush 0 Sent Done Done false
9844 Acked Read 1 Sent Done Done false
9845 Acked Write 3 Sent Done Done false
9846 Acked Read 8 Sent Done Done false
9847 Acked Read 8 Sent Done Done false
9848 Acked Flush 0 Sent Done Done false
9849 Acked Write 5 Sent Done Done false
9850 Acked Write 8 Sent Done Done false
9851 Acked Read 9 Sent Done Done false
9852 Acked Write 8 Sent Done Done false
9853 Acked Write 6 Sent Done Done false
9854 Acked Write 6 Sent Done Done false
9855 Acked Read 6 Sent Done Done false
9856 Acked Flush 0 Sent Done Done false
9857 Acked Write 3 Sent Done Done false
9858 Acked Read 3 Sent Done Done false
9859 Acked Write 3 Sent Done Done false
9860 Acked Write 6 Sent Done Done false
9861 Acked Read 10 Sent Done Done false
9862 Acked Write 2 Sent Done Done false
9863 Acked Read 10 Sent Done Done false
9864 Acked Write 9 Sent Done Done false
9865 Acked Read 7 Sent Done Done false
9866 Acked Write 9 Sent Done Done false
9867 Acked Write 9 Sent Done Done false
9868 Acked Read 10 Sent Done Done false
9869 Acked Write 4 Sent Done Done false
9870 Acked Read 6 Sent Done Done false
9871 Acked Read 10 Sent Done Done false
9872 Acked Read 7 Sent Done Done false
9873 Acked Read 9 Sent Done Done false
9874 Acked Read 10 Sent Done Done false
9875 Acked Read 6 Sent Done Done false
9876 Acked Write 6 Sent Done Done false
9877 Acked Read 2 Sent Done Done false
9878 Acked Read 6 Sent Done Done false
9879 Acked Write 5 Sent Done Done false
9880 Acked Write 7 Sent Done Done false
9881 Acked Write 3 Sent Done Done false
9882 Acked Read 7 Sent Done Done false
9883 Acked Read 10 Sent Done Done false
9884 Acked Read 1 Sent Done Done false
9885 Acked Flush 0 Sent Done Done false
9886 Acked Write 1 Sent Done Done false
9887 Acked Read 5 Sent Done Done false
9888 Acked Read 7 Sent Done Done false
9889 Acked Read 6 Sent Done Done false
9890 Acked Flush 0 Sent Done Done false
9891 Acked Read 5 Sent Done Done false
9892 Acked Read 2 Sent Done Done false
9893 Acked Read 4 Sent Done Done false
9894 Acked Write 10 Sent Done Done false
9895 Acked Write 3 Sent Done Done false
9896 Acked Flush 0 Sent Done Done false
9897 Acked Read 8 Sent Done Done false
9898 Acked Flush 0 Sent Done Done false
9899 Acked Write 2 Sent Done Done false
9900 Acked Read 9 Sent Done Done false
9901 Acked Write 10 Sent Done Done false
9902 Acked Write 9 Sent Done Done false
9903 Acked Flush 0 Sent Done Done false
9904 Acked Read 7 Sent Done Done false
9905 Acked Read 6 Sent Done Done false
9906 Acked Read 6 Sent Done Done false
9907 Acked Write 9 Sent Done Done false
9908 Acked Write 2 Sent Done Done false
9909 Acked Write 8 Sent Done Done false
9910 Acked Read 3 Sent Done Done false
9911 Acked Write 10 Sent Done Done false
9912 Acked Read 3 Sent Done Done false
9913 Acked Write 10 Sent Done Done false
9914 Acked Flush 0 Sent Done Done false
9915 Acked Read 10 Sent Done Done false
9916 Acked Write 1 Sent Done Done false
9917 Acked Read 2 Sent Done Done false
9918 Acked Write 7 Sent Done Done false
9919 Acked Write 1 Sent Done Done false
9920 Acked Flush 0 Sent Done Done false
9921 Acked Read 5 Sent Done Done false
9922 Acked Write 6 Sent Done Done false
9923 Acked Read 6 Sent Done Done false
9924 Acked Read 9 Sent Done Done false
9925 Acked Read 5 Sent Done Done false
9926 Acked Read 5 Sent Done Done false
9927 Acked Write 5 Sent Done Done false
9928 Acked Write 5 Sent Done Done false
9929 Acked Write 3 Sent Done Done false
9930 Acked Flush 0 Sent Done Done false
9931 Acked Write 9 Sent Done Done false
9932 Acked Read 9 Sent Done Done false
9933 Acked Read 6 Sent Done Done false
9934 Acked Write 2 Sent Done Done false
9935 Acked Write 3 Sent Done Done false
9936 Acked Write 9 Sent Done Done false
9937 Acked Read 8 Sent Done Done false
9938 Acked Read 8 Sent Done Done false
9939 Acked Write 7 Sent Done Done false
9940 Acked Write 8 Sent Done Done false
9941 Acked Read 5 Sent Done Done false
9942 Acked Read 1 Sent Done Done false
9943 Acked Write 8 Sent Done Done false
9944 Acked Read 2 Sent Done Done false
9945 Acked Read 10 Sent Done Done false
9946 Acked Read 7 Sent Done Done false
9947 Acked Read 3 Sent Done Done false
9948 Acked Write 2 Sent Done Done false
9949 Acked Read 9 Sent Done Done false
9950 Acked Read 1 Sent Done Done false
9951 Acked Write 9 Sent Done Done false
9952 Acked Write 8 Sent Done Done false
9953 Acked Read 9 Sent Done Done false
9954 Acked Read 8 Sent Done Done false
9955 Acked Write 9 Sent Done Done false
9956 Acked Write 5 Sent Done Done false
9957 Acked Read 9 Sent Done Done false
9958 Acked Read 9 Sent Done Done false
9959 Acked Write 2 Sent Done Done false
9960 Acked Flush 0 Sent Done Done false
9961 Acked Read 7 Sent Done Done false
9962 Acked Read 7 Sent Done Done false
9963 Acked Read 1 Sent Done Done false
9964 Acked Write 4 Sent Done Done false
9965 Acked Write 3 Sent Done Done false
9966 Acked Read 10 Sent Done Done false
9967 Acked Write 5 Sent Done Done false
9968 Acked Read 7 Sent Done Done false
9969 Acked Write 4 Sent Done Done false
9970 Acked Read 3 Sent Done Done false
9971 Acked Flush 0 Sent Done Done false
9972 Acked Read 8 Sent Done Done false
9973 Acked Write 7 Sent Done Done false
9974 Acked Write 8 Sent Done Done false
9975 Acked Read 1 Sent Done Done false
9976 Acked Write 8 Sent Done Done false
9977 Acked Read 2 Sent Done Done false
9978 Acked Read 10 Sent Done Done false
9979 Acked Read 6 Sent Done Done false
9980 Acked Write 10 Sent Done Done false
9981 Acked Read 6 Sent Done Done false
9982 Acked Write 10 Sent Done Done false
9983 Acked Read 2 Sent Done Done false
9984 Acked Read 10 Sent Done Done false
9985 Acked Read 10 Sent Done Done false
9986 Acked Read 6 Sent Done Done false
9987 Acked Read 9 Sent Done Done false
9988 Acked Read 1 Sent Done Done false
9989 Acked Read 9 Sent Done Done false
9990 Acked Write 1 Sent Done Done false
9991 Acked Read 8 Sent Done Done false
9992 Acked Read 4 Sent Done Done false
9993 Acked Write 1 Sent Done Done false
9994 Acked Write 10 Sent Done Done false
9995 Acked Read 10 Sent Done Done false
9996 Acked Read 10 Sent Done Done false
9997 Acked Write 8 Sent Done Done false
9998 Acked Read 6 Sent Done Done false
9999 Acked Flush 0 Sent Done Done false
10000 Acked Read 6 Sent Done Done false
10001 Acked Read 5 Sent Done Done false
10002 Acked Read 9 Sent Done Done false
10003 Acked Flush 0 Sent Done Done false
10004 Acked Write 4 Sent Done Done false
10005 Acked Write 4 Sent Done Done false
10006 Acked Flush 0 Sent Done Done false
10007 Acked Write 5 Sent Done Done false
10008 Acked Read 4 Sent Done Done false
10009 Acked Write 7 Sent Done Done false
10010 Acked Read 3 Sent Done Done false
10011 Acked Write 10 Sent Done Done false
10012 Acked Write 8 Sent Done Done false
10013 Acked Write 1 Sent Done Done false
10014 Acked Write 4 Sent Done Done false
10015 Acked Read 2 Sent Done Done false
10016 Acked Read 4 Sent Done Done false
10017 Acked Flush 0 Sent Done Done false
10018 Acked Read 1 Sent Done Done false
10019 Acked Read 3 Sent Done Done false
10020 Acked Write 6 Sent Done Done false
10021 Acked Read 1 Sent Done Done false
10022 Acked Read 8 Sent Done Done false
10023 Acked Read 2 Sent Done Done false
10024 Acked Read 2 Sent Done Done false
10025 Acked Read 6 Sent Done Done false
10026 Acked Write 9 Sent Done Done false
10027 Acked Flush 0 Sent Done Done false
10028 Acked Read 10 Sent Done Done false
10029 Acked Flush 0 Sent Done Done false
10030 Acked Read 2 Sent Done Done false
10031 Acked Read 7 Sent Done Done false
10032 Acked Read 6 Sent Done Done false
10033 Acked Write 6 Sent Done Done false
10034 Acked Read 4 Sent Done Done false
10035 Acked Read 10 Sent Done Done false
10036 Acked Read 4 Sent Done Done false
10037 Acked Write 4 Sent Done Done false
10038 Acked Read 3 Sent Done Done false
10039 Acked Flush 0 Sent Done Done false
10040 Acked Write 4 Sent Done Done false
10041 Acked Read 1 Sent Done Done false
10042 Acked Read 6 Sent Done Done false
10043 Acked Write 3 Sent Done Done false
10044 Acked Read 5 Sent Done Done false
10045 Acked Write 3 Sent Done Done false
10046 Acked Write 1 Sent Done Done false
10047 Acked Write 6 Sent Done Done false
10048 Acked Read 2 Sent Done Done false
10049 Acked Write 8 Sent Done Done false
10050 Acked Read 8 Sent Done Done false
10051 Acked Read 2 Sent Done Done false
10052 Acked Read 1 Sent Done Done false
10053 Acked Read 4 Sent Done Done false
10054 Acked Read 2 Sent Done Done false
10055 Acked Flush 0 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 302 0 0 302
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9753 10055 10055
Downstairs last five completed: 9753 9752 9751 9750 9749
Upstairs last five completed: 10055 10054 10053 10052 10051
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.098622012Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58381)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.098734169Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.099934429Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.100570815Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.100595312Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 9753","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.10061041Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 9753","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.100881748Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"0","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] client re-new 302 jobs since flush 9753","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.100905005Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[0] Replayed 302 jobs since flush: 9753","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.101146927Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10055 10054 10053 10052 10051
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Jan 27 19:43:28.343 INFO request completed, latency_us: 387, response_code:[8830] Got stop action so:false kr:false
204, uri: /stop/cid/2, method: POST, req_id: c3062292-3136-44b9-acfa-a9f551aa0004, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:28.343852794Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:28.343903758Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:28.343921846Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:28.343952892Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
Jan 27 19:43:28.343 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/2, method: GET, req_id: b7f504cb-cb5a-4099-ac22-a9f219068f3a, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
Jan 27 19:43:32.346 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3e402c90-a33b-4cec-8551-f0e1f3cdbaf6, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
001/300 Write block 1251 len 12288 data: 6 5 4
002/300 Write block 2214 len 12288 data: 3 3 4
003/300 Read block 1227 len 28672
004/300 Read block 2621 len 36864
005/300 Read block 1676 len 20480
006/300 Write block 2257 len 36864 data: 8 9 7 7 6 6 4 4 3
007/300 Write block 2860 len 20480 data: 6 6 7 7 7
008/300 Write block 1166 len 12288 data: 5 6 6
009/300 Flush
010/300 Read block 3564 len 20480
011/300 Write block 2287 len 32768 data: 9 12 11 11 8 8 3 3
012/300 Write block 2133 len 36864 data: 7 7 6 7 7 6 5 6 7
013/300 Write block 2411 len 8192 data: 9 8
014/300 Read block 2508 len 4096
015/300 Write block 2867 len 4096 data: 6
016/300 Flush
017/300 Read block 1224 len 36864
018/300 Write block 1307 len 16384 data: 9 7 5 5
019/300 Read block 2496 len 16384
020/300 Read block 2270 len 28672
021/300 Flush
022/300 Read block 1254 len 24576
023/300 Write block 1493 len 36864 data: 7 7 6 5 6 5 5 5 4
024/300 Write block 254 len 28672 data: 8 7 7 8 7 7 7
025/300 Read block 3754 len 40960
026/300 Read block 1832 len 12288
027/300 Write block 4964 len 16384 data: 7 4 4 7
028/300 Write block 3512 len 40960 data: 4 3 4 3 3 4 5 7 7 7
029/300 Write block 4740 len 28672 data: 8 8 7 6 6 6 7
030/300 Flush
031/300 Write block 3967 len 8192 data: 8 8
032/300 Write block 3234 len 32768 data: 7 8 6 6 5 6 6 7
033/300 Read block 3071 len 8192
034/300 Read block 4941 len 40960
035/300 Read block 191 len 20480
036/300 Read block 2176 len 8192
037/300 Write block 3088 len 16384 data: 6 5 7 8
038/300 Write block 4323 len 20480 data: 6 6 6 4 4
039/300 Read block 55 len 12288
040/300 Write block 2800 len 36864 data: 6 8 7 7 7 4 5 5 6
041/300 Read block 1176 len 28672
042/300 Read block 1219 len 28672
043/300 Read block 853 len 32768
044/300 Write block 3960 len 8192 data: 6 6
045/300 Read block 2090 len 36864
046/300 Write block 1934 len 20480 data: 6 6 4 5 4
047/300 Read block 1085 len 8192
048/300 Read block 2688 len 12288
049/300 Flush
050/300 Read block 3842 len 20480
051/300 Write block 2929 len 36864 data: 11 9 9 9 9 8 8 9 7
052/300 Read block 2242 len 4096
053/300 Read block 3908 len 32768
054/300 Read block 1960 len 32768
055/300 Write block 2518 len 36864 data: 6 5 6 5 5 5 6 5 4
056/300 Write block 231 len 12288 data: 3 3 5
057/300 Read block 4345 len 32768
058/300 Read block 3385 len 20480
059/300 Write block 1251 len 20480 data: 7 6 5 6 5
060/300 Read block 3800 len 32768
061/300 Flush
062/300 Flush
063/300 Flush
064/300 Write block 2899 len 12288 data: 2 2 2
065/300 Flush
066/300 Read block 4727 len 28672
067/300 Read block 144 len 16384
068/300 Write block 1560 len 12288 data: 6 4 3
069/300 Read block 1336 len 4096
070/300 Write block 3165 len 16384 data: 3 3 3 3
071/300 Read block 2859 len 4096
072/300 Read block 2159 len 28672
073/300 Flush
074/300 Write block 4800 len 4096 data: 8
075/300 Write block 2020 len 20480 data: 6 6 6 6 5
076/300 Read block 3879 len 24576
077/300 Read block 1142 len 24576
078/300 Read block 611 len 36864
079/300 Flush
080/300 Read block 1500 len 32768
081/300 Read block 4365 len 28672
082/300 Write block 2257 len 4096 data: 9
083/300 Write block 123 len 4096 data: 8
084/300 Write block 3361 len 12288 data: 8 7 6
085/300 Read block 2134 len 28672
086/300 Read block 2999 len 24576
087/300 Flush
088/300 Read block 3197 len 12288
089/300 Read block 1138 len 12288
090/300 Read block 3338 len 12288
091/300 Read block 4820 len 20480
092/300 Write block 1438 len 20480 data: 7 7 7 8 8
093/300 Read block 1337 len 24576
094/300 Flush
095/300 Write block 3547 len 40960 data: 4 4 5 7 7 6 7 5 5 8
096/300 Read block 4599 len 28672
097/300 Read block 4635 len 36864
098/300 Write block 643 len 28672 data: 3 5 5 5 5 8 6
099/300 Read block 2825 len 4096
100/300 Read block 4165 len 28672
101/300 Read block 275 len 12288
102/300 Write block 2951 len 20480 data: 6 8 7 7 6
103/300 Read block 2431 len 20480
104/300 Flush
105/300 Read block 3568 len 32768
106/300 Write block 1752 len 16384 data: 3 4 6 5
107/300 Read block 4873 len 32768
108/300 Write block 2550 len 36864 data: 6 6 6 6 6 8 8 10 9
109/300 Write block 3974 len 12288 data: 4 4 4
110/300 Read block 2673 len 12288
111/300 Read block 4280 len 24576
112/300 Read block 1375 len 8192
113/300 Read block 2199 len 40960
114/300 Write block 1068 len 40960 data: 6 5 6 6 8 8 7 7 7 7
115/300 Write block 4716 len 40960 data: 7 7 9 7 4 4 5 5 5 5
116/300 Write block 566 len 4096 data: 3
117/300 Read block 3988 len 36864
118/300 Write block 4825 len 24576 data: 6 7 6 4 8 9
119/300 Read block 606 len 24576
120/300 Write block 4906 len 32768 data: 5 7 8 8 8 8 8 7
121/300 Read block 2733 len 40960
122/300 Read block 726 len 8192
123/300 Read block 3885 len 28672
124/300 Read block 3645 len 12288
125/300 Read block 3935 len 24576
126/300 Read block 3908 len 16384
127/300 Read block 3584 len 36864
128/300 Write block 1457 len 4096 data: 9
129/300 Write block 1913 len 8192 data: 2 2
130/300 Read block 1626 len 28672
131/300 Write block 682 len 20480 data: 7 7 8 8 9
132/300 Write block 1584 len 8192 data: 4 4
133/300 Write block 4942 len 32768 data: 7 8 8 6 5 5 3 5
134/300 Read block 1654 len 32768
135/300 Flush
136/300 Read block 4874 len 28672
137/300 Flush
138/300 Read block 797 len 8192
139/300 Write block 846 len 16384 data: 7 5 7 6
140/300 Read block 1674 len 28672
141/300 Write block 1927 len 32768 data: 6 6 6 6 4 4 5 7
142/300 Write block 4168 len 32768 data: 10 9 8 7 7 9 6 6
143/300 Read block 2405 len 32768
144/300 Write block 3886 len 32768 data: 5 5 7 7 5 4 4 6
145/300 Read block 4777 len 24576
146/300 Write block 330 len 8192 data: 7 6
147/300 Write block 3638 len 12288 data: 6 4 4
148/300 Write block 1672 len 12288 data: 5 6 9
149/300 Write block 1700 len 12288 data: 5 5 5
150/300 Read block 3691 len 16384
151/300 Read block 3514 len 40960
152/300 Write block 3846 len 8192 data: 7 6
153/300 Read block 3685 len 20480
154/300 Read block 870 len 24576
155/300 Read block 2009 len 12288
156/300 Read block 1303 len 8192
157/300 Write block 1506 len 8192 data: 4 5
158/300 Write block 4567 len 36864 data: 7 8 6 4 4 4 3 3 5
159/300 Read block 930 len 16384
160/300 Read block 27 len 24576
161/300 Write block 2235 len 16384 data: 4 6 6 5
162/300 Write block 4418 len 12288 data: 5 4 2
163/300 Write block 4502 len 8192 data: 7 9
164/300 Read block 133 len 12288
165/300 Read block 2933 len 8192
166/300 Write block 2779 len 20480 data: 7 6 6 8 8
167/300 Read block 527 len 20480
168/300 Read block 3791 len 32768
169/300 Write block 2134 len 12288 data: 8 7 8
170/300 Write block 2024 len 12288 data: 6 4 6
171/300 Read block 3432 len 40960
172/300 Write block 3597 len 16384 data: 5 4 6 7
173/300 Flush
174/300 Write block 3415 len 40960 data: 3 3 2 2 2 2 2 4 4 5
175/300 Read block 49 len 40960
176/300 Read block 3258 len 40960
177/300 Flush
178/300 Write block 2488 len 32768 data: 5 5 7 7 7 7 7 6
179/300 Write block 2435 len 4096 data: 3
180/300 Read block 3490 len 12288
181/300 Read block 2324 len 40960
182/300 Write block 609 len 24576 data: 6 6 6 5 6 7
183/300 Write block 2496 len 36864 data: 5 4 6 6 5 6 6 5 3
184/300 Write block 2505 len 16384 data: 3 5 6 8
185/300 Read block 1152 len 40960
186/300 Write block 4950 len 36864 data: 5 5 4 3 5 5 5 6 8
187/300 Read block 1318 len 36864
188/300 Write block 4113 len 4096 data: 7
189/300 Write block 4074 len 32768 data: 7 5 3 5 5 4 6 6
190/300 Read block 2725 len 24576
191/300 Read block 1258 len 40960
192/300 Read block 1114 len 40960
193/300 Write block 4662 len 36864 data: 4 4 3 3 4 5 4 4 4
194/300 Read block 858 len 28672
195/300 Write block 4303 len 36864 data: 5 6 5 4 4 5 5 5 4
196/300 Write block 2345 len 8192 data: 6 7
197/300 Write block 4370 len 20480 data: 6 5 5 4 4
198/300 Write block 826 len 16384 data: 7 6 8 7
199/300 Read block 4021 len 32768
200/300 Write block 16 len 24576 data: 6 6 4 4 4 4
201/300 Read block 2233 len 4096
202/300 Write block 4018 len 32768 data: 5 6 6 6 4 5 3 3
203/300 Write block 1595 len 28672 data: 8 10 12 13 9 9 6
204/300 Read block 3871 len 12288
205/300 Write block 195 len 40960 data: 5 6 6 6 8 7 5 4 4 5
206/300 Write block 2873 len 20480 data: 6 5 5 5 5
207/300 Read block 754 len 28672
208/300 Read block 1662 len 28672
209/300 Read block 260 len 12288
210/300 Flush
211/300 Write block 2974 len 16384 data: 9 7 6 6
212/300 Read block 1180 len 12288
213/300 Write block 2818 len 20480 data: 5 5 6 5 5
214/300 Flush
215/300 Write block 3099 len 4096 data: 9
216/300 Write block 75 len 32768 data: 4 5 5 6 7 6 6 5
217/300 Write block 1292 len 8192 data: 5 5
218/300 Flush
219/300 Read block 382 len 8192
220/300 Write block 2838 len 40960 data: 5 7 7 7 6 5 7 8 9 7
221/300 Write block 4684 len 32768 data: 6 6 7 7 6 7 6 7
222/300 Read block 3302 len 24576
223/300 Write block 979 len 12288 data: 3 4 2
224/300 Read block 3855 len 8192
225/300 Read block 3648 len 40960
226/300 Read block 4787 len 36864
227/300 Write block 1360 len 4096 data: 10
228/300 Read block 4101 len 12288
229/300 Read block 4799 len 8192
230/300 Write block 4913 len 12288 data: 8 6 5
231/300 Write block 2791 len 20480 data: 6 5 7 8 7
232/300 Write block 4066 len 8192 data: 5 4
233/300 Read block 1607 len 4096
234/300 Flush
235/300 Read block 1389 len 40960
236/300 Read block 4442 len 40960
237/300 Read block 1467 len 32768
238/300 Read block 2718 len 36864
239/300 Write block 3827 len 4096 data: 5
240/300 Flush
241/300 Read block 2538 len 16384
242/300 Read block 2356 len 20480
243/300 Flush
244/300 Write block 4466 len 8192 data: 6 5
245/300 Write block 3386 len 28672 data: 6 7 8 9 8 7 6
246/300 Write block 3778 len 20480 data: 7 7 7 6 6
247/300 Read block 3023 len 32768
248/300 Write block 1687 len 8192 data: 5 5
249/300 Read block 4834 len 20480
250/300 Write block 3021 len 16384 data: 7 7 8 9
251/300 Flush
252/300 Write block 2146 len 8192 data: 8 6
253/300 Write block 2960 len 12288 data: 5 5 7
254/300 Flush
255/300 Read block 2383 len 32768
256/300 Read block 4579 len 16384
257/300 Read block 420 len 32768
258/300 Read block 2812 len 36864
259/300 Write block 1240 len 4096 data: 8
260/300 Read block 3282 len 36864
261/300 Write block 3621 len 4096 data: 10
262/300 Read block 4935 len 8192
263/300 Read block 2788 len 12288
264/300 Read block 3085 len 36864
265/300 Flush
266/300 Read block 4286 len 32768
267/300 Write block 2137 len 12288 data: 8 7 6
268/300 Read block 3681 len 28672
269/300 Write block 870 len 32768 data: 6 8 7 8 6 7 7 6
270/300 Flush
271/300 Read block 2075 len 40960
272/300 Write block 4876 len 28672 data: 3 3 3 4 4 4 6
273/300 Write block 2158 len 32768 data: 11 9 8 9 9 7 7 6
274/300 Read block 447 len 4096
275/300 Read block 3813 len 12288
276/300 Read block 42 len 16384
277/300 Read block 2076 len 12288
278/300 Read block 2433 len 36864
279/300 Write block 2174 len 24576 data: 9 10 11 8 8 7
280/300 Read block 1950 len 28672
281/300 Read block 648 len 4096
282/300 Flush
283/300 Write block 1565 len 40960 data: 4 3 3 4 6 6 7 8 9 8
284/300 Read block 4414 len 24576
285/300 Write block 508 len 24576 data: 6 5 7 9 8 6
286/300 Read block 1686 len 32768
287/300 Write block 3154 len 12288 data: 4 4 4
288/300 Write block 1249 len 20480 data: 5 4 8 7 6
289/300 Read block 1586 len 8192
290/300 Write block 4961 len 4096 data: 10
291/300 Write block 380 len 36864 data: 5 4 6 5 4 4 3 3 2
292/300 Write block 22 len 40960 data: 3 4 5 5 4 4 6 5 5 4
293/300 Read block 1757 len 36864
294/300 Write block 3891 len 36864 data: 5 5 7 8 6 6 6 6 5
295/300 Write block 4544 len 36864 data: 5 5 5 4 3 4 4 5 5
296/300 Write block 3496 len 12288 data: 6 5 4
297/300 Read block 2164 len 8192
298/300 Write block 2294 len 28672 data: 4 4 4 4 6 7 6
299/300 Write block 1198 len 16384 data: 9 7 6 6
300/300 Write block 4538 len 32768 data: 6 4 5 5 4 4 6 6
301/300 Read block 2520 len 40960
Main task has work to do, go find it
got dsc Start(2)
start 2
Jan 27 19:43:32.581 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/2, method: POST, req_id: bf9b5e89-a792-44e7-9d64-3d2c07686bd9, remote_addr: 127.0.0.1:55047, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10056 Acked Write 3 Done Done Sent false
10057 Acked Write 3 Done Done Sent false
10058 Acked Read 7 Done Done Sent false
10059 Acked Read 9 Done Done Sent false
10060 Acked Read 5 Done Done Sent false
10061 Acked Write 9 Done Done Sent false
10062 Acked Write 5 Done Done Sent false
10063 Acked Write 3 Done Done Sent false
10064 Acked Flush 0 Done Done Sent false
10065 Acked Read 5 Done Done Sent false
10066 Acked Write 8 Done Done Sent false
10067 Acked Write 9 Done Done Sent false
10068 Acked Write 2 Done Done Sent false
10069 Acked Read 1 Done Done Sent false
10070 Acked Write 1 Done Done Sent false
10071 Acked Flush 0 Done Done Sent false
10072 Acked Read 9 Done Done Sent false
10073 Acked Write 4 Done Done Sent false
10074 Acked Read 4 Done Done Sent false
10075 Acked Read 7 Done Done Sent false
10076 Acked Flush 0 Done Done Sent false
10077 Acked Read 6 Done Done Sent false
10078 Acked Write 9 Done Done Sent false
10079 Acked Write 7 Done Done Sent false
10080 Acked Read 10 Done Done Sent false
10081 Acked Read 3 Done Done Sent false
10082 Acked Write 4 Done Done Sent false
10083 Acked Write 10 Done Done Sent false
10084 Acked Write 7 Done Done Sent false
10085 Acked Flush 0 Done Done Sent false
10086 Acked Write 2 Done Done Sent false
10087 Acked Write 8 Done Done Sent false
10088 Acked Read 2 Done Done Sent false
10089 Acked Read 10 Done Done Sent false
10090 Acked Read 5 Done Done Sent false
10091 Acked Read 2 Done Done Sent false
10092 Acked Write 4 Done Done Sent false
10093 Acked Write 5 Done Done Sent false
10094 Acked Read 3 Done Done Sent false
10095 Acked Write 9 Done Done Sent false
10096 Acked Read 7 Done Done Sent false
10097 Acked Read 7 Done Done Sent false
10098 Acked Read 8 Done Done Sent false
10099 Acked Write 2 Done Done Sent false
10100 Acked Read 9 Done Done Sent false
10101 Acked Write 5 Done Done Sent false
10102 Acked Read 2 Done Done Sent false
10103 Acked Read 3 Done Done Sent false
10104 Acked Flush 0 Done Done Sent false
10105 Acked Read 5 Done Done Sent false
10106 Acked Write 9 Done Done Sent false
10107 Acked Read 1 Done Done Sent false
10108 Acked Read 8 Done Done Sent false
10109 Acked Read 8 Done Done Sent false
10110 Acked Write 9 Done Done Sent false
10111 Acked Write 3 Done Done Sent false
10112 Acked Read 8 Done Done Sent false
10113 Acked Read 5 Done Done Sent false
10114 Acked Write 5 Done Done Sent false
10115 Acked Read 8 Done Done Sent false
10116 Acked Flush 0 Done Done Sent false
10117 Acked Flush 0 Done Done Sent false
10118 Acked Flush 0 Done Done Sent false
10119 Acked Write 3 Done Done Sent false
10120 Acked Flush 0 Done Done Sent false
10121 Acked Read 7 Done Done Sent false
10122 Acked Read 4 Done Done Sent false
10123 Acked Write 3 Done Done Sent false
10124 Acked Read 1 Done Done Sent false
10125 Acked Write 4 Done Done Sent false
10126 Acked Read 1 Done Done Sent false
10127 Acked Read 7 Done Done Sent false
10128 Acked Flush 0 Done Done Sent false
10129 Acked Write 1 Done Done Sent false
10130 Acked Write 5 Done Done Sent false
10131 Acked Read 6 Done Done Sent false
10132 Acked Read 6 Done Done Sent false
10133 Acked Read 9 Done Done Sent false
10134 Acked Flush 0 Done Done Sent false
10135 Acked Read 8 Done Done Sent false
10136 Acked Read 7 Done Done Sent false
10137 Acked Write 1 Done Done Sent false
10138 Acked Write 1 Done Done Sent false
10139 Acked Write 3 Done Done Sent false
10140 Acked Read 7 Done Done Sent false
10141 Acked Read 6 Done Done Sent false
10142 Acked Flush 0 Done Done Sent false
10143 Acked Read 3 Done Done Sent false
10144 Acked Read 3 Done Done Sent false
10145 Acked Read 3 Done Done Sent false
10146 Acked Read 5 Done Done Sent false
10147 Acked Write 5 Done Done Sent false
10148 Acked Read 6 Done Done Sent false
10149 Acked Flush 0 Done Done Sent false
10150 Acked Write 10 Done Done Sent false
10151 Acked Read 7 Done Done Sent false
10152 Acked Read 9 Done Done Sent false
10153 Acked Write 7 Done Done Sent false
10154 Acked Read 1 Done Done Sent false
10155 Acked Read 7 Done Done Sent false
10156 Acked Read 3 Done Done Sent false
10157 Acked Write 5 Done Done Sent false
10158 Acked Read 5 Done Done Sent false
10159 Acked Flush 0 Done Done Sent false
10160 Acked Read 8 Done Done Sent false
10161 Acked Write 4 Done Done Sent false
10162 Acked Read 8 Done Done Sent false
10163 Acked Write 9 Done Done Sent false
10164 Acked Write 3 Done Done Sent false
10165 Acked Read 3 Done Done Sent false
10166 Acked Read 6 Done Done Sent false
10167 Acked Read 2 Done Done Sent false
10168 Acked Read 10 Done Done Sent false
10169 Acked Write 10 Done Done Sent false
10170 Acked Write 10 Done Done Sent false
10171 Acked Write 1 Done Done Sent false
10172 Acked Read 9 Done Done Sent false
10173 Acked Write 6 Done Done Sent false
10174 Acked Read 6 Done Done Sent false
10175 Acked Write 8 Done Done Sent false
10176 Acked Read 10 Done Done Sent false
10177 Acked Read 2 Done Done Sent false
10178 Acked Read 7 Done Done Sent false
10179 Acked Read 3 Done Done Sent false
10180 Acked Read 6 Done Done Sent false
10181 Acked Read 4 Done Done Sent false
10182 Acked Read 9 Done Done Sent false
10183 Acked Write 1 Done Done Sent false
10184 Acked Write 2 Done Done Sent false
10185 Acked Read 7 Done Done Sent false
10186 Acked Write 5 Done Done Sent false
10187 Acked Write 2 Done Done Sent false
10188 Acked Write 8 Done Done Sent false
10189 Acked Read 8 Done Done Sent false
10190 Acked Flush 0 Done Done Sent false
10191 Acked Read 7 Done Done Sent false
10192 Acked Flush 0 Done Done Sent false
10193 Acked Read 2 Done Done Sent false
10194 Acked Write 4 Done Done Sent false
10195 Acked Read 7 Done Done Sent false
10196 Acked Write 8 Done Done Sent false
10197 Acked Write 8 Done Done Sent false
10198 Acked Read 8 Done Done Sent false
10199 Acked Write 8 Done Done Sent false
10200 Acked Read 6 Done Done Sent false
10201 Acked Write 2 Done Done Sent false
10202 Acked Write 3 Done Done Sent false
10203 Acked Write 3 Done Done Sent false
10204 Acked Write 3 Done Done Sent false
10205 Acked Read 4 Done Done Sent false
10206 Acked Read 10 Done Done Sent false
10207 Acked Write 2 Done Done Sent false
10208 Acked Read 5 Done Done Sent false
10209 Acked Read 6 Done Done Sent false
10210 Acked Read 3 Done Done Sent false
10211 Acked Read 2 Done Done Sent false
10212 Acked Write 2 Done Done Sent false
10213 Acked Write 9 Done Done Sent false
10214 Acked Read 4 Done Done Sent false
10215 Acked Read 6 Done Done Sent false
10216 Acked Write 4 Done Done Sent false
10217 Acked Write 3 Done Done Sent false
10218 Acked Write 2 Done Done Sent false
10219 Acked Read 3 Done Done Sent false
10220 Acked Read 2 Done Done Sent false
10221 Acked Write 5 Done Done Sent false
10222 Acked Read 5 Done Done Sent false
10223 Acked Read 8 Done Done Sent false
10224 Acked Write 3 Done Done Sent false
10225 Acked Write 3 Done Done Sent false
10226 Acked Read 10 Done Done Sent false
10227 Acked Write 4 Done Done Sent false
10228 Acked Flush 0 Done Done Sent false
10229 Acked Write 10 Done Done Sent false
10230 Acked Read 10 Done Done Sent false
10231 Acked Read 10 Done Done Sent false
10232 Acked Flush 0 Done Done Sent false
10233 Acked Write 8 Done Done Sent false
10234 Acked Write 1 Done Done Sent false
10235 Acked Read 3 Done Done Sent false
10236 Acked Read 10 Done Done Sent false
10237 Acked Write 6 Done Done Sent false
10238 Acked Write 9 Done Done Sent false
10239 Acked Write 4 Done Done Sent false
10240 Acked Read 10 Done Done Sent false
10241 Acked Write 9 Done Done Sent false
10242 Acked Read 9 Done Done Sent false
10243 Acked Write 1 Done Done Sent false
10244 Acked Write 8 Done Done Sent false
10245 Acked Read 6 Done Done Sent false
10246 Acked Read 10 Done Done Sent false
10247 Acked Read 10 Done Done Sent false
10248 Acked Write 9 Done Done Sent false
10249 Acked Read 7 Done Done Sent false
10250 Acked Write 9 Done Done Sent false
10251 Acked Write 2 Done Done Sent false
10252 Acked Write 5 Done Done Sent false
10253 Acked Write 4 Done Done Sent false
10254 Acked Read 8 Done Done Sent false
10255 Acked Write 6 Done Done Sent false
10256 Acked Read 1 Done Done Sent false
10257 Acked Write 8 Done Done Sent false
10258 Acked Write 7 Done Done Sent false
10259 Acked Read 3 Done Done Sent false
10260 Acked Write 10 Done Done Sent false
10261 Acked Write 5 Done Done Sent false
10262 Acked Read 7 Done Done Sent false
10263 Acked Read 7 Done Done Sent false
10264 Acked Read 3 Done Done Sent false
10265 Acked Flush 0 Done Done Sent false
10266 Acked Write 4 Done Done Sent false
10267 Acked Read 3 Done Done Sent false
10268 Acked Write 5 Done Done Sent false
10269 Acked Flush 0 Done Done Sent false
10270 Acked Write 1 Done Done Sent false
10271 Acked Write 8 Done Done Sent false
10272 Acked Write 2 Done Done Sent false
10273 Acked Flush 0 Done Done Sent false
10274 Acked Read 2 Done Done Sent false
10275 Acked Write 10 Done Done Sent false
10276 Acked Write 8 Done Done Sent false
10277 Acked Read 6 Done Done Sent false
10278 Acked Write 3 Done Done Sent false
10279 Acked Read 2 Done Done Sent false
10280 Acked Read 10 Done Done Sent false
10281 Acked Read 9 Done Done Sent false
10282 Acked Write 1 Done Done Sent false
10283 Acked Read 3 Done Done Sent false
10284 Acked Read 2 Done Done Sent false
10285 Acked Write 3 Done Done Sent false
10286 Acked Write 5 Done Done Sent false
10287 Acked Write 2 Done Done Sent false
10288 Acked Read 1 Done Done Sent false
10289 Acked Flush 0 Done Done Sent false
10290 Acked Read 10 Done Done Sent false
10291 Acked Read 10 Done Done Sent false
10292 Acked Read 8 Done Done Sent false
10293 Acked Read 9 Done Done Sent false
10294 Acked Write 1 Done Done Sent false
10295 Acked Flush 0 Done Done Sent false
10296 Acked Read 4 Done Done Sent false
10297 Acked Read 5 Done Done Sent false
10298 Acked Flush 0 Done Done Sent false
10299 Acked Write 2 Done Done Sent false
10300 Acked Write 7 Done Done Sent false
10301 Acked Write 5 Done Done Sent false
10302 Acked Read 8 Done Done Sent false
10303 Acked Write 2 Done Done Sent false
10304 Acked Read 5 Done Done Sent false
10305 Acked Write 4 Done Done Sent false
10306 Acked Flush 0 Done Done Sent false
10307 Acked Write 2 Done Done Sent false
10308 Acked Write 3 Done Done Sent false
10309 Acked Flush 0 Done Done Sent false
10310 Acked Read 8 Done Done Sent false
10311 Acked Read 4 Done Done Sent false
10312 Acked Read 8 Done Done Sent false
10313 Acked Read 9 Done Done Sent false
10314 Acked Write 1 Done Done Sent false
10315 Acked Read 9 Done Done Sent false
10316 Acked Write 1 Done Done Sent false
10317 Acked Read 2 Done Done Sent false
10318 Acked Read 3 Done Done Sent false
10319 Acked Read 9 Done Done Sent false
10320 Acked Flush 0 Done Done Sent false
10321 Acked Read 8 Done Done Sent false
10322 Acked Write 3 Done Done Sent false
10323 Acked Read 7 Done Done Sent false
10324 Acked Write 8 Done Done Sent false
10325 Acked Flush 0 Done Done Sent false
10326 Acked Read 10 Done Done Sent false
10327 Acked Write 7 Done Done Sent false
10328 Acked Write 8 Done Done Sent false
10329 Acked Read 1 Done Done Sent false
10330 Acked Read 3 Done Done Sent false
10331 Acked Read 4 Done Done Sent false
10332 Acked Read 3 Done Done Sent false
10333 Acked Read 9 Done Done Sent false
10334 Acked Write 6 Done Done Sent false
10335 Acked Read 7 Done Done Sent false
10336 Acked Read 1 Done Done Sent false
10337 Acked Flush 0 Done Done Sent false
10338 Acked Write 10 Done Done Sent false
10339 Acked Read 6 Done Done Sent false
10340 Acked Write 6 Done Done Sent false
10341 Acked Read 8 Done Done Sent false
10342 Acked Write 3 Done Done Sent false
10343 Acked Write 5 Done Done Sent false
10344 Acked Read 2 Done Done Sent false
10345 Acked Write 1 Done Done Sent false
10346 Acked Write 9 Done Done Sent false
10347 Acked Write 10 Done Done Sent false
10348 Acked Read 9 Done Done Sent false
10349 Acked Write 9 Done Done Sent false
10350 Acked Write 9 Done Done Sent false
10351 Acked Write 3 Done Done Sent false
10352 Acked Read 2 Done Done Sent false
10353 Acked Write 7 Done Done Sent false
10354 Acked Write 4 Done Done Sent false
10355 Acked Write 8 Done Done Sent false
10356 Acked Read 10 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 301 301
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 10337 10337 10055
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10356 10355 10354 10353 10352
CLIENT: Up:0 ds:301 act:2
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10056 Acked Write 3 Done Done Sent false
10057 Acked Write 3 Done Done Sent false
10058 Acked Read 7 Done Done Sent false
10059 Acked Read 9 Done Done Sent false
10060 Acked Read 5 Done Done Sent false
10061 Acked Write 9 Done Done Sent false
10062 Acked Write 5 Done Done Sent false
10063 Acked Write 3 Done Done Sent false
10064 Acked Flush 0 Done Done Sent false
10065 Acked Read 5 Done Done Sent false
10066 Acked Write 8 Done Done Sent false
10067 Acked Write 9 Done Done Sent false
10068 Acked Write 2 Done Done Sent false
10069 Acked Read 1 Done Done Sent false
10070 Acked Write 1 Done Done Sent false
10071 Acked Flush 0 Done Done Sent false
10072 Acked Read 9 Done Done Sent false
10073 Acked Write 4 Done Done Sent false
10074 Acked Read 4 Done Done Sent false
10075 Acked Read 7 Done Done Sent false
10076 Acked Flush 0 Done Done Sent false
10077 Acked Read 6 Done Done Sent false
10078 Acked Write 9 Done Done Sent false
10079 Acked Write 7 Done Done Sent false
10080 Acked Read 10 Done Done Sent false
10081 Acked Read 3 Done Done Sent false
10082 Acked Write 4 Done Done Sent false
10083 Acked Write 10 Done Done Sent false
10084 Acked Write 7 Done Done Sent false
10085 Acked Flush 0 Done Done Sent false
10086 Acked Write 2 Done Done Sent false
10087 Acked Write 8 Done Done Sent false
10088 Acked Read 2 Done Done Sent false
10089 Acked Read 10 Done Done Sent false
10090 Acked Read 5 Done Done Sent false
10091 Acked Read 2 Done Done Sent false
10092 Acked Write 4 Done Done Sent false
10093 Acked Write 5 Done Done Sent false
10094 Acked Read 3 Done Done Sent false
10095 Acked Write 9 Done Done Sent false
10096 Acked Read 7 Done Done Sent false
10097 Acked Read 7 Done Done Sent false
10098 Acked Read 8 Done Done Sent false
10099 Acked Write 2 Done Done Sent false
10100 Acked Read 9 Done Done Sent false
10101 Acked Write 5 Done Done Sent false
10102 Acked Read 2 Done Done Sent false
10103 Acked Read 3 Done Done Sent false
10104 Acked Flush 0 Done Done Sent false
10105 Acked Read 5 Done Done Sent false
10106 Acked Write 9 Done Done Sent false
10107 Acked Read 1 Done Done Sent false
10108 Acked Read 8 Done Done Sent false
10109 Acked Read 8 Done Done Sent false
10110 Acked Write 9 Done Done Sent false
10111 Acked Write 3 Done Done Sent false
10112 Acked Read 8 Done Done Sent false
10113 Acked Read 5 Done Done Sent false
10114 Acked Write 5 Done Done Sent false
10115 Acked Read 8 Done Done Sent false
10116 Acked Flush 0 Done Done Sent false
10117 Acked Flush 0 Done Done Sent false
10118 Acked Flush 0 Done Done Sent false
10119 Acked Write 3 Done Done Sent false
10120 Acked Flush 0 Done Done Sent false
10121 Acked Read 7 Done Done Sent false
10122 Acked Read 4 Done Done Sent false
10123 Acked Write 3 Done Done Sent false
10124 Acked Read 1 Done Done Sent false
10125 Acked Write 4 Done Done Sent false
10126 Acked Read 1 Done Done Sent false
10127 Acked Read 7 Done Done Sent false
10128 Acked Flush 0 Done Done Sent false
10129 Acked Write 1 Done Done Sent false
10130 Acked Write 5 Done Done Sent false
10131 Acked Read 6 Done Done Sent false
10132 Acked Read 6 Done Done Sent false
10133 Acked Read 9 Done Done Sent false
10134 Acked Flush 0 Done Done Sent false
10135 Acked Read 8 Done Done Sent false
10136 Acked Read 7 Done Done Sent false
10137 Acked Write 1 Done Done Sent false
10138 Acked Write 1 Done Done Sent false
10139 Acked Write 3 Done Done Sent false
10140 Acked Read 7 Done Done Sent false
10141 Acked Read 6 Done Done Sent false
10142 Acked Flush 0 Done Done Sent false
10143 Acked Read 3 Done Done Sent false
10144 Acked Read 3 Done Done Sent false
10145 Acked Read 3 Done Done Sent false
10146 Acked Read 5 Done Done Sent false
10147 Acked Write 5 Done Done Sent false
10148 Acked Read 6 Done Done Sent false
10149 Acked Flush 0 Done Done Sent false
10150 Acked Write 10 Done Done Sent false
10151 Acked Read 7 Done Done Sent false
10152 Acked Read 9 Done Done Sent false
10153 Acked Write 7 Done Done Sent false
10154 Acked Read 1 Done Done Sent false
10155 Acked Read 7 Done Done Sent false
10156 Acked Read 3 Done Done Sent false
10157 Acked Write 5 Done Done Sent false
10158 Acked Read 5 Done Done Sent false
10159 Acked Flush 0 Done Done Sent false
10160 Acked Read 8 Done Done Sent false
10161 Acked Write 4 Done Done Sent false
10162 Acked Read 8 Done Done Sent false
10163 Acked Write 9 Done Done Sent false
10164 Acked Write 3 Done Done Sent false
10165 Acked Read 3 Done Done Sent false
10166 Acked Read 6 Done Done Sent false
10167 Acked Read 2 Done Done Sent false
10168 Acked Read 10 Done Done Sent false
10169 Acked Write 10 Done Done Sent false
10170 Acked Write 10 Done Done Sent false
10171 Acked Write 1 Done Done Sent false
10172 Acked Read 9 Done Done Sent false
10173 Acked Write 6 Done Done Sent false
10174 Acked Read 6 Done Done Sent false
10175 Acked Write 8 Done Done Sent false
10176 Acked Read 10 Done Done Sent false
10177 Acked Read 2 Done Done Sent false
10178 Acked Read 7 Done Done Sent false
10179 Acked Read 3 Done Done Sent false
10180 Acked Read 6 Done Done Sent false
10181 Acked Read 4 Done Done Sent false
10182 Acked Read 9 Done Done Sent false
10183 Acked Write 1 Done Done Sent false
10184 Acked Write 2 Done Done Sent false
10185 Acked Read 7 Done Done Sent false
10186 Acked Write 5 Done Done Sent false
10187 Acked Write 2 Done Done Sent false
10188 Acked Write 8 Done Done Sent false
10189 Acked Read 8 Done Done Sent false
10190 Acked Flush 0 Done Done Sent false
10191 Acked Read 7 Done Done Sent false
10192 Acked Flush 0 Done Done Sent false
10193 Acked Read 2 Done Done Sent false
10194 Acked Write 4 Done Done Sent false
10195 Acked Read 7 Done Done Sent false
10196 Acked Write 8 Done Done Sent false
10197 Acked Write 8 Done Done Sent false
10198 Acked Read 8 Done Done Sent false
10199 Acked Write 8 Done Done Sent false
10200 Acked Read 6 Done Done Sent false
10201 Acked Write 2 Done Done Sent false
10202 Acked Write 3 Done Done Sent false
10203 Acked Write 3 Done Done Sent false
10204 Acked Write 3 Done Done Sent false
10205 Acked Read 4 Done Done Sent false
10206 Acked Read 10 Done Done Sent false
10207 Acked Write 2 Done Done Sent false
10208 Acked Read 5 Done Done Sent false
10209 Acked Read 6 Done Done Sent false
10210 Acked Read 3 Done Done Sent false
10211 Acked Read 2 Done Done Sent false
10212 Acked Write 2 Done Done Sent false
10213 Acked Write 9 Done Done Sent false
10214 Acked Read 4 Done Done Sent false
10215 Acked Read 6 Done Done Sent false
10216 Acked Write 4 Done Done Sent false
10217 Acked Write 3 Done Done Sent false
10218 Acked Write 2 Done Done Sent false
10219 Acked Read 3 Done Done Sent false
10220 Acked Read 2 Done Done Sent false
10221 Acked Write 5 Done Done Sent false
10222 Acked Read 5 Done Done Sent false
10223 Acked Read 8 Done Done Sent false
10224 Acked Write 3 Done Done Sent false
10225 Acked Write 3 Done Done Sent false
10226 Acked Read 10 Done Done Sent false
10227 Acked Write 4 Done Done Sent false
10228 Acked Flush 0 Done Done Sent false
10229 Acked Write 10 Done Done Sent false
10230 Acked Read 10 Done Done Sent false
10231 Acked Read 10 Done Done Sent false
10232 Acked Flush 0 Done Done Sent false
10233 Acked Write 8 Done Done Sent false
10234 Acked Write 1 Done Done Sent false
10235 Acked Read 3 Done Done Sent false
10236 Acked Read 10 Done Done Sent false
10237 Acked Write 6 Done Done Sent false
10238 Acked Write 9 Done Done Sent false
10239 Acked Write 4 Done Done Sent false
10240 Acked Read 10 Done Done Sent false
10241 Acked Write 9 Done Done Sent false
10242 Acked Read 9 Done Done Sent false
10243 Acked Write 1 Done Done Sent false
10244 Acked Write 8 Done Done Sent false
10245 Acked Read 6 Done Done Sent false
10246 Acked Read 10 Done Done Sent false
10247 Acked Read 10 Done Done Sent false
10248 Acked Write 9 Done Done Sent false
10249 Acked Read 7 Done Done Sent false
10250 Acked Write 9 Done Done Sent false
10251 Acked Write 2 Done Done Sent false
10252 Acked Write 5 Done Done Sent false
10253 Acked Write 4 Done Done Sent false
10254 Acked Read 8 Done Done Sent false
10255 Acked Write 6 Done Done Sent false
10256 Acked Read 1 Done Done Sent false
10257 Acked Write 8 Done Done Sent false
10258 Acked Write 7 Done Done Sent false
10259 Acked Read 3 Done Done Sent false
10260 Acked Write 10 Done Done Sent false
10261 Acked Write 5 Done Done Sent false
10262 Acked Read 7 Done Done Sent false
10263 Acked Read 7 Done Done Sent false
10264 Acked Read 3 Done Done Sent false
10265 Acked Flush 0 Done Done Sent false
10266 Acked Write 4 Done Done Sent false
10267 Acked Read 3 Done Done Sent false
10268 Acked Write 5 Done Done Sent false
10269 Acked Flush 0 Done Done Sent false
10270 Acked Write 1 Done Done Sent false
10271 Acked Write 8 Done Done Sent false
10272 Acked Write 2 Done Done Sent false
10273 Acked Flush 0 Done Done Sent false
10274 Acked Read 2 Done Done Sent false
10275 Acked Write 10 Done Done Sent false
10276 Acked Write 8 Done Done Sent false
10277 Acked Read 6 Done Done Sent false
10278 Acked Write 3 Done Done Sent false
10279 Acked Read 2 Done Done Sent false
10280 Acked Read 10 Done Done Sent false
10281 Acked Read 9 Done Done Sent false
10282 Acked Write 1 Done Done Sent false
10283 Acked Read 3 Done Done Sent false
10284 Acked Read 2 Done Done Sent false
10285 Acked Write 3 Done Done Sent false
10286 Acked Write 5 Done Done Sent false
10287 Acked Write 2 Done Done Sent false
10288 Acked Read 1 Done Done Sent false
10289 Acked Flush 0 Done Done Sent false
10290 Acked Read 10 Done Done Sent false
10291 Acked Read 10 Done Done Sent false
10292 Acked Read 8 Done Done Sent false
10293 Acked Read 9 Done Done Sent false
10294 Acked Write 1 Done Done Sent false
10295 Acked Flush 0 Done Done Sent false
10296 Acked Read 4 Done Done Sent false
10297 Acked Read 5 Done Done Sent false
10298 Acked Flush 0 Done Done Sent false
10299 Acked Write 2 Done Done Sent false
10300 Acked Write 7 Done Done Sent false
10301 Acked Write 5 Done Done Sent false
10302 Acked Read 8 Done Done Sent false
10303 Acked Write 2 Done Done Sent false
10304 Acked Read 5 Done Done Sent false
10305 Acked Write 4 Done Done Sent false
10306 Acked Flush 0 Done Done Sent false
10307 Acked Write 2 Done Done Sent false
10308 Acked Write 3 Done Done Sent false
10309 Acked Flush 0 Done Done Sent false
10310 Acked Read 8 Done Done Sent false
10311 Acked Read 4 Done Done Sent false
10312 Acked Read 8 Done Done Sent false
10313 Acked Read 9 Done Done Sent false
10314 Acked Write 1 Done Done Sent false
10315 Acked Read 9 Done Done Sent false
10316 Acked Write 1 Done Done Sent false
10317 Acked Read 2 Done Done Sent false
10318 Acked Read 3 Done Done Sent false
10319 Acked Read 9 Done Done Sent false
10320 Acked Flush 0 Done Done Sent false
10321 Acked Read 8 Done Done Sent false
10322 Acked Write 3 Done Done Sent false
10323 Acked Read 7 Done Done Sent false
10324 Acked Write 8 Done Done Sent false
10325 Acked Flush 0 Done Done Sent false
10326 Acked Read 10 Done Done Sent false
10327 Acked Write 7 Done Done Sent false
10328 Acked Write 8 Done Done Sent false
10329 Acked Read 1 Done Done Sent false
10330 Acked Read 3 Done Done Sent false
10331 Acked Read 4 Done Done Sent false
10332 Acked Read 3 Done Done Sent false
10333 Acked Read 9 Done Done Sent false
10334 Acked Write 6 Done Done Sent false
10335 Acked Read 7 Done Done Sent false
10336 Acked Read 1 Done Done Sent false
10337 Acked Flush 0 Done Done Sent false
10338 Acked Write 10 Done Done Sent false
10339 Acked Read 6 Done Done Sent false
10340 Acked Write 6 Done Done Sent false
10341 Acked Read 8 Done Done Sent false
10342 Acked Write 3 Done Done Sent false
10343 Acked Write 5 Done Done Sent false
10344 Acked Read 2 Done Done Sent false
10345 Acked Write 1 Done Done Sent false
10346 Acked Write 9 Done Done Sent false
10347 Acked Write 10 Done Done Sent false
10348 Acked Read 9 Done Done Sent false
10349 Acked Write 9 Done Done Sent false
10350 Acked Write 9 Done Done Sent false
10351 Acked Write 3 Done Done Sent false
10352 Acked Read 2 Done Done Sent false
10353 Acked Write 7 Done Done Sent false
10354 Acked Write 4 Done Done Sent false
10355 Acked Write 8 Done Done Sent false
10356 Acked Read 10 Done Done Sent false
10357 Acked Flush 0 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 302 302
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 10357 10357 10055
Downstairs last five completed: 10055 10054 10053 10052 10051
Upstairs last five completed: 10357 10356 10355 10354 10353
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.345848705Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38327)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.346116414Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: Offline }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347146164Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347688861Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347710678Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"send last flush ID to this DS: 10055","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347724986Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"Replied this last flush ID: 10055","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347957639Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"client":"2","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] client re-new 302 jobs since flush 10055","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.347980547Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
{"msg":"[2] Replayed 302 jobs since flush: 10055","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:38.348221299Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200,"":"downstairs","":"downstairs","session_id":"edf4bf43-2393-40de-98dd-2a3c00d4e1dd"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10357 10356 10355 10354 10353
Upstairs last five completed: 10357 10356 10355 10354 10353
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished
Test replay has completed
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10357 10356 10355 10354 10353
Upstairs last five completed: 10357 10356 10355 10354 10353
CLIENT: Up:0 ds:0 act:3
CLIENT: All jobs finished, all DS active.
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:40.589053374Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1200}
Replays:30 time: 7:16
Do final verify
Jan 27 19:43:40.677 INFO accepted connection, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
Jan 27 19:43:40.677 INFO request completed, latency_us: 209, response_code: 200, uri: /regioninfo, method: GET, req_id: 7d8b5577-b640-42d6-8b56-be4688cb1480, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
Jan 27 19:43:40.678 INFO request completed, latency_us:{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.678510355Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"task":"crutest"}
90, response_code: 200, uri: /regioncount, method: GET, req_id: 1a83dc71-810f-425f-811c-701907a30946, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.679061031Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"task":"crutest"}
Jan 27 19:43:40.679 INFO request completed, latency_us: 101, response_code: 200, uri: /port/cid/0, method: GET, req_id: 03b188d9-997e-4c55-b1ff-8766c8cdfdb5, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
Jan 27 19:43:40.679 INFO request completed, latency_us: 106, response_code: 200, uri: /port/cid/1, method: GET, req_id: a0b08fe8-f673-4155-b1a0-2c2ee2a7b1d1, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
Jan 27 19:43:40.680 INFO request completed, latency_us: 129, response_code: 200, uri: /port/cid/2, method: GET, req_id: 813ef9da-9ccc-4455-ac10-2986f590cff6, remote_addr: 127.0.0.1:39659, local_addr: 127.0.0.1:9998
{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68070205Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.680880909Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.680923474Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.680940212Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68095766Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
{"msg":"Crucible f799725d-62e8-4ece-8332-dabeb84162bf has session id: 066847eb-0730-4bf3-a1b4-47ac637a7749","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.681559979Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Upstairs opts: Upstairs UUID: f799725d-62e8-4ece-8332-dabeb84162bf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.681609714Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Crucible stats registered with UUID: f799725d-62e8-4ece-8332-dabeb84162bf","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68163998Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:40.681667117Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682452515Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"task":"crutest"}
{"msg":"The guest has requested activation with gen:3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682485782Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
{"msg":"f799725d-62e8-4ece-8332-dabeb84162bf active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682577041Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682612917Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682657531Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682688398Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682716775Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682740722Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682766349Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682790756Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682815263Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68284079Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40944)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682865277Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37755)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.682888764Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58030)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.683025139Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"io task","client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68352832Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.683559106Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.683574144Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.684174065Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68421376Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.684243417Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.684917528Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685044683Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[0]R flush_numbers[0..12]: [941, 930, 930, 941, 904, 941, 930, 911, 939, 933, 926, 941]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685079629Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685095137Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685109445Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[1]R flush_numbers[0..12]: [941, 930, 930, 941, 904, 941, 930, 911, 939, 933, 926, 941]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685123134Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685136492Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685150421Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[2]R flush_numbers[0..12]: [941, 930, 930, 941, 904, 941, 930, 911, 939, 933, 926, 941]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685163999Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685181437Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685195565Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685213223Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:43:40.685236881Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Next flush: 942","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685265387Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685283725Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"":"downstairs","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685303233Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.68532547Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"0","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685348928Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"1","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msgDisk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685377454Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"client":"2","":"downstairs","session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685399671Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685420619Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"f799725d-62e8-4ece-8332-dabeb84162bf is now active with session: 066847eb-0730-4bf3-a1b4-47ac637a7749","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685434207Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685445916Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236,"session_id":"066847eb-0730-4bf3-a1b4-47ac637a7749"}
{"msg":"The guest has finished waiting for activation with:3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:40.685457905Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
Loading write count information from file "/tmp/test_replay_verify.log"
Read and Verify all blocks (0..5000 range:false)
Wrote out file "/tmp/test_replay_verify.log"
Verify test completed
Wrote out file "/tmp/test_replay_verify.log"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Read 256 Done Done Done false
1001 Acked Read 256 Done Done Done false
1002 Acked Read 256 Done Done Done false
1003 Acked Read 256 Done Done Done false
1004 Acked Read 256 Done Done Done false
1005 Acked Read 256 Done Done Done false
1006 Acked Read 256 Done Done Done false
1007 Acked Read 256 Done Done Done false
1008 Acked Read 256 Done Done Done false
1009 Acked Read 256 Done Done Done false
1010 Acked Read 256 Done Done Done false
1011 Acked Read 256 Done Done Done false
1012 Acked Read 256 Done Done Done false
1013 Acked Read 256 Done Done Done false
1014 Acked Read 256 Done Done Done false
1015 Acked Read 256 Done Done Done false
1016 Acked Read 256 Done Done Done false
1017 Acked Read 256 Done Done Done false
1018 Acked Read 136 Done Done Done false
1019 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1019 1018 1017 1016 1015
CLIENT: Up:0 ds:20 act:3
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1020 1019 1018 1017 1016
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:45.093493906Z","hostname":"w-01JJMM51R1R6GNKMPF161F4D5M","pid":1236}
Jan 27 19:43:45.157 INFO accepted connection, remote_addr: 127.0.0.1:51249, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Shutdown
Shutdown
Jan 27 19:43:45.158 INFO request completed, latency_us: 161Shut it down
, [8810] Disable keep_running
response_code: 204, uri:[8810] Got stop action so:false kr:false
/shutdown, method: POST, req_id: 390aa63e-68c3-4fa3-b9c2-1de197ac803b, remote_addr: 127.0.0.1:51249, local_addr: 127.0.0.1:9998
[8820] Disable keep_running
[8820] Got stop action so:false kr:false
[8830] Disable keep_running
[8830] Got stop action so:false kr:false
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
[8810][0] reports Exit
January 27, 2025 at 07:43:50 PM UTC Test ends with 0