starting November 27, 2024 at 01:41:21 AM UTC Creating 3 downstairs regions Creating region directory at: /var/tmp/dsc/region Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2692403 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.021012528 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.021911448 Created 3 regions Starting 3 downstairs Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }) start ds: 8810 start access at:127.0.0.1:9998 start ds: 8820 start ds: 8830 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8820.txt" Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1193) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1194) [8820][1] initial start wait reports Running All downstairs are running Nov 27 01:41:21.851 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Running initial fill Nov 27 01:41:26.913 INFO accepted connection, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 Nov 27 01:41:26.915 INFO request completed, latency_us: 561, response_code: 200, uri: /regioninfo, method: GET, req_id: af99fc45-5167-482d-acc1-113908286cb3, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 Nov 27 01:41:26.917 INFO request completed, latency_us: 111, response_code: 200, uri: /regioncount, method: GET, req_id: 884e4227-4d3f-4d3d-b367-2431f9ecb940, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 {"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.916911116Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"task":"crutest"} {"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.917408622Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"task":"crutest"} Nov 27 01:41:26.917 INFO request completed, latency_us: 171, response_code: 200, uri: /port/cid/0, method: GET, req_id: 8c2d0ac9-7455-4c96-b732-6d2eb8b4e37b, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 Nov 27 01:41:26.918 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: 8ff65ff9-8bf2-4043-a7d0-c29653fb2b6e, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 Nov 27 01:41:26.918 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/2, method: GET, req_id: f23b1d2a-ac5b-48bd-9eec-c01d3fc8532d, remote_addr: 127.0.0.1:51292, local_addr: 127.0.0.1:9998 {"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.91920611Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.919732048Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"7dc0be16221dc0ce43144ff676801ae1255fa766\",\n git_commit_timestamp: \"2024-11-27T01:26:03.000000000Z\",\n git_branch: \"mkeeter/deactivate-need-flush\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.919781435Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.919804389Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.919824564Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} {"msg":"Crucible d2e0cd1f-8fb5-42a6-a08d-1e462f085d9c has session id: d88a5b6e-8f8e-4be3-8316-f4b25566aa21","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.920595727Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Upstairs opts: Upstairs UUID: d2e0cd1f-8fb5-42a6-a08d-1e462f085d9c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.920653001Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Crucible stats registered with UUID: d2e0cd1f-8fb5-42a6-a08d-1e462f085d9c","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.920670507Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955242046Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955307778Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} {"msg":"d2e0cd1f-8fb5-42a6-a08d-1e462f085d9c active request set","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955343688Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955407121Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955472634Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955500566Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955518121Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955542185Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955568248Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955589752Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955618474Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955664902Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_connection connected from Ok(127.0.0.1:45560)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955696214Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_connection connected from Ok(127.0.0.1:52681)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955723746Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_connection connected from Ok(127.0.0.1:62727)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.955748Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957859713Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957888855Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957911689Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957937702Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957957877Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.957978841Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.958894735Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.958954069Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959006395Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959851748Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959894566Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959912972Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959927058Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959942154Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.9599574Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959970636Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959984882Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.959999478Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960013914Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960029061Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960114958Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960135302Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960150328Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960164254Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.96017773Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960195026Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960209832Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960224378Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960238104Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960303817Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"":"downstairs","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960323631Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960337088Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"0","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960350844Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"1","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960364171Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"client":"2","":"downstairs","session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960378267Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960391193Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"d2e0cd1f-8fb5-42a6-a08d-1e462f085d9c is now active with session: d88a5b6e-8f8e-4be3-8316-f4b25566aa21","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.96040481Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","Activate successful pidWait for a query_work_queue command to finish before sending IO ":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960423055Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198,"session_id":"d88a5b6e-8f8e-4be3-8316-f4b25566aa21"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:26.960435701Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..5000 range:false) Wrote out file "/tmp/test_replay_verify.log" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1021 Acked Read 256 Done Done Done false 1022 Acked Read 256 Done Done Done false 1023 Acked Read 256 Done Done Done false 1024 Acked Read 256 Done Done Done false 1025 Acked Read 256 Done Done Done false 1026 Acked Read 256 Done Done Done false 1027 Acked Read 256 Done Done Done false 1028 Acked Read 256 Done Done Done false 1029 Acked Read 256 Done Done Done false 1030 Acked Read 256 Done Done Done false 1031 Acked Read 256 Done Done Done false 1032 Acked Read 256 Done Done Done false 1033 Acked Read 256 Done Done Done false 1034 Acked Read 256 Done Done Done false 1035 Acked Read 256 Done Done Done false 1036 Acked Read 256 Done Done Done false 1037 Acked Read 256 Done Done Done false 1038 Acked Read 256 Done Done Done false 1039 Acked Read 256 Done Done Done false 1040 Acked Read 136 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 20 20 20 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 1040 1039 1038 1037 1036 CLIENT: Up:0 ds:20 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1041 1040 1039 1038 1037 Upstairs last five completed: 1041 1040 1039 1038 1037 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:31.527373721Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1198} Replay loop starts now November 27, 2024 at 01:41:31 AM UTC Nov 27 01:41:31.602 INFO accepted connection, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 Nov 27 01:41:31.603 INFO request completed, latency_us: 158, response_code: 200, uri: /regioninfo, method: GET, req_id: 5b83d88c-2166-4db0-a72d-f463fc81e3bd, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 {"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.603969318Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"task":"crutest"} Nov 27 01:41:31.604 INFO request completed, latency_us: 97, response_code: 200, uri: /regioncount, method: GET, req_id: 3789992f-bdd0-4865-8f8d-87794fe3e954, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 {"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.604506614Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"task":"crutest"} Nov 27 01:41:31.604 INFO request completed, latency_us: 115, response_code: 200, uri: /port/cid/0, method: GET, req_id: 786c9497-c821-419c-8bb5-011c06515066, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 Nov 27 01:41:31.605 INFO request completed, latency_us: 93, response_code: 200, uri: /port/cid/1, method: GET, req_id: d8d0dee9-15df-46bf-aea7-bce9152db198, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 Nov 27 01:41:31.605 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/2, method: GET, req_id: f5cf7348-dc6a-47c7-b421-8a68a57ff2ed, remote_addr: 127.0.0.1:63353, local_addr: 127.0.0.1:9998 {"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606093937Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.60626901Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"7dc0be16221dc0ce43144ff676801ae1255fa766\",\n git_commit_timestamp: \"2024-11-27T01:26:03.000000000Z\",\n git_branch: \"mkeeter/deactivate-need-flush\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606298842Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606317337Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606332773Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} {"msg":"Crucible 0dd66b09-f555-4dbf-9293-7654bc03f3f7 has session id: 94dfdc19-74c7-449c-a147-e4ca5570ea17","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606746192Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Upstairs opts: Upstairs UUID: 0dd66b09-f555-4dbf-9293-7654bc03f3f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.60679129Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Crucible stats registered with UUID: 0dd66b09-f555-4dbf-9293-7654bc03f3f7","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.606811195Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641653731Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641682953Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} {"msg":"0dd66b09-f555-4dbf-9293-7654bc03f3f7 active request set","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641760513Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641781737Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.64180744Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641822566Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641837032Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641852428Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641878401Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641910312Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641936235Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.641964598Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:57354)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642043737Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:52520)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642086545Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:38011)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642319932Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642729462Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642791396Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642829635Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642876993Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642899097Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.642914213Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.643411839Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.643455847Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.643529128Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.64407809Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644105643Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644120229Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644213294Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644246105Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644271148Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644284765Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644298051Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644311788Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644325444Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.64433965Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644353546Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644366893Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644384718Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644410331Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30Activate successful Wait for a query_work_queue command to finish before sending IO ,"time":"2024-11-27T01:41:31.644442153Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644531299Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644551283Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.64456487Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644577716Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644590733Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644603869Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644615876Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644628883Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644641949Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644655165Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644667442Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"0dd66b09-f555-4dbf-9293-7654bc03f3f7 is now active with session: 94dfdc19-74c7-449c-a147-e4ca5570ea17","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644680279Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644692286Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.644704442Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202} Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Loading write count information from file "/tmp/test_replay_verify.log" Nov 27 01:41:31.678 INFO accepted connection, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:41:31.678 INFO request completed, latency_us: 170, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 8634d4ec-1e57-4d7f-9bb4-62d7ea0e976a, remote_addr: 127.0.0.1:[8810] Got stop action so:false kr:false 51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:31.679292567Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:31.679422162Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:31.679446485Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.679460682Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.679475797Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:41:31.679 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: 520eb9c4-4d06-43ea-b243-901e0a9fe462, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:31.679551567Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:41:35.680 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3ab8b971-1763-4285-a753-988ee5867d71, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 1044 len 24576 data: 2 2 2 2 2 2 002/300 Read block 1081 len 24576 003/300 Read block 1532 len 40960 004/300 Read block 932 len 28672 005/300 Write block 1165 len 16384 data: 2 2 2 2 006/300 Write block 3620 len 20480 data: 2 2 2 2 2 007/300 Read block 3957 len 28672 008/300 Write block 2244 len 36864 data: 2 2 2 2 2 2 2 2 2 009/300 Read block 1632 len 36864 010/300 Flush 011/300 Flush 012/300 Write block 2104 len 28672 data: 2 2 2 2 2 2 2 013/300 Flush 014/300 Write block 4719 len 32768 data: 2 2 2 2 2 2 2 2 015/300 Read block 3222 len 16384 016/300 Read block 981 len 24576 017/300 Read block 2142 len 28672 018/300 Read block 4815 len 32768 019/300 Read block 1060 len 4096 020/300 Flush 021/300 Write block 1406 len 28672 data: 2 2 2 2 2 2 2 022/300 Write block 4859 len 16384 data: 2 2 2 2 023/300 Write block 4753 len 40960 data: 2 2 2 2 2 2 2 2 2 2 024/300 Read block 2150 len 4096 025/300 Write block 2320 len 32768 data: 2 2 2 2 2 2 2 2 026/300 Read block 3584 len 8192 027/300 Write block 2961 len 28672 data: 2 2 2 2 2 2 2 028/300 Read block 948 len 28672 029/300 Read block 4332 len 40960 030/300 Read block 2937 len 24576 031/300 Read block 2390 len 8192 032/300 Read block 1242 len 24576 033/300 Write block 1644 len 36864 data: 2 2 2 2 2 2 2 2 2 034/300 Read block 4666 len 16384 035/300 Write block 3207 len 12288 data: 2 2 2 036/300 Write block 1991 len 12288 data: 2 2 2 037/300 Read block 2258 len 4096 038/300 Write block 4609 len 8192 data: 2 2 039/300 Read block 4169 len 16384 040/300 Read block 2316 len 24576 041/300 Read block 4649 len 16384 042/300 Read block 2312 len 4096 043/300 Write block 481 len 16384 data: 2 2 2 2 044/300 Write block 4607 len 4096 data: 2 045/300 Flush 046/300 Write block 4204 len 40960 data: 2 2 2 2 2 2 2 2 2 2 047/300 Read block 1783 len 4096 048/300 Read block 2087 len 24576 049/300 Read block 4474 len 32768 050/300 Write block 3675 len 36864 data: 2 2 2 2 2 2 2 2 2 051/300 Write block 414 len 4096 data: 2 052/300 Read block 2441 len 16384 053/300 Read block 2592 len 40960 054/300 Flush 055/300 Read block 4563 len 32768 056/300 Read block 4919 len 12288 057/300 Read block 1873 len 32768 058/300 Write block 3934 len 4096 data: 2 059/300 Flush 060/300 Write block 1284 len 28672 data: 2 2 2 2 2 2 2 061/300 Read block 804 len 16384 062/300 Write block 2859 len 16384 data: 2 2 2 2 063/300 Read block 2912 len 32768 064/300 Write block 2361 len 8192 data: 2 2 065/300 Write block 1996 len 24576 data: 2 2 2 2 2 2 066/300 Read block 3045 len 4096 067/300 Read block 1226 len 32768 068/300 Write block 3025 len 16384 data: 2 2 2 2 069/300 Read block 2917 len 32768 070/300 Write block 4466 len 12288 data: 2 2 2 071/300 Read block 2203 len 4096 072/300 Read block 6 len 4096 073/300 Read block 1043 len 32768 074/300 Write block 4577 len 36864 data: 2 2 2 2 2 2 2 2 2 075/300 Flush 076/300 Write block 783 len 12288 data: 2 2 2 077/300 Read block 1408 len 16384 078/300 Read block 2379 len 32768 079/300 Write block 2547 len 28672 data: 2 2 2 2 2 2 2 080/300 Read block 608 len 16384 081/300 Read block 4785 len 32768 082/300 Read block 1898 len 32768 083/300 Write block 1179 len 36864 data: 2 2 2 2 2 2 2 2 2 084/300 Write block 3729 len 40960 data: 2 2 2 2 2 2 2 2 2 2 085/300 Read block 2213 len 20480 086/300 Write block 4329 len 32768 data: 2 2 2 2 2 2 2 2 087/300 Write block 1092 len 16384 data: 2 2 2 2 088/300 Read block 4838 len 4096 089/300 Read block 3748 len 32768 090/300 Write block 3088 len 32768 data: 2 2 2 2 2 2 2 2 091/300 Read block 2203 len 40960 092/300 Read block 4172 len 12288 093/300 Write block 1714 len 20480 data: 2 2 2 2 2 094/300 Read block 3832 len 40960 095/300 Write block 4639 len 36864 data: 2 2 2 2 2 2 2 2 2 096/300 Flush 097/300 Read block 2218 len 36864 098/300 Read block 2168 len 24576 099/300 Read block 10 len 16384 100/300 Read block 3857 len 40960 101/300 Read block 464 len 36864 102/300 Read block 192 len 8192 103/300 Read block 4359 len 16384 104/300 Read block 2785 len 36864 105/300 Write block 4666 len 36864 data: 2 2 2 2 2 2 2 2 2 106/300 Write block 4484 len 4096 data: 2 107/300 Write block 1133 len 40960 data: 2 2 2 2 2 2 2 2 2 2 108/300 Read block 5 len 4096 109/300 Read block 630 len 16384 110/300 Write block 2333 len 20480 data: 2 2 2 2 2 111/300 Read block 3739 len 24576 112/300 Read block 2877 len 8192 113/300 Read block 2274 len 24576 114/300 Write block 4688 len 24576 data: 2 2 2 2 2 2 115/300 Write block 1824 len 24576 data: 2 2 2 2 2 2 116/300 Write block 3901 len 8192 data: 2 2 117/300 Read block 2612 len 20480 118/300 Flush 119/300 Read block 767 len 16384 120/300 Read block 1591 len 28672 121/300 Flush 122/300 Write block 29 len 28672 data: 2 2 2 2 2 2 2 123/300 Write block 933 len 8192 data: 2 2 124/300 Write block 3165 len 24576 data: 2 2 2 2 2 2 125/300 Read block 2246 len 4096 126/300 Read block 879 len 36864 127/300 Read block 1151 len 4096 128/300 Read block 4187 len 24576 129/300 Write block 3760 len 12288 data: 2 2 2 130/300 Read block 1201 len 20480 131/300 Write block 4152 len 12288 data: 2 2 2 132/300 Write block 3524 len 24576 data: 2 2 2 2 2 2 133/300 Read block 3309 len 36864 134/300 Read block 3657 len 8192 135/300 Write block 1903 len 20480 data: 2 2 2 2 2 136/300 Read block 1275 len 24576 137/300 Read block 4288 len 4096 138/300 Read block 3536 len 40960 139/300 Read block 1649 len 36864 140/300 Write block 905 len 4096 data: 2 141/300 Flush 142/300 Write block 797 len 40960 data: 2 2 2 2 2 2 2 2 2 2 143/300 Write block 2404 len 36864 data: 2 2 2 2 2 2 2 2 2 144/300 Read block 1565 len 12288 145/300 Write block 291 len 12288 data: 2 2 2 146/300 Read block 4589 len 28672 147/300 Write block 4478 len 32768 data: 2 2 2 2 2 2 3 2 148/300 Write block 1045 len 20480 data: 3 3 3 3 3 149/300 Read block 1124 len 40960 150/300 Read block 154 len 36864 151/300 Read block 3312 len 8192 152/300 Write block 669 len 12288 data: 2 2 2 153/300 Read block 2965 len 36864 154/300 Read block 1277 len 12288 155/300 Read block 352 len 4096 156/300 Write block 3699 len 32768 data: 2 2 2 2 2 2 2 2 157/300 Write block 3555 len 24576 data: 2 2 2 2 2 2 158/300 Read block 1213 len 4096 159/300 Read block 2569 len 40960 160/300 Read block 3102 len 40960 161/300 Read block 1134 len 8192 162/300 Write block 4388 len 8192 data: 2 2 163/300 Write block 4194 len 32768 data: 2 2 2 2 2 2 2 2 164/300 Read block 85 len 40960 165/300 Read block 4809 len 32768 166/300 Write block 992 len 40960 data: 2 2 2 2 2 2 2 2 2 2 167/300 Flush 168/300 Read block 903 len 32768 169/300 Flush 170/300 Read block 450 len 32768 171/300 Read block 3695 len 24576 172/300 Read block 4887 len 4096 173/300 Read block 2267 len 32768 174/300 Write block 4383 len 32768 data: 2 2 2 2 2 3 3 2 175/300 Write block 2459 len 40960 data: 2 2 2 2 2 2 2 2 2 2 176/300 Write block 1467 len 8192 data: 2 2 177/300 Read block 672 len 8192 178/300 Write block 1288 len 40960 data: 3 3 3 2 2 2 2 2 2 2 179/300 Read block 3413 len 20480 180/300 Flush 181/300 Flush 182/300 Flush 183/300 Read block 3622 len 36864 184/300 Read block 1073 len 20480 185/300 Read block 4103 len 8192 186/300 Write block 2849 len 32768 data: 2 2 2 2 2 2 2 2 187/300 Flush 188/300 Read block 3521 len 12288 189/300 Read block 3075 len 24576 190/300 Write block 65 len 32768 data: 2 2 2 2 2 2 2 2 191/300 Read block 4390 len 24576 192/300 Write block 4594 len 16384 data: 2 2 2 2 193/300 Write block 207 len 32768 data: 2 2 2 2 2 2 2 2 194/300 Read block 2239 len 40960 195/300 Read block 692 len 12288 196/300 Write block 3745 len 8192 data: 2 2 197/300 Write block 4842 len 20480 data: 2 2 2 2 2 198/300 Read block 4897 len 32768 199/300 Read block 2013 len 36864 200/300 Read block 340 len 32768 201/300 Read block 2455 len 28672 202/300 Read block 2511 len 36864 203/300 Read block 198 len 8192 204/300 Read block 313 len 32768 205/300 Read block 1882 len 36864 206/300 Write block 220 len 8192 data: 2 2 207/300 Write block 528 len 32768 data: 2 2 2 2 2 2 2 2 208/300 Flush 209/300 Read block 4871 len 12288 210/300 Read block 3670 len 16384 211/300 Read block 2934 len 32768 212/300 Flush 213/300 Read block 2317 len 12288 214/300 Read block 3902 len 40960 215/300 Write block 3141 len 4096 data: 2 216/300 Read block 802 len 32768 217/300 Read block 1870 len 12288 218/300 Write block 3653 len 24576 data: 2 2 2 2 2 2 219/300 Read block 2669 len 20480 220/300 Read block 234 len 16384 221/300 Flush 222/300 Read block 666 len 28672 223/300 Read block 1550 len 28672 224/300 Write block 2699 len 12288 data: 2 2 2 225/300 Read block 2118 len 32768 226/300 Read block 1958 len 40960 227/300 Read block 3960 len 36864 228/300 Read block 733 len 12288 229/300 Write block 1390 len 16384 data: 2 2 2 2 230/300 Write block 2138 len 28672 data: 2 2 2 2 2 2 2 231/300 Flush 232/300 Write block 2331 len 40960 data: 2 2 3 3 3 3 3 2 2 2 233/300 Read block 3407 len 28672 234/300 Write block 4219 len 8192 data: 2 2 235/300 Write block 1692 len 16384 data: 2 2 2 2 236/300 Read block 1219 len 4096 237/300 Write block 3005 len 12288 data: 2 2 2 238/300 Read block 1494 len 36864 239/300 Read block 4713 len 28672 240/300 Write block 3147 len 4096 data: 2 241/300 Read block 699 len 12288 242/300 Write block 108 len 24576 data: 2 2 2 2 2 2 243/300 Read block 2442 len 12288 244/300 Read block 33 len 20480 245/300 Write block 3770 len 36864 data: 2 2 2 2 2 2 2 2 2 246/300 Read block 3332 len 36864 247/300 Flush 248/300 Read block 581 len 4096 249/300 Read block 3423 len 40960 250/300 Write block 464 len 4096 data: 2 251/300 Read block 1345 len 16384 252/300 Write block 116 len 36864 data: 2 2 2 2 2 2 2 2 2 253/300 Flush 254/300 Write block 1003 len 40960 data: 2 2 2 2 2 2 2 2 2 2 255/300 Read block 1368 len 8192 256/300 Read block 4887 len 4096 257/300 Write block 1792 len 28672 data: 2 2 2 2 2 2 2 258/300 Write block 4657 len 8192 data: 2 2 259/300 Read block 334 len 4096 260/300 Read block 1225 len 16384 261/300 Write block 3154 len 24576 data: 2 2 2 2 2 2 262/300 Read block 911 len 28672 263/300 Write block 2735 len 36864 data: 2 2 2 2 2 2 2 2 2 264/300 Read block 971 len 28672 265/300 Write block 2814 len 36864 data: 2 2 2 2 2 2 2 2 2 266/300 Read block 159 len 20480 267/300 Write block 3119 len 8192 data: 2 2 268/300 Write block 1900 len 20480 data: 2 2 2 3 3 269/300 Read block 4419 len 32768 270/300 Read block 3619 len 4096 271/300 Read block 1880 len 4096 272/300 Read block 3156 len 24576 273/300 Write block 1814 len 40960 data: 2 2 2 2 2 2 2 2 2 2 274/300 Read block 128 len 28672 275/300 Write block 658 len 28672 data: 2 2 2 2 2 2 2 276/300 Read block 4032 len 20480 277/300 Read block 3995 len 36864 278/300 Read block 1262 len 28672 279/300 Read block 66 len 32768 280/300 Write block 507 len 24576 data: 2 2 2 2 2 2 281/300 Read block 3918 len 28672 282/300 Write block 4393 len 8192 data: 2 2 283/300 Read block 3685 len 28672 284/300 Flush 285/300 Write block 3905 len 20480 data: 2 2 2 2 2 286/300 Write block 4255 len 28672 data: 2 2 2 2 2 2 2 287/300 Write block 4286 len 28672 data: 2 2 2 2 2 2 2 288/300 Write block 4838 len 20480 data: 2 2 2 2 3 289/300 Write block 49 len 24576 data: 2 2 2 2 2 2 290/300 Read block 4152 len 16384 291/300 Write block 1962 len 28672 data: 2 2 2 2 2 2 2 292/300 Flush 293/300 Read block 3822 len 12288 294/300 Read block 283 len 40960 295/300 Read block 827 len 40960 296/300 Read block 2466 len 24576 297/300 Read block 708 len 40960 298/300 Read block 4451 len 40960 299/300 Read block 3597 len 36864 300/300 Write block 2345 len 32768 data: 2 2 2 2 2 2 2 2 301/300 Read block 4244 len 28672 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:41:35.932 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/0, method: POST, req_id: a503b2ca-dd2a-4fff-9cce-15b483026e02, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 6 Sent Done Done false 1030 Acked Read 2 Sent Done Done false 1031 Acked Read 6 Sent Done Done false 1032 Acked Write 9 Sent Done Done false 1033 Acked Read 4 Sent Done Done false 1034 Acked Write 3 Sent Done Done false 1035 Acked Write 3 Sent Done Done false 1036 Acked Read 1 Sent Done Done false 1037 Acked Write 2 Sent Done Done false 1038 Acked Read 4 Sent Done Done false 1039 Acked Read 6 Sent Done Done false 1040 Acked Read 4 Sent Done Done false 1041 Acked Read 1 Sent Done Done false 1042 Acked Write 4 Sent Done Done false 1043 Acked Write 1 Sent Done Done false 1044 Acked Flush 0 Sent Done Done false 1045 Acked Write 10 Sent Done Done false 1046 Acked Read 1 Sent Done Done false 1047 Acked Read 6 Sent Done Done false 1048 Acked Read 8 Sent Done Done false 1049 Acked Write 9 Sent Done Done false 1050 Acked Write 1 Sent Done Done false 1051 Acked Read 4 Sent Done Done false 1052 Acked Read 10 Sent Done Done false 1053 Acked Flush 0 Sent Done Done false 1054 Acked Read 8 Sent Done Done false 1055 Acked Read 3 Sent Done Done false 1056 Acked Read 8 Sent Done Done false 1057 Acked Write 1 Sent Done Done false 1058 Acked Flush 0 Sent Done Done false 1059 Acked Write 7 Sent Done Done false 1060 Acked Read 4 Sent Done Done false 1061 Acked Write 4 Sent Done Done false 1062 Acked Read 8 Sent Done Done false 1063 Acked Write 2 Sent Done Done false 1064 Acked Write 6 Sent Done Done false 1065 Acked Read 1 Sent Done Done false 1066 Acked Read 8 Sent Done Done false 1067 Acked Write 4 Sent Done Done false 1068 Acked Read 8 Sent Done Done false 1069 Acked Write 3 Sent Done Done false 1070 Acked Read 1 Sent Done Done false 1071 Acked Read 1 Sent Done Done false 1072 Acked Read 8 Sent Done Done false 1073 Acked Write 9 Sent Done Done false 1074 Acked Flush 0 Sent Done Done false 1075 Acked Write 3 Sent Done Done false 1076 Acked Read 4 Sent Done Done false 1077 Acked Read 8 Sent Done Done false 1078 Acked Write 7 Sent Done Done false 1079 Acked Read 4 Sent Done Done false 1080 Acked Read 8 Sent Done Done false 1081 Acked Read 8 Sent Done Done false 1082 Acked Write 9 Sent Done Done false 1083 Acked Write 10 Sent Done Done false 1084 Acked Read 5 Sent Done Done false 1085 Acked Write 8 Sent Done Done false 1086 Acked Write 4 Sent Done Done false 1087 Acked Read 1 Sent Done Done false 1088 Acked Read 8 Sent Done Done false 1089 Acked Write 8 Sent Done Done false 1090 Acked Read 10 Sent Done Done false 1091 Acked Read 3 Sent Done Done false 1092 Acked Write 5 Sent Done Done false 1093 Acked Read 10 Sent Done Done false 1094 Acked Write 9 Sent Done Done false 1095 Acked Flush 0 Sent Done Done false 1096 Acked Read 9 Sent Done Done false 1097 Acked Read 6 Sent Done Done false 1098 Acked Read 4 Sent Done Done false 1099 Acked Read 10 Sent Done Done false 1100 Acked Read 9 Sent Done Done false 1101 Acked Read 2 Sent Done Done false 1102 Acked Read 4 Sent Done Done false 1103 Acked Read 9 Sent Done Done false 1104 Acked Write 9 Sent Done Done false 1105 Acked Write 1 Sent Done Done false 1106 Acked Write 10 Sent Done Done false 1107 Acked Read 1 Sent Done Done false 1108 Acked Read 4 Sent Done Done false 1109 Acked Write 5 Sent Done Done false 1110 Acked Read 6 Sent Done Done false 1111 Acked Read 2 Sent Done Done false 1112 Acked Read 6 Sent Done Done false 1113 Acked Write 6 Sent Done Done false 1114 Acked Write 6 Sent Done Done false 1115 Acked Write 2 Sent Done Done false 1116 Acked Read 5 Sent Done Done false 1117 Acked Flush 0 Sent Done Done false 1118 Acked Read 4 Sent Done Done false 1119 Acked Read 7 Sent Done Done false 1120 Acked Flush 0 Sent Done Done false 1121 Acked Write 7 Sent Done Done false 1122 Acked Write 2 Sent Done Done false 1123 Acked Write 6 Sent Done Done false 1124 Acked Read 1 Sent Done Done false 1125 Acked Read 9 Sent Done Done false 1126 Acked Read 1 Sent Done Done false 1127 Acked Read 6 Sent Done Done false 1128 Acked Write 3 Sent Done Done false 1129 Acked Read 5 Sent Done Done false 1130 Acked Write 3 Sent Done Done false 1131 Acked Write 6 Sent Done Done false 1132 Acked Read 9 Sent Done Done false 1133 Acked Read 2 Sent Done Done false 1134 Acked Write 5 Sent Done Done false 1135 Acked Read 6 Sent Done Done false 1136 Acked Read 1 Sent Done Done false 1137 Acked Read 10 Sent Done Done false 1138 Acked Read 9 Sent Done Done false 1139 Acked Write 1 Sent Done Done false 1140 Acked Flush 0 Sent Done Done false 1141 Acked Write 10 Sent Done Done false 1142 Acked Write 9 Sent Done Done false 1143 Acked Read 3 Sent Done Done false 1144 Acked Write 3 Sent Done Done false 1145 Acked Read 7 Sent Done Done false 1146 Acked Write 8 Sent Done Done false 1147 Acked Write 5 Sent Done Done false 1148 Acked Read 10 Sent Done Done false 1149 Acked Read 9 Sent Done Done false 1150 Acked Read 2 Sent Done Done false 1151 Acked Write 3 Sent Done Done false 1152 Acked Read 9 Sent Done Done false 1153 Acked Read 3 Sent Done Done false 1154 Acked Read 1 Sent Done Done false 1155 Acked Write 8 Sent Done Done false 1156 Acked Write 6 Sent Done Done false 1157 Acked Read 1 Sent Done Done false 1158 Acked Read 10 Sent Done Done false 1159 Acked Read 10 Sent Done Done false 1160 Acked Read 2 Sent Done Done false 1161 Acked Write 2 Sent Done Done false 1162 Acked Write 8 Sent Done Done false 1163 Acked Read 10 Sent Done Done false 1164 Acked Read 8 Sent Done Done false 1165 Acked Write 10 Sent Done Done false 1166 Acked Flush 0 Sent Done Done false 1167 Acked Read 8 Sent Done Done false 1168 Acked Flush 0 Sent Done Done false 1169 Acked Read 8 Sent Done Done false 1170 Acked Read 6 Sent Done Done false 1171 Acked Read 1 Sent Done Done false 1172 Acked Read 8 Sent Done Done false 1173 Acked Write 8 Sent Done Done false 1174 Acked Write 10 Sent Done Done false 1175 Acked Write 2 Sent Done Done false 1176 Acked Read 2 Sent Done Done false 1177 Acked Write 10 Sent Done Done false 1178 Acked Read 5 Sent Done Done false 1179 Acked Flush 0 Sent Done Done false 1180 Acked Flush 0 Sent Done Done false 1181 Acked Flush 0 Sent Done Done false 1182 Acked Read 9 Sent Done Done false 1183 Acked Read 5 Sent Done Done false 1184 Acked Read 2 Sent Done Done false 1185 Acked Write 8 Sent Done Done false 1186 Acked Flush 0 Sent Done Done false 1187 Acked Read 3 Sent Done Done false 1188 Acked Read 6 Sent Done Done false 1189 Acked Write 8 Sent Done Done false 1190 Acked Read 6 Sent Done Done false 1191 Acked Write 4 Sent Done Done false 1192 Acked Write 8 Sent Done Done false 1193 Acked Read 10 Sent Done Done false 1194 Acked Read 3 Sent Done Done false 1195 Acked Write 2 Sent Done Done false 1196 Acked Write 5 Sent Done Done false 1197 Acked Read 8 Sent Done Done false 1198 Acked Read 9 Sent Done Done false 1199 Acked Read 8 Sent Done Done false 1200 Acked Read 7 Sent Done Done false 1201 Acked Read 9 Sent Done Done false 1202 Acked Read 2 Sent Done Done false 1203 Acked Read 8 Sent Done Done false 1204 Acked Read 9 Sent Done Done false 1205 Acked Write 2 Sent Done Done false 1206 Acked Write 8 Sent Done Done false 1207 Acked Flush 0 Sent Done Done false 1208 Acked Read 3 Sent Done Done false 1209 Acked Read 4 Sent Done Done false 1210 Acked Read 8 Sent Done Done false 1211 Acked Flush 0 Sent Done Done false 1212 Acked Read 3 Sent Done Done false 1213 Acked Read 10 Sent Done Done false 1214 Acked Write 1 Sent Done Done false 1215 Acked Read 8 Sent Done Done false 1216 Acked Read 3 Sent Done Done false 1217 Acked Write 6 Sent Done Done false 1218 Acked Read 5 Sent Done Done false 1219 Acked Read 4 Sent Done Done false 1220 Acked Flush 0 Sent Done Done false 1221 Acked Read 7 Sent Done Done false 1222 Acked Read 7 Sent Done Done false 1223 Acked Write 3 Sent Done Done false 1224 Acked Read 8 Sent Done Done false 1225 Acked Read 10 Sent Done Done false 1226 Acked Read 9 Sent Done Done false 1227 Acked Read 3 Sent Done Done false 1228 Acked Write 4 Sent Done Done false 1229 Acked Write 7 Sent Done Done false 1230 Acked Flush 0 Sent Done Done false 1231 Acked Write 10 Sent Done Done false 1232 Acked Read 7 Sent Done Done false 1233 Acked Write 2 Sent Done Done false 1234 Acked Write 4 Sent Done Done false 1235 Acked Read 1 Sent Done Done false 1236 Acked Write 3 Sent Done Done false 1237 Acked Read 9 Sent Done Done false 1238 Acked Read 7 Sent Done Done false 1239 Acked Write 1 Sent Done Done false 1240 Acked Read 3 Sent Done Done false 1241 Acked Write 6 Sent Done Done false 1242 Acked Read 3 Sent Done Done false 1243 Acked Read 5 Sent Done Done false 1244 Acked Write 9 Sent Done Done false 1245 Acked Read 9 Sent Done Done false 1246 Acked Flush 0 Sent Done Done false 1247 Acked Read 1 Sent Done Done false 1248 Acked Read 10 Sent Done Done false 1249 Acked Write 1 Sent Done Done false 1250 Acked Read 4 Sent Done Done false 1251 Acked Write 9 Sent Done Done false 1252 Acked Flush 0 Sent Done Done false 1253 Acked Write 10 Sent Done Done false 1254 Acked Read 2 Sent Done Done false 1255 Acked Read 1 Sent Done Done false 1256 Acked Write 7 Sent Done Done false 1257 Acked Write 2 Sent Done Done false 1258 Acked Read 1 Sent Done Done false 1259 Acked Read 4 Sent Done Done false 1260 Acked Write 6 Sent Done Done false 1261 Acked Read 7 Sent Done Done false 1262 Acked Write 9 Sent Done Done false 1263 Acked Read 7 Sent Done Done false 1264 Acked Write 9 Sent Done Done false 1265 Acked Read 5 Sent Done Done false 1266 Acked Write 2 Sent Done Done false 1267 Acked Write 5 Sent Done Done false 1268 Acked Read 8 Sent Done Done false 1269 Acked Read 1 Sent Done Done false 1270 Acked Read 1 Sent Done Done false 1271 Acked Read 6 Sent Done Done false 1272 Acked Write 10 Sent Done Done false 1273 Acked Read 7 Sent Done Done false 1274 Acked Write 7 Sent Done Done false 1275 Acked Read 5 Sent Done Done false 1276 Acked Read 9 Sent Done Done false 1277 Acked Read 7 Sent Done Done false 1278 Acked Read 8 Sent Done Done false 1279 Acked Write 6 Sent Done Done false 1280 Acked Read 7 Sent Done Done false 1281 Acked Write 2 Sent Done Done false 1282 Acked Read 7 Sent Done Done false 1283 Acked Flush 0 Sent Done Done false 1284 Acked Write 5 Sent Done Done false 1285 Acked Write 7 Sent Done Done false 1286 Acked Write 7 Sent Done Done false 1287 Acked Write 5 Sent Done Done false 1288 Acked Write 6 Sent Done Done false 1289 Acked Read 4 Sent Done Done false 1290 Acked Write 7 Sent Done Done false 1291 Acked Flush 0 Sent Done Done false 1292 Acked Read 3 Sent Done Done false 1293 Acked Read 10 Sent Done Done false 1294 Acked Read 10 Sent Done Done false 1295 Acked Read 6 Sent Done Done false 1296 Acked Read 10 Sent Done Done false 1297 Acked Read 10 Sent Done Done false 1298 Acked Read 9 Sent Done Done false 1299 Acked Write 8 Sent Done Done false 1300 Acked Read 7 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 1291 1291 Downstairs last five completed: Upstairs last five completed: 1300 1299 1298 1297 1296 CLIENT: Up:0 ds:301 act:2 [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1203) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 6 Sent Done Done false 1030 Acked Read 2 Sent Done Done false 1031 Acked Read 6 Sent Done Done false 1032 Acked Write 9 Sent Done Done false 1033 Acked Read 4 Sent Done Done false 1034 Acked Write 3 Sent Done Done false 1035 Acked Write 3 Sent Done Done false 1036 Acked Read 1 Sent Done Done false 1037 Acked Write 2 Sent Done Done false 1038 Acked Read 4 Sent Done Done false 1039 Acked Read 6 Sent Done Done false 1040 Acked Read 4 Sent Done Done false 1041 Acked Read 1 Sent Done Done false 1042 Acked Write 4 Sent Done Done false 1043 Acked Write 1 Sent Done Done false 1044 Acked Flush 0 Sent Done Done false 1045 Acked Write 10 Sent Done Done false 1046 Acked Read 1 Sent Done Done false 1047 Acked Read 6 Sent Done Done false 1048 Acked Read 8 Sent Done Done false 1049 Acked Write 9 Sent Done Done false 1050 Acked Write 1 Sent Done Done false 1051 Acked Read 4 Sent Done Done false 1052 Acked Read 10 Sent Done Done false 1053 Acked Flush 0 Sent Done Done false 1054 Acked Read 8 Sent Done Done false 1055 Acked Read 3 Sent Done Done false 1056 Acked Read 8 Sent Done Done false 1057 Acked Write 1 Sent Done Done false 1058 Acked Flush 0 Sent Done Done false 1059 Acked Write 7 Sent Done Done false 1060 Acked Read 4 Sent Done Done false 1061 Acked Write 4 Sent Done Done false 1062 Acked Read 8 Sent Done Done false 1063 Acked Write 2 Sent Done Done false 1064 Acked Write 6 Sent Done Done false 1065 Acked Read 1 Sent Done Done false 1066 Acked Read 8 Sent Done Done false 1067 Acked Write 4 Sent Done Done false 1068 Acked Read 8 Sent Done Done false 1069 Acked Write 3 Sent Done Done false 1070 Acked Read 1 Sent Done Done false 1071 Acked Read 1 Sent Done Done false 1072 Acked Read 8 Sent Done Done false 1073 Acked Write 9 Sent Done Done false 1074 Acked Flush 0 Sent Done Done false 1075 Acked Write 3 Sent Done Done false 1076 Acked Read 4 Sent Done Done false 1077 Acked Read 8 Sent Done Done false 1078 Acked Write 7 Sent Done Done false 1079 Acked Read 4 Sent Done Done false 1080 Acked Read 8 Sent Done Done false 1081 Acked Read 8 Sent Done Done false 1082 Acked Write 9 Sent Done Done false 1083 Acked Write 10 Sent Done Done false 1084 Acked Read 5 Sent Done Done false 1085 Acked Write 8 Sent Done Done false 1086 Acked Write 4 Sent Done Done false 1087 Acked Read 1 Sent Done Done false 1088 Acked Read 8 Sent Done Done false 1089 Acked Write 8 Sent Done Done false 1090 Acked Read 10 Sent Done Done false 1091 Acked Read 3 Sent Done Done false 1092 Acked Write 5 Sent Done Done false 1093 Acked Read 10 Sent Done Done false 1094 Acked Write 9 Sent Done Done false 1095 Acked Flush 0 Sent Done Done false 1096 Acked Read 9 Sent Done Done false 1097 Acked Read 6 Sent Done Done false 1098 Acked Read 4 Sent Done Done false 1099 Acked Read 10 Sent Done Done false 1100 Acked Read 9 Sent Done Done false 1101 Acked Read 2 Sent Done Done false 1102 Acked Read 4 Sent Done Done false 1103 Acked Read 9 Sent Done Done false 1104 Acked Write 9 Sent Done Done false 1105 Acked Write 1 Sent Done Done false 1106 Acked Write 10 Sent Done Done false 1107 Acked Read 1 Sent Done Done false 1108 Acked Read 4 Sent Done Done false 1109 Acked Write 5 Sent Done Done false 1110 Acked Read 6 Sent Done Done false 1111 Acked Read 2 Sent Done Done false 1112 Acked Read 6 Sent Done Done false 1113 Acked Write 6 Sent Done Done false 1114 Acked Write 6 Sent Done Done false 1115 Acked Write 2 Sent Done Done false 1116 Acked Read 5 Sent Done Done false 1117 Acked Flush 0 Sent Done Done false 1118 Acked Read 4 Sent Done Done false 1119 Acked Read 7 Sent Done Done false 1120 Acked Flush 0 Sent Done Done false 1121 Acked Write 7 Sent Done Done false 1122 Acked Write 2 Sent Done Done false 1123 Acked Write 6 Sent Done Done false 1124 Acked Read 1 Sent Done Done false 1125 Acked Read 9 Sent Done Done false 1126 Acked Read 1 Sent Done Done false 1127 Acked Read 6 Sent Done Done false 1128 Acked Write 3 Sent Done Done false 1129 Acked Read 5 Sent Done Done false 1130 Acked Write 3 Sent Done Done false 1131 Acked Write 6 Sent Done Done false 1132 Acked Read 9 Sent Done Done false 1133 Acked Read 2 Sent Done Done false 1134 Acked Write 5 Sent Done Done false 1135 Acked Read 6 Sent Done Done false 1136 Acked Read 1 Sent Done Done false 1137 Acked Read 10 Sent Done Done false 1138 Acked Read 9 Sent Done Done false 1139 Acked Write 1 Sent Done Done false 1140 Acked Flush 0 Sent Done Done false 1141 Acked Write 10 Sent Done Done false 1142 Acked Write 9 Sent Done Done false 1143 Acked Read 3 Sent Done Done false 1144 Acked Write 3 Sent Done Done false 1145 Acked Read 7 Sent Done Done false 1146 Acked Write 8 Sent Done Done false 1147 Acked Write 5 Sent Done Done false 1148 Acked Read 10 Sent Done Done false 1149 Acked Read 9 Sent Done Done false 1150 Acked Read 2 Sent Done Done false 1151 Acked Write 3 Sent Done Done false 1152 Acked Read 9 Sent Done Done false 1153 Acked Read 3 Sent Done Done false 1154 Acked Read 1 Sent Done Done false 1155 Acked Write 8 Sent Done Done false 1156 Acked Write 6 Sent Done Done false 1157 Acked Read 1 Sent Done Done false 1158 Acked Read 10 Sent Done Done false 1159 Acked Read 10 Sent Done Done false 1160 Acked Read 2 Sent Done Done false 1161 Acked Write 2 Sent Done Done false 1162 Acked Write 8 Sent Done Done false 1163 Acked Read 10 Sent Done Done false 1164 Acked Read 8 Sent Done Done false 1165 Acked Write 10 Sent Done Done false 1166 Acked Flush 0 Sent Done Done false 1167 Acked Read 8 Sent Done Done false 1168 Acked Flush 0 Sent Done Done false 1169 Acked Read 8 Sent Done Done false 1170 Acked Read 6 Sent Done Done false 1171 Acked Read 1 Sent Done Done false 1172 Acked Read 8 Sent Done Done false 1173 Acked Write 8 Sent Done Done false 1174 Acked Write 10 Sent Done Done false 1175 Acked Write 2 Sent Done Done false 1176 Acked Read 2 Sent Done Done false 1177 Acked Write 10 Sent Done Done false 1178 Acked Read 5 Sent Done Done false 1179 Acked Flush 0 Sent Done Done false 1180 Acked Flush 0 Sent Done Done false 1181 Acked Flush 0 Sent Done Done false 1182 Acked Read 9 Sent Done Done false 1183 Acked Read 5 Sent Done Done false 1184 Acked Read 2 Sent Done Done false 1185 Acked Write 8 Sent Done Done false 1186 Acked Flush 0 Sent Done Done false 1187 Acked Read 3 Sent Done Done false 1188 Acked Read 6 Sent Done Done false 1189 Acked Write 8 Sent Done Done false 1190 Acked Read 6 Sent Done Done false 1191 Acked Write 4 Sent Done Done false 1192 Acked Write 8 Sent Done Done false 1193 Acked Read 10 Sent Done Done false 1194 Acked Read 3 Sent Done Done false 1195 Acked Write 2 Sent Done Done false 1196 Acked Write 5 Sent Done Done false 1197 Acked Read 8 Sent Done Done false 1198 Acked Read 9 Sent Done Done false 1199 Acked Read 8 Sent Done Done false 1200 Acked Read 7 Sent Done Done false 1201 Acked Read 9 Sent Done Done false 1202 Acked Read 2 Sent Done Done false 1203 Acked Read 8 Sent Done Done false 1204 Acked Read 9 Sent Done Done false 1205 Acked Write 2 Sent Done Done false 1206 Acked Write 8 Sent Done Done false 1207 Acked Flush 0 Sent Done Done false 1208 Acked Read 3 Sent Done Done false 1209 Acked Read 4 Sent Done Done false 1210 Acked Read 8 Sent Done Done false 1211 Acked Flush 0 Sent Done Done false 1212 Acked Read 3 Sent Done Done false 1213 Acked Read 10 Sent Done Done false 1214 Acked Write 1 Sent Done Done false 1215 Acked Read 8 Sent Done Done false 1216 Acked Read 3 Sent Done Done false 1217 Acked Write 6 Sent Done Done false 1218 Acked Read 5 Sent Done Done false 1219 Acked Read 4 Sent Done Done false 1220 Acked Flush 0 Sent Done Done false 1221 Acked Read 7 Sent Done Done false 1222 Acked Read 7 Sent Done Done false 1223 Acked Write 3 Sent Done Done false 1224 Acked Read 8 Sent Done Done false 1225 Acked Read 10 Sent Done Done false 1226 Acked Read 9 Sent Done Done false 1227 Acked Read 3 Sent Done Done false 1228 Acked Write 4 Sent Done Done false 1229 Acked Write 7 Sent Done Done false 1230 Acked Flush 0 Sent Done Done false 1231 Acked Write 10 Sent Done Done false 1232 Acked Read 7 Sent Done Done false 1233 Acked Write 2 Sent Done Done false 1234 Acked Write 4 Sent Done Done false 1235 Acked Read 1 Sent Done Done false 1236 Acked Write 3 Sent Done Done false 1237 Acked Read 9 Sent Done Done false 1238 Acked Read 7 Sent Done Done false 1239 Acked Write 1 Sent Done Done false 1240 Acked Read 3 Sent Done Done false 1241 Acked Write 6 Sent Done Done false 1242 Acked Read 3 Sent Done Done false 1243 Acked Read 5 Sent Done Done false 1244 Acked Write 9 Sent Done Done false 1245 Acked Read 9 Sent Done Done false 1246 Acked Flush 0 Sent Done Done false 1247 Acked Read 1 Sent Done Done false 1248 Acked Read 10 Sent Done Done false 1249 Acked Write 1 Sent Done Done false 1250 Acked Read 4 Sent Done Done false 1251 Acked Write 9 Sent Done Done false 1252 Acked Flush 0 Sent Done Done false 1253 Acked Write 10 Sent Done Done false 1254 Acked Read 2 Sent Done Done false 1255 Acked Read 1 Sent Done Done false 1256 Acked Write 7 Sent Done Done false 1257 Acked Write 2 Sent Done Done false 1258 Acked Read 1 Sent Done Done false 1259 Acked Read 4 Sent Done Done false 1260 Acked Write 6 Sent Done Done false 1261 Acked Read 7 Sent Done Done false 1262 Acked Write 9 Sent Done Done false 1263 Acked Read 7 Sent Done Done false 1264 Acked Write 9 Sent Done Done false 1265 Acked Read 5 Sent Done Done false 1266 Acked Write 2 Sent Done Done false 1267 Acked Write 5 Sent Done Done false 1268 Acked Read 8 Sent Done Done false 1269 Acked Read 1 Sent Done Done false 1270 Acked Read 1 Sent Done Done false 1271 Acked Read 6 Sent Done Done false 1272 Acked Write 10 Sent Done Done false 1273 Acked Read 7 Sent Done Done false 1274 Acked Write 7 Sent Done Done false 1275 Acked Read 5 Sent Done Done false 1276 Acked Read 9 Sent Done Done false 1277 Acked Read 7 Sent Done Done false 1278 Acked Read 8 Sent Done Done false 1279 Acked Write 6 Sent Done Done false 1280 Acked Read 7 Sent Done Done false 1281 Acked Write 2 Sent Done Done false 1282 Acked Read 7 Sent Done Done false 1283 Acked Flush 0 Sent Done Done false 1284 Acked Write 5 Sent Done Done false 1285 Acked Write 7 Sent Done Done false 1286 Acked Write 7 Sent Done Done false 1287 Acked Write 5 Sent Done Done false 1288 Acked Write 6 Sent Done Done false 1289 Acked Read 4 Sent Done Done false 1290 Acked Write 7 Sent Done Done false 1291 Acked Flush 0 Sent Done Done false 1292 Acked Read 3 Sent Done Done false 1293 Acked Read 10 Sent Done Done false 1294 Acked Read 10 Sent Done Done false 1295 Acked Read 6 Sent Done Done false 1296 Acked Read 10 Sent Done Done false 1297 Acked Read 10 Sent Done Done false 1298 Acked Read 9 Sent Done Done false 1299 Acked Write 8 Sent Done Done false 1300 Acked Read 7 Sent Done Done false 1301 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 1301 1301 Downstairs last five completed: Upstairs last five completed: 1301 1300 1299 1298 1297 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.680238803Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:39243)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.680391882Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:41.681306116Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.681854919Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.681884701Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 0","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.681901506Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 0","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.682120567Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.682150399Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.682166575Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.682180921Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 0","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:41.682479171Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1301 1300 1299 1298 1297 Upstairs last five completed: 1301 1300 1299 1298 1297 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:41:43.936 INFO [8820] Got stop action so:false kr:false request completed, latency_us: 322, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 59144eb6-0539-4915-9adf-87fd4adeb14b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:43.937572743Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:43.937630768Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:43.937653022Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:43.937690192Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:41:43.937 {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:43.937708587Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/1, method: GET, req_id: f201a7b7-cc43-4988-9df5-142d9bfcae5b, remote_addr: 127.0.0.1:51788, local_addr: 127{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:43.937766171Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:41:47.940 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: d796acf2-b920-4fc7-96b9-8183308503f2, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 4913 len 28672 002/300 Write block 4241 len 24576 data: 2 2 2 2 2 2 003/300 Write block 14 len 32768 data: 2 2 2 2 2 2 2 2 004/300 Read block 720 len 24576 005/300 Read block 2953 len 16384 006/300 Read block 3847 len 8192 007/300 Write block 3730 len 16384 data: 3 3 3 3 008/300 Read block 3309 len 4096 009/300 Flush 010/300 Write block 2106 len 32768 data: 3 3 3 3 3 2 2 2 011/300 Write block 749 len 32768 data: 2 2 2 2 2 2 2 2 012/300 Read block 4675 len 36864 013/300 Read block 372 len 16384 014/300 Read block 2306 len 16384 015/300 Read block 4276 len 36864 016/300 Read block 3689 len 12288 017/300 Read block 3883 len 16384 018/300 Write block 406 len 24576 data: 2 2 2 2 2 2 019/300 Flush 020/300 Read block 3057 len 40960 021/300 Read block 3709 len 32768 022/300 Write block 2273 len 28672 data: 2 2 2 2 2 2 2 023/300 Read block 74 len 24576 024/300 Read block 2119 len 16384 025/300 Write block 272 len 16384 data: 2 2 2 2 026/300 Read block 4792 len 36864 027/300 Read block 4775 len 24576 028/300 Write block 4578 len 8192 data: 3 3 029/300 Write block 1883 len 36864 data: 2 2 2 2 2 2 2 2 2 030/300 Write block 3213 len 40960 data: 2 2 2 2 2 2 2 2 2 2 031/300 Write block 4730 len 32768 data: 2 2 2 2 2 2 2 2 032/300 Read block 3150 len 36864 033/300 Write block 2636 len 16384 data: 2 2 2 2 034/300 Write block 1507 len 36864 data: 2 2 2 2 2 2 2 2 2 035/300 Write block 253 len 20480 data: 2 2 2 2 2 036/300 Read block 3431 len 8192 037/300 Write block 3338 len 8192 data: 2 2 038/300 Write block 2541 len 28672 data: 2 2 2 2 2 2 3 039/300 Read block 1730 len 28672 040/300 Write block 873 len 36864 data: 2 2 2 2 2 2 2 2 2 041/300 Read block 4778 len 32768 042/300 Write block 2523 len 32768 data: 2 2 2 2 2 2 2 2 043/300 Write block 2437 len 40960 data: 2 2 2 2 2 2 2 2 2 2 044/300 Read block 3030 len 4096 045/300 Flush 046/300 Read block 1919 len 12288 047/300 Read block 2342 len 40960 048/300 Write block 3312 len 40960 data: 2 2 2 2 2 2 2 2 2 2 049/300 Flush 050/300 Read block 1513 len 16384 051/300 Read block 1854 len 36864 052/300 Write block 1170 len 16384 data: 2 2 2 2 053/300 Read block 3650 len 20480 054/300 Write block 4439 len 24576 data: 2 2 2 2 2 2 055/300 Read block 4807 len 36864 056/300 Write block 4857 len 28672 data: 2 2 3 3 3 3 2 057/300 Read block 2722 len 40960 058/300 Read block 2362 len 36864 059/300 Write block 3694 len 40960 data: 2 2 2 2 2 3 3 3 3 3 060/300 Read block 3515 len 40960 061/300 Write block 525 len 36864 data: 2 2 2 3 3 3 3 3 3 062/300 Read block 1914 len 16384 063/300 Write block 2424 len 12288 data: 2 2 2 064/300 Read block 610 len 36864 065/300 Read block 2592 len 28672 066/300 Write block 2350 len 8192 data: 3 3 067/300 Write block 4978 len 28672 data: 2 2 2 2 2 2 2 068/300 Write block 2241 len 36864 data: 2 2 2 3 3 3 3 3 3 069/300 Read block 1833 len 20480 070/300 Read block 4367 len 32768 071/300 Flush 072/300 Write block 1243 len 4096 data: 2 073/300 Read block 297 len 16384 074/300 Write block 3012 len 36864 data: 2 2 2 2 2 2 2 2 2 075/300 Read block 423 len 16384 076/300 Write block 4954 len 16384 data: 2 2 2 2 077/300 Write block 536 len 40960 data: 2 2 2 2 2 2 2 2 2 2 078/300 Write block 4399 len 20480 data: 2 2 2 2 2 079/300 Write block 3878 len 20480 data: 2 2 2 2 2 080/300 Write block 275 len 28672 data: 3 2 2 2 2 2 2 081/300 Flush 082/300 Write block 1619 len 4096 data: 2 083/300 Write block 3811 len 20480 data: 2 2 2 2 2 084/300 Read block 1843 len 32768 085/300 Read block 362 len 32768 086/300 Write block 2168 len 8192 data: 2 2 087/300 Write block 1721 len 36864 data: 2 2 2 2 2 2 2 2 2 088/300 Flush 089/300 Read block 3388 len 20480 090/300 Write block 2864 len 32768 data: 2 2 2 2 2 2 2 2 091/300 Write block 3489 len 12288 data: 2 2 2 092/300 Flush 093/300 Read block 1245 len 24576 094/300 Write block 1195 len 24576 data: 2 2 2 2 2 2 095/300 Read block 1829 len 40960 096/300 Write block 4707 len 32768 data: 2 2 2 2 2 2 2 2 097/300 Read block 769 len 8192 098/300 Write block 1082 len 16384 data: 2 2 2 2 099/300 Write block 3140 len 20480 data: 2 3 2 2 2 100/300 Read block 2297 len 8192 101/300 Read block 3886 len 36864 102/300 Read block 1172 len 20480 103/300 Read block 782 len 20480 104/300 Read block 3825 len 36864 105/300 Write block 4686 len 24576 data: 2 2 3 3 3 3 106/300 Read block 2378 len 32768 107/300 Write block 2467 len 20480 data: 3 3 2 2 2 108/300 Write block 467 len 28672 data: 2 2 2 2 2 2 2 109/300 Write block 3738 len 40960 data: 3 2 2 2 2 2 2 3 3 2 110/300 Write block 1573 len 20480 data: 2 2 2 2 2 111/300 Write block 1801 len 16384 data: 2 2 2 2 112/300 Read block 2705 len 20480 113/300 Read block 1006 len 40960 114/300 Write block 1672 len 20480 data: 2 2 2 2 2 115/300 Read block 4829 len 8192 116/300 Flush 117/300 Read block 3319 len 16384 118/300 Read block 1741 len 16384 119/300 Read block 2454 len 4096 120/300 Read block 2722 len 24576 121/300 Read block 2387 len 24576 122/300 Read block 4676 len 16384 123/300 Read block 4465 len 16384 124/300 Read block 999 len 24576 125/300 Write block 695 len 20480 data: 2 2 2 2 2 126/300 Read block 4892 len 28672 127/300 Flush 128/300 Write block 2142 len 16384 data: 3 3 3 2 129/300 Read block 241 len 16384 130/300 Read block 4858 len 32768 131/300 Write block 1133 len 32768 data: 3 3 3 3 3 3 3 3 132/300 Flush 133/300 Read block 878 len 28672 134/300 Write block 2453 len 20480 data: 2 2 2 2 2 135/300 Write block 1540 len 24576 data: 2 2 2 2 2 2 136/300 Write block 4928 len 20480 data: 2 2 2 2 2 137/300 Write block 4669 len 12288 data: 3 3 3 138/300 Write block 433 len 12288 data: 2 2 2 139/300 Read block 2478 len 24576 140/300 Read block 1294 len 8192 141/300 Read block 3301 len 40960 142/300 Read block 255 len 28672 143/300 Read block 337 len 24576 144/300 Write block 4512 len 28672 data: 2 2 2 2 2 2 2 145/300 Read block 1181 len 12288 146/300 Read block 2895 len 36864 147/300 Read block 1653 len 40960 148/300 Write block 4134 len 28672 data: 2 2 2 2 2 2 2 149/300 Flush 150/300 Read block 4975 len 36864 151/300 Read block 4043 len 40960 152/300 Write block 504 len 4096 data: 2 153/300 Read block 2762 len 12288 154/300 Write block 4841 len 24576 data: 3 4 3 3 3 3 155/300 Read block 1226 len 24576 156/300 Read block 3253 len 32768 157/300 Read block 2998 len 8192 158/300 Write block 1995 len 40960 data: 2 3 3 3 3 3 3 2 2 2 159/300 Flush 160/300 Read block 2904 len 4096 161/300 Write block 3406 len 28672 data: 2 2 2 2 2 2 2 162/300 Read block 3314 len 28672 163/300 Write block 1171 len 36864 data: 3 3 3 2 2 2 2 2 3 164/300 Write block 2098 len 32768 data: 2 2 2 2 2 2 3 3 165/300 Write block 2853 len 36864 data: 3 3 3 3 2 2 3 3 3 166/300 Flush 167/300 Read block 3109 len 4096 168/300 Read block 1832 len 28672 169/300 Read block 4725 len 24576 170/300 Write block 2043 len 28672 data: 2 2 2 2 2 2 2 171/300 Write block 4172 len 20480 data: 2 2 2 2 2 172/300 Read block 501 len 28672 173/300 Write block 3197 len 8192 data: 2 2 174/300 Read block 4561 len 4096 175/300 Read block 4442 len 4096 176/300 Read block 941 len 28672 177/300 Read block 4288 len 16384 178/300 Read block 2314 len 12288 179/300 Read block 2564 len 32768 180/300 Write block 1094 len 16384 data: 3 3 2 2 181/300 Write block 3019 len 20480 data: 3 3 2 2 2 182/300 Write block 2285 len 20480 data: 2 2 2 2 2 183/300 Write block 1500 len 24576 data: 2 2 2 2 2 2 184/300 Read block 1907 len 32768 185/300 Write block 3753 len 16384 data: 2 2 2 2 186/300 Write block 1539 len 20480 data: 2 3 3 3 3 187/300 Read block 2524 len 32768 188/300 Read block 3386 len 20480 189/300 Flush 190/300 Flush 191/300 Read block 4036 len 32768 192/300 Read block 389 len 36864 193/300 Flush 194/300 Flush 195/300 Write block 3233 len 40960 data: 2 2 2 2 2 2 2 2 2 2 196/300 Read block 3677 len 24576 197/300 Read block 463 len 32768 198/300 Write block 4260 len 32768 data: 3 3 2 2 2 2 2 2 199/300 Read block 4236 len 12288 200/300 Write block 4657 len 16384 data: 3 3 2 2 201/300 Read block 4801 len 4096 202/300 Write block 4331 len 12288 data: 3 3 3 203/300 Read block 461 len 28672 204/300 Read block 3534 len 24576 205/300 Read block 3759 len 32768 206/300 Write block 568 len 40960 data: 2 2 2 2 2 2 2 2 2 2 207/300 Write block 1798 len 32768 data: 3 2 2 3 3 3 3 2 208/300 Read block 1750 len 28672 209/300 Read block 1095 len 36864 210/300 Read block 2562 len 32768 211/300 Read block 3654 len 16384 212/300 Write block 1041 len 4096 data: 2 213/300 Read block 3966 len 8192 214/300 Read block 1614 len 28672 215/300 Read block 3345 len 16384 216/300 Read block 295 len 36864 217/300 Flush 218/300 Write block 3574 len 32768 data: 2 2 2 2 2 2 2 2 219/300 Write block 1907 len 16384 data: 3 2 2 2 220/300 Flush 221/300 Read block 1094 len 28672 222/300 Read block 4582 len 32768 223/300 Write block 2648 len 40960 data: 2 2 2 2 2 2 2 2 2 2 224/300 Read block 2945 len 16384 225/300 Write block 3014 len 12288 data: 3 3 3 226/300 Write block 251 len 28672 data: 2 2 3 3 3 3 3 227/300 Write block 1620 len 16384 data: 2 2 2 2 228/300 Write block 804 len 20480 data: 3 3 3 2 2 229/300 Read block 1036 len 40960 230/300 Write block 1438 len 8192 data: 2 2 231/300 Read block 1080 len 32768 232/300 Flush 233/300 Flush 234/300 Write block 153 len 20480 data: 2 2 2 2 2 235/300 Read block 36 len 4096 236/300 Write block 1085 len 20480 data: 3 2 2 2 2 237/300 Write block 1222 len 36864 data: 2 2 2 2 2 2 2 2 2 238/300 Read block 1974 len 4096 239/300 Write block 2305 len 40960 data: 2 2 2 2 2 2 2 2 2 2 240/300 Write block 2835 len 36864 data: 2 2 2 2 2 2 2 2 2 241/300 Read block 3092 len 28672 242/300 Read block 1472 len 28672 243/300 Write block 4658 len 28672 data: 4 3 3 2 2 2 2 244/300 Read block 2912 len 40960 245/300 Read block 3611 len 40960 246/300 Write block 4887 len 16384 data: 2 2 2 2 247/300 Read block 1022 len 28672 248/300 Write block 317 len 4096 data: 2 249/300 Write block 886 len 16384 data: 2 2 2 2 250/300 Read block 2122 len 4096 251/300 Write block 2044 len 8192 data: 3 3 252/300 Read block 2597 len 8192 253/300 Write block 1905 len 36864 data: 3 3 4 3 3 3 2 2 2 254/300 Write block 4356 len 16384 data: 2 2 2 2 255/300 Read block 4871 len 16384 256/300 Flush 257/300 Read block 2921 len 32768 258/300 Write block 1799 len 12288 data: 3 3 4 259/300 Read block 4737 len 24576 260/300 Read block 564 len 4096 261/300 Write block 1447 len 36864 data: 2 2 2 2 2 2 2 2 2 262/300 Read block 3865 len 32768 263/300 Read block 32 len 4096 264/300 Write block 3781 len 36864 data: 2 2 2 2 2 2 2 2 2 265/300 Read block 3059 len 28672 266/300 Write block 4763 len 24576 data: 2 2 2 2 2 2 267/300 Read block 4165 len 40960 268/300 Read block 4553 len 8192 269/300 Write block 2056 len 8192 data: 2 2 270/300 Write block 4422 len 32768 data: 2 2 2 2 2 2 2 2 271/300 Flush 272/300 Read block 3565 len 32768 273/300 Flush 274/300 Write block 2498 len 40960 data: 2 2 2 2 2 2 2 2 2 2 275/300 Write block 4708 len 32768 data: 3 3 3 3 3 3 3 2 276/300 Read block 1983 len 8192 277/300 Read block 3136 len 12288 278/300 Write block 4982 len 12288 data: 3 3 3 279/300 Read block 4670 len 12288 280/300 Write block 4950 len 28672 data: 2 2 2 2 3 3 3 281/300 Write block 3613 len 8192 data: 2 2 282/300 Read block 3564 len 4096 283/300 Write block 752 len 24576 data: 3 3 3 3 3 2 284/300 Read block 2171 len 40960 285/300 Flush 286/300 Read block 900 len 16384 287/300 Read block 4986 len 16384 288/300 Write block 3690 len 32768 data: 2 2 2 2 3 3 3 3 289/300 Read block 19 len 24576 290/300 Read block 3333 len 32768 291/300 Read block 4695 len 36864 292/300 Write block 2554 len 36864 data: 2 2 2 2 2 2 2 2 2 293/300 Read block 228 len 12288 294/300 Flush 295/300 Read block 3329 len 16384 296/300 Read block 4774 len 12288 297/300 Flush 298/300 Write block 4975 len 16384 data: 2 2 2 3 299/300 Flush 300/300 Write block 4678 len 20480 data: 2 2 2 2 2 301/300 Write block 1439 len 24576 data: 3 2 2 2 2 2 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:41:48.179 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 72a4db61-3751-4872-9fcb-678349100d64, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1302 Acked Read 7 Done Sent Done false 1303 Acked Write 6 Done Sent Done false 1304 Acked Write 8 Done Sent Done false 1305 Acked Read 6 Done Sent Done false 1306 Acked Read 4 Done Sent Done false 1307 Acked Read 2 Done Sent Done false 1308 Acked Write 4 Done Sent Done false 1309 Acked Read 1 Done Sent Done false 1310 Acked Flush 0 Done Sent Done false 1311 Acked Write 8 Done Sent Done false 1312 Acked Write 8 Done Sent Done false 1313 Acked Read 9 Done Sent Done false 1314 Acked Read 4 Done Sent Done false 1315 Acked Read 4 Done Sent Done false 1316 Acked Read 9 Done Sent Done false 1317 Acked Read 3 Done Sent Done false 1318 Acked Read 4 Done Sent Done false 1319 Acked Write 6 Done Sent Done false 1320 Acked Flush 0 Done Sent Done false 1321 Acked Read 10 Done Sent Done false 1322 Acked Read 8 Done Sent Done false 1323 Acked Write 7 Done Sent Done false 1324 Acked Read 6 Done Sent Done false 1325 Acked Read 4 Done Sent Done false 1326 Acked Write 4 Done Sent Done false 1327 Acked Read 9 Done Sent Done false 1328 Acked Read 6 Done Sent Done false 1329 Acked Write 2 Done Sent Done false 1330 Acked Write 9 Done Sent Done false 1331 Acked Write 10 Done Sent Done false 1332 Acked Write 8 Done Sent Done false 1333 Acked Read 9 Done Sent Done false 1334 Acked Write 4 Done Sent Done false 1335 Acked Write 9 Done Sent Done false 1336 Acked Write 5 Done Sent Done false 1337 Acked Read 2 Done Sent Done false 1338 Acked Write 2 Done Sent Done false 1339 Acked Write 7 Done Sent Done false 1340 Acked Read 7 Done Sent Done false 1341 Acked Write 9 Done Sent Done false 1342 Acked Read 8 Done Sent Done false 1343 Acked Write 8 Done Sent Done false 1344 Acked Write 10 Done Sent Done false 1345 Acked Read 1 Done Sent Done false 1346 Acked Flush 0 Done Sent Done false 1347 Acked Read 3 Done Sent Done false 1348 Acked Read 10 Done Sent Done false 1349 Acked Write 10 Done Sent Done false 1350 Acked Flush 0 Done Sent Done false 1351 Acked Read 4 Done Sent Done false 1352 Acked Read 9 Done Sent Done false 1353 Acked Write 4 Done Sent Done false 1354 Acked Read 5 Done Sent Done false 1355 Acked Write 6 Done Sent Done false 1356 Acked Read 9 Done Sent Done false 1357 Acked Write 7 Done Sent Done false 1358 Acked Read 10 Done Sent Done false 1359 Acked Read 9 Done Sent Done false 1360 Acked Write 10 Done Sent Done false 1361 Acked Read 10 Done Sent Done false 1362 Acked Write 9 Done Sent Done false 1363 Acked Read 4 Done Sent Done false 1364 Acked Write 3 Done Sent Done false 1365 Acked Read 9 Done Sent Done false 1366 Acked Read 7 Done Sent Done false 1367 Acked Write 2 Done Sent Done false 1368 Acked Write 7 Done Sent Done false 1369 Acked Write 9 Done Sent Done false 1370 Acked Read 5 Done Sent Done false 1371 Acked Read 8 Done Sent Done false 1372 Acked Flush 0 Done Sent Done false 1373 Acked Write 1 Done Sent Done false 1374 Acked Read 4 Done Sent Done false 1375 Acked Write 9 Done Sent Done false 1376 Acked Read 4 Done Sent Done false 1377 Acked Write 4 Done Sent Done false 1378 Acked Write 10 Done Sent Done false 1379 Acked Write 5 Done Sent Done false 1380 Acked Write 5 Done Sent Done false 1381 Acked Write 7 Done Sent Done false 1382 Acked Flush 0 Done Sent Done false 1383 Acked Write 1 Done Sent Done false 1384 Acked Write 5 Done Sent Done false 1385 Acked Read 8 Done Sent Done false 1386 Acked Read 8 Done Sent Done false 1387 Acked Write 2 Done Sent Done false 1388 Acked Write 9 Done Sent Done false 1389 Acked Flush 0 Done Sent Done false 1390 Acked Read 5 Done Sent Done false 1391 Acked Write 8 Done Sent Done false 1392 Acked Write 3 Done Sent Done false 1393 Acked Flush 0 Done Sent Done false 1394 Acked Read 6 Done Sent Done false 1395 Acked Write 6 Done Sent Done false 1396 Acked Read 10 Done Sent Done false 1397 Acked Write 8 Done Sent Done false 1398 Acked Read 2 Done Sent Done false 1399 Acked Write 4 Done Sent Done false 1400 Acked Write 5 Done Sent Done false 1401 Acked Read 2 Done Sent Done false 1402 Acked Read 9 Done Sent Done false 1403 Acked Read 5 Done Sent Done false 1404 Acked Read 5 Done Sent Done false 1405 Acked Read 9 Done Sent Done false 1406 Acked Write 6 Done Sent Done false 1407 Acked Read 8 Done Sent Done false 1408 Acked Write 5 Done Sent Done false 1409 Acked Write 7 Done Sent Done false 1410 Acked Write 10 Done Sent Done false 1411 Acked Write 5 Done Sent Done false 1412 Acked Write 4 Done Sent Done false 1413 Acked Read 5 Done Sent Done false 1414 Acked Read 10 Done Sent Done false 1415 Acked Write 5 Done Sent Done false 1416 Acked Read 2 Done Sent Done false 1417 Acked Flush 0 Done Sent Done false 1418 Acked Read 4 Done Sent Done false 1419 Acked Read 4 Done Sent Done false 1420 Acked Read 1 Done Sent Done false 1421 Acked Read 6 Done Sent Done false 1422 Acked Read 6 Done Sent Done false 1423 Acked Read 4 Done Sent Done false 1424 Acked Read 4 Done Sent Done false 1425 Acked Read 6 Done Sent Done false 1426 Acked Write 5 Done Sent Done false 1427 Acked Read 7 Done Sent Done false 1428 Acked Flush 0 Done Sent Done false 1429 Acked Write 4 Done Sent Done false 1430 Acked Read 4 Done Sent Done false 1431 Acked Read 8 Done Sent Done false 1432 Acked Write 8 Done Sent Done false 1433 Acked Flush 0 Done Sent Done false 1434 Acked Read 7 Done Sent Done false 1435 Acked Write 5 Done Sent Done false 1436 Acked Write 6 Done Sent Done false 1437 Acked Write 5 Done Sent Done false 1438 Acked Write 3 Done Sent Done false 1439 Acked Write 3 Done Sent Done false 1440 Acked Read 6 Done Sent Done false 1441 Acked Read 2 Done Sent Done false 1442 Acked Read 10 Done Sent Done false 1443 Acked Read 7 Done Sent Done false 1444 Acked Read 6 Done Sent Done false 1445 Acked Write 7 Done Sent Done false 1446 Acked Read 3 Done Sent Done false 1447 Acked Read 9 Done Sent Done false 1448 Acked Read 10 Done Sent Done false 1449 Acked Write 7 Done Sent Done false 1450 Acked Flush 0 Done Sent Done false 1451 Acked Read 9 Done Sent Done false 1452 Acked Read 10 Done Sent Done false 1453 Acked Write 1 Done Sent Done false 1454 Acked Read 3 Done Sent Done false 1455 Acked Write 6 Done Sent Done false 1456 Acked Read 6 Done Sent Done false 1457 Acked Read 8 Done Sent Done false 1458 Acked Read 2 Done Sent Done false 1459 Acked Write 10 Done Sent Done false 1460 Acked Flush 0 Done Sent Done false 1461 Acked Read 1 Done Sent Done false 1462 Acked Write 7 Done Sent Done false 1463 Acked Read 7 Done Sent Done false 1464 Acked Write 9 Done Sent Done false 1465 Acked Write 8 Done Sent Done false 1466 Acked Write 9 Done Sent Done false 1467 Acked Flush 0 Done Sent Done false 1468 Acked Read 1 Done Sent Done false 1469 Acked Read 7 Done Sent Done false 1470 Acked Read 6 Done Sent Done false 1471 Acked Write 7 Done Sent Done false 1472 Acked Write 5 Done Sent Done false 1473 Acked Read 7 Done Sent Done false 1474 Acked Write 2 Done Sent Done false 1475 Acked Read 1 Done Sent Done false 1476 Acked Read 1 Done Sent Done false 1477 Acked Read 7 Done Sent Done false 1478 Acked Read 4 Done Sent Done false 1479 Acked Read 3 Done Sent Done false 1480 Acked Read 8 Done Sent Done false 1481 Acked Write 4 Done Sent Done false 1482 Acked Write 5 Done Sent Done false 1483 Acked Write 5 Done Sent Done false 1484 Acked Write 6 Done Sent Done false 1485 Acked Read 8 Done Sent Done false 1486 Acked Write 4 Done Sent Done false 1487 Acked Write 5 Done Sent Done false 1488 Acked Read 8 Done Sent Done false 1489 Acked Read 5 Done Sent Done false 1490 Acked Flush 0 Done Sent Done false 1491 Acked Flush 0 Done Sent Done false 1492 Acked Read 8 Done Sent Done false 1493 Acked Read 9 Done Sent Done false 1494 Acked Flush 0 Done Sent Done false 1495 Acked Flush 0 Done Sent Done false 1496 Acked Write 10 Done Sent Done false 1497 Acked Read 6 Done Sent Done false 1498 Acked Read 8 Done Sent Done false 1499 Acked Write 8 Done Sent Done false 1500 Acked Read 3 Done Sent Done false 1501 Acked Write 4 Done Sent Done false 1502 Acked Read 1 Done Sent Done false 1503 Acked Write 3 Done Sent Done false 1504 Acked Read 7 Done Sent Done false 1505 Acked Read 6 Done Sent Done false 1506 Acked Read 8 Done Sent Done false 1507 Acked Write 10 Done Sent Done false 1508 Acked Write 8 Done Sent Done false 1509 Acked Read 7 Done Sent Done false 1510 Acked Read 9 Done Sent Done false 1511 Acked Read 8 Done Sent Done false 1512 Acked Read 4 Done Sent Done false 1513 Acked Write 1 Done Sent Done false 1514 Acked Read 2 Done Sent Done false 1515 Acked Read 7 Done Sent Done false 1516 Acked Read 4 Done Sent Done false 1517 Acked Read 9 Done Sent Done false 1518 Acked Flush 0 Done Sent Done false 1519 Acked Write 8 Done Sent Done false 1520 Acked Write 4 Done Sent Done false 1521 Acked Flush 0 Done Sent Done false 1522 Acked Read 7 Done Sent Done false 1523 Acked Read 8 Done Sent Done false 1524 Acked Write 10 Done Sent Done false 1525 Acked Read 4 Done Sent Done false 1526 Acked Write 3 Done Sent Done false 1527 Acked Write 7 Done Sent Done false 1528 Acked Write 4 Done Sent Done false 1529 Acked Write 5 Done Sent Done false 1530 Acked Read 10 Done Sent Done false 1531 Acked Write 2 Done Sent Done false 1532 Acked Read 8 Done Sent Done false 1533 Acked Flush 0 Done Sent Done false 1534 Acked Flush 0 Done Sent Done false 1535 Acked Write 5 Done Sent Done false 1536 Acked Read 1 Done Sent Done false 1537 Acked Write 5 Done Sent Done false 1538 Acked Write 9 Done Sent Done false 1539 Acked Read 1 Done Sent Done false 1540 Acked Write 10 Done Sent Done false 1541 Acked Write 9 Done Sent Done false 1542 Acked Read 7 Done Sent Done false 1543 Acked Read 7 Done Sent Done false 1544 Acked Write 7 Done Sent Done false 1545 Acked Read 10 Done Sent Done false 1546 Acked Read 10 Done Sent Done false 1547 Acked Write 4 Done Sent Done false 1548 Acked Read 7 Done Sent Done false 1549 Acked Write 1 Done Sent Done false 1550 Acked Write 4 Done Sent Done false 1551 Acked Read 1 Done Sent Done false 1552 Acked Write 2 Done Sent Done false 1553 Acked Read 2 Done Sent Done false 1554 Acked Write 9 Done Sent Done false 1555 Acked Write 4 Done Sent Done false 1556 Acked Read 4 Done Sent Done false 1557 Acked Flush 0 Done Sent Done false 1558 Acked Read 8 Done Sent Done false 1559 Acked Write 3 Done Sent Done false 1560 Acked Read 6 Done Sent Done false 1561 Acked Read 1 Done Sent Done false 1562 Acked Write 9 Done Sent Done false 1563 Acked Read 8 Done Sent Done false 1564 Acked Read 1 Done Sent Done false 1565 Acked Write 9 Done Sent Done false 1566 Acked Read 7 Done Sent Done false 1567 Acked Write 6 Done Sent Done false 1568 Acked Read 10 Done Sent Done false 1569 Acked Read 2 Done Sent Done false 1570 Acked Write 2 Done Sent Done false 1571 Acked Write 8 Done Sent Done false 1572 Acked Flush 0 Done Sent Done false 1573 Acked Read 8 Done Sent Done false 1574 Acked Flush 0 Done Sent Done false 1575 Acked Write 10 Done Sent Done false 1576 Acked Write 8 Done Sent Done false 1577 Acked Read 2 Done Sent Done false 1578 Acked Read 3 Done Sent Done false 1579 Acked Write 3 Done Sent Done false 1580 Acked Read 3 Done Sent Done false 1581 Acked Write 7 Done Sent Done false 1582 Acked Write 2 Done Sent Done false 1583 Acked Read 1 Done Sent Done false 1584 Acked Write 6 Done Sent Done false 1585 Acked Read 10 Done Sent Done false 1586 Acked Flush 0 Done Sent Done false 1587 Acked Read 4 Done Sent Done false 1588 Acked Read 4 Done Sent Done false 1589 Acked Write 8 Done Sent Done false 1590 Acked Read 6 Done Sent Done false 1591 Acked Read 8 Done Sent Done false 1592 Acked Read 9 Done Sent Done false 1593 Acked Write 9 Done Sent Done false 1594 Acked Read 3 Done Sent Done false 1595 Acked Flush 0 Done Sent Done false 1596 Acked Read 4 Done Sent Done false 1597 Acked Read 3 Done Sent Done false 1598 Acked Flush 0 Done Sent Done false 1599 Acked Write 4 Done Sent Done false 1600 Acked Flush 0 Done Sent Done false 1601 Acked Write 5 Done Sent Done false 1602 Acked Write 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1600 1301 1600 Downstairs last five completed: 1301 1300 1299 1298 1297 Upstairs last five completed: 1602 1601 1600 1599 1598 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1204) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1302 Acked Read 7 Done Sent Done false 1303 Acked Write 6 Done Sent Done false 1304 Acked Write 8 Done Sent Done false 1305 Acked Read 6 Done Sent Done false 1306 Acked Read 4 Done Sent Done false 1307 Acked Read 2 Done Sent Done false 1308 Acked Write 4 Done Sent Done false 1309 Acked Read 1 Done Sent Done false 1310 Acked Flush 0 Done Sent Done false 1311 Acked Write 8 Done Sent Done false 1312 Acked Write 8 Done Sent Done false 1313 Acked Read 9 Done Sent Done false 1314 Acked Read 4 Done Sent Done false 1315 Acked Read 4 Done Sent Done false 1316 Acked Read 9 Done Sent Done false 1317 Acked Read 3 Done Sent Done false 1318 Acked Read 4 Done Sent Done false 1319 Acked Write 6 Done Sent Done false 1320 Acked Flush 0 Done Sent Done false 1321 Acked Read 10 Done Sent Done false 1322 Acked Read 8 Done Sent Done false 1323 Acked Write 7 Done Sent Done false 1324 Acked Read 6 Done Sent Done false 1325 Acked Read 4 Done Sent Done false 1326 Acked Write 4 Done Sent Done false 1327 Acked Read 9 Done Sent Done false 1328 Acked Read 6 Done Sent Done false 1329 Acked Write 2 Done Sent Done false 1330 Acked Write 9 Done Sent Done false 1331 Acked Write 10 Done Sent Done false 1332 Acked Write 8 Done Sent Done false 1333 Acked Read 9 Done Sent Done false 1334 Acked Write 4 Done Sent Done false 1335 Acked Write 9 Done Sent Done false 1336 Acked Write 5 Done Sent Done false 1337 Acked Read 2 Done Sent Done false 1338 Acked Write 2 Done Sent Done false 1339 Acked Write 7 Done Sent Done false 1340 Acked Read 7 Done Sent Done false 1341 Acked Write 9 Done Sent Done false 1342 Acked Read 8 Done Sent Done false 1343 Acked Write 8 Done Sent Done false 1344 Acked Write 10 Done Sent Done false 1345 Acked Read 1 Done Sent Done false 1346 Acked Flush 0 Done Sent Done false 1347 Acked Read 3 Done Sent Done false 1348 Acked Read 10 Done Sent Done false 1349 Acked Write 10 Done Sent Done false 1350 Acked Flush 0 Done Sent Done false 1351 Acked Read 4 Done Sent Done false 1352 Acked Read 9 Done Sent Done false 1353 Acked Write 4 Done Sent Done false 1354 Acked Read 5 Done Sent Done false 1355 Acked Write 6 Done Sent Done false 1356 Acked Read 9 Done Sent Done false 1357 Acked Write 7 Done Sent Done false 1358 Acked Read 10 Done Sent Done false 1359 Acked Read 9 Done Sent Done false 1360 Acked Write 10 Done Sent Done false 1361 Acked Read 10 Done Sent Done false 1362 Acked Write 9 Done Sent Done false 1363 Acked Read 4 Done Sent Done false 1364 Acked Write 3 Done Sent Done false 1365 Acked Read 9 Done Sent Done false 1366 Acked Read 7 Done Sent Done false 1367 Acked Write 2 Done Sent Done false 1368 Acked Write 7 Done Sent Done false 1369 Acked Write 9 Done Sent Done false 1370 Acked Read 5 Done Sent Done false 1371 Acked Read 8 Done Sent Done false 1372 Acked Flush 0 Done Sent Done false 1373 Acked Write 1 Done Sent Done false 1374 Acked Read 4 Done Sent Done false 1375 Acked Write 9 Done Sent Done false 1376 Acked Read 4 Done Sent Done false 1377 Acked Write 4 Done Sent Done false 1378 Acked Write 10 Done Sent Done false 1379 Acked Write 5 Done Sent Done false 1380 Acked Write 5 Done Sent Done false 1381 Acked Write 7 Done Sent Done false 1382 Acked Flush 0 Done Sent Done false 1383 Acked Write 1 Done Sent Done false 1384 Acked Write 5 Done Sent Done false 1385 Acked Read 8 Done Sent Done false 1386 Acked Read 8 Done Sent Done false 1387 Acked Write 2 Done Sent Done false 1388 Acked Write 9 Done Sent Done false 1389 Acked Flush 0 Done Sent Done false 1390 Acked Read 5 Done Sent Done false 1391 Acked Write 8 Done Sent Done false 1392 Acked Write 3 Done Sent Done false 1393 Acked Flush 0 Done Sent Done false 1394 Acked Read 6 Done Sent Done false 1395 Acked Write 6 Done Sent Done false 1396 Acked Read 10 Done Sent Done false 1397 Acked Write 8 Done Sent Done false 1398 Acked Read 2 Done Sent Done false 1399 Acked Write 4 Done Sent Done false 1400 Acked Write 5 Done Sent Done false 1401 Acked Read 2 Done Sent Done false 1402 Acked Read 9 Done Sent Done false 1403 Acked Read 5 Done Sent Done false 1404 Acked Read 5 Done Sent Done false 1405 Acked Read 9 Done Sent Done false 1406 Acked Write 6 Done Sent Done false 1407 Acked Read 8 Done Sent Done false 1408 Acked Write 5 Done Sent Done false 1409 Acked Write 7 Done Sent Done false 1410 Acked Write 10 Done Sent Done false 1411 Acked Write 5 Done Sent Done false 1412 Acked Write 4 Done Sent Done false 1413 Acked Read 5 Done Sent Done false 1414 Acked Read 10 Done Sent Done false 1415 Acked Write 5 Done Sent Done false 1416 Acked Read 2 Done Sent Done false 1417 Acked Flush 0 Done Sent Done false 1418 Acked Read 4 Done Sent Done false 1419 Acked Read 4 Done Sent Done false 1420 Acked Read 1 Done Sent Done false 1421 Acked Read 6 Done Sent Done false 1422 Acked Read 6 Done Sent Done false 1423 Acked Read 4 Done Sent Done false 1424 Acked Read 4 Done Sent Done false 1425 Acked Read 6 Done Sent Done false 1426 Acked Write 5 Done Sent Done false 1427 Acked Read 7 Done Sent Done false 1428 Acked Flush 0 Done Sent Done false 1429 Acked Write 4 Done Sent Done false 1430 Acked Read 4 Done Sent Done false 1431 Acked Read 8 Done Sent Done false 1432 Acked Write 8 Done Sent Done false 1433 Acked Flush 0 Done Sent Done false 1434 Acked Read 7 Done Sent Done false 1435 Acked Write 5 Done Sent Done false 1436 Acked Write 6 Done Sent Done false 1437 Acked Write 5 Done Sent Done false 1438 Acked Write 3 Done Sent Done false 1439 Acked Write 3 Done Sent Done false 1440 Acked Read 6 Done Sent Done false 1441 Acked Read 2 Done Sent Done false 1442 Acked Read 10 Done Sent Done false 1443 Acked Read 7 Done Sent Done false 1444 Acked Read 6 Done Sent Done false 1445 Acked Write 7 Done Sent Done false 1446 Acked Read 3 Done Sent Done false 1447 Acked Read 9 Done Sent Done false 1448 Acked Read 10 Done Sent Done false 1449 Acked Write 7 Done Sent Done false 1450 Acked Flush 0 Done Sent Done false 1451 Acked Read 9 Done Sent Done false 1452 Acked Read 10 Done Sent Done false 1453 Acked Write 1 Done Sent Done false 1454 Acked Read 3 Done Sent Done false 1455 Acked Write 6 Done Sent Done false 1456 Acked Read 6 Done Sent Done false 1457 Acked Read 8 Done Sent Done false 1458 Acked Read 2 Done Sent Done false 1459 Acked Write 10 Done Sent Done false 1460 Acked Flush 0 Done Sent Done false 1461 Acked Read 1 Done Sent Done false 1462 Acked Write 7 Done Sent Done false 1463 Acked Read 7 Done Sent Done false 1464 Acked Write 9 Done Sent Done false 1465 Acked Write 8 Done Sent Done false 1466 Acked Write 9 Done Sent Done false 1467 Acked Flush 0 Done Sent Done false 1468 Acked Read 1 Done Sent Done false 1469 Acked Read 7 Done Sent Done false 1470 Acked Read 6 Done Sent Done false 1471 Acked Write 7 Done Sent Done false 1472 Acked Write 5 Done Sent Done false 1473 Acked Read 7 Done Sent Done false 1474 Acked Write 2 Done Sent Done false 1475 Acked Read 1 Done Sent Done false 1476 Acked Read 1 Done Sent Done false 1477 Acked Read 7 Done Sent Done false 1478 Acked Read 4 Done Sent Done false 1479 Acked Read 3 Done Sent Done false 1480 Acked Read 8 Done Sent Done false 1481 Acked Write 4 Done Sent Done false 1482 Acked Write 5 Done Sent Done false 1483 Acked Write 5 Done Sent Done false 1484 Acked Write 6 Done Sent Done false 1485 Acked Read 8 Done Sent Done false 1486 Acked Write 4 Done Sent Done false 1487 Acked Write 5 Done Sent Done false 1488 Acked Read 8 Done Sent Done false 1489 Acked Read 5 Done Sent Done false 1490 Acked Flush 0 Done Sent Done false 1491 Acked Flush 0 Done Sent Done false 1492 Acked Read 8 Done Sent Done false 1493 Acked Read 9 Done Sent Done false 1494 Acked Flush 0 Done Sent Done false 1495 Acked Flush 0 Done Sent Done false 1496 Acked Write 10 Done Sent Done false 1497 Acked Read 6 Done Sent Done false 1498 Acked Read 8 Done Sent Done false 1499 Acked Write 8 Done Sent Done false 1500 Acked Read 3 Done Sent Done false 1501 Acked Write 4 Done Sent Done false 1502 Acked Read 1 Done Sent Done false 1503 Acked Write 3 Done Sent Done false 1504 Acked Read 7 Done Sent Done false 1505 Acked Read 6 Done Sent Done false 1506 Acked Read 8 Done Sent Done false 1507 Acked Write 10 Done Sent Done false 1508 Acked Write 8 Done Sent Done false 1509 Acked Read 7 Done Sent Done false 1510 Acked Read 9 Done Sent Done false 1511 Acked Read 8 Done Sent Done false 1512 Acked Read 4 Done Sent Done false 1513 Acked Write 1 Done Sent Done false 1514 Acked Read 2 Done Sent Done false 1515 Acked Read 7 Done Sent Done false 1516 Acked Read 4 Done Sent Done false 1517 Acked Read 9 Done Sent Done false 1518 Acked Flush 0 Done Sent Done false 1519 Acked Write 8 Done Sent Done false 1520 Acked Write 4 Done Sent Done false 1521 Acked Flush 0 Done Sent Done false 1522 Acked Read 7 Done Sent Done false 1523 Acked Read 8 Done Sent Done false 1524 Acked Write 10 Done Sent Done false 1525 Acked Read 4 Done Sent Done false 1526 Acked Write 3 Done Sent Done false 1527 Acked Write 7 Done Sent Done false 1528 Acked Write 4 Done Sent Done false 1529 Acked Write 5 Done Sent Done false 1530 Acked Read 10 Done Sent Done false 1531 Acked Write 2 Done Sent Done false 1532 Acked Read 8 Done Sent Done false 1533 Acked Flush 0 Done Sent Done false 1534 Acked Flush 0 Done Sent Done false 1535 Acked Write 5 Done Sent Done false 1536 Acked Read 1 Done Sent Done false 1537 Acked Write 5 Done Sent Done false 1538 Acked Write 9 Done Sent Done false 1539 Acked Read 1 Done Sent Done false 1540 Acked Write 10 Done Sent Done false 1541 Acked Write 9 Done Sent Done false 1542 Acked Read 7 Done Sent Done false 1543 Acked Read 7 Done Sent Done false 1544 Acked Write 7 Done Sent Done false 1545 Acked Read 10 Done Sent Done false 1546 Acked Read 10 Done Sent Done false 1547 Acked Write 4 Done Sent Done false 1548 Acked Read 7 Done Sent Done false 1549 Acked Write 1 Done Sent Done false 1550 Acked Write 4 Done Sent Done false 1551 Acked Read 1 Done Sent Done false 1552 Acked Write 2 Done Sent Done false 1553 Acked Read 2 Done Sent Done false 1554 Acked Write 9 Done Sent Done false 1555 Acked Write 4 Done Sent Done false 1556 Acked Read 4 Done Sent Done false 1557 Acked Flush 0 Done Sent Done false 1558 Acked Read 8 Done Sent Done false 1559 Acked Write 3 Done Sent Done false 1560 Acked Read 6 Done Sent Done false 1561 Acked Read 1 Done Sent Done false 1562 Acked Write 9 Done Sent Done false 1563 Acked Read 8 Done Sent Done false 1564 Acked Read 1 Done Sent Done false 1565 Acked Write 9 Done Sent Done false 1566 Acked Read 7 Done Sent Done false 1567 Acked Write 6 Done Sent Done false 1568 Acked Read 10 Done Sent Done false 1569 Acked Read 2 Done Sent Done false 1570 Acked Write 2 Done Sent Done false 1571 Acked Write 8 Done Sent Done false 1572 Acked Flush 0 Done Sent Done false 1573 Acked Read 8 Done Sent Done false 1574 Acked Flush 0 Done Sent Done false 1575 Acked Write 10 Done Sent Done false 1576 Acked Write 8 Done Sent Done false 1577 Acked Read 2 Done Sent Done false 1578 Acked Read 3 Done Sent Done false 1579 Acked Write 3 Done Sent Done false 1580 Acked Read 3 Done Sent Done false 1581 Acked Write 7 Done Sent Done false 1582 Acked Write 2 Done Sent Done false 1583 Acked Read 1 Done Sent Done false 1584 Acked Write 6 Done Sent Done false 1585 Acked Read 10 Done Sent Done false 1586 Acked Flush 0 Done Sent Done false 1587 Acked Read 4 Done Sent Done false 1588 Acked Read 4 Done Sent Done false 1589 Acked Write 8 Done Sent Done false 1590 Acked Read 6 Done Sent Done false 1591 Acked Read 8 Done Sent Done false 1592 Acked Read 9 Done Sent Done false 1593 Acked Write 9 Done Sent Done false 1594 Acked Read 3 Done Sent Done false 1595 Acked Flush 0 Done Sent Done false 1596 Acked Read 4 Done Sent Done false 1597 Acked Read 3 Done Sent Done false 1598 Acked Flush 0 Done Sent Done false 1599 Acked Write 4 Done Sent Done false 1600 Acked Flush 0 Done Sent Done false 1601 Acked Write 5 Done Sent Done false 1602 Acked Write 6 Done Sent Done false 1603 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1603 1301 1603 Downstairs last five completed: 1301 1300 1299 1298 1297 Upstairs last five completed: 1603 1602 1601 1600 1599 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.938905695Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:47283)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.938985294Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:53.939914575Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.940420479Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.9404505Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 1301","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.940467756Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 1301","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.940666562Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.940686577Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.940701293Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 1301","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.94071512Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 1301","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:53.941037213Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1603 1602 1601 1600 1599 Upstairs last five completed: 1603 1602 1601 1600 1599 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:41:56.184[8830] Got stop action so:false kr:false INFO request completed, latency_us: 328, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 927c957b-782e-40ab-a5b4-3f149982f621, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:41:56.184 INFO request completed, latency_us: 149{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:56.18486574Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} , response_code: 200, uri: /state/cid/2, method: GET, req_id: 87dafd7f-f595-4319-b22b-ee9b74d45cd4, remote_addr: 127.0.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:56.184959175Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:41:56.185027386Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:56.185065156Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:56.185094688Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:41:56.185121171Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:42:00.186 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2bc1f9fe-29d8-4685-a7fb-2556a49f284d, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 2991 len 16384 002/300 Read block 4812 len 8192 003/300 Write block 2215 len 12288 data: 2 2 2 004/300 Write block 4336 len 4096 data: 3 005/300 Write block 981 len 20480 data: 2 2 2 2 2 006/300 Flush 007/300 Read block 363 len 36864 008/300 Read block 2516 len 16384 009/300 Read block 1978 len 16384 010/300 Read block 1203 len 36864 011/300 Read block 2398 len 36864 012/300 Read block 2838 len 8192 013/300 Read block 2784 len 8192 014/300 Read block 4803 len 12288 015/300 Write block 1625 len 28672 data: 2 2 2 2 2 2 2 016/300 Read block 1626 len 12288 017/300 Write block 4853 len 16384 data: 2 2 2 2 018/300 Read block 753 len 24576 019/300 Write block 2603 len 36864 data: 2 2 2 2 2 2 2 2 2 020/300 Read block 98 len 20480 021/300 Read block 805 len 32768 022/300 Read block 4496 len 8192 023/300 Read block 1039 len 40960 024/300 Read block 3504 len 8192 025/300 Read block 4213 len 8192 026/300 Write block 4772 len 36864 data: 2 2 2 2 2 2 2 2 2 027/300 Read block 1935 len 24576 028/300 Read block 4712 len 40960 029/300 Read block 2824 len 16384 030/300 Read block 1048 len 36864 031/300 Write block 4072 len 32768 data: 2 2 2 2 2 2 2 2 032/300 Read block 689 len 12288 033/300 Read block 2296 len 28672 034/300 Write block 4016 len 12288 data: 2 2 2 035/300 Flush 036/300 Write block 4862 len 4096 data: 4 037/300 Read block 3708 len 4096 038/300 Read block 648 len 4096 039/300 Write block 3886 len 24576 data: 2 2 2 2 2 2 040/300 Write block 2020 len 40960 data: 2 2 2 2 2 2 2 2 2 2 041/300 Read block 452 len 4096 042/300 Read block 252 len 4096 043/300 Write block 3962 len 8192 data: 2 2 044/300 Read block 179 len 32768 045/300 Read block 4944 len 28672 046/300 Read block 3439 len 36864 047/300 Flush 048/300 Flush 049/300 Read block 2230 len 36864 050/300 Read block 124 len 28672 051/300 Flush 052/300 Write block 3017 len 4096 data: 3 053/300 Flush 054/300 Read block 1853 len 24576 055/300 Read block 1650 len 24576 056/300 Flush 057/300 Read block 4378 len 20480 058/300 Write block 3183 len 4096 data: 2 059/300 Read block 630 len 32768 060/300 Write block 579 len 36864 data: 2 2 2 2 2 2 2 2 2 061/300 Read block 2837 len 16384 062/300 Read block 2297 len 32768 063/300 Read block 2797 len 32768 064/300 Write block 584 len 16384 data: 3 3 3 3 065/300 Read block 1978 len 8192 066/300 Read block 2235 len 4096 067/300 Read block 4951 len 36864 068/300 Write block 1403 len 12288 data: 2 2 2 069/300 Flush 070/300 Write block 2814 len 24576 data: 3 3 3 3 3 3 071/300 Write block 1997 len 36864 data: 4 4 4 4 4 3 3 3 2 072/300 Read block 3856 len 20480 073/300 Write block 4826 len 36864 data: 2 2 2 2 2 2 2 2 2 074/300 Write block 355 len 4096 data: 2 075/300 Read block 2287 len 12288 076/300 Write block 2682 len 28672 data: 2 2 2 2 2 2 2 077/300 Read block 4433 len 24576 078/300 Read block 4943 len 20480 079/300 Write block 2558 len 40960 data: 3 3 3 3 3 2 2 2 2 2 080/300 Flush 081/300 Write block 3700 len 32768 data: 4 4 4 4 3 3 3 2 082/300 Read block 4223 len 32768 083/300 Read block 4785 len 16384 084/300 Flush 085/300 Read block 3545 len 32768 086/300 Write block 4549 len 20480 data: 2 2 2 2 2 087/300 Flush 088/300 Read block 1023 len 40960 089/300 Write block 3937 len 24576 data: 2 2 2 2 2 2 090/300 Write block 738 len 24576 data: 2 2 2 2 2 2 091/300 Read block 2061 len 4096 092/300 Write block 2345 len 40960 data: 3 3 3 3 3 4 4 3 2 2 093/300 Write block 1463 len 12288 data: 2 2 2 094/300 Write block 1826 len 4096 data: 3 095/300 Read block 1496 len 28672 096/300 Write block 2457 len 8192 data: 3 2 097/300 Read block 3717 len 40960 098/300 Read block 2850 len 12288 099/300 Write block 4166 len 4096 data: 2 100/300 Write block 3793 len 16384 data: 2 2 2 2 101/300 Read block 4258 len 24576 102/300 Flush 103/300 Write block 1863 len 36864 data: 2 2 2 2 2 2 2 2 2 104/300 Read block 3462 len 24576 105/300 Write block 1800 len 32768 data: 4 5 4 4 4 3 2 2 106/300 Write block 1540 len 24576 data: 4 4 4 4 3 3 107/300 Write block 3965 len 8192 data: 2 2 108/300 Read block 2802 len 32768 109/300 Read block 1389 len 12288 110/300 Read block 389 len 28672 111/300 Write block 4725 len 4096 data: 3 112/300 Write block 2946 len 8192 data: 2 2 113/300 Write block 2584 len 32768 data: 2 2 2 2 2 2 2 2 114/300 Read block 4003 len 16384 115/300 Write block 806 len 28672 data: 4 3 3 2 2 2 2 116/300 Flush 117/300 Read block 4688 len 40960 118/300 Flush 119/300 Read block 4927 len 24576 120/300 Write block 2675 len 16384 data: 2 2 2 2 121/300 Write block 1314 len 20480 data: 2 2 2 2 2 122/300 Write block 4118 len 8192 data: 2 2 123/300 Flush 124/300 Write block 903 len 20480 data: 2 2 3 2 2 125/300 Write block 2039 len 32768 data: 2 2 2 2 3 4 4 3 126/300 Write block 4089 len 28672 data: 2 2 2 2 2 2 2 127/300 Write block 1175 len 20480 data: 3 3 3 3 4 128/300 Write block 3379 len 36864 data: 2 2 2 2 2 2 2 2 2 129/300 Write block 1708 len 16384 data: 2 2 2 2 130/300 Read block 1966 len 4096 131/300 Read block 3991 len 32768 132/300 Write block 3759 len 32768 data: 2 3 3 3 2 2 2 2 133/300 Read block 3486 len 32768 134/300 Read block 634 len 4096 135/300 Read block 1615 len 28672 136/300 Flush 137/300 Read block 2798 len 32768 138/300 Read block 1124 len 40960 139/300 Write block 2340 len 16384 data: 3 2 2 2 140/300 Read block 3743 len 32768 141/300 Write block 2493 len 40960 data: 2 2 2 2 2 3 3 3 3 3 142/300 Read block 3826 len 40960 143/300 Flush 144/300 Write block 4470 len 20480 data: 2 2 2 2 2 145/300 Read block 2671 len 24576 146/300 Write block 1989 len 20480 data: 2 2 3 3 3 147/300 Write block 2801 len 24576 data: 2 2 2 2 2 2 148/300 Read block 3174 len 40960 149/300 Read block 2998 len 20480 150/300 Read block 670 len 32768 151/300 Read block 4795 len 28672 152/300 Read block 4674 len 36864 153/300 Read block 3222 len 20480 154/300 Write block 2396 len 12288 data: 2 2 2 155/300 Read block 4215 len 32768 156/300 Read block 549 len 8192 157/300 Write block 1471 len 28672 data: 2 2 2 2 2 2 2 158/300 Write block 4115 len 16384 data: 2 2 2 3 159/300 Read block 527 len 32768 160/300 Write block 4697 len 8192 data: 2 2 161/300 Write block 865 len 24576 data: 2 2 2 2 2 2 162/300 Write block 125 len 36864 data: 2 2 2 2 2 2 2 2 2 163/300 Write block 320 len 8192 data: 2 2 164/300 Write block 2479 len 12288 data: 2 2 2 165/300 Write block 4443 len 12288 data: 3 3 2 166/300 Write block 2933 len 12288 data: 2 2 2 167/300 Read block 1164 len 40960 168/300 Flush 169/300 Write block 2540 len 24576 data: 2 3 3 3 3 3 170/300 Write block 1311 len 20480 data: 2 2 2 3 3 171/300 Write block 3549 len 32768 data: 2 2 2 2 2 2 3 3 172/300 Read block 3673 len 36864 173/300 Read block 2355 len 20480 174/300 Write block 2806 len 20480 data: 3 2 2 2 2 175/300 Write block 655 len 28672 data: 2 2 2 3 3 3 3 176/300 Read block 3524 len 32768 177/300 Write block 2413 len 16384 data: 2 2 2 2 178/300 Write block 3677 len 28672 data: 3 3 3 3 3 3 3 179/300 Read block 374 len 40960 180/300 Flush 181/300 Flush 182/300 Write block 2784 len 32768 data: 2 2 2 2 2 2 2 2 183/300 Write block 2333 len 4096 data: 4 184/300 Write block 4583 len 12288 data: 3 3 3 185/300 Read block 3354 len 24576 186/300 Read block 1621 len 24576 187/300 Read block 4127 len 24576 188/300 Read block 2641 len 32768 189/300 Write block 206 len 24576 data: 2 3 3 3 3 3 190/300 Read block 2174 len 32768 191/300 Read block 2032 len 36864 192/300 Write block 746 len 32768 data: 2 2 2 3 3 3 4 4 193/300 Write block 2186 len 12288 data: 2 2 2 194/300 Read block 4732 len 16384 195/300 Write block 4419 len 32768 data: 2 2 2 3 3 3 3 3 196/300 Read block 426 len 16384 197/300 Write block 2014 len 20480 data: 2 2 2 2 2 198/300 Read block 3786 len 20480 199/300 Write block 146 len 12288 data: 2 2 2 200/300 Read block 2322 len 20480 201/300 Write block 333 len 20480 data: 2 2 2 2 2 202/300 Read block 4691 len 36864 203/300 Read block 501 len 8192 204/300 Read block 1111 len 16384 205/300 Read block 1928 len 32768 206/300 Write block 798 len 4096 data: 3 207/300 Write block 3938 len 4096 data: 3 208/300 Read block 3348 len 32768 209/300 Write block 3034 len 16384 data: 2 2 2 2 210/300 Write block 4951 len 16384 data: 3 3 3 4 211/300 Read block 83 len 12288 212/300 Flush 213/300 Read block 814 len 12288 214/300 Write block 4771 len 20480 data: 2 3 3 3 3 215/300 Read block 1131 len 36864 216/300 Write block 4417 len 28672 data: 2 2 3 3 3 4 4 217/300 Read block 258 len 20480 218/300 Read block 4181 len 28672 219/300 Write block 889 len 24576 data: 3 2 2 2 2 2 220/300 Read block 2944 len 32768 221/300 Write block 1989 len 24576 data: 3 3 4 4 4 2 222/300 Write block 2112 len 40960 data: 3 3 2 2 2 2 2 2 2 2 223/300 Write block 756 len 16384 data: 4 3 2 2 224/300 Write block 2373 len 12288 data: 2 2 2 225/300 Read block 4972 len 4096 226/300 Read block 4788 len 24576 227/300 Write block 512 len 28672 data: 3 2 2 2 2 2 2 228/300 Read block 4265 len 20480 229/300 Write block 3673 len 12288 data: 2 2 3 230/300 Read block 1862 len 28672 231/300 Read block 1192 len 4096 232/300 Write block 2362 len 36864 data: 3 2 2 2 2 2 2 2 2 233/300 Write block 176 len 36864 data: 2 2 2 2 2 2 2 2 2 234/300 Read block 4537 len 4096 235/300 Write block 4621 len 28672 data: 2 2 2 2 2 2 2 236/300 Read block 1289 len 28672 237/300 Flush 238/300 Write block 491 len 40960 data: 2 2 2 2 2 2 2 2 2 2 239/300 Write block 1038 len 4096 data: 2 240/300 Read block 3948 len 32768 241/300 Write block 187 len 24576 data: 2 2 2 2 2 2 242/300 Read block 2307 len 24576 243/300 Read block 3902 len 32768 244/300 Write block 1607 len 12288 data: 2 2 2 245/300 Write block 946 len 40960 data: 2 2 2 2 2 2 2 2 2 2 246/300 Flush 247/300 Flush 248/300 Write block 2701 len 20480 data: 3 2 2 2 2 249/300 Write block 2936 len 32768 data: 2 2 2 2 2 2 2 2 250/300 Read block 4716 len 16384 251/300 Flush 252/300 Write block 1772 len 24576 data: 2 2 2 2 2 2 253/300 Write block 1816 len 32768 data: 3 3 3 3 3 3 3 3 254/300 Write block 4030 len 24576 data: 2 2 2 2 2 2 255/300 Flush 256/300 Write block 3753 len 4096 data: 3 257/300 Flush 258/300 Flush 259/300 Write block 2761 len 32768 data: 2 2 2 2 2 2 2 2 260/300 Write block 1217 len 40960 data: 2 2 2 2 2 3 3 3 3 3 261/300 Read block 1304 len 24576 262/300 Read block 1041 len 20480 263/300 Write block 4571 len 28672 data: 2 2 2 2 2 2 3 264/300 Read block 3301 len 40960 265/300 Read block 4711 len 12288 266/300 Write block 1911 len 28672 data: 3 3 3 2 2 2 2 267/300 Flush 268/300 Write block 3085 len 12288 data: 2 2 2 269/300 Read block 3234 len 24576 270/300 Write block 3959 len 36864 data: 2 2 2 3 3 2 3 3 2 271/300 Read block 3672 len 4096 272/300 Write block 3357 len 28672 data: 2 2 2 2 2 2 2 273/300 Write block 1483 len 8192 data: 2 2 274/300 Write block 1807 len 40960 data: 3 2 2 2 2 2 2 3 3 4 275/300 Read block 3087 len 16384 276/300 Write block 4045 len 36864 data: 2 2 2 2 2 2 2 2 2 277/300 Flush 278/300 Write block 1924 len 28672 data: 2 2 2 2 2 2 2 279/300 Read block 4859 len 16384 280/300 Write block 1468 len 12288 data: 3 2 2 281/300 Write block 2885 len 28672 data: 2 2 2 2 2 2 2 282/300 Write block 3104 len 36864 data: 2 2 2 2 2 2 2 2 2 283/300 Flush 284/300 Flush 285/300 Read block 28 len 4096 286/300 Read block 775 len 4096 287/300 Read block 2419 len 32768 288/300 Read block 3742 len 12288 289/300 Read block 1965 len 24576 290/300 Write block 27 len 4096 data: 2 291/300 Read block 2849 len 8192 292/300 Write block 3749 len 8192 data: 2 2 293/300 Read block 1916 len 28672 294/300 Read block 431 len 4096 295/300 Write block 2437 len 24576 data: 3 3 3 3 3 3 296/300 Read block 1644 len 40960 297/300 Write block 3697 len 24576 data: 4 3 4 5 5 5 298/300 Write block 1739 len 20480 data: 2 2 2 2 2 299/300 Read block 2503 len 24576 300/300 Flush 301/300 Write block 1116 len 20480 data: 2 2 2 2 2 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:42:00.414 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8889e506-c368-4ef5-aebd-9ce569fc2678, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1604 Acked Read 4 Done Done Sent false 1605 Acked Read 2 Done Done Sent false 1606 Acked Write 3 Done Done Sent false 1607 Acked Write 1 Done Done Sent false 1608 Acked Write 5 Done Done Sent false 1609 Acked Flush 0 Done Done Sent false 1610 Acked Read 9 Done Done Sent false 1611 Acked Read 4 Done Done Sent false 1612 Acked Read 4 Done Done Sent false 1613 Acked Read 9 Done Done Sent false 1614 Acked Read 9 Done Done Sent false 1615 Acked Read 2 Done Done Sent false 1616 Acked Read 2 Done Done Sent false 1617 Acked Read 3 Done Done Sent false 1618 Acked Write 7 Done Done Sent false 1619 Acked Read 3 Done Done Sent false 1620 Acked Write 4 Done Done Sent false 1621 Acked Read 6 Done Done Sent false 1622 Acked Write 9 Done Done Sent false 1623 Acked Read 5 Done Done Sent false 1624 Acked Read 8 Done Done Sent false 1625 Acked Read 2 Done Done Sent false 1626 Acked Read 10 Done Done Sent false 1627 Acked Read 2 Done Done Sent false 1628 Acked Read 2 Done Done Sent false 1629 Acked Write 9 Done Done Sent false 1630 Acked Read 6 Done Done Sent false 1631 Acked Read 10 Done Done Sent false 1632 Acked Read 4 Done Done Sent false 1633 Acked Read 9 Done Done Sent false 1634 Acked Write 8 Done Done Sent false 1635 Acked Read 3 Done Done Sent false 1636 Acked Read 7 Done Done Sent false 1637 Acked Write 3 Done Done Sent false 1638 Acked Flush 0 Done Done Sent false 1639 Acked Write 1 Done Done Sent false 1640 Acked Read 1 Done Done Sent false 1641 Acked Read 1 Done Done Sent false 1642 Acked Write 6 Done Done Sent false 1643 Acked Write 10 Done Done Sent false 1644 Acked Read 1 Done Done Sent false 1645 Acked Read 1 Done Done Sent false 1646 Acked Write 2 Done Done Sent false 1647 Acked Read 8 Done Done Sent false 1648 Acked Read 7 Done Done Sent false 1649 Acked Read 9 Done Done Sent false 1650 Acked Flush 0 Done Done Sent false 1651 Acked Flush 0 Done Done Sent false 1652 Acked Read 9 Done Done Sent false 1653 Acked Read 7 Done Done Sent false 1654 Acked Flush 0 Done Done Sent false 1655 Acked Write 1 Done Done Sent false 1656 Acked Flush 0 Done Done Sent false 1657 Acked Read 6 Done Done Sent false 1658 Acked Read 6 Done Done Sent false 1659 Acked Flush 0 Done Done Sent false 1660 Acked Read 5 Done Done Sent false 1661 Acked Write 1 Done Done Sent false 1662 Acked Read 8 Done Done Sent false 1663 Acked Write 9 Done Done Sent false 1664 Acked Read 4 Done Done Sent false 1665 Acked Read 8 Done Done Sent false 1666 Acked Read 8 Done Done Sent false 1667 Acked Write 4 Done Done Sent false 1668 Acked Read 2 Done Done Sent false 1669 Acked Read 1 Done Done Sent false 1670 Acked Read 9 Done Done Sent false 1671 Acked Write 3 Done Done Sent false 1672 Acked Flush 0 Done Done Sent false 1673 Acked Write 6 Done Done Sent false 1674 Acked Write 9 Done Done Sent false 1675 Acked Read 5 Done Done Sent false 1676 Acked Write 9 Done Done Sent false 1677 Acked Write 1 Done Done Sent false 1678 Acked Read 3 Done Done Sent false 1679 Acked Write 7 Done Done Sent false 1680 Acked Read 6 Done Done Sent false 1681 Acked Read 5 Done Done Sent false 1682 Acked Write 10 Done Done Sent false 1683 Acked Flush 0 Done Done Sent false 1684 Acked Write 8 Done Done Sent false 1685 Acked Read 8 Done Done Sent false 1686 Acked Read 4 Done Done Sent false 1687 Acked Flush 0 Done Done Sent false 1688 Acked Read 8 Done Done Sent false 1689 Acked Write 5 Done Done Sent false 1690 Acked Flush 0 Done Done Sent false 1691 Acked Read 10 Done Done Sent false 1692 Acked Write 6 Done Done Sent false 1693 Acked Write 6 Done Done Sent false 1694 Acked Read 1 Done Done Sent false 1695 Acked Write 10 Done Done Sent false 1696 Acked Write 3 Done Done Sent false 1697 Acked Write 1 Done Done Sent false 1698 Acked Read 7 Done Done Sent false 1699 Acked Write 2 Done Done Sent false 1700 Acked Read 10 Done Done Sent false 1701 Acked Read 3 Done Done Sent false 1702 Acked Write 1 Done Done Sent false 1703 Acked Write 4 Done Done Sent false 1704 Acked Read 6 Done Done Sent false 1705 Acked Flush 0 Done Done Sent false 1706 Acked Write 9 Done Done Sent false 1707 Acked Read 6 Done Done Sent false 1708 Acked Write 8 Done Done Sent false 1709 Acked Write 6 Done Done Sent false 1710 Acked Write 2 Done Done Sent false 1711 Acked Read 8 Done Done Sent false 1712 Acked Read 3 Done Done Sent false 1713 Acked Read 7 Done Done Sent false 1714 Acked Write 1 Done Done Sent false 1715 Acked Write 2 Done Done Sent false 1716 Acked Write 8 Done Done Sent false 1717 Acked Read 4 Done Done Sent false 1718 Acked Write 7 Done Done Sent false 1719 Acked Flush 0 Done Done Sent false 1720 Acked Read 10 Done Done Sent false 1721 Acked Flush 0 Done Done Sent false 1722 Acked Read 6 Done Done Sent false 1723 Acked Write 4 Done Done Sent false 1724 Acked Write 5 Done Done Sent false 1725 Acked Write 2 Done Done Sent false 1726 Acked Flush 0 Done Done Sent false 1727 Acked Write 5 Done Done Sent false 1728 Acked Write 8 Done Done Sent false 1729 Acked Write 7 Done Done Sent false 1730 Acked Write 5 Done Done Sent false 1731 Acked Write 9 Done Done Sent false 1732 Acked Write 4 Done Done Sent false 1733 Acked Read 1 Done Done Sent false 1734 Acked Read 8 Done Done Sent false 1735 Acked Write 8 Done Done Sent false 1736 Acked Read 8 Done Done Sent false 1737 Acked Read 1 Done Done Sent false 1738 Acked Read 7 Done Done Sent false 1739 Acked Flush 0 Done Done Sent false 1740 Acked Read 8 Done Done Sent false 1741 Acked Read 10 Done Done Sent false 1742 Acked Write 4 Done Done Sent false 1743 Acked Read 8 Done Done Sent false 1744 Acked Write 10 Done Done Sent false 1745 Acked Read 10 Done Done Sent false 1746 Acked Flush 0 Done Done Sent false 1747 Acked Write 5 Done Done Sent false 1748 Acked Read 6 Done Done Sent false 1749 Acked Write 5 Done Done Sent false 1750 Acked Write 6 Done Done Sent false 1751 Acked Read 10 Done Done Sent false 1752 Acked Read 5 Done Done Sent false 1753 Acked Read 8 Done Done Sent false 1754 Acked Read 7 Done Done Sent false 1755 Acked Read 9 Done Done Sent false 1756 Acked Read 5 Done Done Sent false 1757 Acked Write 3 Done Done Sent false 1758 Acked Read 8 Done Done Sent false 1759 Acked Read 2 Done Done Sent false 1760 Acked Write 7 Done Done Sent false 1761 Acked Write 4 Done Done Sent false 1762 Acked Read 8 Done Done Sent false 1763 Acked Write 2 Done Done Sent false 1764 Acked Write 6 Done Done Sent false 1765 Acked Write 9 Done Done Sent false 1766 Acked Write 2 Done Done Sent false 1767 Acked Write 3 Done Done Sent false 1768 Acked Write 3 Done Done Sent false 1769 Acked Write 3 Done Done Sent false 1770 Acked Read 10 Done Done Sent false 1771 Acked Flush 0 Done Done Sent false 1772 Acked Write 6 Done Done Sent false 1773 Acked Write 5 Done Done Sent false 1774 Acked Write 8 Done Done Sent false 1775 Acked Read 9 Done Done Sent false 1776 Acked Read 5 Done Done Sent false 1777 Acked Write 5 Done Done Sent false 1778 Acked Write 7 Done Done Sent false 1779 Acked Read 8 Done Done Sent false 1780 Acked Write 4 Done Done Sent false 1781 Acked Write 7 Done Done Sent false 1782 Acked Read 10 Done Done Sent false 1783 Acked Flush 0 Done Done Sent false 1784 Acked Flush 0 Done Done Sent false 1785 Acked Write 8 Done Done Sent false 1786 Acked Write 1 Done Done Sent false 1787 Acked Write 3 Done Done Sent false 1788 Acked Read 6 Done Done Sent false 1789 Acked Read 6 Done Done Sent false 1790 Acked Read 6 Done Done Sent false 1791 Acked Read 8 Done Done Sent false 1792 Acked Write 6 Done Done Sent false 1793 Acked Read 8 Done Done Sent false 1794 Acked Read 9 Done Done Sent false 1795 Acked Write 8 Done Done Sent false 1796 Acked Write 3 Done Done Sent false 1797 Acked Read 4 Done Done Sent false 1798 Acked Write 8 Done Done Sent false 1799 Acked Read 4 Done Done Sent false 1800 Acked Write 5 Done Done Sent false 1801 Acked Read 5 Done Done Sent false 1802 Acked Write 3 Done Done Sent false 1803 Acked Read 5 Done Done Sent false 1804 Acked Write 5 Done Done Sent false 1805 Acked Read 9 Done Done Sent false 1806 Acked Read 2 Done Done Sent false 1807 Acked Read 4 Done Done Sent false 1808 Acked Read 8 Done Done Sent false 1809 Acked Write 1 Done Done Sent false 1810 Acked Write 1 Done Done Sent false 1811 Acked Read 8 Done Done Sent false 1812 Acked Write 4 Done Done Sent false 1813 Acked Write 4 Done Done Sent false 1814 Acked Read 3 Done Done Sent false 1815 Acked Flush 0 Done Done Sent false 1816 Acked Read 3 Done Done Sent false 1817 Acked Write 5 Done Done Sent false 1818 Acked Read 9 Done Done Sent false 1819 Acked Write 7 Done Done Sent false 1820 Acked Read 5 Done Done Sent false 1821 Acked Read 7 Done Done Sent false 1822 Acked Write 6 Done Done Sent false 1823 Acked Read 8 Done Done Sent false 1824 Acked Write 6 Done Done Sent false 1825 Acked Write 10 Done Done Sent false 1826 Acked Write 4 Done Done Sent false 1827 Acked Write 3 Done Done Sent false 1828 Acked Read 1 Done Done Sent false 1829 Acked Read 6 Done Done Sent false 1830 Acked Write 7 Done Done Sent false 1831 Acked Read 5 Done Done Sent false 1832 Acked Write 3 Done Done Sent false 1833 Acked Read 7 Done Done Sent false 1834 Acked Read 1 Done Done Sent false 1835 Acked Write 9 Done Done Sent false 1836 Acked Write 9 Done Done Sent false 1837 Acked Read 1 Done Done Sent false 1838 Acked Write 7 Done Done Sent false 1839 Acked Read 7 Done Done Sent false 1840 Acked Flush 0 Done Done Sent false 1841 Acked Write 10 Done Done Sent false 1842 Acked Write 1 Done Done Sent false 1843 Acked Read 8 Done Done Sent false 1844 Acked Write 6 Done Done Sent false 1845 Acked Read 6 Done Done Sent false 1846 Acked Read 8 Done Done Sent false 1847 Acked Write 3 Done Done Sent false 1848 Acked Write 10 Done Done Sent false 1849 Acked Flush 0 Done Done Sent false 1850 Acked Flush 0 Done Done Sent false 1851 Acked Write 5 Done Done Sent false 1852 Acked Write 8 Done Done Sent false 1853 Acked Read 4 Done Done Sent false 1854 Acked Flush 0 Done Done Sent false 1855 Acked Write 6 Done Done Sent false 1856 Acked Write 8 Done Done Sent false 1857 Acked Write 6 Done Done Sent false 1858 Acked Flush 0 Done Done Sent false 1859 Acked Write 1 Done Done Sent false 1860 Acked Flush 0 Done Done Sent false 1861 Acked Flush 0 Done Done Sent false 1862 Acked Write 8 Done Done Sent false 1863 Acked Write 10 Done Done Sent false 1864 Acked Read 6 Done Done Sent false 1865 Acked Read 5 Done Done Sent false 1866 Acked Write 7 Done Done Sent false 1867 Acked Read 10 Done Done Sent false 1868 Acked Read 3 Done Done Sent false 1869 Acked Write 7 Done Done Sent false 1870 Acked Flush 0 Done Done Sent false 1871 Acked Write 3 Done Done Sent false 1872 Acked Read 6 Done Done Sent false 1873 Acked Write 9 Done Done Sent false 1874 Acked Read 1 Done Done Sent false 1875 Acked Write 7 Done Done Sent false 1876 Acked Write 2 Done Done Sent false 1877 Acked Write 10 Done Done Sent false 1878 Acked Read 4 Done Done Sent false 1879 Acked Write 9 Done Done Sent false 1880 Acked Flush 0 Done Done Sent false 1881 Acked Write 7 Done Done Sent false 1882 Acked Read 4 Done Done Sent false 1883 Acked Write 3 Done Done Sent false 1884 Acked Write 7 Done Done Sent false 1885 Acked Write 9 Done Done Sent false 1886 Acked Flush 0 Done Done Sent false 1887 Acked Flush 0 Done Done Sent false 1888 Acked Read 1 Done Done Sent false 1889 Acked Read 1 Done Done Sent false 1890 Acked Read 8 Done Done Sent false 1891 Acked Read 3 Done Done Sent false 1892 Acked Read 6 Done Done Sent false 1893 Acked Write 1 Done Done Sent false 1894 Acked Read 2 Done Done Sent false 1895 Acked Write 2 Done Done Sent false 1896 Acked Read 7 Done Done Sent false 1897 Acked Read 1 Done Done Sent false 1898 Acked Write 6 Done Done Sent false 1899 Acked Read 10 Done Done Sent false 1900 Acked Write 6 Done Done Sent false 1901 Acked Write 5 Done Done Sent false 1902 Acked Read 6 Done Done Sent false 1903 Acked Flush 0 Done Done Sent false 1904 Acked Write 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1903 1903 1603 Downstairs last five completed: 1603 1602 1601 1600 1599 Upstairs last five completed: 1904 1903 1902 1901 1900 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1604 Acked Read 4 Done Done Sent false 1605 Acked Read 2 Done Done Sent false 1606 Acked Write 3 Done Done Sent false 1607 Acked Write 1 Done Done Sent false 1608 Acked Write 5 Done Done Sent false 1609 Acked Flush 0 Done Done Sent false 1610 Acked Read 9 Done Done Sent false 1611 Acked Read 4 Done Done Sent false 1612 Acked Read 4 Done Done Sent false 1613 Acked Read 9 Done Done Sent false 1614 Acked Read 9 Done Done Sent false 1615 Acked Read 2 Done Done Sent false 1616 Acked Read 2 Done Done Sent false 1617 Acked Read 3 Done Done Sent false 1618 Acked Write 7 Done Done Sent false 1619 Acked Read 3 Done Done Sent false 1620 Acked Write 4 Done Done Sent false 1621 Acked Read 6 Done Done Sent false 1622 Acked Write 9 Done Done Sent false 1623 Acked Read 5 Done Done Sent false 1624 Acked Read 8 Done Done Sent false 1625 Acked Read 2 Done Done Sent false 1626 Acked Read 10 Done Done Sent false 1627 Acked Read 2 Done Done Sent false 1628 Acked Read 2 Done Done Sent false 1629 Acked Write 9 Done Done Sent false 1630 Acked Read 6 Done Done Sent false 1631 Acked Read 10 Done Done Sent false 1632 Acked Read 4 Done Done Sent false 1633 Acked Read 9 Done Done Sent false 1634 Acked Write 8 Done Done Sent false 1635 Acked Read 3 Done Done Sent false 1636 Acked Read 7 Done Done Sent false 1637 Acked Write 3 Done Done Sent false 1638 Acked Flush 0 Done Done Sent false 1639 Acked Write 1 Done Done Sent false 1640 Acked Read 1 Done Done Sent false 1641 Acked Read 1 Done Done Sent false 1642 Acked Write 6 Done Done Sent false 1643 Acked Write 10 Done Done Sent false 1644 Acked Read 1 Done Done Sent false 1645 Acked Read 1 Done Done Sent false 1646 Acked Write 2 Done Done Sent false 1647 Acked Read 8 Done Done Sent false 1648 Acked Read 7 Done Done Sent false 1649 Acked Read 9 Done Done Sent false 1650 Acked Flush 0 Done Done Sent false 1651 Acked Flush 0 Done Done Sent false 1652 Acked Read 9 Done Done Sent false 1653 Acked Read 7 Done Done Sent false 1654 Acked Flush 0 Done Done Sent false 1655 Acked Write 1 Done Done Sent false 1656 Acked Flush 0 Done Done Sent false 1657 Acked Read 6 Done Done Sent false 1658 Acked Read 6 Done Done Sent false 1659 Acked Flush 0 Done Done Sent false 1660 Acked Read 5 Done Done Sent false 1661 Acked Write 1 Done Done Sent false 1662 Acked Read 8 Done Done Sent false 1663 Acked Write 9 Done Done Sent false 1664 Acked Read 4 Done Done Sent false 1665 Acked Read 8 Done Done Sent false 1666 Acked Read 8 Done Done Sent false 1667 Acked Write 4 Done Done Sent false 1668 Acked Read 2 Done Done Sent false 1669 Acked Read 1 Done Done Sent false 1670 Acked Read 9 Done Done Sent false 1671 Acked Write 3 Done Done Sent false 1672 Acked Flush 0 Done Done Sent false 1673 Acked Write 6 Done Done Sent false 1674 Acked Write 9 Done Done Sent false 1675 Acked Read 5 Done Done Sent false 1676 Acked Write 9 Done Done Sent false 1677 Acked Write 1 Done Done Sent false 1678 Acked Read 3 Done Done Sent false 1679 Acked Write 7 Done Done Sent false 1680 Acked Read 6 Done Done Sent false 1681 Acked Read 5 Done Done Sent false 1682 Acked Write 10 Done Done Sent false 1683 Acked Flush 0 Done Done Sent false 1684 Acked Write 8 Done Done Sent false 1685 Acked Read 8 Done Done Sent false 1686 Acked Read 4 Done Done Sent false 1687 Acked Flush 0 Done Done Sent false 1688 Acked Read 8 Done Done Sent false 1689 Acked Write 5 Done Done Sent false 1690 Acked Flush 0 Done Done Sent false 1691 Acked Read 10 Done Done Sent false 1692 Acked Write 6 Done Done Sent false 1693 Acked Write 6 Done Done Sent false 1694 Acked Read 1 Done Done Sent false 1695 Acked Write 10 Done Done Sent false 1696 Acked Write 3 Done Done Sent false 1697 Acked Write 1 Done Done Sent false 1698 Acked Read 7 Done Done Sent false 1699 Acked Write 2 Done Done Sent false 1700 Acked Read 10 Done Done Sent false 1701 Acked Read 3 Done Done Sent false 1702 Acked Write 1 Done Done Sent false 1703 Acked Write 4 Done Done Sent false 1704 Acked Read 6 Done Done Sent false 1705 Acked Flush 0 Done Done Sent false 1706 Acked Write 9 Done Done Sent false 1707 Acked Read 6 Done Done Sent false 1708 Acked Write 8 Done Done Sent false 1709 Acked Write 6 Done Done Sent false 1710 Acked Write 2 Done Done Sent false 1711 Acked Read 8 Done Done Sent false 1712 Acked Read 3 Done Done Sent false 1713 Acked Read 7 Done Done Sent false 1714 Acked Write 1 Done Done Sent false 1715 Acked Write 2 Done Done Sent false 1716 Acked Write 8 Done Done Sent false 1717 Acked Read 4 Done Done Sent false 1718 Acked Write 7 Done Done Sent false 1719 Acked Flush 0 Done Done Sent false 1720 Acked Read 10 Done Done Sent false 1721 Acked Flush 0 Done Done Sent false 1722 Acked Read 6 Done Done Sent false 1723 Acked Write 4 Done Done Sent false 1724 Acked Write 5 Done Done Sent false 1725 Acked Write 2 Done Done Sent false 1726 Acked Flush 0 Done Done Sent false 1727 Acked Write 5 Done Done Sent false 1728 Acked Write 8 Done Done Sent false 1729 Acked Write 7 Done Done Sent false 1730 Acked Write 5 Done Done Sent false 1731 Acked Write 9 Done Done Sent false 1732 Acked Write 4 Done Done Sent false 1733 Acked Read 1 Done Done Sent false 1734 Acked Read 8 Done Done Sent false 1735 Acked Write 8 Done Done Sent false 1736 Acked Read 8 Done Done Sent false 1737 Acked Read 1 Done Done Sent false 1738 Acked Read 7 Done Done Sent false 1739 Acked Flush 0 Done Done Sent false 1740 Acked Read 8 Done Done Sent false 1741 Acked Read 10 Done Done Sent false 1742 Acked Write 4 Done Done Sent false 1743 Acked Read 8 Done Done Sent false 1744 Acked Write 10 Done Done Sent false 1745 Acked Read 10 Done Done Sent false 1746 Acked Flush 0 Done Done Sent false 1747 Acked Write 5 Done Done Sent false 1748 Acked Read 6 Done Done Sent false 1749 Acked Write 5 Done Done Sent false 1750 Acked Write 6 Done Done Sent false 1751 Acked Read 10 Done Done Sent false 1752 Acked Read 5 Done Done Sent false 1753 Acked Read 8 Done Done Sent false 1754 Acked Read 7 Done Done Sent false 1755 Acked Read 9 Done Done Sent false 1756 Acked Read 5 Done Done Sent false 1757 Acked Write 3 Done Done Sent false 1758 Acked Read 8 Done Done Sent false 1759 Acked Read 2 Done Done Sent false 1760 Acked Write 7 Done Done Sent false 1761 Acked Write 4 Done Done Sent false 1762 Acked Read 8 Done Done Sent false 1763 Acked Write 2 Done Done Sent false 1764 Acked Write 6 Done Done Sent false 1765 Acked Write 9 Done Done Sent false 1766 Acked Write 2 Done Done Sent false 1767 Acked Write 3 Done Done Sent false 1768 Acked Write 3 Done Done Sent false 1769 Acked Write 3 Done Done Sent false 1770 Acked Read 10 Done Done Sent false 1771 Acked Flush 0 Done Done Sent false 1772 Acked Write 6 Done Done Sent false 1773 Acked Write 5 Done Done Sent false 1774 Acked Write 8 Done Done Sent false 1775 Acked Read 9 Done Done Sent false 1776 Acked Read 5 Done Done Sent false 1777 Acked Write 5 Done Done Sent false 1778 Acked Write 7 Done Done Sent false 1779 Acked Read 8 Done Done Sent false 1780 Acked Write 4 Done Done Sent false 1781 Acked Write 7 Done Done Sent false 1782 Acked Read 10 Done Done Sent false 1783 Acked Flush 0 Done Done Sent false 1784 Acked Flush 0 Done Done Sent false 1785 Acked Write 8 Done Done Sent false 1786 Acked Write 1 Done Done Sent false 1787 Acked Write 3 Done Done Sent false 1788 Acked Read 6 Done Done Sent false 1789 Acked Read 6 Done Done Sent false 1790 Acked Read 6 Done Done Sent false 1791 Acked Read 8 Done Done Sent false 1792 Acked Write 6 Done Done Sent false 1793 Acked Read 8 Done Done Sent false 1794 Acked Read 9 Done Done Sent false 1795 Acked Write 8 Done Done Sent false 1796 Acked Write 3 Done Done Sent false 1797 Acked Read 4 Done Done Sent false 1798 Acked Write 8 Done Done Sent false 1799 Acked Read 4 Done Done Sent false 1800 Acked Write 5 Done Done Sent false 1801 Acked Read 5 Done Done Sent false 1802 Acked Write 3 Done Done Sent false 1803 Acked Read 5 Done Done Sent false 1804 Acked Write 5 Done Done Sent false 1805 Acked Read 9 Done Done Sent false 1806 Acked Read 2 Done Done Sent false 1807 Acked Read 4 Done Done Sent false 1808 Acked Read 8 Done Done Sent false 1809 Acked Write 1 Done Done Sent false 1810 Acked Write 1 Done Done Sent false 1811 Acked Read 8 Done Done Sent false 1812 Acked Write 4 Done Done Sent false 1813 Acked Write 4 Done Done Sent false 1814 Acked Read 3 Done Done Sent false 1815 Acked Flush 0 Done Done Sent false 1816 Acked Read 3 Done Done Sent false 1817 Acked Write 5 Done Done Sent false 1818 Acked Read 9 Done Done Sent false 1819 Acked Write 7 Done Done Sent false 1820 Acked Read 5 Done Done Sent false 1821 Acked Read 7 Done Done Sent false 1822 Acked Write 6 Done Done Sent false 1823 Acked Read 8 Done Done Sent false 1824 Acked Write 6 Done Done Sent false 1825 Acked Write 10 Done Done Sent false 1826 Acked Write 4 Done Done Sent false 1827 Acked Write 3 Done Done Sent false 1828 Acked Read 1 Done Done Sent false 1829 Acked Read 6 Done Done Sent false 1830 Acked Write 7 Done Done Sent false 1831 Acked Read 5 Done Done Sent false 1832 Acked Write 3 Done Done Sent false 1833 Acked Read 7 Done Done Sent false 1834 Acked Read 1 Done Done Sent false 1835 Acked Write 9 Done Done Sent false 1836 Acked Write 9 Done Done Sent false 1837 Acked Read 1 Done Done Sent false 1838 Acked Write 7 Done Done Sent false 1839 Acked Read 7 Done Done Sent false 1840 Acked Flush 0 Done Done Sent false 1841 Acked Write 10 Done Done Sent false 1842 Acked Write 1 Done Done Sent false 1843 Acked Read 8 Done Done Sent false 1844 Acked Write 6 Done Done Sent false 1845 Acked Read 6 Done Done Sent false 1846 Acked Read 8 Done Done Sent false 1847 Acked Write 3 Done Done Sent false 1848 Acked Write 10 Done Done Sent false 1849 Acked Flush 0 Done Done Sent false 1850 Acked Flush 0 Done Done Sent false 1851 Acked Write 5 Done Done Sent false 1852 Acked Write 8 Done Done Sent false 1853 Acked Read 4 Done Done Sent false 1854 Acked Flush 0 Done Done Sent false 1855 Acked Write 6 Done Done Sent false 1856 Acked Write 8 Done Done Sent false 1857 Acked Write 6 Done Done Sent false 1858 Acked Flush 0 Done Done Sent false 1859 Acked Write 1 Done Done Sent false 1860 Acked Flush 0 Done Done Sent false 1861 Acked Flush 0 Done Done Sent false 1862 Acked Write 8 Done Done Sent false 1863 Acked Write 10 Done Done Sent false 1864 Acked Read 6 Done Done Sent false 1865 Acked Read 5 Done Done Sent false 1866 Acked Write 7 Done Done Sent false 1867 Acked Read 10 Done Done Sent false 1868 Acked Read 3 Done Done Sent false 1869 Acked Write 7 Done Done Sent false 1870 Acked Flush 0 Done Done Sent false 1871 Acked Write 3 Done Done Sent false 1872 Acked Read 6 Done Done Sent false 1873 Acked Write 9 Done Done Sent false 1874 Acked Read 1 Done Done Sent false 1875 Acked Write 7 Done Done Sent false 1876 Acked Write 2 Done Done Sent false 1877 Acked Write 10 Done Done Sent false 1878 Acked Read 4 Done Done Sent false 1879 Acked Write 9 Done Done Sent false 1880 Acked Flush 0 Done Done Sent false 1881 Acked Write 7 Done Done Sent false 1882 Acked Read 4 Done Done Sent false 1883 Acked Write 3 Done Done Sent false 1884 Acked Write 7 Done Done Sent false 1885 Acked Write 9 Done Done Sent false 1886 Acked Flush 0 Done Done Sent false 1887 Acked Flush 0 Done Done Sent false 1888 Acked Read 1 Done Done Sent false 1889 Acked Read 1 Done Done Sent false 1890 Acked Read 8 Done Done Sent false 1891 Acked Read 3 Done Done Sent false 1892 Acked Read 6 Done Done Sent false 1893 Acked Write 1 Done Done Sent false 1894 Acked Read 2 Done Done Sent false 1895 Acked Write 2 Done Done Sent false 1896 Acked Read 7 Done Done Sent false 1897 Acked Read 1 Done Done Sent false 1898 Acked Write 6 Done Done Sent false 1899 Acked Read 10 Done Done Sent false 1900 Acked Write 6 Done Done Sent false 1901 Acked Write 5 Done Done Sent false 1902 Acked Read 6 Done Done Sent false 1903 Acked Flush 0 Done Done Sent false 1904 Acked Write 5 Done Done Sent false 1905 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1905 1905 1603 Downstairs last five completed: 1603 1602 1601 1600 1599 Upstairs last five completed: 1905 1904 1903 1902 1901 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:06.186365307Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:06.186570932Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:06.186597035Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:06.18661457Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:06.186721251Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1205) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1604 Acked Read 4 Done Done Sent false 1605 Acked Read 2 Done Done Sent false 1606 Acked Write 3 Done Done Sent false 1607 Acked Write 1 Done Done Sent false 1608 Acked Write 5 Done Done Sent false 1609 Acked Flush 0 Done Done Sent false 1610 Acked Read 9 Done Done Sent false 1611 Acked Read 4 Done Done Sent false 1612 Acked Read 4 Done Done Sent false 1613 Acked Read 9 Done Done Sent false 1614 Acked Read 9 Done Done Sent false 1615 Acked Read 2 Done Done Sent false 1616 Acked Read 2 Done Done Sent false 1617 Acked Read 3 Done Done Sent false 1618 Acked Write 7 Done Done Sent false 1619 Acked Read 3 Done Done Sent false 1620 Acked Write 4 Done Done Sent false 1621 Acked Read 6 Done Done Sent false 1622 Acked Write 9 Done Done Sent false 1623 Acked Read 5 Done Done Sent false 1624 Acked Read 8 Done Done Sent false 1625 Acked Read 2 Done Done Sent false 1626 Acked Read 10 Done Done Sent false 1627 Acked Read 2 Done Done Sent false 1628 Acked Read 2 Done Done Sent false 1629 Acked Write 9 Done Done Sent false 1630 Acked Read 6 Done Done Sent false 1631 Acked Read 10 Done Done Sent false 1632 Acked Read 4 Done Done Sent false 1633 Acked Read 9 Done Done Sent false 1634 Acked Write 8 Done Done Sent false 1635 Acked Read 3 Done Done Sent false 1636 Acked Read 7 Done Done Sent false 1637 Acked Write 3 Done Done Sent false 1638 Acked Flush 0 Done Done Sent false 1639 Acked Write 1 Done Done Sent false 1640 Acked Read 1 Done Done Sent false 1641 Acked Read 1 Done Done Sent false 1642 Acked Write 6 Done Done Sent false 1643 Acked Write 10 Done Done Sent false 1644 Acked Read 1 Done Done Sent false 1645 Acked Read 1 Done Done Sent false 1646 Acked Write 2 Done Done Sent false 1647 Acked Read 8 Done Done Sent false 1648 Acked Read 7 Done Done Sent false 1649 Acked Read 9 Done Done Sent false 1650 Acked Flush 0 Done Done Sent false 1651 Acked Flush 0 Done Done Sent false 1652 Acked Read 9 Done Done Sent false 1653 Acked Read 7 Done Done Sent false 1654 Acked Flush 0 Done Done Sent false 1655 Acked Write 1 Done Done Sent false 1656 Acked Flush 0 Done Done Sent false 1657 Acked Read 6 Done Done Sent false 1658 Acked Read 6 Done Done Sent false 1659 Acked Flush 0 Done Done Sent false 1660 Acked Read 5 Done Done Sent false 1661 Acked Write 1 Done Done Sent false 1662 Acked Read 8 Done Done Sent false 1663 Acked Write 9 Done Done Sent false 1664 Acked Read 4 Done Done Sent false 1665 Acked Read 8 Done Done Sent false 1666 Acked Read 8 Done Done Sent false 1667 Acked Write 4 Done Done Sent false 1668 Acked Read 2 Done Done Sent false 1669 Acked Read 1 Done Done Sent false 1670 Acked Read 9 Done Done Sent false 1671 Acked Write 3 Done Done Sent false 1672 Acked Flush 0 Done Done Sent false 1673 Acked Write 6 Done Done Sent false 1674 Acked Write 9 Done Done Sent false 1675 Acked Read 5 Done Done Sent false 1676 Acked Write 9 Done Done Sent false 1677 Acked Write 1 Done Done Sent false 1678 Acked Read 3 Done Done Sent false 1679 Acked Write 7 Done Done Sent false 1680 Acked Read 6 Done Done Sent false 1681 Acked Read 5 Done Done Sent false 1682 Acked Write 10 Done Done Sent false 1683 Acked Flush 0 Done Done Sent false 1684 Acked Write 8 Done Done Sent false 1685 Acked Read 8 Done Done Sent false 1686 Acked Read 4 Done Done Sent false 1687 Acked Flush 0 Done Done Sent false 1688 Acked Read 8 Done Done Sent false 1689 Acked Write 5 Done Done Sent false 1690 Acked Flush 0 Done Done Sent false 1691 Acked Read 10 Done Done Sent false 1692 Acked Write 6 Done Done Sent false 1693 Acked Write 6 Done Done Sent false 1694 Acked Read 1 Done Done Sent false 1695 Acked Write 10 Done Done Sent false 1696 Acked Write 3 Done Done Sent false 1697 Acked Write 1 Done Done Sent false 1698 Acked Read 7 Done Done Sent false 1699 Acked Write 2 Done Done Sent false 1700 Acked Read 10 Done Done Sent false 1701 Acked Read 3 Done Done Sent false 1702 Acked Write 1 Done Done Sent false 1703 Acked Write 4 Done Done Sent false 1704 Acked Read 6 Done Done Sent false 1705 Acked Flush 0 Done Done Sent false 1706 Acked Write 9 Done Done Sent false 1707 Acked Read 6 Done Done Sent false 1708 Acked Write 8 Done Done Sent false 1709 Acked Write 6 Done Done Sent false 1710 Acked Write 2 Done Done Sent false 1711 Acked Read 8 Done Done Sent false 1712 Acked Read 3 Done Done Sent false 1713 Acked Read 7 Done Done Sent false 1714 Acked Write 1 Done Done Sent false 1715 Acked Write 2 Done Done Sent false 1716 Acked Write 8 Done Done Sent false 1717 Acked Read 4 Done Done Sent false 1718 Acked Write 7 Done Done Sent false 1719 Acked Flush 0 Done Done Sent false 1720 Acked Read 10 Done Done Sent false 1721 Acked Flush 0 Done Done Sent false 1722 Acked Read 6 Done Done Sent false 1723 Acked Write 4 Done Done Sent false 1724 Acked Write 5 Done Done Sent false 1725 Acked Write 2 Done Done Sent false 1726 Acked Flush 0 Done Done Sent false 1727 Acked Write 5 Done Done Sent false 1728 Acked Write 8 Done Done Sent false 1729 Acked Write 7 Done Done Sent false 1730 Acked Write 5 Done Done Sent false 1731 Acked Write 9 Done Done Sent false 1732 Acked Write 4 Done Done Sent false 1733 Acked Read 1 Done Done Sent false 1734 Acked Read 8 Done Done Sent false 1735 Acked Write 8 Done Done Sent false 1736 Acked Read 8 Done Done Sent false 1737 Acked Read 1 Done Done Sent false 1738 Acked Read 7 Done Done Sent false 1739 Acked Flush 0 Done Done Sent false 1740 Acked Read 8 Done Done Sent false 1741 Acked Read 10 Done Done Sent false 1742 Acked Write 4 Done Done Sent false 1743 Acked Read 8 Done Done Sent false 1744 Acked Write 10 Done Done Sent false 1745 Acked Read 10 Done Done Sent false 1746 Acked Flush 0 Done Done Sent false 1747 Acked Write 5 Done Done Sent false 1748 Acked Read 6 Done Done Sent false 1749 Acked Write 5 Done Done Sent false 1750 Acked Write 6 Done Done Sent false 1751 Acked Read 10 Done Done Sent false 1752 Acked Read 5 Done Done Sent false 1753 Acked Read 8 Done Done Sent false 1754 Acked Read 7 Done Done Sent false 1755 Acked Read 9 Done Done Sent false 1756 Acked Read 5 Done Done Sent false 1757 Acked Write 3 Done Done Sent false 1758 Acked Read 8 Done Done Sent false 1759 Acked Read 2 Done Done Sent false 1760 Acked Write 7 Done Done Sent false 1761 Acked Write 4 Done Done Sent false 1762 Acked Read 8 Done Done Sent false 1763 Acked Write 2 Done Done Sent false 1764 Acked Write 6 Done Done Sent false 1765 Acked Write 9 Done Done Sent false 1766 Acked Write 2 Done Done Sent false 1767 Acked Write 3 Done Done Sent false 1768 Acked Write 3 Done Done Sent false 1769 Acked Write 3 Done Done Sent false 1770 Acked Read 10 Done Done Sent false 1771 Acked Flush 0 Done Done Sent false 1772 Acked Write 6 Done Done Sent false 1773 Acked Write 5 Done Done Sent false 1774 Acked Write 8 Done Done Sent false 1775 Acked Read 9 Done Done Sent false 1776 Acked Read 5 Done Done Sent false 1777 Acked Write 5 Done Done Sent false 1778 Acked Write 7 Done Done Sent false 1779 Acked Read 8 Done Done Sent false 1780 Acked Write 4 Done Done Sent false 1781 Acked Write 7 Done Done Sent false 1782 Acked Read 10 Done Done Sent false 1783 Acked Flush 0 Done Done Sent false 1784 Acked Flush 0 Done Done Sent false 1785 Acked Write 8 Done Done Sent false 1786 Acked Write 1 Done Done Sent false 1787 Acked Write 3 Done Done Sent false 1788 Acked Read 6 Done Done Sent false 1789 Acked Read 6 Done Done Sent false 1790 Acked Read 6 Done Done Sent false 1791 Acked Read 8 Done Done Sent false 1792 Acked Write 6 Done Done Sent false 1793 Acked Read 8 Done Done Sent false 1794 Acked Read 9 Done Done Sent false 1795 Acked Write 8 Done Done Sent false 1796 Acked Write 3 Done Done Sent false 1797 Acked Read 4 Done Done Sent false 1798 Acked Write 8 Done Done Sent false 1799 Acked Read 4 Done Done Sent false 1800 Acked Write 5 Done Done Sent false 1801 Acked Read 5 Done Done Sent false 1802 Acked Write 3 Done Done Sent false 1803 Acked Read 5 Done Done Sent false 1804 Acked Write 5 Done Done Sent false 1805 Acked Read 9 Done Done Sent false 1806 Acked Read 2 Done Done Sent false 1807 Acked Read 4 Done Done Sent false 1808 Acked Read 8 Done Done Sent false 1809 Acked Write 1 Done Done Sent false 1810 Acked Write 1 Done Done Sent false 1811 Acked Read 8 Done Done Sent false 1812 Acked Write 4 Done Done Sent false 1813 Acked Write 4 Done Done Sent false 1814 Acked Read 3 Done Done Sent false 1815 Acked Flush 0 Done Done Sent false 1816 Acked Read 3 Done Done Sent false 1817 Acked Write 5 Done Done Sent false 1818 Acked Read 9 Done Done Sent false 1819 Acked Write 7 Done Done Sent false 1820 Acked Read 5 Done Done Sent false 1821 Acked Read 7 Done Done Sent false 1822 Acked Write 6 Done Done Sent false 1823 Acked Read 8 Done Done Sent false 1824 Acked Write 6 Done Done Sent false 1825 Acked Write 10 Done Done Sent false 1826 Acked Write 4 Done Done Sent false 1827 Acked Write 3 Done Done Sent false 1828 Acked Read 1 Done Done Sent false 1829 Acked Read 6 Done Done Sent false 1830 Acked Write 7 Done Done Sent false 1831 Acked Read 5 Done Done Sent false 1832 Acked Write 3 Done Done Sent false 1833 Acked Read 7 Done Done Sent false 1834 Acked Read 1 Done Done Sent false 1835 Acked Write 9 Done Done Sent false 1836 Acked Write 9 Done Done Sent false 1837 Acked Read 1 Done Done Sent false 1838 Acked Write 7 Done Done Sent false 1839 Acked Read 7 Done Done Sent false 1840 Acked Flush 0 Done Done Sent false 1841 Acked Write 10 Done Done Sent false 1842 Acked Write 1 Done Done Sent false 1843 Acked Read 8 Done Done Sent false 1844 Acked Write 6 Done Done Sent false 1845 Acked Read 6 Done Done Sent false 1846 Acked Read 8 Done Done Sent false 1847 Acked Write 3 Done Done Sent false 1848 Acked Write 10 Done Done Sent false 1849 Acked Flush 0 Done Done Sent false 1850 Acked Flush 0 Done Done Sent false 1851 Acked Write 5 Done Done Sent false 1852 Acked Write 8 Done Done Sent false 1853 Acked Read 4 Done Done Sent false 1854 Acked Flush 0 Done Done Sent false 1855 Acked Write 6 Done Done Sent false 1856 Acked Write 8 Done Done Sent false 1857 Acked Write 6 Done Done Sent false 1858 Acked Flush 0 Done Done Sent false 1859 Acked Write 1 Done Done Sent false 1860 Acked Flush 0 Done Done Sent false 1861 Acked Flush 0 Done Done Sent false 1862 Acked Write 8 Done Done Sent false 1863 Acked Write 10 Done Done Sent false 1864 Acked Read 6 Done Done Sent false 1865 Acked Read 5 Done Done Sent false 1866 Acked Write 7 Done Done Sent false 1867 Acked Read 10 Done Done Sent false 1868 Acked Read 3 Done Done Sent false 1869 Acked Write 7 Done Done Sent false 1870 Acked Flush 0 Done Done Sent false 1871 Acked Write 3 Done Done Sent false 1872 Acked Read 6 Done Done Sent false 1873 Acked Write 9 Done Done Sent false 1874 Acked Read 1 Done Done Sent false 1875 Acked Write 7 Done Done Sent false 1876 Acked Write 2 Done Done Sent false 1877 Acked Write 10 Done Done Sent false 1878 Acked Read 4 Done Done Sent false 1879 Acked Write 9 Done Done Sent false 1880 Acked Flush 0 Done Done Sent false 1881 Acked Write 7 Done Done Sent false 1882 Acked Read 4 Done Done Sent false 1883 Acked Write 3 Done Done Sent false 1884 Acked Write 7 Done Done Sent false 1885 Acked Write 9 Done Done Sent false 1886 Acked Flush 0 Done Done Sent false 1887 Acked Flush 0 Done Done Sent false 1888 Acked Read 1 Done Done Sent false 1889 Acked Read 1 Done Done Sent false 1890 Acked Read 8 Done Done Sent false 1891 Acked Read 3 Done Done Sent false 1892 Acked Read 6 Done Done Sent false 1893 Acked Write 1 Done Done Sent false 1894 Acked Read 2 Done Done Sent false 1895 Acked Write 2 Done Done Sent false 1896 Acked Read 7 Done Done Sent false 1897 Acked Read 1 Done Done Sent false 1898 Acked Write 6 Done Done Sent false 1899 Acked Read 10 Done Done Sent false 1900 Acked Write 6 Done Done Sent false 1901 Acked Write 5 Done Done Sent false 1902 Acked Read 6 Done Done Sent false 1903 Acked Flush 0 Done Done Sent false 1904 Acked Write 5 Done Done Sent false 1905 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1905 1905 1603 Downstairs last five completed: 1603 1602 1601 1600 1599 Upstairs last five completed: 1905 1904 1903 1902 1901 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1604 Acked Read 4 Done Done Sent false 1605 Acked Read 2 Done Done Sent false 1606 Acked Write 3 Done Done Sent false 1607 Acked Write 1 Done Done Sent false 1608 Acked Write 5 Done Done Sent false 1609 Acked Flush 0 Done Done Sent false 1610 Acked Read 9 Done Done Sent false 1611 Acked Read 4 Done Done Sent false 1612 Acked Read 4 Done Done Sent false 1613 Acked Read 9 Done Done Sent false 1614 Acked Read 9 Done Done Sent false 1615 Acked Read 2 Done Done Sent false 1616 Acked Read 2 Done Done Sent false 1617 Acked Read 3 Done Done Sent false 1618 Acked Write 7 Done Done Sent false 1619 Acked Read 3 Done Done Sent false 1620 Acked Write 4 Done Done Sent false 1621 Acked Read 6 Done Done Sent false 1622 Acked Write 9 Done Done Sent false 1623 Acked Read 5 Done Done Sent false 1624 Acked Read 8 Done Done Sent false 1625 Acked Read 2 Done Done Sent false 1626 Acked Read 10 Done Done Sent false 1627 Acked Read 2 Done Done Sent false 1628 Acked Read 2 Done Done Sent false 1629 Acked Write 9 Done Done Sent false 1630 Acked Read 6 Done Done Sent false 1631 Acked Read 10 Done Done Sent false 1632 Acked Read 4 Done Done Sent false 1633 Acked Read 9 Done Done Sent false 1634 Acked Write 8 Done Done Sent false 1635 Acked Read 3 Done Done Sent false 1636 Acked Read 7 Done Done Sent false 1637 Acked Write 3 Done Done Sent false 1638 Acked Flush 0 Done Done Sent false 1639 Acked Write 1 Done Done Sent false 1640 Acked Read 1 Done Done Sent false 1641 Acked Read 1 Done Done Sent false 1642 Acked Write 6 Done Done Sent false 1643 Acked Write 10 Done Done Sent false 1644 Acked Read 1 Done Done Sent false 1645 Acked Read 1 Done Done Sent false 1646 Acked Write 2 Done Done Sent false 1647 Acked Read 8 Done Done Sent false 1648 Acked Read 7 Done Done Sent false 1649 Acked Read 9 Done Done Sent false 1650 Acked Flush 0 Done Done Sent false 1651 Acked Flush 0 Done Done Sent false 1652 Acked Read 9 Done Done Sent false 1653 Acked Read 7 Done Done Sent false 1654 Acked Flush 0 Done Done Sent false 1655 Acked Write 1 Done Done Sent false 1656 Acked Flush 0 Done Done Sent false 1657 Acked Read 6 Done Done Sent false 1658 Acked Read 6 Done Done Sent false 1659 Acked Flush 0 Done Done Sent false 1660 Acked Read 5 Done Done Sent false 1661 Acked Write 1 Done Done Sent false 1662 Acked Read 8 Done Done Sent false 1663 Acked Write 9 Done Done Sent false 1664 Acked Read 4 Done Done Sent false 1665 Acked Read 8 Done Done Sent false 1666 Acked Read 8 Done Done Sent false 1667 Acked Write 4 Done Done Sent false 1668 Acked Read 2 Done Done Sent false 1669 Acked Read 1 Done Done Sent false 1670 Acked Read 9 Done Done Sent false 1671 Acked Write 3 Done Done Sent false 1672 Acked Flush 0 Done Done Sent false 1673 Acked Write 6 Done Done Sent false 1674 Acked Write 9 Done Done Sent false 1675 Acked Read 5 Done Done Sent false 1676 Acked Write 9 Done Done Sent false 1677 Acked Write 1 Done Done Sent false 1678 Acked Read 3 Done Done Sent false 1679 Acked Write 7 Done Done Sent false 1680 Acked Read 6 Done Done Sent false 1681 Acked Read 5 Done Done Sent false 1682 Acked Write 10 Done Done Sent false 1683 Acked Flush 0 Done Done Sent false 1684 Acked Write 8 Done Done Sent false 1685 Acked Read 8 Done Done Sent false 1686 Acked Read 4 Done Done Sent false 1687 Acked Flush 0 Done Done Sent false 1688 Acked Read 8 Done Done Sent false 1689 Acked Write 5 Done Done Sent false 1690 Acked Flush 0 Done Done Sent false 1691 Acked Read 10 Done Done Sent false 1692 Acked Write 6 Done Done Sent false 1693 Acked Write 6 Done Done Sent false 1694 Acked Read 1 Done Done Sent false 1695 Acked Write 10 Done Done Sent false 1696 Acked Write 3 Done Done Sent false 1697 Acked Write 1 Done Done Sent false 1698 Acked Read 7 Done Done Sent false 1699 Acked Write 2 Done Done Sent false 1700 Acked Read 10 Done Done Sent false 1701 Acked Read 3 Done Done Sent false 1702 Acked Write 1 Done Done Sent false 1703 Acked Write 4 Done Done Sent false 1704 Acked Read 6 Done Done Sent false 1705 Acked Flush 0 Done Done Sent false 1706 Acked Write 9 Done Done Sent false 1707 Acked Read 6 Done Done Sent false 1708 Acked Write 8 Done Done Sent false 1709 Acked Write 6 Done Done Sent false 1710 Acked Write 2 Done Done Sent false 1711 Acked Read 8 Done Done Sent false 1712 Acked Read 3 Done Done Sent false 1713 Acked Read 7 Done Done Sent false 1714 Acked Write 1 Done Done Sent false 1715 Acked Write 2 Done Done Sent false 1716 Acked Write 8 Done Done Sent false 1717 Acked Read 4 Done Done Sent false 1718 Acked Write 7 Done Done Sent false 1719 Acked Flush 0 Done Done Sent false 1720 Acked Read 10 Done Done Sent false 1721 Acked Flush 0 Done Done Sent false 1722 Acked Read 6 Done Done Sent false 1723 Acked Write 4 Done Done Sent false 1724 Acked Write 5 Done Done Sent false 1725 Acked Write 2 Done Done Sent false 1726 Acked Flush 0 Done Done Sent false 1727 Acked Write 5 Done Done Sent false 1728 Acked Write 8 Done Done Sent false 1729 Acked Write 7 Done Done Sent false 1730 Acked Write 5 Done Done Sent false 1731 Acked Write 9 Done Done Sent false 1732 Acked Write 4 Done Done Sent false 1733 Acked Read 1 Done Done Sent false 1734 Acked Read 8 Done Done Sent false 1735 Acked Write 8 Done Done Sent false 1736 Acked Read 8 Done Done Sent false 1737 Acked Read 1 Done Done Sent false 1738 Acked Read 7 Done Done Sent false 1739 Acked Flush 0 Done Done Sent false 1740 Acked Read 8 Done Done Sent false 1741 Acked Read 10 Done Done Sent false 1742 Acked Write 4 Done Done Sent false 1743 Acked Read 8 Done Done Sent false 1744 Acked Write 10 Done Done Sent false 1745 Acked Read 10 Done Done Sent false 1746 Acked Flush 0 Done Done Sent false 1747 Acked Write 5 Done Done Sent false 1748 Acked Read 6 Done Done Sent false 1749 Acked Write 5 Done Done Sent false 1750 Acked Write 6 Done Done Sent false 1751 Acked Read 10 Done Done Sent false 1752 Acked Read 5 Done Done Sent false 1753 Acked Read 8 Done Done Sent false 1754 Acked Read 7 Done Done Sent false 1755 Acked Read 9 Done Done Sent false 1756 Acked Read 5 Done Done Sent false 1757 Acked Write 3 Done Done Sent false 1758 Acked Read 8 Done Done Sent false 1759 Acked Read 2 Done Done Sent false 1760 Acked Write 7 Done Done Sent false 1761 Acked Write 4 Done Done Sent false 1762 Acked Read 8 Done Done Sent false 1763 Acked Write 2 Done Done Sent false 1764 Acked Write 6 Done Done Sent false 1765 Acked Write 9 Done Done Sent false 1766 Acked Write 2 Done Done Sent false 1767 Acked Write 3 Done Done Sent false 1768 Acked Write 3 Done Done Sent false 1769 Acked Write 3 Done Done Sent false 1770 Acked Read 10 Done Done Sent false 1771 Acked Flush 0 Done Done Sent false 1772 Acked Write 6 Done Done Sent false 1773 Acked Write 5 Done Done Sent false 1774 Acked Write 8 Done Done Sent false 1775 Acked Read 9 Done Done Sent false 1776 Acked Read 5 Done Done Sent false 1777 Acked Write 5 Done Done Sent false 1778 Acked Write 7 Done Done Sent false 1779 Acked Read 8 Done Done Sent false 1780 Acked Write 4 Done Done Sent false 1781 Acked Write 7 Done Done Sent false 1782 Acked Read 10 Done Done Sent false 1783 Acked Flush 0 Done Done Sent false 1784 Acked Flush 0 Done Done Sent false 1785 Acked Write 8 Done Done Sent false 1786 Acked Write 1 Done Done Sent false 1787 Acked Write 3 Done Done Sent false 1788 Acked Read 6 Done Done Sent false 1789 Acked Read 6 Done Done Sent false 1790 Acked Read 6 Done Done Sent false 1791 Acked Read 8 Done Done Sent false 1792 Acked Write 6 Done Done Sent false 1793 Acked Read 8 Done Done Sent false 1794 Acked Read 9 Done Done Sent false 1795 Acked Write 8 Done Done Sent false 1796 Acked Write 3 Done Done Sent false 1797 Acked Read 4 Done Done Sent false 1798 Acked Write 8 Done Done Sent false 1799 Acked Read 4 Done Done Sent false 1800 Acked Write 5 Done Done Sent false 1801 Acked Read 5 Done Done Sent false 1802 Acked Write 3 Done Done Sent false 1803 Acked Read 5 Done Done Sent false 1804 Acked Write 5 Done Done Sent false 1805 Acked Read 9 Done Done Sent false 1806 Acked Read 2 Done Done Sent false 1807 Acked Read 4 Done Done Sent false 1808 Acked Read 8 Done Done Sent false 1809 Acked Write 1 Done Done Sent false 1810 Acked Write 1 Done Done Sent false 1811 Acked Read 8 Done Done Sent false 1812 Acked Write 4 Done Done Sent false 1813 Acked Write 4 Done Done Sent false 1814 Acked Read 3 Done Done Sent false 1815 Acked Flush 0 Done Done Sent false 1816 Acked Read 3 Done Done Sent false 1817 Acked Write 5 Done Done Sent false 1818 Acked Read 9 Done Done Sent false 1819 Acked Write 7 Done Done Sent false 1820 Acked Read 5 Done Done Sent false 1821 Acked Read 7 Done Done Sent false 1822 Acked Write 6 Done Done Sent false 1823 Acked Read 8 Done Done Sent false 1824 Acked Write 6 Done Done Sent false 1825 Acked Write 10 Done Done Sent false 1826 Acked Write 4 Done Done Sent false 1827 Acked Write 3 Done Done Sent false 1828 Acked Read 1 Done Done Sent false 1829 Acked Read 6 Done Done Sent false 1830 Acked Write 7 Done Done Sent false 1831 Acked Read 5 Done Done Sent false 1832 Acked Write 3 Done Done Sent false 1833 Acked Read 7 Done Done Sent false 1834 Acked Read 1 Done Done Sent false 1835 Acked Write 9 Done Done Sent false 1836 Acked Write 9 Done Done Sent false 1837 Acked Read 1 Done Done Sent false 1838 Acked Write 7 Done Done Sent false 1839 Acked Read 7 Done Done Sent false 1840 Acked Flush 0 Done Done Sent false 1841 Acked Write 10 Done Done Sent false 1842 Acked Write 1 Done Done Sent false 1843 Acked Read 8 Done Done Sent false 1844 Acked Write 6 Done Done Sent false 1845 Acked Read 6 Done Done Sent false 1846 Acked Read 8 Done Done Sent false 1847 Acked Write 3 Done Done Sent false 1848 Acked Write 10 Done Done Sent false 1849 Acked Flush 0 Done Done Sent false 1850 Acked Flush 0 Done Done Sent false 1851 Acked Write 5 Done Done Sent false 1852 Acked Write 8 Done Done Sent false 1853 Acked Read 4 Done Done Sent false 1854 Acked Flush 0 Done Done Sent false 1855 Acked Write 6 Done Done Sent false 1856 Acked Write 8 Done Done Sent false 1857 Acked Write 6 Done Done Sent false 1858 Acked Flush 0 Done Done Sent false 1859 Acked Write 1 Done Done Sent false 1860 Acked Flush 0 Done Done Sent false 1861 Acked Flush 0 Done Done Sent false 1862 Acked Write 8 Done Done Sent false 1863 Acked Write 10 Done Done Sent false 1864 Acked Read 6 Done Done Sent false 1865 Acked Read 5 Done Done Sent false 1866 Acked Write 7 Done Done Sent false 1867 Acked Read 10 Done Done Sent false 1868 Acked Read 3 Done Done Sent false 1869 Acked Write 7 Done Done Sent false 1870 Acked Flush 0 Done Done Sent false 1871 Acked Write 3 Done Done Sent false 1872 Acked Read 6 Done Done Sent false 1873 Acked Write 9 Done Done Sent false 1874 Acked Read 1 Done Done Sent false 1875 Acked Write 7 Done Done Sent false 1876 Acked Write 2 Done Done Sent false 1877 Acked Write 10 Done Done Sent false 1878 Acked Read 4 Done Done Sent false 1879 Acked Write 9 Done Done Sent false 1880 Acked Flush 0 Done Done Sent false 1881 Acked Write 7 Done Done Sent false 1882 Acked Read 4 Done Done Sent false 1883 Acked Write 3 Done Done Sent false 1884 Acked Write 7 Done Done Sent false 1885 Acked Write 9 Done Done Sent false 1886 Acked Flush 0 Done Done Sent false 1887 Acked Flush 0 Done Done Sent false 1888 Acked Read 1 Done Done Sent false 1889 Acked Read 1 Done Done Sent false 1890 Acked Read 8 Done Done Sent false 1891 Acked Read 3 Done Done Sent false 1892 Acked Read 6 Done Done Sent false 1893 Acked Write 1 Done Done Sent false 1894 Acked Read 2 Done Done Sent false 1895 Acked Write 2 Done Done Sent false 1896 Acked Read 7 Done Done Sent false 1897 Acked Read 1 Done Done Sent false 1898 Acked Write 6 Done Done Sent false 1899 Acked Read 10 Done Done Sent false 1900 Acked Write 6 Done Done Sent false 1901 Acked Write 5 Done Done Sent false 1902 Acked Read 6 Done Done Sent false 1903 Acked Flush 0 Done Done Sent false 1904 Acked Write 5 Done Done Sent false 1905 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1905 1905 1603 Downstairs last five completed: 1603 1602 1601 1600 1599 Upstairs last five completed: 1905 1904 1903 1902 1901 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.187148167Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:34612)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.187288619Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:16.188308525Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.188819548Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.188844671Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 1603","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.188859857Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 1603","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.189075799Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.189099813Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.189114329Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 1603","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.189128205Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 1603","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.189415368Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1905 1904 1903 1902 1901 Upstairs last five completed: 1905 1904 1903 1902 1901 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:42:16.422 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false method: POST, req_id: 3569995c-2ad8-4ccf-ac60-d74e89d1edd2, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:16.422913924Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:16.422951174Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:16.422968679Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.422983025Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.4230016Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:16.423102563Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:42:16.423 INFO request completed, latency_us: 151, response_code: 200, uri: /state/cid/2, method: GET, req_id: 24cc8373-b86b-46ef-980f-4f71f995b4aa, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:42:20.424 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: 45830a78-bff1-45c5-8257-5209615935e5, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 1326 len 4096 002/300 Read block 1477 len 36864 003/300 Read block 838 len 32768 004/300 Read block 4515 len 20480 005/300 Read block 2538 len 16384 006/300 Write block 2349 len 28672 data: 4 5 5 4 3 3 2 007/300 Flush 008/300 Read block 3330 len 28672 009/300 Read block 2024 len 12288 010/300 Flush 011/300 Read block 3474 len 32768 012/300 Write block 4968 len 12288 data: 2 2 2 013/300 Read block 515 len 40960 014/300 Read block 4885 len 40960 015/300 Read block 1519 len 12288 016/300 Flush 017/300 Read block 129 len 8192 018/300 Read block 1175 len 24576 019/300 Read block 190 len 12288 020/300 Write block 2399 len 40960 data: 2 2 2 2 2 3 3 3 3 3 021/300 Read block 319 len 4096 022/300 Read block 4984 len 24576 023/300 Read block 4458 len 32768 024/300 Read block 3307 len 28672 025/300 Write block 1452 len 40960 data: 3 3 3 3 2 2 2 2 2 2 026/300 Read block 3590 len 12288 027/300 Write block 20 len 36864 data: 3 3 2 2 2 2 2 3 2 028/300 Read block 694 len 36864 029/300 Flush 030/300 Read block 4809 len 16384 031/300 Read block 2884 len 40960 032/300 Read block 2943 len 4096 033/300 Read block 3969 len 4096 034/300 Read block 847 len 8192 035/300 Read block 694 len 12288 036/300 Read block 2797 len 12288 037/300 Flush 038/300 Read block 3281 len 24576 039/300 Write block 2688 len 36864 data: 3 2 2 2 2 2 2 2 2 040/300 Write block 705 len 4096 data: 2 041/300 Write block 4094 len 12288 data: 3 3 2 042/300 Read block 4368 len 16384 043/300 Read block 743 len 36864 044/300 Write block 2440 len 16384 data: 4 4 4 3 045/300 Read block 4817 len 28672 046/300 Write block 4980 len 36864 data: 3 3 4 4 4 2 2 2 2 047/300 Read block 1425 len 16384 048/300 Read block 3785 len 24576 049/300 Read block 1275 len 40960 050/300 Read block 3342 len 12288 051/300 Write block 1578 len 16384 data: 2 2 2 2 052/300 Read block 3867 len 32768 053/300 Read block 589 len 24576 054/300 Read block 4404 len 4096 055/300 Read block 595 len 4096 056/300 Flush 057/300 Read block 4702 len 40960 058/300 Write block 4278 len 12288 data: 2 2 2 059/300 Write block 1222 len 36864 data: 4 4 4 4 4 3 3 3 3 060/300 Write block 220 len 32768 data: 3 3 2 2 2 2 2 2 061/300 Read block 2725 len 8192 062/300 Write block 1573 len 16384 data: 3 3 3 3 063/300 Read block 1087 len 12288 064/300 Read block 2548 len 4096 065/300 Write block 2560 len 40960 data: 4 4 4 3 3 3 3 3 2 2 066/300 Read block 3791 len 8192 067/300 Read block 138 len 12288 068/300 Flush 069/300 Read block 635 len 4096 070/300 Read block 277 len 8192 071/300 Read block 312 len 24576 072/300 Write block 2383 len 36864 data: 2 2 2 2 2 2 2 2 2 073/300 Read block 2279 len 36864 074/300 Read block 2606 len 20480 075/300 Flush 076/300 Read block 367 len 16384 077/300 Flush 078/300 Write block 2665 len 12288 data: 2 2 2 079/300 Read block 3565 len 24576 080/300 Read block 4510 len 12288 081/300 Write block 3319 len 28672 data: 3 3 3 2 2 2 2 082/300 Flush 083/300 Read block 4059 len 16384 084/300 Write block 2796 len 32768 data: 2 2 2 2 2 3 3 3 085/300 Read block 1634 len 12288 086/300 Read block 4240 len 20480 087/300 Write block 3491 len 4096 data: 3 088/300 Read block 4214 len 28672 089/300 Read block 3363 len 28672 090/300 Read block 2618 len 40960 091/300 Write block 1626 len 40960 data: 3 3 3 3 3 3 2 2 2 2 092/300 Read block 787 len 8192 093/300 Read block 4130 len 16384 094/300 Write block 2957 len 4096 data: 2 095/300 Read block 4212 len 20480 096/300 Read block 608 len 28672 097/300 Read block 2965 len 40960 098/300 Write block 3726 len 20480 data: 2 2 2 3 4 099/300 Read block 3219 len 20480 100/300 Write block 4200 len 12288 data: 3 3 2 101/300 Read block 3306 len 4096 102/300 Read block 3959 len 28672 103/300 Read block 1171 len 12288 104/300 Write block 2563 len 32768 data: 4 4 4 4 4 3 3 2 105/300 Read block 1950 len 24576 106/300 Write block 519 len 8192 data: 2 2 107/300 Write block 428 len 12288 data: 2 2 2 108/300 Write block 1091 len 36864 data: 2 3 3 4 4 3 3 2 2 109/300 Write block 3157 len 40960 data: 3 3 3 2 2 2 2 2 3 3 110/300 Write block 3830 len 40960 data: 2 2 2 2 2 2 2 2 2 2 111/300 Read block 963 len 20480 112/300 Read block 1381 len 16384 113/300 Write block 4336 len 20480 data: 4 2 2 2 2 114/300 Write block 79 len 20480 data: 2 2 2 2 2 115/300 Read block 4898 len 28672 116/300 Write block 4525 len 36864 data: 2 2 2 2 2 2 2 2 2 117/300 Read block 3285 len 32768 118/300 Read block 2164 len 36864 119/300 Read block 1800 len 12288 120/300 Flush 121/300 Write block 711 len 8192 data: 2 2 122/300 Read block 4384 len 36864 123/300 Write block 1019 len 24576 data: 2 2 2 2 2 2 124/300 Write block 2058 len 36864 data: 2 2 2 2 2 2 2 2 2 125/300 Read block 2906 len 4096 126/300 Read block 1886 len 4096 127/300 Write block 3011 len 12288 data: 2 3 3 128/300 Read block 2329 len 4096 129/300 Write block 4938 len 20480 data: 2 2 2 2 2 130/300 Write block 2566 len 32768 data: 5 5 4 4 3 2 2 2 131/300 Write block 552 len 4096 data: 2 132/300 Write block 1040 len 28672 data: 2 3 2 2 3 4 4 133/300 Write block 1681 len 40960 data: 2 2 2 2 2 2 2 2 2 2 134/300 Read block 1442 len 20480 135/300 Read block 598 len 28672 136/300 Read block 2161 len 4096 137/300 Read block 2074 len 8192 138/300 Read block 3504 len 40960 139/300 Read block 4368 len 40960 140/300 Read block 202 len 4096 141/300 Read block 3219 len 20480 142/300 Write block 1279 len 24576 data: 2 2 2 2 2 3 143/300 Read block 4696 len 4096 144/300 Read block 629 len 40960 145/300 Read block 3924 len 8192 146/300 Write block 2398 len 24576 data: 3 3 3 3 3 3 147/300 Read block 3544 len 40960 148/300 Write block 2762 len 16384 data: 3 3 3 3 149/300 Read block 2153 len 16384 150/300 Read block 3636 len 16384 151/300 Read block 1268 len 28672 152/300 Write block 4849 len 12288 data: 2 2 2 153/300 Read block 114 len 16384 154/300 Read block 973 len 36864 155/300 Write block 1592 len 4096 data: 2 156/300 Flush 157/300 Read block 893 len 32768 158/300 Write block 2644 len 36864 data: 2 2 2 2 3 3 3 3 3 159/300 Read block 2155 len 40960 160/300 Read block 933 len 40960 161/300 Write block 824 len 12288 data: 2 2 2 162/300 Read block 3010 len 40960 163/300 Write block 4154 len 12288 data: 3 2 2 164/300 Read block 2061 len 32768 165/300 Write block 4030 len 4096 data: 3 166/300 Write block 1830 len 16384 data: 2 2 2 2 167/300 Write block 4196 len 20480 data: 3 3 3 3 4 168/300 Write block 4500 len 36864 data: 2 2 2 2 2 2 2 2 2 169/300 Read block 1157 len 28672 170/300 Flush 171/300 Write block 4636 len 8192 data: 2 2 172/300 Read block 2384 len 32768 173/300 Write block 3562 len 16384 data: 2 2 2 2 174/300 Write block 341 len 28672 data: 2 2 2 2 2 2 2 175/300 Write block 4921 len 28672 data: 2 2 2 2 2 2 2 176/300 Read block 2493 len 12288 177/300 Read block 477 len 12288 178/300 Read block 2955 len 16384 179/300 Read block 1381 len 4096 180/300 Read block 3216 len 32768 181/300 Flush 182/300 Read block 3521 len 20480 183/300 Write block 4093 len 36864 data: 3 4 4 3 2 2 2 2 2 184/300 Read block 2091 len 24576 185/300 Write block 2333 len 8192 data: 5 4 186/300 Read block 1752 len 40960 187/300 Write block 445 len 12288 data: 2 2 2 188/300 Write block 3327 len 24576 data: 2 2 2 2 2 2 189/300 Flush 190/300 Read block 3319 len 24576 191/300 Flush 192/300 Write block 3459 len 16384 data: 2 2 2 2 193/300 Write block 697 len 8192 data: 3 3 194/300 Read block 3531 len 36864 195/300 Read block 722 len 16384 196/300 Write block 1404 len 20480 data: 3 3 3 3 3 197/300 Read block 1119 len 8192 198/300 Read block 3173 len 8192 199/300 Flush 200/300 Read block 4409 len 24576 201/300 Read block 4482 len 36864 202/300 Read block 898 len 12288 203/300 Write block 64 len 12288 data: 2 3 3 204/300 Read block 4975 len 20480 205/300 Flush 206/300 Write block 2795 len 12288 data: 2 3 3 207/300 Write block 789 len 12288 data: 2 2 2 208/300 Read block 1002 len 36864 209/300 Flush 210/300 Write block 4124 len 28672 data: 2 2 2 2 2 2 2 211/300 Read block 4294 len 28672 212/300 Write block 2867 len 20480 data: 3 3 3 3 3 213/300 Read block 381 len 28672 214/300 Write block 1695 len 24576 data: 3 2 2 2 2 2 215/300 Read block 2903 len 24576 216/300 Read block 714 len 4096 217/300 Read block 1212 len 16384 218/300 Write block 744 len 36864 data: 2 2 3 3 3 4 4 4 5 219/300 Write block 2402 len 20480 data: 4 4 4 4 4 220/300 Read block 305 len 16384 221/300 Read block 3562 len 24576 222/300 Write block 4976 len 40960 data: 3 3 4 3 4 4 5 5 5 3 223/300 Write block 3072 len 4096 data: 2 224/300 Read block 2510 len 36864 225/300 Read block 4352 len 4096 226/300 Read block 4255 len 40960 227/300 Write block 4676 len 32768 data: 2 2 3 3 3 3 3 2 228/300 Read block 52 len 24576 229/300 Read block 476 len 28672 230/300 Write block 4548 len 32768 data: 2 3 3 3 3 3 2 2 231/300 Read block 3781 len 12288 232/300 Read block 2620 len 12288 233/300 Read block 4865 len 20480 234/300 Read block 499 len 32768 235/300 Write block 3301 len 20480 data: 2 2 2 2 2 236/300 Read block 1429 len 12288 237/300 Read block 1693 len 4096 238/300 Write block 4899 len 20480 data: 2 2 2 2 2 239/300 Read block 211 len 4096 240/300 Write block 1150 len 12288 data: 2 2 2 241/300 Read block 3050 len 4096 242/300 Flush 243/300 Flush 244/300 Write block 1771 len 28672 data: 2 3 3 3 3 3 3 245/300 Write block 2983 len 40960 data: 2 2 2 2 2 2 2 2 2 2 246/300 Read block 4182 len 40960 247/300 Read block 2416 len 32768 248/300 Read block 1526 len 40960 249/300 Read block 2191 len 16384 250/300 Write block 722 len 8192 data: 2 2 251/300 Flush 252/300 Write block 2936 len 28672 data: 3 3 3 3 3 3 3 253/300 Read block 4780 len 32768 254/300 Write block 432 len 32768 data: 2 3 3 3 2 2 2 2 255/300 Read block 849 len 40960 256/300 Flush 257/300 Read block 2788 len 16384 258/300 Read block 2766 len 28672 259/300 Read block 2550 len 24576 260/300 Read block 4957 len 16384 261/300 Write block 4896 len 32768 data: 2 2 2 3 3 3 3 3 262/300 Write block 4362 len 32768 data: 2 2 2 2 2 2 2 2 263/300 Write block 2778 len 8192 data: 2 2 264/300 Write block 3773 len 20480 data: 3 3 3 3 3 265/300 Read block 2499 len 4096 266/300 Read block 1586 len 32768 267/300 Read block 4734 len 24576 268/300 Read block 4024 len 12288 269/300 Read block 4279 len 12288 270/300 Write block 4354 len 4096 data: 2 271/300 Write block 3462 len 36864 data: 3 2 2 2 2 2 2 2 2 272/300 Read block 1079 len 20480 273/300 Write block 4937 len 36864 data: 2 3 3 3 3 3 2 2 2 274/300 Flush 275/300 Write block 2115 len 32768 data: 3 3 3 3 3 3 3 2 276/300 Read block 4540 len 4096 277/300 Read block 227 len 12288 278/300 Flush 279/300 Write block 2823 len 40960 data: 2 2 2 2 2 2 2 2 2 2 280/300 Read block 2246 len 20480 281/300 Write block 650 len 24576 data: 2 2 2 2 2 3 282/300 Write block 3200 len 12288 data: 2 2 2 283/300 Read block 3251 len 40960 284/300 Read block 3586 len 36864 285/300 Read block 4853 len 16384 286/300 Read block 1980 len 16384 287/300 Write block 1614 len 8192 data: 2 2 288/300 Flush 289/300 Read block 3961 len 28672 290/300 Read block 4053 len 40960 291/300 Read block 4031 len 4096 292/300 Write block 4371 len 16384 data: 2 2 2 2 293/300 Write block 1822 len 24576 data: 4 4 3 3 4 3 294/300 Read block 10 len 20480 295/300 Read block 3439 len 4096 296/300 Read block 297 len 16384 297/300 Read block 4257 len 16384 298/300 Read block 4015 len 24576 299/300 Write block 4962 len 12288 data: 2 2 2 300/300 Write block 714 len 28672 data: 2 2 2 2 2 2 2 301/300 Flush Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:42:20.667 INFO request completed, latency_us: 229, response_code: 204, uri: /start/cid/2, method: POST, req_id: 614da3f9-1e38-4a24-8a7c-bdacb776b9f2, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1906 Acked Read 1 Done Done Sent false 1907 Acked Read 9 Done Done Sent false 1908 Acked Read 8 Done Done Sent false 1909 Acked Read 5 Done Done Sent false 1910 Acked Read 4 Done Done Sent false 1911 Acked Write 7 Done Done Sent false 1912 Acked Flush 0 Done Done Sent false 1913 Acked Read 7 Done Done Sent false 1914 Acked Read 3 Done Done Sent false 1915 Acked Flush 0 Done Done Sent false 1916 Acked Read 8 Done Done Sent false 1917 Acked Write 3 Done Done Sent false 1918 Acked Read 10 Done Done Sent false 1919 Acked Read 10 Done Done Sent false 1920 Acked Read 3 Done Done Sent false 1921 Acked Flush 0 Done Done Sent false 1922 Acked Read 2 Done Done Sent false 1923 Acked Read 6 Done Done Sent false 1924 Acked Read 3 Done Done Sent false 1925 Acked Write 10 Done Done Sent false 1926 Acked Read 1 Done Done Sent false 1927 Acked Read 6 Done Done Sent false 1928 Acked Read 8 Done Done Sent false 1929 Acked Read 7 Done Done Sent false 1930 Acked Write 10 Done Done Sent false 1931 Acked Read 3 Done Done Sent false 1932 Acked Write 9 Done Done Sent false 1933 Acked Read 9 Done Done Sent false 1934 Acked Flush 0 Done Done Sent false 1935 Acked Read 4 Done Done Sent false 1936 Acked Read 10 Done Done Sent false 1937 Acked Read 1 Done Done Sent false 1938 Acked Read 1 Done Done Sent false 1939 Acked Read 2 Done Done Sent false 1940 Acked Read 3 Done Done Sent false 1941 Acked Read 3 Done Done Sent false 1942 Acked Flush 0 Done Done Sent false 1943 Acked Read 6 Done Done Sent false 1944 Acked Write 9 Done Done Sent false 1945 Acked Write 1 Done Done Sent false 1946 Acked Write 3 Done Done Sent false 1947 Acked Read 4 Done Done Sent false 1948 Acked Read 9 Done Done Sent false 1949 Acked Write 4 Done Done Sent false 1950 Acked Read 7 Done Done Sent false 1951 Acked Write 9 Done Done Sent false 1952 Acked Read 4 Done Done Sent false 1953 Acked Read 6 Done Done Sent false 1954 Acked Read 10 Done Done Sent false 1955 Acked Read 3 Done Done Sent false 1956 Acked Write 4 Done Done Sent false 1957 Acked Read 8 Done Done Sent false 1958 Acked Read 6 Done Done Sent false 1959 Acked Read 1 Done Done Sent false 1960 Acked Read 1 Done Done Sent false 1961 Acked Flush 0 Done Done Sent false 1962 Acked Read 10 Done Done Sent false 1963 Acked Write 3 Done Done Sent false 1964 Acked Write 9 Done Done Sent false 1965 Acked Write 8 Done Done Sent false 1966 Acked Read 2 Done Done Sent false 1967 Acked Write 4 Done Done Sent false 1968 Acked Read 3 Done Done Sent false 1969 Acked Read 1 Done Done Sent false 1970 Acked Write 10 Done Done Sent false 1971 Acked Read 2 Done Done Sent false 1972 Acked Read 3 Done Done Sent false 1973 Acked Flush 0 Done Done Sent false 1974 Acked Read 1 Done Done Sent false 1975 Acked Read 2 Done Done Sent false 1976 Acked Read 6 Done Done Sent false 1977 Acked Write 9 Done Done Sent false 1978 Acked Read 9 Done Done Sent false 1979 Acked Read 5 Done Done Sent false 1980 Acked Flush 0 Done Done Sent false 1981 Acked Read 4 Done Done Sent false 1982 Acked Flush 0 Done Done Sent false 1983 Acked Write 3 Done Done Sent false 1984 Acked Read 6 Done Done Sent false 1985 Acked Read 3 Done Done Sent false 1986 Acked Write 7 Done Done Sent false 1987 Acked Flush 0 Done Done Sent false 1988 Acked Read 4 Done Done Sent false 1989 Acked Write 8 Done Done Sent false 1990 Acked Read 3 Done Done Sent false 1991 Acked Read 5 Done Done Sent false 1992 Acked Write 1 Done Done Sent false 1993 Acked Read 7 Done Done Sent false 1994 Acked Read 7 Done Done Sent false 1995 Acked Read 10 Done Done Sent false 1996 Acked Write 10 Done Done Sent false 1997 Acked Read 2 Done Done Sent false 1998 Acked Read 4 Done Done Sent false 1999 Acked Write 1 Done Done Sent false 2000 Acked Read 5 Done Done Sent false 2001 Acked Read 7 Done Done Sent false 2002 Acked Read 10 Done Done Sent false 2003 Acked Write 5 Done Done Sent false 2004 Acked Read 5 Done Done Sent false 2005 Acked Write 3 Done Done Sent false 2006 Acked Read 1 Done Done Sent false 2007 Acked Read 7 Done Done Sent false 2008 Acked Read 3 Done Done Sent false 2009 Acked Write 8 Done Done Sent false 2010 Acked Read 6 Done Done Sent false 2011 Acked Write 2 Done Done Sent false 2012 Acked Write 3 Done Done Sent false 2013 Acked Write 9 Done Done Sent false 2014 Acked Write 10 Done Done Sent false 2015 Acked Write 10 Done Done Sent false 2016 Acked Read 5 Done Done Sent false 2017 Acked Read 4 Done Done Sent false 2018 Acked Write 5 Done Done Sent false 2019 Acked Write 5 Done Done Sent false 2020 Acked Read 7 Done Done Sent false 2021 Acked Write 9 Done Done Sent false 2022 Acked Read 8 Done Done Sent false 2023 Acked Read 9 Done Done Sent false 2024 Acked Read 3 Done Done Sent false 2025 Acked Flush 0 Done Done Sent false 2026 Acked Write 2 Done Done Sent false 2027 Acked Read 9 Done Done Sent false 2028 Acked Write 6 Done Done Sent false 2029 Acked Write 9 Done Done Sent false 2030 Acked Read 1 Done Done Sent false 2031 Acked Read 1 Done Done Sent false 2032 Acked Write 3 Done Done Sent false 2033 Acked Read 1 Done Done Sent false 2034 Acked Write 5 Done Done Sent false 2035 Acked Write 8 Done Done Sent false 2036 Acked Write 1 Done Done Sent false 2037 Acked Write 7 Done Done Sent false 2038 Acked Write 10 Done Done Sent false 2039 Acked Read 5 Done Done Sent false 2040 Acked Read 7 Done Done Sent false 2041 Acked Read 1 Done Done Sent false 2042 Acked Read 2 Done Done Sent false 2043 Acked Read 10 Done Done Sent false 2044 Acked Read 10 Done Done Sent false 2045 Acked Read 1 Done Done Sent false 2046 Acked Read 5 Done Done Sent false 2047 Acked Write 6 Done Done Sent false 2048 Acked Read 1 Done Done Sent false 2049 Acked Read 10 Done Done Sent false 2050 Acked Read 2 Done Done Sent false 2051 Acked Write 6 Done Done Sent false 2052 Acked Read 10 Done Done Sent false 2053 Acked Write 4 Done Done Sent false 2054 Acked Read 4 Done Done Sent false 2055 Acked Read 4 Done Done Sent false 2056 Acked Read 7 Done Done Sent false 2057 Acked Write 3 Done Done Sent false 2058 Acked Read 4 Done Done Sent false 2059 Acked Read 9 Done Done Sent false 2060 Acked Write 1 Done Done Sent false 2061 Acked Flush 0 Done Done Sent false 2062 Acked Read 8 Done Done Sent false 2063 Acked Write 9 Done Done Sent false 2064 Acked Read 10 Done Done Sent false 2065 Acked Read 10 Done Done Sent false 2066 Acked Write 3 Done Done Sent false 2067 Acked Read 10 Done Done Sent false 2068 Acked Write 3 Done Done Sent false 2069 Acked Read 8 Done Done Sent false 2070 Acked Write 1 Done Done Sent false 2071 Acked Write 4 Done Done Sent false 2072 Acked Write 5 Done Done Sent false 2073 Acked Write 9 Done Done Sent false 2074 Acked Read 7 Done Done Sent false 2075 Acked Flush 0 Done Done Sent false 2076 Acked Write 2 Done Done Sent false 2077 Acked Read 8 Done Done Sent false 2078 Acked Write 4 Done Done Sent false 2079 Acked Write 7 Done Done Sent false 2080 Acked Write 7 Done Done Sent false 2081 Acked Read 3 Done Done Sent false 2082 Acked Read 3 Done Done Sent false 2083 Acked Read 4 Done Done Sent false 2084 Acked Read 1 Done Done Sent false 2085 Acked Read 8 Done Done Sent false 2086 Acked Flush 0 Done Done Sent false 2087 Acked Read 5 Done Done Sent false 2088 Acked Write 9 Done Done Sent false 2089 Acked Read 6 Done Done Sent false 2090 Acked Write 2 Done Done Sent false 2091 Acked Read 10 Done Done Sent false 2092 Acked Write 3 Done Done Sent false 2093 Acked Write 6 Done Done Sent false 2094 Acked Flush 0 Done Done Sent false 2095 Acked Read 6 Done Done Sent false 2096 Acked Flush 0 Done Done Sent false 2097 Acked Write 4 Done Done Sent false 2098 Acked Write 2 Done Done Sent false 2099 Acked Read 9 Done Done Sent false 2100 Acked Read 4 Done Done Sent false 2101 Acked Write 5 Done Done Sent false 2102 Acked Read 2 Done Done Sent false 2103 Acked Read 2 Done Done Sent false 2104 Acked Flush 0 Done Done Sent false 2105 Acked Read 6 Done Done Sent false 2106 Acked Read 9 Done Done Sent false 2107 Acked Read 3 Done Done Sent false 2108 Acked Write 3 Done Done Sent false 2109 Acked Read 5 Done Done Sent false 2110 Acked Flush 0 Done Done Sent false 2111 Acked Write 3 Done Done Sent false 2112 Acked Write 3 Done Done Sent false 2113 Acked Read 9 Done Done Sent false 2114 Acked Flush 0 Done Done Sent false 2115 Acked Write 7 Done Done Sent false 2116 Acked Read 7 Done Done Sent false 2117 Acked Write 5 Done Done Sent false 2118 Acked Read 7 Done Done Sent false 2119 Acked Write 6 Done Done Sent false 2120 Acked Read 6 Done Done Sent false 2121 Acked Read 1 Done Done Sent false 2122 Acked Read 4 Done Done Sent false 2123 Acked Write 9 Done Done Sent false 2124 Acked Write 5 Done Done Sent false 2125 Acked Read 4 Done Done Sent false 2126 Acked Read 6 Done Done Sent false 2127 Acked Write 10 Done Done Sent false 2128 Acked Write 1 Done Done Sent false 2129 Acked Read 9 Done Done Sent false 2130 Acked Read 1 Done Done Sent false 2131 Acked Read 10 Done Done Sent false 2132 Acked Write 8 Done Done Sent false 2133 Acked Read 6 Done Done Sent false 2134 Acked Read 7 Done Done Sent false 2135 Acked Write 8 Done Done Sent false 2136 Acked Read 3 Done Done Sent false 2137 Acked Read 3 Done Done Sent false 2138 Acked Read 5 Done Done Sent false 2139 Acked Read 8 Done Done Sent false 2140 Acked Write 5 Done Done Sent false 2141 Acked Read 3 Done Done Sent false 2142 Acked Read 1 Done Done Sent false 2143 Acked Write 5 Done Done Sent false 2144 Acked Read 1 Done Done Sent false 2145 Acked Write 3 Done Done Sent false 2146 Acked Read 1 Done Done Sent false 2147 Acked Flush 0 Done Done Sent false 2148 Acked Flush 0 Done Done Sent false 2149 Acked Write 7 Done Done Sent false 2150 Acked Write 10 Done Done Sent false 2151 Acked Read 10 Done Done Sent false 2152 Acked Read 8 Done Done Sent false 2153 Acked Read 10 Done Done Sent false 2154 Acked Read 4 Done Done Sent false 2155 Acked Write 2 Done Done Sent false 2156 Acked Flush 0 Done Done Sent false 2157 Acked Write 7 Done Done Sent false 2158 Acked Read 8 Done Done Sent false 2159 Acked Write 8 Done Done Sent false 2160 Acked Read 10 Done Done Sent false 2161 Acked Flush 0 Done Done Sent false 2162 Acked Read 4 Done Done Sent false 2163 Acked Read 7 Done Done Sent false 2164 Acked Read 6 Done Done Sent false 2165 Acked Read 4 Done Done Sent false 2166 Acked Write 8 Done Done Sent false 2167 Acked Write 8 Done Done Sent false 2168 Acked Write 2 Done Done Sent false 2169 Acked Write 5 Done Done Sent false 2170 Acked Read 1 Done Done Sent false 2171 Acked Read 8 Done Done Sent false 2172 Acked Read 6 Done Done Sent false 2173 Acked Read 3 Done Done Sent false 2174 Acked Read 3 Done Done Sent false 2175 Acked Write 1 Done Done Sent false 2176 Acked Write 9 Done Done Sent false 2177 Acked Read 5 Done Done Sent false 2178 Acked Write 9 Done Done Sent false 2179 Acked Flush 0 Done Done Sent false 2180 Acked Write 8 Done Done Sent false 2181 Acked Read 1 Done Done Sent false 2182 Acked Read 3 Done Done Sent false 2183 Acked Flush 0 Done Done Sent false 2184 Acked Write 10 Done Done Sent false 2185 Acked Read 5 Done Done Sent false 2186 Acked Write 6 Done Done Sent false 2187 Acked Write 3 Done Done Sent false 2188 Acked Read 10 Done Done Sent false 2189 Acked Read 9 Done Done Sent false 2190 Acked Read 4 Done Done Sent false 2191 Acked Read 4 Done Done Sent false 2192 Acked Write 2 Done Done Sent false 2193 Acked Flush 0 Done Done Sent false 2194 Acked Read 7 Done Done Sent false 2195 Acked Read 10 Done Done Sent false 2196 Acked Read 1 Done Done Sent false 2197 Acked Write 4 Done Done Sent false 2198 Acked Write 6 Done Done Sent false 2199 Acked Read 5 Done Done Sent false 2200 Acked Read 1 Done Done Sent false 2201 Acked Read 4 Done Done Sent false 2202 Acked Read 4 Done Done Sent false 2203 Acked Read 6 Done Done Sent false 2204 Acked Write 3 Done Done Sent false 2205 Acked Write 7 Done Done Sent false 2206 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2206 2206 1905 Downstairs last five completed: 1905 1904 1903 1902 1901 Upstairs last five completed: 2206 2205 2204 2203 2202 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1206) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1906 Acked Read 1 Done Done Sent false 1907 Acked Read 9 Done Done Sent false 1908 Acked Read 8 Done Done Sent false 1909 Acked Read 5 Done Done Sent false 1910 Acked Read 4 Done Done Sent false 1911 Acked Write 7 Done Done Sent false 1912 Acked Flush 0 Done Done Sent false 1913 Acked Read 7 Done Done Sent false 1914 Acked Read 3 Done Done Sent false 1915 Acked Flush 0 Done Done Sent false 1916 Acked Read 8 Done Done Sent false 1917 Acked Write 3 Done Done Sent false 1918 Acked Read 10 Done Done Sent false 1919 Acked Read 10 Done Done Sent false 1920 Acked Read 3 Done Done Sent false 1921 Acked Flush 0 Done Done Sent false 1922 Acked Read 2 Done Done Sent false 1923 Acked Read 6 Done Done Sent false 1924 Acked Read 3 Done Done Sent false 1925 Acked Write 10 Done Done Sent false 1926 Acked Read 1 Done Done Sent false 1927 Acked Read 6 Done Done Sent false 1928 Acked Read 8 Done Done Sent false 1929 Acked Read 7 Done Done Sent false 1930 Acked Write 10 Done Done Sent false 1931 Acked Read 3 Done Done Sent false 1932 Acked Write 9 Done Done Sent false 1933 Acked Read 9 Done Done Sent false 1934 Acked Flush 0 Done Done Sent false 1935 Acked Read 4 Done Done Sent false 1936 Acked Read 10 Done Done Sent false 1937 Acked Read 1 Done Done Sent false 1938 Acked Read 1 Done Done Sent false 1939 Acked Read 2 Done Done Sent false 1940 Acked Read 3 Done Done Sent false 1941 Acked Read 3 Done Done Sent false 1942 Acked Flush 0 Done Done Sent false 1943 Acked Read 6 Done Done Sent false 1944 Acked Write 9 Done Done Sent false 1945 Acked Write 1 Done Done Sent false 1946 Acked Write 3 Done Done Sent false 1947 Acked Read 4 Done Done Sent false 1948 Acked Read 9 Done Done Sent false 1949 Acked Write 4 Done Done Sent false 1950 Acked Read 7 Done Done Sent false 1951 Acked Write 9 Done Done Sent false 1952 Acked Read 4 Done Done Sent false 1953 Acked Read 6 Done Done Sent false 1954 Acked Read 10 Done Done Sent false 1955 Acked Read 3 Done Done Sent false 1956 Acked Write 4 Done Done Sent false 1957 Acked Read 8 Done Done Sent false 1958 Acked Read 6 Done Done Sent false 1959 Acked Read 1 Done Done Sent false 1960 Acked Read 1 Done Done Sent false 1961 Acked Flush 0 Done Done Sent false 1962 Acked Read 10 Done Done Sent false 1963 Acked Write 3 Done Done Sent false 1964 Acked Write 9 Done Done Sent false 1965 Acked Write 8 Done Done Sent false 1966 Acked Read 2 Done Done Sent false 1967 Acked Write 4 Done Done Sent false 1968 Acked Read 3 Done Done Sent false 1969 Acked Read 1 Done Done Sent false 1970 Acked Write 10 Done Done Sent false 1971 Acked Read 2 Done Done Sent false 1972 Acked Read 3 Done Done Sent false 1973 Acked Flush 0 Done Done Sent false 1974 Acked Read 1 Done Done Sent false 1975 Acked Read 2 Done Done Sent false 1976 Acked Read 6 Done Done Sent false 1977 Acked Write 9 Done Done Sent false 1978 Acked Read 9 Done Done Sent false 1979 Acked Read 5 Done Done Sent false 1980 Acked Flush 0 Done Done Sent false 1981 Acked Read 4 Done Done Sent false 1982 Acked Flush 0 Done Done Sent false 1983 Acked Write 3 Done Done Sent false 1984 Acked Read 6 Done Done Sent false 1985 Acked Read 3 Done Done Sent false 1986 Acked Write 7 Done Done Sent false 1987 Acked Flush 0 Done Done Sent false 1988 Acked Read 4 Done Done Sent false 1989 Acked Write 8 Done Done Sent false 1990 Acked Read 3 Done Done Sent false 1991 Acked Read 5 Done Done Sent false 1992 Acked Write 1 Done Done Sent false 1993 Acked Read 7 Done Done Sent false 1994 Acked Read 7 Done Done Sent false 1995 Acked Read 10 Done Done Sent false 1996 Acked Write 10 Done Done Sent false 1997 Acked Read 2 Done Done Sent false 1998 Acked Read 4 Done Done Sent false 1999 Acked Write 1 Done Done Sent false 2000 Acked Read 5 Done Done Sent false 2001 Acked Read 7 Done Done Sent false 2002 Acked Read 10 Done Done Sent false 2003 Acked Write 5 Done Done Sent false 2004 Acked Read 5 Done Done Sent false 2005 Acked Write 3 Done Done Sent false 2006 Acked Read 1 Done Done Sent false 2007 Acked Read 7 Done Done Sent false 2008 Acked Read 3 Done Done Sent false 2009 Acked Write 8 Done Done Sent false 2010 Acked Read 6 Done Done Sent false 2011 Acked Write 2 Done Done Sent false 2012 Acked Write 3 Done Done Sent false 2013 Acked Write 9 Done Done Sent false 2014 Acked Write 10 Done Done Sent false 2015 Acked Write 10 Done Done Sent false 2016 Acked Read 5 Done Done Sent false 2017 Acked Read 4 Done Done Sent false 2018 Acked Write 5 Done Done Sent false 2019 Acked Write 5 Done Done Sent false 2020 Acked Read 7 Done Done Sent false 2021 Acked Write 9 Done Done Sent false 2022 Acked Read 8 Done Done Sent false 2023 Acked Read 9 Done Done Sent false 2024 Acked Read 3 Done Done Sent false 2025 Acked Flush 0 Done Done Sent false 2026 Acked Write 2 Done Done Sent false 2027 Acked Read 9 Done Done Sent false 2028 Acked Write 6 Done Done Sent false 2029 Acked Write 9 Done Done Sent false 2030 Acked Read 1 Done Done Sent false 2031 Acked Read 1 Done Done Sent false 2032 Acked Write 3 Done Done Sent false 2033 Acked Read 1 Done Done Sent false 2034 Acked Write 5 Done Done Sent false 2035 Acked Write 8 Done Done Sent false 2036 Acked Write 1 Done Done Sent false 2037 Acked Write 7 Done Done Sent false 2038 Acked Write 10 Done Done Sent false 2039 Acked Read 5 Done Done Sent false 2040 Acked Read 7 Done Done Sent false 2041 Acked Read 1 Done Done Sent false 2042 Acked Read 2 Done Done Sent false 2043 Acked Read 10 Done Done Sent false 2044 Acked Read 10 Done Done Sent false 2045 Acked Read 1 Done Done Sent false 2046 Acked Read 5 Done Done Sent false 2047 Acked Write 6 Done Done Sent false 2048 Acked Read 1 Done Done Sent false 2049 Acked Read 10 Done Done Sent false 2050 Acked Read 2 Done Done Sent false 2051 Acked Write 6 Done Done Sent false 2052 Acked Read 10 Done Done Sent false 2053 Acked Write 4 Done Done Sent false 2054 Acked Read 4 Done Done Sent false 2055 Acked Read 4 Done Done Sent false 2056 Acked Read 7 Done Done Sent false 2057 Acked Write 3 Done Done Sent false 2058 Acked Read 4 Done Done Sent false 2059 Acked Read 9 Done Done Sent false 2060 Acked Write 1 Done Done Sent false 2061 Acked Flush 0 Done Done Sent false 2062 Acked Read 8 Done Done Sent false 2063 Acked Write 9 Done Done Sent false 2064 Acked Read 10 Done Done Sent false 2065 Acked Read 10 Done Done Sent false 2066 Acked Write 3 Done Done Sent false 2067 Acked Read 10 Done Done Sent false 2068 Acked Write 3 Done Done Sent false 2069 Acked Read 8 Done Done Sent false 2070 Acked Write 1 Done Done Sent false 2071 Acked Write 4 Done Done Sent false 2072 Acked Write 5 Done Done Sent false 2073 Acked Write 9 Done Done Sent false 2074 Acked Read 7 Done Done Sent false 2075 Acked Flush 0 Done Done Sent false 2076 Acked Write 2 Done Done Sent false 2077 Acked Read 8 Done Done Sent false 2078 Acked Write 4 Done Done Sent false 2079 Acked Write 7 Done Done Sent false 2080 Acked Write 7 Done Done Sent false 2081 Acked Read 3 Done Done Sent false 2082 Acked Read 3 Done Done Sent false 2083 Acked Read 4 Done Done Sent false 2084 Acked Read 1 Done Done Sent false 2085 Acked Read 8 Done Done Sent false 2086 Acked Flush 0 Done Done Sent false 2087 Acked Read 5 Done Done Sent false 2088 Acked Write 9 Done Done Sent false 2089 Acked Read 6 Done Done Sent false 2090 Acked Write 2 Done Done Sent false 2091 Acked Read 10 Done Done Sent false 2092 Acked Write 3 Done Done Sent false 2093 Acked Write 6 Done Done Sent false 2094 Acked Flush 0 Done Done Sent false 2095 Acked Read 6 Done Done Sent false 2096 Acked Flush 0 Done Done Sent false 2097 Acked Write 4 Done Done Sent false 2098 Acked Write 2 Done Done Sent false 2099 Acked Read 9 Done Done Sent false 2100 Acked Read 4 Done Done Sent false 2101 Acked Write 5 Done Done Sent false 2102 Acked Read 2 Done Done Sent false 2103 Acked Read 2 Done Done Sent false 2104 Acked Flush 0 Done Done Sent false 2105 Acked Read 6 Done Done Sent false 2106 Acked Read 9 Done Done Sent false 2107 Acked Read 3 Done Done Sent false 2108 Acked Write 3 Done Done Sent false 2109 Acked Read 5 Done Done Sent false 2110 Acked Flush 0 Done Done Sent false 2111 Acked Write 3 Done Done Sent false 2112 Acked Write 3 Done Done Sent false 2113 Acked Read 9 Done Done Sent false 2114 Acked Flush 0 Done Done Sent false 2115 Acked Write 7 Done Done Sent false 2116 Acked Read 7 Done Done Sent false 2117 Acked Write 5 Done Done Sent false 2118 Acked Read 7 Done Done Sent false 2119 Acked Write 6 Done Done Sent false 2120 Acked Read 6 Done Done Sent false 2121 Acked Read 1 Done Done Sent false 2122 Acked Read 4 Done Done Sent false 2123 Acked Write 9 Done Done Sent false 2124 Acked Write 5 Done Done Sent false 2125 Acked Read 4 Done Done Sent false 2126 Acked Read 6 Done Done Sent false 2127 Acked Write 10 Done Done Sent false 2128 Acked Write 1 Done Done Sent false 2129 Acked Read 9 Done Done Sent false 2130 Acked Read 1 Done Done Sent false 2131 Acked Read 10 Done Done Sent false 2132 Acked Write 8 Done Done Sent false 2133 Acked Read 6 Done Done Sent false 2134 Acked Read 7 Done Done Sent false 2135 Acked Write 8 Done Done Sent false 2136 Acked Read 3 Done Done Sent false 2137 Acked Read 3 Done Done Sent false 2138 Acked Read 5 Done Done Sent false 2139 Acked Read 8 Done Done Sent false 2140 Acked Write 5 Done Done Sent false 2141 Acked Read 3 Done Done Sent false 2142 Acked Read 1 Done Done Sent false 2143 Acked Write 5 Done Done Sent false 2144 Acked Read 1 Done Done Sent false 2145 Acked Write 3 Done Done Sent false 2146 Acked Read 1 Done Done Sent false 2147 Acked Flush 0 Done Done Sent false 2148 Acked Flush 0 Done Done Sent false 2149 Acked Write 7 Done Done Sent false 2150 Acked Write 10 Done Done Sent false 2151 Acked Read 10 Done Done Sent false 2152 Acked Read 8 Done Done Sent false 2153 Acked Read 10 Done Done Sent false 2154 Acked Read 4 Done Done Sent false 2155 Acked Write 2 Done Done Sent false 2156 Acked Flush 0 Done Done Sent false 2157 Acked Write 7 Done Done Sent false 2158 Acked Read 8 Done Done Sent false 2159 Acked Write 8 Done Done Sent false 2160 Acked Read 10 Done Done Sent false 2161 Acked Flush 0 Done Done Sent false 2162 Acked Read 4 Done Done Sent false 2163 Acked Read 7 Done Done Sent false 2164 Acked Read 6 Done Done Sent false 2165 Acked Read 4 Done Done Sent false 2166 Acked Write 8 Done Done Sent false 2167 Acked Write 8 Done Done Sent false 2168 Acked Write 2 Done Done Sent false 2169 Acked Write 5 Done Done Sent false 2170 Acked Read 1 Done Done Sent false 2171 Acked Read 8 Done Done Sent false 2172 Acked Read 6 Done Done Sent false 2173 Acked Read 3 Done Done Sent false 2174 Acked Read 3 Done Done Sent false 2175 Acked Write 1 Done Done Sent false 2176 Acked Write 9 Done Done Sent false 2177 Acked Read 5 Done Done Sent false 2178 Acked Write 9 Done Done Sent false 2179 Acked Flush 0 Done Done Sent false 2180 Acked Write 8 Done Done Sent false 2181 Acked Read 1 Done Done Sent false 2182 Acked Read 3 Done Done Sent false 2183 Acked Flush 0 Done Done Sent false 2184 Acked Write 10 Done Done Sent false 2185 Acked Read 5 Done Done Sent false 2186 Acked Write 6 Done Done Sent false 2187 Acked Write 3 Done Done Sent false 2188 Acked Read 10 Done Done Sent false 2189 Acked Read 9 Done Done Sent false 2190 Acked Read 4 Done Done Sent false 2191 Acked Read 4 Done Done Sent false 2192 Acked Write 2 Done Done Sent false 2193 Acked Flush 0 Done Done Sent false 2194 Acked Read 7 Done Done Sent false 2195 Acked Read 10 Done Done Sent false 2196 Acked Read 1 Done Done Sent false 2197 Acked Write 4 Done Done Sent false 2198 Acked Write 6 Done Done Sent false 2199 Acked Read 5 Done Done Sent false 2200 Acked Read 1 Done Done Sent false 2201 Acked Read 4 Done Done Sent false 2202 Acked Read 4 Done Done Sent false 2203 Acked Read 6 Done Done Sent false 2204 Acked Write 3 Done Done Sent false 2205 Acked Write 7 Done Done Sent false 2206 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2206 2206 1905 Downstairs last five completed: 1905 1904 1903 1902 1901 Upstairs last five completed: 2206 2205 2204 2203 2202 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.423937089Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:56041)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.42404326Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:26.424921795Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425430268Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425449583Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 1905","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425463859Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 1905","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425678421Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425697436Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425711473Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 301 jobs since flush 1905","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.425725009Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 301 jobs since flush: 1905","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:26.4259465Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 2206 2205 2204 2203 2202 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:42:28.672 INFO request completed, [8820] Got stop action so:false kr:false latency_us: 338, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 30d2bfee-ea14-4321-bd1e-dc4e592069e9, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:28.67277601Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:28.672832905Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:42:28.672 INFO request completed{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:28.6728523Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} , latency_us: 153, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9812f428-0814-45d0-86ea-8342242be83d, remote_addr: 127.0.0.1:51788, local_addr{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:28.672877793Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} : 127.0.0.1:{"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:28.672895808Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 9998 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:28.67296308Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:42:32.675 INFO request completed, latency_us: 372, response_code: 200, uri: /state/cid/1, method: GET, req_id: b6f7b108-f3e7-48ac-b4e7-9609c0b98bde, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 1241 len 20480 002/300 Read block 3289 len 36864 003/300 Write block 3178 len 12288 data: 2 2 2 004/300 Read block 2496 len 8192 005/300 Write block 4564 len 32768 data: 2 2 2 2 2 2 2 3 006/300 Read block 4618 len 12288 007/300 Write block 2039 len 12288 data: 3 3 3 008/300 Flush 009/300 Read block 4588 len 12288 010/300 Read block 4649 len 16384 011/300 Flush 012/300 Read block 2026 len 4096 013/300 Write block 3366 len 16384 data: 2 2 2 2 014/300 Write block 987 len 36864 data: 2 2 2 2 2 3 3 3 3 015/300 Write block 2373 len 40960 data: 3 3 3 2 2 2 2 2 2 2 016/300 Read block 4793 len 36864 017/300 Read block 1607 len 32768 018/300 Read block 756 len 28672 019/300 Write block 849 len 28672 data: 2 2 2 2 2 2 2 020/300 Write block 2388 len 32768 data: 3 3 3 3 2 2 2 2 021/300 Read block 2346 len 28672 022/300 Write block 3897 len 24576 data: 2 2 2 2 3 3 023/300 Read block 151 len 28672 024/300 Read block 2487 len 28672 025/300 Read block 2809 len 4096 026/300 Write block 1074 len 12288 data: 2 2 2 027/300 Write block 355 len 4096 data: 3 028/300 Write block 2518 len 8192 data: 2 2 029/300 Read block 4562 len 24576 030/300 Write block 2851 len 36864 data: 3 3 4 4 4 4 3 3 4 031/300 Read block 3129 len 40960 032/300 Write block 354 len 8192 data: 2 4 033/300 Flush 034/300 Write block 3163 len 40960 data: 3 3 4 4 3 3 3 3 2 2 035/300 Write block 3563 len 4096 data: 3 036/300 Write block 1361 len 40960 data: 2 2 2 2 2 2 2 2 2 2 037/300 Write block 4794 len 32768 data: 2 2 2 2 2 2 2 2 038/300 Write block 3948 len 28672 data: 2 2 2 2 2 2 2 039/300 Write block 1987 len 24576 data: 2 2 4 4 5 5 040/300 Write block 4626 len 20480 data: 3 3 2 2 2 041/300 Flush 042/300 Write block 236 len 32768 data: 2 2 2 2 2 2 2 2 043/300 Read block 1500 len 32768 044/300 Flush 045/300 Read block 4339 len 32768 046/300 Read block 1413 len 20480 047/300 Write block 3536 len 40960 data: 2 2 2 2 2 2 2 2 2 2 048/300 Read block 3846 len 16384 049/300 Read block 1943 len 12288 050/300 Read block 4105 len 32768 051/300 Write block 1169 len 12288 data: 2 3 4 052/300 Flush 053/300 Write block 4202 len 40960 data: 3 2 3 3 3 3 3 3 3 3 054/300 Write block 2527 len 12288 data: 3 3 3 055/300 Read block 506 len 8192 056/300 Write block 4828 len 4096 data: 3 057/300 Read block 3268 len 16384 058/300 Write block 1067 len 8192 data: 2 2 059/300 Write block 261 len 24576 data: 2 2 2 2 2 2 060/300 Write block 2478 len 12288 data: 2 3 3 061/300 Flush 062/300 Flush 063/300 Read block 4432 len 24576 064/300 Read block 1171 len 40960 065/300 Flush 066/300 Write block 1598 len 28672 data: 2 2 2 2 2 2 2 067/300 Write block 2893 len 36864 data: 2 2 2 2 2 2 2 2 2 068/300 Read block 3185 len 4096 069/300 Read block 1215 len 8192 070/300 Read block 3471 len 36864 071/300 Write block 3135 len 28672 data: 2 2 2 2 2 3 4 072/300 Read block 167 len 4096 073/300 Read block 2434 len 32768 074/300 Read block 3411 len 36864 075/300 Write block 3257 len 4096 data: 2 076/300 Read block 1352 len 4096 077/300 Write block 437 len 20480 data: 3 3 3 2 2 078/300 Read block 1579 len 8192 079/300 Write block 1385 len 36864 data: 2 2 2 2 2 3 3 3 3 080/300 Read block 4327 len 24576 081/300 Write block 1938 len 16384 data: 2 2 2 2 082/300 Flush 083/300 Read block 476 len 8192 084/300 Write block 1885 len 40960 data: 3 3 3 3 3 3 3 2 2 2 085/300 Read block 647 len 12288 086/300 Read block 4635 len 8192 087/300 Read block 4494 len 4096 088/300 Read block 3254 len 24576 089/300 Read block 2223 len 40960 090/300 Write block 2740 len 28672 data: 3 3 3 3 2 2 2 091/300 Flush 092/300 Write block 662 len 28672 data: 3 3 3 2 2 2 2 093/300 Write block 3954 len 24576 data: 3 2 2 2 2 3 094/300 Write block 2256 len 20480 data: 2 2 2 2 2 095/300 Read block 3990 len 40960 096/300 Read block 2782 len 4096 097/300 Read block 2361 len 4096 098/300 Read block 2481 len 32768 099/300 Write block 4860 len 28672 data: 4 4 5 3 2 2 2 100/300 Write block 4726 len 32768 data: 3 2 2 2 3 3 3 3 101/300 Read block 4566 len 12288 102/300 Read block 2896 len 36864 103/300 Read block 4553 len 20480 104/300 Read block 3548 len 40960 105/300 Write block 3128 len 12288 data: 2 2 2 106/300 Read block 2419 len 36864 107/300 Read block 4357 len 16384 108/300 Write block 4066 len 12288 data: 2 2 2 109/300 Write block 108 len 24576 data: 3 3 3 3 3 3 110/300 Write block 29 len 24576 data: 3 3 3 3 3 3 111/300 Write block 4304 len 36864 data: 2 2 2 2 2 2 2 2 2 112/300 Read block 4830 len 12288 113/300 Flush 114/300 Read block 1275 len 28672 115/300 Write block 2665 len 4096 data: 3 116/300 Read block 3747 len 4096 117/300 Read block 641 len 28672 118/300 Read block 1167 len 20480 119/300 Write block 755 len 28672 data: 4 5 4 3 3 2 2 120/300 Write block 2297 len 12288 data: 2 2 2 121/300 Read block 4304 len 4096 122/300 Flush 123/300 Write block 3198 len 8192 data: 3 2 124/300 Read block 1547 len 12288 125/300 Read block 2708 len 40960 126/300 Read block 1176 len 32768 127/300 Read block 2395 len 24576 128/300 Read block 3796 len 24576 129/300 Write block 4742 len 28672 data: 2 2 2 2 2 2 2 130/300 Flush 131/300 Read block 713 len 16384 132/300 Write block 1423 len 24576 data: 2 2 2 2 2 2 133/300 Read block 307 len 28672 134/300 Write block 2475 len 32768 data: 2 2 2 3 4 4 3 2 135/300 Write block 3361 len 8192 data: 3 3 136/300 Write block 68 len 36864 data: 3 3 3 3 3 2 2 2 2 137/300 Write block 4522 len 12288 data: 2 2 2 138/300 Write block 2792 len 16384 data: 2 2 2 3 139/300 Write block 2220 len 40960 data: 2 2 2 2 2 2 2 2 2 2 140/300 Write block 4883 len 28672 data: 2 2 2 2 3 3 3 141/300 Write block 4179 len 36864 data: 2 2 2 2 2 2 2 2 2 142/300 Read block 3358 len 12288 143/300 Write block 1899 len 4096 data: 2 144/300 Read block 2058 len 8192 145/300 Read block 4203 len 36864 146/300 Write block 2581 len 40960 data: 2 2 2 3 3 3 3 3 3 3 147/300 Read block 3340 len 20480 148/300 Flush 149/300 Write block 3552 len 4096 data: 3 150/300 Read block 2679 len 40960 151/300 Read block 3100 len 8192 152/300 Write block 3421 len 8192 data: 2 2 153/300 Write block 1032 len 12288 data: 2 2 2 154/300 Write block 4742 len 36864 data: 3 3 3 3 3 3 3 2 2 155/300 Write block 4182 len 36864 data: 3 3 3 3 3 3 2 2 2 156/300 Read block 996 len 32768 157/300 Read block 984 len 4096 158/300 Read block 2802 len 20480 159/300 Write block 4844 len 36864 data: 4 4 4 2 2 3 3 3 2 160/300 Read block 1669 len 20480 161/300 Read block 918 len 20480 162/300 Read block 1020 len 12288 163/300 Write block 726 len 8192 data: 2 2 164/300 Read block 1044 len 8192 165/300 Write block 3890 len 28672 data: 3 3 2 2 2 2 2 166/300 Read block 1629 len 24576 167/300 Read block 280 len 28672 168/300 Write block 3122 len 20480 data: 2 2 2 2 2 169/300 Flush 170/300 Write block 591 len 28672 data: 2 2 2 2 2 2 2 171/300 Read block 3752 len 28672 172/300 Write block 1434 len 8192 data: 2 2 173/300 Write block 4738 len 28672 data: 2 2 2 2 4 4 4 174/300 Flush 175/300 Read block 1741 len 20480 176/300 Write block 4436 len 32768 data: 2 2 2 3 3 3 3 4 177/300 Write block 1892 len 12288 data: 3 3 3 178/300 Read block 3335 len 36864 179/300 Read block 4001 len 36864 180/300 Read block 3192 len 4096 181/300 Read block 1289 len 20480 182/300 Write block 1559 len 24576 data: 2 2 2 2 2 2 183/300 Read block 2646 len 16384 184/300 Read block 2947 len 24576 185/300 Read block 653 len 32768 186/300 Write block 338 len 24576 data: 2 2 2 3 3 3 187/300 Read block 2775 len 8192 188/300 Read block 3958 len 40960 189/300 Read block 4763 len 12288 190/300 Read block 3548 len 36864 191/300 Write block 2100 len 16384 data: 3 3 3 3 192/300 Flush 193/300 Write block 1836 len 20480 data: 2 2 2 2 2 194/300 Write block 1820 len 16384 data: 4 4 5 5 195/300 Read block 2116 len 36864 196/300 Read block 4483 len 12288 197/300 Write block 3370 len 40960 data: 2 2 2 2 2 2 2 2 2 3 198/300 Write block 4885 len 4096 data: 3 199/300 Read block 4719 len 4096 200/300 Read block 2919 len 20480 201/300 Write block 1557 len 36864 data: 2 2 3 3 3 3 3 3 2 202/300 Read block 3564 len 4096 203/300 Write block 3377 len 32768 data: 3 3 4 3 3 3 3 3 204/300 Write block 1368 len 24576 data: 3 3 3 2 2 2 205/300 Read block 3183 len 40960 206/300 Read block 4865 len 40960 207/300 Write block 2970 len 24576 data: 2 2 2 2 2 2 208/300 Write block 4985 len 4096 data: 4 209/300 Read block 4744 len 40960 210/300 Write block 2653 len 12288 data: 3 3 3 211/300 Read block 4466 len 32768 212/300 Write block 4855 len 24576 data: 3 3 3 3 4 5 213/300 Read block 2174 len 36864 214/300 Read block 614 len 8192 215/300 Read block 4697 len 40960 216/300 Flush 217/300 Write block 1873 len 4096 data: 2 218/300 Read block 3913 len 36864 219/300 Write block 4186 len 4096 data: 4 220/300 Read block 1063 len 40960 221/300 Flush 222/300 Write block 2551 len 12288 data: 3 3 3 223/300 Read block 1882 len 28672 224/300 Write block 3817 len 24576 data: 2 2 2 2 2 2 225/300 Read block 921 len 24576 226/300 Write block 1331 len 4096 data: 2 227/300 Read block 4023 len 20480 228/300 Write block 4994 len 16384 data: 2 2 2 2 229/300 Write block 3436 len 20480 data: 2 2 2 2 2 230/300 Write block 939 len 16384 data: 2 2 2 2 231/300 Read block 1375 len 8192 232/300 Read block 2483 len 16384 233/300 Write block 4772 len 12288 data: 4 4 4 234/300 Write block 857 len 4096 data: 2 235/300 Write block 3185 len 24576 data: 2 2 2 2 2 2 236/300 Read block 1242 len 36864 237/300 Write block 4385 len 36864 data: 3 3 3 4 4 3 2 2 3 238/300 Write block 1831 len 32768 data: 3 3 3 2 2 3 3 3 239/300 Read block 2566 len 32768 240/300 Read block 3658 len 28672 241/300 Read block 1504 len 4096 242/300 Flush 243/300 Read block 4926 len 4096 244/300 Write block 1248 len 36864 data: 2 2 2 2 2 2 2 2 2 245/300 Write block 3284 len 40960 data: 2 2 2 2 2 2 2 2 2 2 246/300 Read block 4027 len 36864 247/300 Read block 3148 len 8192 248/300 Read block 4960 len 12288 249/300 Read block 247 len 28672 250/300 Read block 957 len 36864 251/300 Read block 2610 len 16384 252/300 Read block 3893 len 36864 253/300 Write block 489 len 16384 data: 2 2 3 3 254/300 Read block 974 len 4096 255/300 Read block 4119 len 28672 256/300 Read block 2151 len 20480 257/300 Write block 183 len 12288 data: 3 3 2 258/300 Read block 2701 len 12288 259/300 Write block 3102 len 8192 data: 2 2 260/300 Read block 2220 len 4096 261/300 Write block 4549 len 12288 data: 4 4 4 262/300 Write block 4938 len 24576 data: 4 4 4 4 4 3 263/300 Flush 264/300 Read block 4760 len 4096 265/300 Flush 266/300 Write block 2371 len 40960 data: 2 2 4 4 4 3 3 3 3 3 267/300 Read block 1914 len 40960 268/300 Read block 1324 len 40960 269/300 Read block 3260 len 40960 270/300 Read block 2272 len 40960 271/300 Read block 1668 len 4096 272/300 Write block 263 len 28672 data: 3 3 3 3 2 2 2 273/300 Write block 836 len 20480 data: 2 2 2 2 2 274/300 Read block 3150 len 36864 275/300 Read block 1519 len 8192 276/300 Read block 2263 len 24576 277/300 Write block 523 len 12288 data: 2 2 3 278/300 Read block 1755 len 12288 279/300 Read block 2247 len 24576 280/300 Write block 2572 len 4096 data: 3 281/300 Write block 3485 len 40960 data: 2 2 2 2 3 3 4 2 2 2 282/300 Write block 414 len 28672 data: 3 2 2 2 2 2 2 283/300 Flush 284/300 Read block 4341 len 24576 285/300 Write block 2345 len 28672 data: 4 4 4 4 5 6 6 286/300 Flush 287/300 Write block 635 len 8192 data: 2 2 288/300 Write block 4557 len 36864 data: 2 2 2 2 2 2 2 3 3 289/300 Read block 1671 len 12288 290/300 Write block 104 len 36864 data: 2 2 2 2 4 4 4 4 4 291/300 Read block 4694 len 32768 292/300 Write block 4802 len 28672 data: 2 2 2 2 2 2 2 293/300 Read block 772 len 40960 294/300 Write block 3680 len 36864 data: 4 4 4 4 2 2 2 2 2 295/300 Read block 3240 len 12288 296/300 Read block 2719 len 24576 297/300 Read block 3884 len 4096 298/300 Read block 715 len 28672 299/300 Read block 3442 len 4096 300/300 Read block 4787 len 20480 301/300 Flush Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:42:32.906 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/1, method: POST, req_id: eb96ddac-ec51-4d6e-ab9c-8415e5f6fd09, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2207 Acked Read 5 Done Sent Done false 2208 Acked Read 9 Done Sent Done false 2209 Acked Write 3 Done Sent Done false 2210 Acked Read 2 Done Sent Done false 2211 Acked Write 8 Done Sent Done false 2212 Acked Read 3 Done Sent Done false 2213 Acked Write 3 Done Sent Done false 2214 Acked Flush 0 Done Sent Done false 2215 Acked Read 3 Done Sent Done false 2216 Acked Read 4 Done Sent Done false 2217 Acked Flush 0 Done Sent Done false 2218 Acked Read 1 Done Sent Done false 2219 Acked Write 4 Done Sent Done false 2220 Acked Write 9 Done Sent Done false 2221 Acked Write 10 Done Sent Done false 2222 Acked Read 9 Done Sent Done false 2223 Acked Read 8 Done Sent Done false 2224 Acked Read 7 Done Sent Done false 2225 Acked Write 7 Done Sent Done false 2226 Acked Write 8 Done Sent Done false 2227 Acked Read 7 Done Sent Done false 2228 Acked Write 6 Done Sent Done false 2229 Acked Read 7 Done Sent Done false 2230 Acked Read 7 Done Sent Done false 2231 Acked Read 1 Done Sent Done false 2232 Acked Write 3 Done Sent Done false 2233 Acked Write 1 Done Sent Done false 2234 Acked Write 2 Done Sent Done false 2235 Acked Read 6 Done Sent Done false 2236 Acked Write 9 Done Sent Done false 2237 Acked Read 10 Done Sent Done false 2238 Acked Write 2 Done Sent Done false 2239 Acked Flush 0 Done Sent Done false 2240 Acked Write 10 Done Sent Done false 2241 Acked Write 1 Done Sent Done false 2242 Acked Write 10 Done Sent Done false 2243 Acked Write 8 Done Sent Done false 2244 Acked Write 7 Done Sent Done false 2245 Acked Write 6 Done Sent Done false 2246 Acked Write 5 Done Sent Done false 2247 Acked Flush 0 Done Sent Done false 2248 Acked Write 8 Done Sent Done false 2249 Acked Read 8 Done Sent Done false 2250 Acked Flush 0 Done Sent Done false 2251 Acked Read 8 Done Sent Done false 2252 Acked Read 5 Done Sent Done false 2253 Acked Write 10 Done Sent Done false 2254 Acked Read 4 Done Sent Done false 2255 Acked Read 3 Done Sent Done false 2256 Acked Read 8 Done Sent Done false 2257 Acked Write 3 Done Sent Done false 2258 Acked Flush 0 Done Sent Done false 2259 Acked Write 10 Done Sent Done false 2260 Acked Write 3 Done Sent Done false 2261 Acked Read 2 Done Sent Done false 2262 Acked Write 1 Done Sent Done false 2263 Acked Read 4 Done Sent Done false 2264 Acked Write 2 Done Sent Done false 2265 Acked Write 6 Done Sent Done false 2266 Acked Write 3 Done Sent Done false 2267 Acked Flush 0 Done Sent Done false 2268 Acked Flush 0 Done Sent Done false 2269 Acked Read 6 Done Sent Done false 2270 Acked Read 10 Done Sent Done false 2271 Acked Flush 0 Done Sent Done false 2272 Acked Write 7 Done Sent Done false 2273 Acked Write 9 Done Sent Done false 2274 Acked Read 1 Done Sent Done false 2275 Acked Read 2 Done Sent Done false 2276 Acked Read 9 Done Sent Done false 2277 Acked Write 7 Done Sent Done false 2278 Acked Read 1 Done Sent Done false 2279 Acked Read 8 Done Sent Done false 2280 Acked Read 9 Done Sent Done false 2281 Acked Write 1 Done Sent Done false 2282 Acked Read 1 Done Sent Done false 2283 Acked Write 5 Done Sent Done false 2284 Acked Read 2 Done Sent Done false 2285 Acked Write 9 Done Sent Done false 2286 Acked Read 6 Done Sent Done false 2287 Acked Write 4 Done Sent Done false 2288 Acked Flush 0 Done Sent Done false 2289 Acked Read 2 Done Sent Done false 2290 Acked Write 10 Done Sent Done false 2291 Acked Read 3 Done Sent Done false 2292 Acked Read 2 Done Sent Done false 2293 Acked Read 1 Done Sent Done false 2294 Acked Read 6 Done Sent Done false 2295 Acked Read 10 Done Sent Done false 2296 Acked Write 7 Done Sent Done false 2297 Acked Flush 0 Done Sent Done false 2298 Acked Write 7 Done Sent Done false 2299 Acked Write 6 Done Sent Done false 2300 Acked Write 5 Done Sent Done false 2301 Acked Read 10 Done Sent Done false 2302 Acked Read 1 Done Sent Done false 2303 Acked Read 1 Done Sent Done false 2304 Acked Read 8 Done Sent Done false 2305 Acked Write 7 Done Sent Done false 2306 Acked Write 8 Done Sent Done false 2307 Acked Read 3 Done Sent Done false 2308 Acked Read 9 Done Sent Done false 2309 Acked Read 5 Done Sent Done false 2310 Acked Read 10 Done Sent Done false 2311 Acked Write 3 Done Sent Done false 2312 Acked Read 9 Done Sent Done false 2313 Acked Read 4 Done Sent Done false 2314 Acked Write 3 Done Sent Done false 2315 Acked Write 6 Done Sent Done false 2316 Acked Write 6 Done Sent Done false 2317 Acked Write 9 Done Sent Done false 2318 Acked Read 3 Done Sent Done false 2319 Acked Flush 0 Done Sent Done false 2320 Acked Read 7 Done Sent Done false 2321 Acked Write 1 Done Sent Done false 2322 Acked Read 1 Done Sent Done false 2323 Acked Read 7 Done Sent Done false 2324 Acked Read 5 Done Sent Done false 2325 Acked Write 7 Done Sent Done false 2326 Acked Write 3 Done Sent Done false 2327 Acked Read 1 Done Sent Done false 2328 Acked Flush 0 Done Sent Done false 2329 Acked Write 2 Done Sent Done false 2330 Acked Read 3 Done Sent Done false 2331 Acked Read 10 Done Sent Done false 2332 Acked Read 8 Done Sent Done false 2333 Acked Read 6 Done Sent Done false 2334 Acked Read 6 Done Sent Done false 2335 Acked Write 7 Done Sent Done false 2336 Acked Flush 0 Done Sent Done false 2337 Acked Read 4 Done Sent Done false 2338 Acked Write 6 Done Sent Done false 2339 Acked Read 7 Done Sent Done false 2340 Acked Write 8 Done Sent Done false 2341 Acked Write 2 Done Sent Done false 2342 Acked Write 9 Done Sent Done false 2343 Acked Write 3 Done Sent Done false 2344 Acked Write 4 Done Sent Done false 2345 Acked Write 10 Done Sent Done false 2346 Acked Write 7 Done Sent Done false 2347 Acked Write 9 Done Sent Done false 2348 Acked Read 3 Done Sent Done false 2349 Acked Write 1 Done Sent Done false 2350 Acked Read 2 Done Sent Done false 2351 Acked Read 9 Done Sent Done false 2352 Acked Write 10 Done Sent Done false 2353 Acked Read 5 Done Sent Done false 2354 Acked Flush 0 Done Sent Done false 2355 Acked Write 1 Done Sent Done false 2356 Acked Read 10 Done Sent Done false 2357 Acked Read 2 Done Sent Done false 2358 Acked Write 2 Done Sent Done false 2359 Acked Write 3 Done Sent Done false 2360 Acked Write 9 Done Sent Done false 2361 Acked Write 9 Done Sent Done false 2362 Acked Read 8 Done Sent Done false 2363 Acked Read 1 Done Sent Done false 2364 Acked Read 5 Done Sent Done false 2365 Acked Write 9 Done Sent Done false 2366 Acked Read 5 Done Sent Done false 2367 Acked Read 5 Done Sent Done false 2368 Acked Read 3 Done Sent Done false 2369 Acked Write 2 Done Sent Done false 2370 Acked Read 2 Done Sent Done false 2371 Acked Write 7 Done Sent Done false 2372 Acked Read 6 Done Sent Done false 2373 Acked Read 7 Done Sent Done false 2374 Acked Write 5 Done Sent Done false 2375 Acked Flush 0 Done Sent Done false 2376 Acked Write 7 Done Sent Done false 2377 Acked Read 7 Done Sent Done false 2378 Acked Write 2 Done Sent Done false 2379 Acked Write 7 Done Sent Done false 2380 Acked Flush 0 Done Sent Done false 2381 Acked Read 5 Done Sent Done false 2382 Acked Write 8 Done Sent Done false 2383 Acked Write 3 Done Sent Done false 2384 Acked Read 9 Done Sent Done false 2385 Acked Read 9 Done Sent Done false 2386 Acked Read 1 Done Sent Done false 2387 Acked Read 5 Done Sent Done false 2388 Acked Write 6 Done Sent Done false 2389 Acked Read 4 Done Sent Done false 2390 Acked Read 6 Done Sent Done false 2391 Acked Read 8 Done Sent Done false 2392 Acked Write 6 Done Sent Done false 2393 Acked Read 2 Done Sent Done false 2394 Acked Read 10 Done Sent Done false 2395 Acked Read 3 Done Sent Done false 2396 Acked Read 9 Done Sent Done false 2397 Acked Write 4 Done Sent Done false 2398 Acked Flush 0 Done Sent Done false 2399 Acked Write 5 Done Sent Done false 2400 Acked Write 4 Done Sent Done false 2401 Acked Read 9 Done Sent Done false 2402 Acked Read 3 Done Sent Done false 2403 Acked Write 10 Done Sent Done false 2404 Acked Write 1 Done Sent Done false 2405 Acked Read 1 Done Sent Done false 2406 Acked Read 5 Done Sent Done false 2407 Acked Write 9 Done Sent Done false 2408 Acked Read 1 Done Sent Done false 2409 Acked Write 8 Done Sent Done false 2410 Acked Write 6 Done Sent Done false 2411 Acked Read 10 Done Sent Done false 2412 Acked Read 10 Done Sent Done false 2413 Acked Write 6 Done Sent Done false 2414 Acked Write 1 Done Sent Done false 2415 Acked Read 10 Done Sent Done false 2416 Acked Write 3 Done Sent Done false 2417 Acked Read 8 Done Sent Done false 2418 Acked Write 6 Done Sent Done false 2419 Acked Read 9 Done Sent Done false 2420 Acked Read 2 Done Sent Done false 2421 Acked Read 10 Done Sent Done false 2422 Acked Flush 0 Done Sent Done false 2423 Acked Write 1 Done Sent Done false 2424 Acked Read 9 Done Sent Done false 2425 Acked Write 1 Done Sent Done false 2426 Acked Read 10 Done Sent Done false 2427 Acked Flush 0 Done Sent Done false 2428 Acked Write 3 Done Sent Done false 2429 Acked Read 7 Done Sent Done false 2430 Acked Write 6 Done Sent Done false 2431 Acked Read 6 Done Sent Done false 2432 Acked Write 1 Done Sent Done false 2433 Acked Read 5 Done Sent Done false 2434 Acked Write 4 Done Sent Done false 2435 Acked Write 5 Done Sent Done false 2436 Acked Write 4 Done Sent Done false 2437 Acked Read 2 Done Sent Done false 2438 Acked Read 4 Done Sent Done false 2439 Acked Write 3 Done Sent Done false 2440 Acked Write 1 Done Sent Done false 2441 Acked Write 6 Done Sent Done false 2442 Acked Read 9 Done Sent Done false 2443 Acked Write 9 Done Sent Done false 2444 Acked Write 8 Done Sent Done false 2445 Acked Read 8 Done Sent Done false 2446 Acked Read 7 Done Sent Done false 2447 Acked Read 1 Done Sent Done false 2448 Acked Flush 0 Done Sent Done false 2449 Acked Read 1 Done Sent Done false 2450 Acked Write 9 Done Sent Done false 2451 Acked Write 10 Done Sent Done false 2452 Acked Read 9 Done Sent Done false 2453 Acked Read 2 Done Sent Done false 2454 Acked Read 3 Done Sent Done false 2455 Acked Read 7 Done Sent Done false 2456 Acked Read 9 Done Sent Done false 2457 Acked Read 4 Done Sent Done false 2458 Acked Read 9 Done Sent Done false 2459 Acked Write 4 Done Sent Done false 2460 Acked Read 1 Done Sent Done false 2461 Acked Read 7 Done Sent Done false 2462 Acked Read 5 Done Sent Done false 2463 Acked Write 3 Done Sent Done false 2464 Acked Read 3 Done Sent Done false 2465 Acked Write 2 Done Sent Done false 2466 Acked Read 1 Done Sent Done false 2467 Acked Write 3 Done Sent Done false 2468 Acked Write 6 Done Sent Done false 2469 Acked Flush 0 Done Sent Done false 2470 Acked Read 1 Done Sent Done false 2471 Acked Flush 0 Done Sent Done false 2472 Acked Write 10 Done Sent Done false 2473 Acked Read 10 Done Sent Done false 2474 Acked Read 10 Done Sent Done false 2475 Acked Read 10 Done Sent Done false 2476 Acked Read 10 Done Sent Done false 2477 Acked Read 1 Done Sent Done false 2478 Acked Write 7 Done Sent Done false 2479 Acked Write 5 Done Sent Done false 2480 Acked Read 9 Done Sent Done false 2481 Acked Read 2 Done Sent Done false 2482 Acked Read 6 Done Sent Done false 2483 Acked Write 3 Done Sent Done false 2484 Acked Read 3 Done Sent Done false 2485 Acked Read 6 Done Sent Done false 2486 Acked Write 1 Done Sent Done false 2487 Acked Write 10 Done Sent Done false 2488 Acked Write 7 Done Sent Done false 2489 Acked Flush 0 Done Sent Done false 2490 Acked Read 6 Done Sent Done false 2491 Acked Write 7 Done Sent Done false 2492 Acked Flush 0 Done Sent Done false 2493 Acked Write 2 Done Sent Done false 2494 Acked Write 9 Done Sent Done false 2495 Acked Read 3 Done Sent Done false 2496 Acked Write 9 Done Sent Done false 2497 Acked Read 8 Done Sent Done false 2498 Acked Write 7 Done Sent Done false 2499 Acked Read 10 Done Sent Done false 2500 Acked Write 9 Done Sent Done false 2501 Acked Read 3 Done Sent Done false 2502 Acked Read 6 Done Sent Done false 2503 Acked Read 1 Done Sent Done false 2504 Acked Read 7 Done Sent Done false 2505 Acked Read 1 Done Sent Done false 2506 Acked Read 5 Done Sent Done false 2507 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2507 2206 2507 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 2507 2506 2505 2504 2503 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2207 Acked Read 5 Done Sent Done false 2208 Acked Read 9 Done Sent Done false 2209 Acked Write 3 Done Sent Done false 2210 Acked Read 2 Done Sent Done false 2211 Acked Write 8 Done Sent Done false 2212 Acked Read 3 Done Sent Done false 2213 Acked Write 3 Done Sent Done false 2214 Acked Flush 0 Done Sent Done false 2215 Acked Read 3 Done Sent Done false 2216 Acked Read 4 Done Sent Done false 2217 Acked Flush 0 Done Sent Done false 2218 Acked Read 1 Done Sent Done false 2219 Acked Write 4 Done Sent Done false 2220 Acked Write 9 Done Sent Done false 2221 Acked Write 10 Done Sent Done false 2222 Acked Read 9 Done Sent Done false 2223 Acked Read 8 Done Sent Done false 2224 Acked Read 7 Done Sent Done false 2225 Acked Write 7 Done Sent Done false 2226 Acked Write 8 Done Sent Done false 2227 Acked Read 7 Done Sent Done false 2228 Acked Write 6 Done Sent Done false 2229 Acked Read 7 Done Sent Done false 2230 Acked Read 7 Done Sent Done false 2231 Acked Read 1 Done Sent Done false 2232 Acked Write 3 Done Sent Done false 2233 Acked Write 1 Done Sent Done false 2234 Acked Write 2 Done Sent Done false 2235 Acked Read 6 Done Sent Done false 2236 Acked Write 9 Done Sent Done false 2237 Acked Read 10 Done Sent Done false 2238 Acked Write 2 Done Sent Done false 2239 Acked Flush 0 Done Sent Done false 2240 Acked Write 10 Done Sent Done false 2241 Acked Write 1 Done Sent Done false 2242 Acked Write 10 Done Sent Done false 2243 Acked Write 8 Done Sent Done false 2244 Acked Write 7 Done Sent Done false 2245 Acked Write 6 Done Sent Done false 2246 Acked Write 5 Done Sent Done false 2247 Acked Flush 0 Done Sent Done false 2248 Acked Write 8 Done Sent Done false 2249 Acked Read 8 Done Sent Done false 2250 Acked Flush 0 Done Sent Done false 2251 Acked Read 8 Done Sent Done false 2252 Acked Read 5 Done Sent Done false 2253 Acked Write 10 Done Sent Done false 2254 Acked Read 4 Done Sent Done false 2255 Acked Read 3 Done Sent Done false 2256 Acked Read 8 Done Sent Done false 2257 Acked Write 3 Done Sent Done false 2258 Acked Flush 0 Done Sent Done false 2259 Acked Write 10 Done Sent Done false 2260 Acked Write 3 Done Sent Done false 2261 Acked Read 2 Done Sent Done false 2262 Acked Write 1 Done Sent Done false 2263 Acked Read 4 Done Sent Done false 2264 Acked Write 2 Done Sent Done false 2265 Acked Write 6 Done Sent Done false 2266 Acked Write 3 Done Sent Done false 2267 Acked Flush 0 Done Sent Done false 2268 Acked Flush 0 Done Sent Done false 2269 Acked Read 6 Done Sent Done false 2270 Acked Read 10 Done Sent Done false 2271 Acked Flush 0 Done Sent Done false 2272 Acked Write 7 Done Sent Done false 2273 Acked Write 9 Done Sent Done false 2274 Acked Read 1 Done Sent Done false 2275 Acked Read 2 Done Sent Done false 2276 Acked Read 9 Done Sent Done false 2277 Acked Write 7 Done Sent Done false 2278 Acked Read 1 Done Sent Done false 2279 Acked Read 8 Done Sent Done false 2280 Acked Read 9 Done Sent Done false 2281 Acked Write 1 Done Sent Done false 2282 Acked Read 1 Done Sent Done false 2283 Acked Write 5 Done Sent Done false 2284 Acked Read 2 Done Sent Done false 2285 Acked Write 9 Done Sent Done false 2286 Acked Read 6 Done Sent Done false 2287 Acked Write 4 Done Sent Done false 2288 Acked Flush 0 Done Sent Done false 2289 Acked Read 2 Done Sent Done false 2290 Acked Write 10 Done Sent Done false 2291 Acked Read 3 Done Sent Done false 2292 Acked Read 2 Done Sent Done false 2293 Acked Read 1 Done Sent Done false 2294 Acked Read 6 Done Sent Done false 2295 Acked Read 10 Done Sent Done false 2296 Acked Write 7 Done Sent Done false 2297 Acked Flush 0 Done Sent Done false 2298 Acked Write 7 Done Sent Done false 2299 Acked Write 6 Done Sent Done false 2300 Acked Write 5 Done Sent Done false 2301 Acked Read 10 Done Sent Done false 2302 Acked Read 1 Done Sent Done false 2303 Acked Read 1 Done Sent Done false 2304 Acked Read 8 Done Sent Done false 2305 Acked Write 7 Done Sent Done false 2306 Acked Write 8 Done Sent Done false 2307 Acked Read 3 Done Sent Done false 2308 Acked Read 9 Done Sent Done false 2309 Acked Read 5 Done Sent Done false 2310 Acked Read 10 Done Sent Done false 2311 Acked Write 3 Done Sent Done false 2312 Acked Read 9 Done Sent Done false 2313 Acked Read 4 Done Sent Done false 2314 Acked Write 3 Done Sent Done false 2315 Acked Write 6 Done Sent Done false 2316 Acked Write 6 Done Sent Done false 2317 Acked Write 9 Done Sent Done false 2318 Acked Read 3 Done Sent Done false 2319 Acked Flush 0 Done Sent Done false 2320 Acked Read 7 Done Sent Done false 2321 Acked Write 1 Done Sent Done false 2322 Acked Read 1 Done Sent Done false 2323 Acked Read 7 Done Sent Done false 2324 Acked Read 5 Done Sent Done false 2325 Acked Write 7 Done Sent Done false 2326 Acked Write 3 Done Sent Done false 2327 Acked Read 1 Done Sent Done false 2328 Acked Flush 0 Done Sent Done false 2329 Acked Write 2 Done Sent Done false 2330 Acked Read 3 Done Sent Done false 2331 Acked Read 10 Done Sent Done false 2332 Acked Read 8 Done Sent Done false 2333 Acked Read 6 Done Sent Done false 2334 Acked Read 6 Done Sent Done false 2335 Acked Write 7 Done Sent Done false 2336 Acked Flush 0 Done Sent Done false 2337 Acked Read 4 Done Sent Done false 2338 Acked Write 6 Done Sent Done false 2339 Acked Read 7 Done Sent Done false 2340 Acked Write 8 Done Sent Done false 2341 Acked Write 2 Done Sent Done false 2342 Acked Write 9 Done Sent Done false 2343 Acked Write 3 Done Sent Done false 2344 Acked Write 4 Done Sent Done false 2345 Acked Write 10 Done Sent Done false 2346 Acked Write 7 Done Sent Done false 2347 Acked Write 9 Done Sent Done false 2348 Acked Read 3 Done Sent Done false 2349 Acked Write 1 Done Sent Done false 2350 Acked Read 2 Done Sent Done false 2351 Acked Read 9 Done Sent Done false 2352 Acked Write 10 Done Sent Done false 2353 Acked Read 5 Done Sent Done false 2354 Acked Flush 0 Done Sent Done false 2355 Acked Write 1 Done Sent Done false 2356 Acked Read 10 Done Sent Done false 2357 Acked Read 2 Done Sent Done false 2358 Acked Write 2 Done Sent Done false 2359 Acked Write 3 Done Sent Done false 2360 Acked Write 9 Done Sent Done false 2361 Acked Write 9 Done Sent Done false 2362 Acked Read 8 Done Sent Done false 2363 Acked Read 1 Done Sent Done false 2364 Acked Read 5 Done Sent Done false 2365 Acked Write 9 Done Sent Done false 2366 Acked Read 5 Done Sent Done false 2367 Acked Read 5 Done Sent Done false 2368 Acked Read 3 Done Sent Done false 2369 Acked Write 2 Done Sent Done false 2370 Acked Read 2 Done Sent Done false 2371 Acked Write 7 Done Sent Done false 2372 Acked Read 6 Done Sent Done false 2373 Acked Read 7 Done Sent Done false 2374 Acked Write 5 Done Sent Done false 2375 Acked Flush 0 Done Sent Done false 2376 Acked Write 7 Done Sent Done false 2377 Acked Read 7 Done Sent Done false 2378 Acked Write 2 Done Sent Done false 2379 Acked Write 7 Done Sent Done false 2380 Acked Flush 0 Done Sent Done false 2381 Acked Read 5 Done Sent Done false 2382 Acked Write 8 Done Sent Done false 2383 Acked Write 3 Done Sent Done false 2384 Acked Read 9 Done Sent Done false 2385 Acked Read 9 Done Sent Done false 2386 Acked Read 1 Done Sent Done false 2387 Acked Read 5 Done Sent Done false 2388 Acked Write 6 Done Sent Done false 2389 Acked Read 4 Done Sent Done false 2390 Acked Read 6 Done Sent Done false 2391 Acked Read 8 Done Sent Done false 2392 Acked Write 6 Done Sent Done false 2393 Acked Read 2 Done Sent Done false 2394 Acked Read 10 Done Sent Done false 2395 Acked Read 3 Done Sent Done false 2396 Acked Read 9 Done Sent Done false 2397 Acked Write 4 Done Sent Done false 2398 Acked Flush 0 Done Sent Done false 2399 Acked Write 5 Done Sent Done false 2400 Acked Write 4 Done Sent Done false 2401 Acked Read 9 Done Sent Done false 2402 Acked Read 3 Done Sent Done false 2403 Acked Write 10 Done Sent Done false 2404 Acked Write 1 Done Sent Done false 2405 Acked Read 1 Done Sent Done false 2406 Acked Read 5 Done Sent Done false 2407 Acked Write 9 Done Sent Done false 2408 Acked Read 1 Done Sent Done false 2409 Acked Write 8 Done Sent Done false 2410 Acked Write 6 Done Sent Done false 2411 Acked Read 10 Done Sent Done false 2412 Acked Read 10 Done Sent Done false 2413 Acked Write 6 Done Sent Done false 2414 Acked Write 1 Done Sent Done false 2415 Acked Read 10 Done Sent Done false 2416 Acked Write 3 Done Sent Done false 2417 Acked Read 8 Done Sent Done false 2418 Acked Write 6 Done Sent Done false 2419 Acked Read 9 Done Sent Done false 2420 Acked Read 2 Done Sent Done false 2421 Acked Read 10 Done Sent Done false 2422 Acked Flush 0 Done Sent Done false 2423 Acked Write 1 Done Sent Done false 2424 Acked Read 9 Done Sent Done false 2425 Acked Write 1 Done Sent Done false 2426 Acked Read 10 Done Sent Done false 2427 Acked Flush 0 Done Sent Done false 2428 Acked Write 3 Done Sent Done false 2429 Acked Read 7 Done Sent Done false 2430 Acked Write 6 Done Sent Done false 2431 Acked Read 6 Done Sent Done false 2432 Acked Write 1 Done Sent Done false 2433 Acked Read 5 Done Sent Done false 2434 Acked Write 4 Done Sent Done false 2435 Acked Write 5 Done Sent Done false 2436 Acked Write 4 Done Sent Done false 2437 Acked Read 2 Done Sent Done false 2438 Acked Read 4 Done Sent Done false 2439 Acked Write 3 Done Sent Done false 2440 Acked Write 1 Done Sent Done false 2441 Acked Write 6 Done Sent Done false 2442 Acked Read 9 Done Sent Done false 2443 Acked Write 9 Done Sent Done false 2444 Acked Write 8 Done Sent Done false 2445 Acked Read 8 Done Sent Done false 2446 Acked Read 7 Done Sent Done false 2447 Acked Read 1 Done Sent Done false 2448 Acked Flush 0 Done Sent Done false 2449 Acked Read 1 Done Sent Done false 2450 Acked Write 9 Done Sent Done false 2451 Acked Write 10 Done Sent Done false 2452 Acked Read 9 Done Sent Done false 2453 Acked Read 2 Done Sent Done false 2454 Acked Read 3 Done Sent Done false 2455 Acked Read 7 Done Sent Done false 2456 Acked Read 9 Done Sent Done false 2457 Acked Read 4 Done Sent Done false 2458 Acked Read 9 Done Sent Done false 2459 Acked Write 4 Done Sent Done false 2460 Acked Read 1 Done Sent Done false 2461 Acked Read 7 Done Sent Done false 2462 Acked Read 5 Done Sent Done false 2463 Acked Write 3 Done Sent Done false 2464 Acked Read 3 Done Sent Done false 2465 Acked Write 2 Done Sent Done false 2466 Acked Read 1 Done Sent Done false 2467 Acked Write 3 Done Sent Done false 2468 Acked Write 6 Done Sent Done false 2469 Acked Flush 0 Done Sent Done false 2470 Acked Read 1 Done Sent Done false 2471 Acked Flush 0 Done Sent Done false 2472 Acked Write 10 Done Sent Done false 2473 Acked Read 10 Done Sent Done false 2474 Acked Read 10 Done Sent Done false 2475 Acked Read 10 Done Sent Done false 2476 Acked Read 10 Done Sent Done false 2477 Acked Read 1 Done Sent Done false 2478 Acked Write 7 Done Sent Done false 2479 Acked Write 5 Done Sent Done false 2480 Acked Read 9 Done Sent Done false 2481 Acked Read 2 Done Sent Done false 2482 Acked Read 6 Done Sent Done false 2483 Acked Write 3 Done Sent Done false 2484 Acked Read 3 Done Sent Done false 2485 Acked Read 6 Done Sent Done false 2486 Acked Write 1 Done Sent Done false 2487 Acked Write 10 Done Sent Done false 2488 Acked Write 7 Done Sent Done false 2489 Acked Flush 0 Done Sent Done false 2490 Acked Read 6 Done Sent Done false 2491 Acked Write 7 Done Sent Done false 2492 Acked Flush 0 Done Sent Done false 2493 Acked Write 2 Done Sent Done false 2494 Acked Write 9 Done Sent Done false 2495 Acked Read 3 Done Sent Done false 2496 Acked Write 9 Done Sent Done false 2497 Acked Read 8 Done Sent Done false 2498 Acked Write 7 Done Sent Done false 2499 Acked Read 10 Done Sent Done false 2500 Acked Write 9 Done Sent Done false 2501 Acked Read 3 Done Sent Done false 2502 Acked Read 6 Done Sent Done false 2503 Acked Read 1 Done Sent Done false 2504 Acked Read 7 Done Sent Done false 2505 Acked Read 1 Done Sent Done false 2506 Acked Read 5 Done Sent Done false 2507 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2507 2206 2507 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 2507 2506 2505 2504 2503 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:38.674565029Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:38.674625543Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:38.674644228Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:38.674660873Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:38.674761496Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1207) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2207 Acked Read 5 Done Sent Done false 2208 Acked Read 9 Done Sent Done false 2209 Acked Write 3 Done Sent Done false 2210 Acked Read 2 Done Sent Done false 2211 Acked Write 8 Done Sent Done false 2212 Acked Read 3 Done Sent Done false 2213 Acked Write 3 Done Sent Done false 2214 Acked Flush 0 Done Sent Done false 2215 Acked Read 3 Done Sent Done false 2216 Acked Read 4 Done Sent Done false 2217 Acked Flush 0 Done Sent Done false 2218 Acked Read 1 Done Sent Done false 2219 Acked Write 4 Done Sent Done false 2220 Acked Write 9 Done Sent Done false 2221 Acked Write 10 Done Sent Done false 2222 Acked Read 9 Done Sent Done false 2223 Acked Read 8 Done Sent Done false 2224 Acked Read 7 Done Sent Done false 2225 Acked Write 7 Done Sent Done false 2226 Acked Write 8 Done Sent Done false 2227 Acked Read 7 Done Sent Done false 2228 Acked Write 6 Done Sent Done false 2229 Acked Read 7 Done Sent Done false 2230 Acked Read 7 Done Sent Done false 2231 Acked Read 1 Done Sent Done false 2232 Acked Write 3 Done Sent Done false 2233 Acked Write 1 Done Sent Done false 2234 Acked Write 2 Done Sent Done false 2235 Acked Read 6 Done Sent Done false 2236 Acked Write 9 Done Sent Done false 2237 Acked Read 10 Done Sent Done false 2238 Acked Write 2 Done Sent Done false 2239 Acked Flush 0 Done Sent Done false 2240 Acked Write 10 Done Sent Done false 2241 Acked Write 1 Done Sent Done false 2242 Acked Write 10 Done Sent Done false 2243 Acked Write 8 Done Sent Done false 2244 Acked Write 7 Done Sent Done false 2245 Acked Write 6 Done Sent Done false 2246 Acked Write 5 Done Sent Done false 2247 Acked Flush 0 Done Sent Done false 2248 Acked Write 8 Done Sent Done false 2249 Acked Read 8 Done Sent Done false 2250 Acked Flush 0 Done Sent Done false 2251 Acked Read 8 Done Sent Done false 2252 Acked Read 5 Done Sent Done false 2253 Acked Write 10 Done Sent Done false 2254 Acked Read 4 Done Sent Done false 2255 Acked Read 3 Done Sent Done false 2256 Acked Read 8 Done Sent Done false 2257 Acked Write 3 Done Sent Done false 2258 Acked Flush 0 Done Sent Done false 2259 Acked Write 10 Done Sent Done false 2260 Acked Write 3 Done Sent Done false 2261 Acked Read 2 Done Sent Done false 2262 Acked Write 1 Done Sent Done false 2263 Acked Read 4 Done Sent Done false 2264 Acked Write 2 Done Sent Done false 2265 Acked Write 6 Done Sent Done false 2266 Acked Write 3 Done Sent Done false 2267 Acked Flush 0 Done Sent Done false 2268 Acked Flush 0 Done Sent Done false 2269 Acked Read 6 Done Sent Done false 2270 Acked Read 10 Done Sent Done false 2271 Acked Flush 0 Done Sent Done false 2272 Acked Write 7 Done Sent Done false 2273 Acked Write 9 Done Sent Done false 2274 Acked Read 1 Done Sent Done false 2275 Acked Read 2 Done Sent Done false 2276 Acked Read 9 Done Sent Done false 2277 Acked Write 7 Done Sent Done false 2278 Acked Read 1 Done Sent Done false 2279 Acked Read 8 Done Sent Done false 2280 Acked Read 9 Done Sent Done false 2281 Acked Write 1 Done Sent Done false 2282 Acked Read 1 Done Sent Done false 2283 Acked Write 5 Done Sent Done false 2284 Acked Read 2 Done Sent Done false 2285 Acked Write 9 Done Sent Done false 2286 Acked Read 6 Done Sent Done false 2287 Acked Write 4 Done Sent Done false 2288 Acked Flush 0 Done Sent Done false 2289 Acked Read 2 Done Sent Done false 2290 Acked Write 10 Done Sent Done false 2291 Acked Read 3 Done Sent Done false 2292 Acked Read 2 Done Sent Done false 2293 Acked Read 1 Done Sent Done false 2294 Acked Read 6 Done Sent Done false 2295 Acked Read 10 Done Sent Done false 2296 Acked Write 7 Done Sent Done false 2297 Acked Flush 0 Done Sent Done false 2298 Acked Write 7 Done Sent Done false 2299 Acked Write 6 Done Sent Done false 2300 Acked Write 5 Done Sent Done false 2301 Acked Read 10 Done Sent Done false 2302 Acked Read 1 Done Sent Done false 2303 Acked Read 1 Done Sent Done false 2304 Acked Read 8 Done Sent Done false 2305 Acked Write 7 Done Sent Done false 2306 Acked Write 8 Done Sent Done false 2307 Acked Read 3 Done Sent Done false 2308 Acked Read 9 Done Sent Done false 2309 Acked Read 5 Done Sent Done false 2310 Acked Read 10 Done Sent Done false 2311 Acked Write 3 Done Sent Done false 2312 Acked Read 9 Done Sent Done false 2313 Acked Read 4 Done Sent Done false 2314 Acked Write 3 Done Sent Done false 2315 Acked Write 6 Done Sent Done false 2316 Acked Write 6 Done Sent Done false 2317 Acked Write 9 Done Sent Done false 2318 Acked Read 3 Done Sent Done false 2319 Acked Flush 0 Done Sent Done false 2320 Acked Read 7 Done Sent Done false 2321 Acked Write 1 Done Sent Done false 2322 Acked Read 1 Done Sent Done false 2323 Acked Read 7 Done Sent Done false 2324 Acked Read 5 Done Sent Done false 2325 Acked Write 7 Done Sent Done false 2326 Acked Write 3 Done Sent Done false 2327 Acked Read 1 Done Sent Done false 2328 Acked Flush 0 Done Sent Done false 2329 Acked Write 2 Done Sent Done false 2330 Acked Read 3 Done Sent Done false 2331 Acked Read 10 Done Sent Done false 2332 Acked Read 8 Done Sent Done false 2333 Acked Read 6 Done Sent Done false 2334 Acked Read 6 Done Sent Done false 2335 Acked Write 7 Done Sent Done false 2336 Acked Flush 0 Done Sent Done false 2337 Acked Read 4 Done Sent Done false 2338 Acked Write 6 Done Sent Done false 2339 Acked Read 7 Done Sent Done false 2340 Acked Write 8 Done Sent Done false 2341 Acked Write 2 Done Sent Done false 2342 Acked Write 9 Done Sent Done false 2343 Acked Write 3 Done Sent Done false 2344 Acked Write 4 Done Sent Done false 2345 Acked Write 10 Done Sent Done false 2346 Acked Write 7 Done Sent Done false 2347 Acked Write 9 Done Sent Done false 2348 Acked Read 3 Done Sent Done false 2349 Acked Write 1 Done Sent Done false 2350 Acked Read 2 Done Sent Done false 2351 Acked Read 9 Done Sent Done false 2352 Acked Write 10 Done Sent Done false 2353 Acked Read 5 Done Sent Done false 2354 Acked Flush 0 Done Sent Done false 2355 Acked Write 1 Done Sent Done false 2356 Acked Read 10 Done Sent Done false 2357 Acked Read 2 Done Sent Done false 2358 Acked Write 2 Done Sent Done false 2359 Acked Write 3 Done Sent Done false 2360 Acked Write 9 Done Sent Done false 2361 Acked Write 9 Done Sent Done false 2362 Acked Read 8 Done Sent Done false 2363 Acked Read 1 Done Sent Done false 2364 Acked Read 5 Done Sent Done false 2365 Acked Write 9 Done Sent Done false 2366 Acked Read 5 Done Sent Done false 2367 Acked Read 5 Done Sent Done false 2368 Acked Read 3 Done Sent Done false 2369 Acked Write 2 Done Sent Done false 2370 Acked Read 2 Done Sent Done false 2371 Acked Write 7 Done Sent Done false 2372 Acked Read 6 Done Sent Done false 2373 Acked Read 7 Done Sent Done false 2374 Acked Write 5 Done Sent Done false 2375 Acked Flush 0 Done Sent Done false 2376 Acked Write 7 Done Sent Done false 2377 Acked Read 7 Done Sent Done false 2378 Acked Write 2 Done Sent Done false 2379 Acked Write 7 Done Sent Done false 2380 Acked Flush 0 Done Sent Done false 2381 Acked Read 5 Done Sent Done false 2382 Acked Write 8 Done Sent Done false 2383 Acked Write 3 Done Sent Done false 2384 Acked Read 9 Done Sent Done false 2385 Acked Read 9 Done Sent Done false 2386 Acked Read 1 Done Sent Done false 2387 Acked Read 5 Done Sent Done false 2388 Acked Write 6 Done Sent Done false 2389 Acked Read 4 Done Sent Done false 2390 Acked Read 6 Done Sent Done false 2391 Acked Read 8 Done Sent Done false 2392 Acked Write 6 Done Sent Done false 2393 Acked Read 2 Done Sent Done false 2394 Acked Read 10 Done Sent Done false 2395 Acked Read 3 Done Sent Done false 2396 Acked Read 9 Done Sent Done false 2397 Acked Write 4 Done Sent Done false 2398 Acked Flush 0 Done Sent Done false 2399 Acked Write 5 Done Sent Done false 2400 Acked Write 4 Done Sent Done false 2401 Acked Read 9 Done Sent Done false 2402 Acked Read 3 Done Sent Done false 2403 Acked Write 10 Done Sent Done false 2404 Acked Write 1 Done Sent Done false 2405 Acked Read 1 Done Sent Done false 2406 Acked Read 5 Done Sent Done false 2407 Acked Write 9 Done Sent Done false 2408 Acked Read 1 Done Sent Done false 2409 Acked Write 8 Done Sent Done false 2410 Acked Write 6 Done Sent Done false 2411 Acked Read 10 Done Sent Done false 2412 Acked Read 10 Done Sent Done false 2413 Acked Write 6 Done Sent Done false 2414 Acked Write 1 Done Sent Done false 2415 Acked Read 10 Done Sent Done false 2416 Acked Write 3 Done Sent Done false 2417 Acked Read 8 Done Sent Done false 2418 Acked Write 6 Done Sent Done false 2419 Acked Read 9 Done Sent Done false 2420 Acked Read 2 Done Sent Done false 2421 Acked Read 10 Done Sent Done false 2422 Acked Flush 0 Done Sent Done false 2423 Acked Write 1 Done Sent Done false 2424 Acked Read 9 Done Sent Done false 2425 Acked Write 1 Done Sent Done false 2426 Acked Read 10 Done Sent Done false 2427 Acked Flush 0 Done Sent Done false 2428 Acked Write 3 Done Sent Done false 2429 Acked Read 7 Done Sent Done false 2430 Acked Write 6 Done Sent Done false 2431 Acked Read 6 Done Sent Done false 2432 Acked Write 1 Done Sent Done false 2433 Acked Read 5 Done Sent Done false 2434 Acked Write 4 Done Sent Done false 2435 Acked Write 5 Done Sent Done false 2436 Acked Write 4 Done Sent Done false 2437 Acked Read 2 Done Sent Done false 2438 Acked Read 4 Done Sent Done false 2439 Acked Write 3 Done Sent Done false 2440 Acked Write 1 Done Sent Done false 2441 Acked Write 6 Done Sent Done false 2442 Acked Read 9 Done Sent Done false 2443 Acked Write 9 Done Sent Done false 2444 Acked Write 8 Done Sent Done false 2445 Acked Read 8 Done Sent Done false 2446 Acked Read 7 Done Sent Done false 2447 Acked Read 1 Done Sent Done false 2448 Acked Flush 0 Done Sent Done false 2449 Acked Read 1 Done Sent Done false 2450 Acked Write 9 Done Sent Done false 2451 Acked Write 10 Done Sent Done false 2452 Acked Read 9 Done Sent Done false 2453 Acked Read 2 Done Sent Done false 2454 Acked Read 3 Done Sent Done false 2455 Acked Read 7 Done Sent Done false 2456 Acked Read 9 Done Sent Done false 2457 Acked Read 4 Done Sent Done false 2458 Acked Read 9 Done Sent Done false 2459 Acked Write 4 Done Sent Done false 2460 Acked Read 1 Done Sent Done false 2461 Acked Read 7 Done Sent Done false 2462 Acked Read 5 Done Sent Done false 2463 Acked Write 3 Done Sent Done false 2464 Acked Read 3 Done Sent Done false 2465 Acked Write 2 Done Sent Done false 2466 Acked Read 1 Done Sent Done false 2467 Acked Write 3 Done Sent Done false 2468 Acked Write 6 Done Sent Done false 2469 Acked Flush 0 Done Sent Done false 2470 Acked Read 1 Done Sent Done false 2471 Acked Flush 0 Done Sent Done false 2472 Acked Write 10 Done Sent Done false 2473 Acked Read 10 Done Sent Done false 2474 Acked Read 10 Done Sent Done false 2475 Acked Read 10 Done Sent Done false 2476 Acked Read 10 Done Sent Done false 2477 Acked Read 1 Done Sent Done false 2478 Acked Write 7 Done Sent Done false 2479 Acked Write 5 Done Sent Done false 2480 Acked Read 9 Done Sent Done false 2481 Acked Read 2 Done Sent Done false 2482 Acked Read 6 Done Sent Done false 2483 Acked Write 3 Done Sent Done false 2484 Acked Read 3 Done Sent Done false 2485 Acked Read 6 Done Sent Done false 2486 Acked Write 1 Done Sent Done false 2487 Acked Write 10 Done Sent Done false 2488 Acked Write 7 Done Sent Done false 2489 Acked Flush 0 Done Sent Done false 2490 Acked Read 6 Done Sent Done false 2491 Acked Write 7 Done Sent Done false 2492 Acked Flush 0 Done Sent Done false 2493 Acked Write 2 Done Sent Done false 2494 Acked Write 9 Done Sent Done false 2495 Acked Read 3 Done Sent Done false 2496 Acked Write 9 Done Sent Done false 2497 Acked Read 8 Done Sent Done false 2498 Acked Write 7 Done Sent Done false 2499 Acked Read 10 Done Sent Done false 2500 Acked Write 9 Done Sent Done false 2501 Acked Read 3 Done Sent Done false 2502 Acked Read 6 Done Sent Done false 2503 Acked Read 1 Done Sent Done false 2504 Acked Read 7 Done Sent Done false 2505 Acked Read 1 Done Sent Done false 2506 Acked Read 5 Done Sent Done false 2507 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2507 2206 2507 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 2507 2506 2505 2504 2503 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2207 Acked Read 5 Done Sent Done false 2208 Acked Read 9 Done Sent Done false 2209 Acked Write 3 Done Sent Done false 2210 Acked Read 2 Done Sent Done false 2211 Acked Write 8 Done Sent Done false 2212 Acked Read 3 Done Sent Done false 2213 Acked Write 3 Done Sent Done false 2214 Acked Flush 0 Done Sent Done false 2215 Acked Read 3 Done Sent Done false 2216 Acked Read 4 Done Sent Done false 2217 Acked Flush 0 Done Sent Done false 2218 Acked Read 1 Done Sent Done false 2219 Acked Write 4 Done Sent Done false 2220 Acked Write 9 Done Sent Done false 2221 Acked Write 10 Done Sent Done false 2222 Acked Read 9 Done Sent Done false 2223 Acked Read 8 Done Sent Done false 2224 Acked Read 7 Done Sent Done false 2225 Acked Write 7 Done Sent Done false 2226 Acked Write 8 Done Sent Done false 2227 Acked Read 7 Done Sent Done false 2228 Acked Write 6 Done Sent Done false 2229 Acked Read 7 Done Sent Done false 2230 Acked Read 7 Done Sent Done false 2231 Acked Read 1 Done Sent Done false 2232 Acked Write 3 Done Sent Done false 2233 Acked Write 1 Done Sent Done false 2234 Acked Write 2 Done Sent Done false 2235 Acked Read 6 Done Sent Done false 2236 Acked Write 9 Done Sent Done false 2237 Acked Read 10 Done Sent Done false 2238 Acked Write 2 Done Sent Done false 2239 Acked Flush 0 Done Sent Done false 2240 Acked Write 10 Done Sent Done false 2241 Acked Write 1 Done Sent Done false 2242 Acked Write 10 Done Sent Done false 2243 Acked Write 8 Done Sent Done false 2244 Acked Write 7 Done Sent Done false 2245 Acked Write 6 Done Sent Done false 2246 Acked Write 5 Done Sent Done false 2247 Acked Flush 0 Done Sent Done false 2248 Acked Write 8 Done Sent Done false 2249 Acked Read 8 Done Sent Done false 2250 Acked Flush 0 Done Sent Done false 2251 Acked Read 8 Done Sent Done false 2252 Acked Read 5 Done Sent Done false 2253 Acked Write 10 Done Sent Done false 2254 Acked Read 4 Done Sent Done false 2255 Acked Read 3 Done Sent Done false 2256 Acked Read 8 Done Sent Done false 2257 Acked Write 3 Done Sent Done false 2258 Acked Flush 0 Done Sent Done false 2259 Acked Write 10 Done Sent Done false 2260 Acked Write 3 Done Sent Done false 2261 Acked Read 2 Done Sent Done false 2262 Acked Write 1 Done Sent Done false 2263 Acked Read 4 Done Sent Done false 2264 Acked Write 2 Done Sent Done false 2265 Acked Write 6 Done Sent Done false 2266 Acked Write 3 Done Sent Done false 2267 Acked Flush 0 Done Sent Done false 2268 Acked Flush 0 Done Sent Done false 2269 Acked Read 6 Done Sent Done false 2270 Acked Read 10 Done Sent Done false 2271 Acked Flush 0 Done Sent Done false 2272 Acked Write 7 Done Sent Done false 2273 Acked Write 9 Done Sent Done false 2274 Acked Read 1 Done Sent Done false 2275 Acked Read 2 Done Sent Done false 2276 Acked Read 9 Done Sent Done false 2277 Acked Write 7 Done Sent Done false 2278 Acked Read 1 Done Sent Done false 2279 Acked Read 8 Done Sent Done false 2280 Acked Read 9 Done Sent Done false 2281 Acked Write 1 Done Sent Done false 2282 Acked Read 1 Done Sent Done false 2283 Acked Write 5 Done Sent Done false 2284 Acked Read 2 Done Sent Done false 2285 Acked Write 9 Done Sent Done false 2286 Acked Read 6 Done Sent Done false 2287 Acked Write 4 Done Sent Done false 2288 Acked Flush 0 Done Sent Done false 2289 Acked Read 2 Done Sent Done false 2290 Acked Write 10 Done Sent Done false 2291 Acked Read 3 Done Sent Done false 2292 Acked Read 2 Done Sent Done false 2293 Acked Read 1 Done Sent Done false 2294 Acked Read 6 Done Sent Done false 2295 Acked Read 10 Done Sent Done false 2296 Acked Write 7 Done Sent Done false 2297 Acked Flush 0 Done Sent Done false 2298 Acked Write 7 Done Sent Done false 2299 Acked Write 6 Done Sent Done false 2300 Acked Write 5 Done Sent Done false 2301 Acked Read 10 Done Sent Done false 2302 Acked Read 1 Done Sent Done false 2303 Acked Read 1 Done Sent Done false 2304 Acked Read 8 Done Sent Done false 2305 Acked Write 7 Done Sent Done false 2306 Acked Write 8 Done Sent Done false 2307 Acked Read 3 Done Sent Done false 2308 Acked Read 9 Done Sent Done false 2309 Acked Read 5 Done Sent Done false 2310 Acked Read 10 Done Sent Done false 2311 Acked Write 3 Done Sent Done false 2312 Acked Read 9 Done Sent Done false 2313 Acked Read 4 Done Sent Done false 2314 Acked Write 3 Done Sent Done false 2315 Acked Write 6 Done Sent Done false 2316 Acked Write 6 Done Sent Done false 2317 Acked Write 9 Done Sent Done false 2318 Acked Read 3 Done Sent Done false 2319 Acked Flush 0 Done Sent Done false 2320 Acked Read 7 Done Sent Done false 2321 Acked Write 1 Done Sent Done false 2322 Acked Read 1 Done Sent Done false 2323 Acked Read 7 Done Sent Done false 2324 Acked Read 5 Done Sent Done false 2325 Acked Write 7 Done Sent Done false 2326 Acked Write 3 Done Sent Done false 2327 Acked Read 1 Done Sent Done false 2328 Acked Flush 0 Done Sent Done false 2329 Acked Write 2 Done Sent Done false 2330 Acked Read 3 Done Sent Done false 2331 Acked Read 10 Done Sent Done false 2332 Acked Read 8 Done Sent Done false 2333 Acked Read 6 Done Sent Done false 2334 Acked Read 6 Done Sent Done false 2335 Acked Write 7 Done Sent Done false 2336 Acked Flush 0 Done Sent Done false 2337 Acked Read 4 Done Sent Done false 2338 Acked Write 6 Done Sent Done false 2339 Acked Read 7 Done Sent Done false 2340 Acked Write 8 Done Sent Done false 2341 Acked Write 2 Done Sent Done false 2342 Acked Write 9 Done Sent Done false 2343 Acked Write 3 Done Sent Done false 2344 Acked Write 4 Done Sent Done false 2345 Acked Write 10 Done Sent Done false 2346 Acked Write 7 Done Sent Done false 2347 Acked Write 9 Done Sent Done false 2348 Acked Read 3 Done Sent Done false 2349 Acked Write 1 Done Sent Done false 2350 Acked Read 2 Done Sent Done false 2351 Acked Read 9 Done Sent Done false 2352 Acked Write 10 Done Sent Done false 2353 Acked Read 5 Done Sent Done false 2354 Acked Flush 0 Done Sent Done false 2355 Acked Write 1 Done Sent Done false 2356 Acked Read 10 Done Sent Done false 2357 Acked Read 2 Done Sent Done false 2358 Acked Write 2 Done Sent Done false 2359 Acked Write 3 Done Sent Done false 2360 Acked Write 9 Done Sent Done false 2361 Acked Write 9 Done Sent Done false 2362 Acked Read 8 Done Sent Done false 2363 Acked Read 1 Done Sent Done false 2364 Acked Read 5 Done Sent Done false 2365 Acked Write 9 Done Sent Done false 2366 Acked Read 5 Done Sent Done false 2367 Acked Read 5 Done Sent Done false 2368 Acked Read 3 Done Sent Done false 2369 Acked Write 2 Done Sent Done false 2370 Acked Read 2 Done Sent Done false 2371 Acked Write 7 Done Sent Done false 2372 Acked Read 6 Done Sent Done false 2373 Acked Read 7 Done Sent Done false 2374 Acked Write 5 Done Sent Done false 2375 Acked Flush 0 Done Sent Done false 2376 Acked Write 7 Done Sent Done false 2377 Acked Read 7 Done Sent Done false 2378 Acked Write 2 Done Sent Done false 2379 Acked Write 7 Done Sent Done false 2380 Acked Flush 0 Done Sent Done false 2381 Acked Read 5 Done Sent Done false 2382 Acked Write 8 Done Sent Done false 2383 Acked Write 3 Done Sent Done false 2384 Acked Read 9 Done Sent Done false 2385 Acked Read 9 Done Sent Done false 2386 Acked Read 1 Done Sent Done false 2387 Acked Read 5 Done Sent Done false 2388 Acked Write 6 Done Sent Done false 2389 Acked Read 4 Done Sent Done false 2390 Acked Read 6 Done Sent Done false 2391 Acked Read 8 Done Sent Done false 2392 Acked Write 6 Done Sent Done false 2393 Acked Read 2 Done Sent Done false 2394 Acked Read 10 Done Sent Done false 2395 Acked Read 3 Done Sent Done false 2396 Acked Read 9 Done Sent Done false 2397 Acked Write 4 Done Sent Done false 2398 Acked Flush 0 Done Sent Done false 2399 Acked Write 5 Done Sent Done false 2400 Acked Write 4 Done Sent Done false 2401 Acked Read 9 Done Sent Done false 2402 Acked Read 3 Done Sent Done false 2403 Acked Write 10 Done Sent Done false 2404 Acked Write 1 Done Sent Done false 2405 Acked Read 1 Done Sent Done false 2406 Acked Read 5 Done Sent Done false 2407 Acked Write 9 Done Sent Done false 2408 Acked Read 1 Done Sent Done false 2409 Acked Write 8 Done Sent Done false 2410 Acked Write 6 Done Sent Done false 2411 Acked Read 10 Done Sent Done false 2412 Acked Read 10 Done Sent Done false 2413 Acked Write 6 Done Sent Done false 2414 Acked Write 1 Done Sent Done false 2415 Acked Read 10 Done Sent Done false 2416 Acked Write 3 Done Sent Done false 2417 Acked Read 8 Done Sent Done false 2418 Acked Write 6 Done Sent Done false 2419 Acked Read 9 Done Sent Done false 2420 Acked Read 2 Done Sent Done false 2421 Acked Read 10 Done Sent Done false 2422 Acked Flush 0 Done Sent Done false 2423 Acked Write 1 Done Sent Done false 2424 Acked Read 9 Done Sent Done false 2425 Acked Write 1 Done Sent Done false 2426 Acked Read 10 Done Sent Done false 2427 Acked Flush 0 Done Sent Done false 2428 Acked Write 3 Done Sent Done false 2429 Acked Read 7 Done Sent Done false 2430 Acked Write 6 Done Sent Done false 2431 Acked Read 6 Done Sent Done false 2432 Acked Write 1 Done Sent Done false 2433 Acked Read 5 Done Sent Done false 2434 Acked Write 4 Done Sent Done false 2435 Acked Write 5 Done Sent Done false 2436 Acked Write 4 Done Sent Done false 2437 Acked Read 2 Done Sent Done false 2438 Acked Read 4 Done Sent Done false 2439 Acked Write 3 Done Sent Done false 2440 Acked Write 1 Done Sent Done false 2441 Acked Write 6 Done Sent Done false 2442 Acked Read 9 Done Sent Done false 2443 Acked Write 9 Done Sent Done false 2444 Acked Write 8 Done Sent Done false 2445 Acked Read 8 Done Sent Done false 2446 Acked Read 7 Done Sent Done false 2447 Acked Read 1 Done Sent Done false 2448 Acked Flush 0 Done Sent Done false 2449 Acked Read 1 Done Sent Done false 2450 Acked Write 9 Done Sent Done false 2451 Acked Write 10 Done Sent Done false 2452 Acked Read 9 Done Sent Done false 2453 Acked Read 2 Done Sent Done false 2454 Acked Read 3 Done Sent Done false 2455 Acked Read 7 Done Sent Done false 2456 Acked Read 9 Done Sent Done false 2457 Acked Read 4 Done Sent Done false 2458 Acked Read 9 Done Sent Done false 2459 Acked Write 4 Done Sent Done false 2460 Acked Read 1 Done Sent Done false 2461 Acked Read 7 Done Sent Done false 2462 Acked Read 5 Done Sent Done false 2463 Acked Write 3 Done Sent Done false 2464 Acked Read 3 Done Sent Done false 2465 Acked Write 2 Done Sent Done false 2466 Acked Read 1 Done Sent Done false 2467 Acked Write 3 Done Sent Done false 2468 Acked Write 6 Done Sent Done false 2469 Acked Flush 0 Done Sent Done false 2470 Acked Read 1 Done Sent Done false 2471 Acked Flush 0 Done Sent Done false 2472 Acked Write 10 Done Sent Done false 2473 Acked Read 10 Done Sent Done false 2474 Acked Read 10 Done Sent Done false 2475 Acked Read 10 Done Sent Done false 2476 Acked Read 10 Done Sent Done false 2477 Acked Read 1 Done Sent Done false 2478 Acked Write 7 Done Sent Done false 2479 Acked Write 5 Done Sent Done false 2480 Acked Read 9 Done Sent Done false 2481 Acked Read 2 Done Sent Done false 2482 Acked Read 6 Done Sent Done false 2483 Acked Write 3 Done Sent Done false 2484 Acked Read 3 Done Sent Done false 2485 Acked Read 6 Done Sent Done false 2486 Acked Write 1 Done Sent Done false 2487 Acked Write 10 Done Sent Done false 2488 Acked Write 7 Done Sent Done false 2489 Acked Flush 0 Done Sent Done false 2490 Acked Read 6 Done Sent Done false 2491 Acked Write 7 Done Sent Done false 2492 Acked Flush 0 Done Sent Done false 2493 Acked Write 2 Done Sent Done false 2494 Acked Write 9 Done Sent Done false 2495 Acked Read 3 Done Sent Done false 2496 Acked Write 9 Done Sent Done false 2497 Acked Read 8 Done Sent Done false 2498 Acked Write 7 Done Sent Done false 2499 Acked Read 10 Done Sent Done false 2500 Acked Write 9 Done Sent Done false 2501 Acked Read 3 Done Sent Done false 2502 Acked Read 6 Done Sent Done false 2503 Acked Read 1 Done Sent Done false 2504 Acked Read 7 Done Sent Done false 2505 Acked Read 1 Done Sent Done false 2506 Acked Read 5 Done Sent Done false 2507 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2507 2206 2507 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 2507 2506 2505 2504 2503 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.675605259Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:59308)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.675673391Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:48.676626095Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677149454Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677169049Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 2206","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677183605Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 2206","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677398747Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677417113Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677431079Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 301 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.677444855Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 301 jobs since flush: 2206","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.67768888Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2507 2506 2505 2504 2503 Upstairs last five completed: 2507 2506 2505 2504 2503 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:42:48.915 INFO request completed, latency_us: 365, [8820] Got stop action so:false kr:false response_code: 204, uri: /stop/cid/1, method: POST, req_id: 54b9f378-0b25-4377-b157-591a0f4b22e3, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:42:48.916 INFO accepted connection, remote_addr: 127.0.0.1:42806, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:48.916424658Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:48.916487561Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:48.916519063Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.916547336Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.916563991Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:48.916602281Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:42:48.916 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/1, method: GET, req_id: cae16e2d-f762-4ae4-a0c3-8c2dd05d697f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:42:52.919 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/1, method: GET, req_id: 56381b70-b1d2-4a65-a1d9-58d05bd4887b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 759 len 36864 002/300 Write block 1082 len 32768 data: 3 3 3 4 3 3 3 3 003/300 Read block 1222 len 12288 004/300 Read block 4627 len 8192 005/300 Write block 1246 len 36864 data: 2 2 3 3 3 3 3 3 3 006/300 Write block 2155 len 40960 data: 2 2 2 2 2 2 2 2 2 2 007/300 Read block 2734 len 16384 008/300 Read block 2307 len 20480 009/300 Write block 128 len 12288 data: 3 3 3 010/300 Write block 2923 len 8192 data: 2 2 011/300 Read block 2595 len 32768 012/300 Flush 013/300 Read block 1348 len 12288 014/300 Read block 2471 len 12288 015/300 Read block 2580 len 36864 016/300 Read block 1564 len 8192 017/300 Write block 1946 len 36864 data: 2 2 2 2 2 2 2 2 2 018/300 Write block 3635 len 40960 data: 2 2 2 2 2 2 2 2 2 2 019/300 Write block 4134 len 4096 data: 3 020/300 Read block 2491 len 40960 021/300 Write block 2232 len 4096 data: 2 022/300 Write block 3397 len 24576 data: 2 2 2 2 2 2 023/300 Read block 1756 len 4096 024/300 Flush 025/300 Read block 1966 len 8192 026/300 Read block 1073 len 8192 027/300 Write block 1265 len 32768 data: 2 2 2 2 2 2 2 2 028/300 Write block 232 len 40960 data: 2 2 2 2 3 3 3 3 3 3 029/300 Write block 230 len 28672 data: 2 2 3 3 3 3 4 030/300 Read block 2171 len 8192 031/300 Write block 890 len 40960 data: 3 3 3 3 3 2 2 2 2 2 032/300 Read block 2989 len 20480 033/300 Write block 4197 len 16384 data: 4 4 4 5 034/300 Read block 1944 len 32768 035/300 Read block 2922 len 4096 036/300 Write block 2998 len 8192 data: 2 2 037/300 Read block 3675 len 32768 038/300 Write block 2244 len 24576 data: 4 4 4 4 4 4 039/300 Write block 180 len 24576 data: 3 3 3 4 4 3 040/300 Flush 041/300 Read block 1404 len 32768 042/300 Write block 4031 len 12288 data: 3 3 3 043/300 Flush 044/300 Write block 4655 len 8192 data: 2 2 045/300 Write block 163 len 36864 data: 2 2 2 2 2 2 2 2 2 046/300 Read block 1067 len 16384 047/300 Write block 1190 len 12288 data: 2 2 2 048/300 Write block 2071 len 20480 data: 2 2 2 2 2 049/300 Read block 3345 len 32768 050/300 Read block 3558 len 32768 051/300 Write block 2771 len 12288 data: 2 2 2 052/300 Write block 4958 len 32768 data: 2 2 2 2 3 3 3 2 053/300 Read block 166 len 32768 054/300 Write block 3018 len 28672 data: 3 4 4 3 3 3 2 055/300 Read block 2731 len 28672 056/300 Read block 4923 len 40960 057/300 Write block 1052 len 12288 data: 2 2 2 058/300 Read block 1299 len 4096 059/300 Flush 060/300 Read block 1260 len 36864 061/300 Read block 2777 len 36864 062/300 Write block 3166 len 32768 data: 5 4 4 4 4 3 3 2 063/300 Write block 1081 len 24576 data: 2 4 4 4 5 4 064/300 Write block 1357 len 32768 data: 2 2 2 2 3 3 3 3 065/300 Write block 1525 len 4096 data: 2 066/300 Write block 821 len 40960 data: 2 2 2 3 3 3 2 2 2 2 067/300 Read block 2054 len 4096 068/300 Read block 4027 len 20480 069/300 Read block 4116 len 32768 070/300 Read block 1839 len 12288 071/300 Read block 2555 len 8192 072/300 Read block 1258 len 28672 073/300 Read block 3642 len 40960 074/300 Read block 1220 len 8192 075/300 Write block 631 len 12288 data: 2 2 2 076/300 Read block 366 len 4096 077/300 Write block 626 len 40960 data: 2 2 2 2 2 3 3 3 2 3 078/300 Read block 3611 len 36864 079/300 Write block 4476 len 40960 data: 2 2 3 3 3 3 3 3 4 3 080/300 Write block 1449 len 12288 data: 3 3 3 081/300 Read block 364 len 28672 082/300 Read block 4973 len 4096 083/300 Read block 4695 len 36864 084/300 Write block 3130 len 36864 data: 3 2 2 2 2 3 3 3 3 085/300 Write block 1749 len 24576 data: 2 2 2 2 2 2 086/300 Read block 183 len 28672 087/300 Read block 3897 len 36864 088/300 Write block 4082 len 20480 data: 2 2 2 2 2 089/300 Flush 090/300 Write block 1864 len 36864 data: 3 3 3 3 3 3 3 3 2 091/300 Read block 4170 len 36864 092/300 Read block 3494 len 4096 093/300 Flush 094/300 Flush 095/300 Flush 096/300 Read block 1739 len 8192 097/300 Write block 2306 len 32768 data: 3 3 3 3 3 3 3 3 098/300 Write block 4304 len 24576 data: 3 3 3 3 3 3 099/300 Write block 4352 len 28672 data: 2 2 3 2 3 3 3 100/300 Read block 2932 len 8192 101/300 Read block 1013 len 40960 102/300 Flush 103/300 Write block 428 len 4096 data: 3 104/300 Read block 2308 len 4096 105/300 Read block 3065 len 36864 106/300 Write block 2423 len 16384 data: 2 3 3 3 107/300 Flush 108/300 Write block 3131 len 36864 data: 3 3 3 3 4 4 4 4 3 109/300 Write block 3428 len 32768 data: 2 2 2 2 2 2 2 2 110/300 Read block 4763 len 12288 111/300 Write block 3734 len 32768 data: 3 3 3 3 4 3 3 3 112/300 Read block 1001 len 16384 113/300 Read block 96 len 28672 114/300 Write block 440 len 24576 data: 3 3 2 2 2 3 115/300 Read block 1836 len 16384 116/300 Read block 4951 len 20480 117/300 Read block 830 len 24576 118/300 Flush 119/300 Read block 4882 len 8192 120/300 Read block 2002 len 16384 121/300 Read block 2808 len 8192 122/300 Read block 2590 len 20480 123/300 Write block 2378 len 12288 data: 4 4 4 124/300 Read block 2033 len 8192 125/300 Read block 2573 len 32768 126/300 Read block 4420 len 8192 127/300 Write block 725 len 12288 data: 2 3 3 128/300 Flush 129/300 Write block 3990 len 12288 data: 2 2 2 130/300 Write block 3275 len 12288 data: 2 2 2 131/300 Read block 407 len 8192 132/300 Write block 3580 len 24576 data: 3 3 2 2 2 2 133/300 Write block 1714 len 4096 data: 3 134/300 Read block 860 len 36864 135/300 Write block 1951 len 40960 data: 3 3 3 3 2 2 2 2 2 2 136/300 Flush 137/300 Write block 2462 len 40960 data: 3 3 3 3 3 4 4 3 3 3 138/300 Write block 2957 len 20480 data: 3 2 2 2 3 139/300 Read block 3760 len 36864 140/300 Read block 1698 len 24576 141/300 Write block 1176 len 24576 data: 4 4 4 5 3 3 142/300 Read block 2802 len 28672 143/300 Write block 2490 len 16384 data: 2 2 2 3 144/300 Read block 2343 len 36864 145/300 Read block 4480 len 20480 146/300 Write block 3615 len 32768 data: 2 2 2 2 2 3 3 3 147/300 Read block 2368 len 8192 148/300 Read block 2058 len 28672 149/300 Read block 1832 len 24576 150/300 Read block 913 len 12288 151/300 Read block 1182 len 16384 152/300 Read block 1443 len 32768 153/300 Flush 154/300 Write block 4227 len 28672 data: 2 2 2 2 2 2 2 155/300 Flush 156/300 Write block 1109 len 28672 data: 2 2 2 2 2 2 2 157/300 Flush 158/300 Read block 286 len 36864 159/300 Read block 3216 len 4096 160/300 Read block 4495 len 16384 161/300 Read block 882 len 16384 162/300 Read block 4383 len 36864 163/300 Write block 3680 len 28672 data: 5 5 5 5 3 3 3 164/300 Read block 2484 len 24576 165/300 Read block 2281 len 40960 166/300 Write block 4332 len 16384 data: 4 4 3 3 167/300 Read block 960 len 28672 168/300 Read block 784 len 16384 169/300 Read block 1061 len 24576 170/300 Read block 4922 len 24576 171/300 Read block 2009 len 24576 172/300 Read block 1719 len 24576 173/300 Read block 3118 len 32768 174/300 Write block 2601 len 4096 data: 2 175/300 Read block 311 len 4096 176/300 Read block 389 len 8192 177/300 Write block 1750 len 16384 data: 3 3 3 3 178/300 Write block 1292 len 20480 data: 3 3 3 3 3 179/300 Read block 2631 len 36864 180/300 Read block 2128 len 8192 181/300 Read block 3558 len 28672 182/300 Read block 3394 len 20480 183/300 Read block 1115 len 4096 184/300 Read block 4987 len 40960 185/300 Read block 4648 len 20480 186/300 Write block 2541 len 16384 data: 4 4 4 4 187/300 Write block 3464 len 32768 data: 3 3 3 3 3 3 3 2 188/300 Flush 189/300 Read block 2997 len 8192 190/300 Read block 4932 len 16384 191/300 Write block 3422 len 4096 data: 3 192/300 Read block 1520 len 12288 193/300 Write block 3858 len 16384 data: 2 2 2 2 194/300 Read block 4192 len 40960 195/300 Read block 3316 len 16384 196/300 Read block 623 len 12288 197/300 Write block 2056 len 12288 data: 3 3 3 198/300 Write block 3667 len 20480 data: 2 2 2 2 2 199/300 Write block 2619 len 16384 data: 2 2 2 2 200/300 Read block 1067 len 20480 201/300 Read block 3792 len 8192 202/300 Read block 3475 len 28672 203/300 Write block 3077 len 4096 data: 2 204/300 Write block 1033 len 8192 data: 3 3 205/300 Read block 4165 len 40960 206/300 Write block 2124 len 12288 data: 2 2 2 207/300 Write block 1767 len 4096 data: 2 208/300 Read block 4920 len 24576 209/300 Read block 753 len 16384 210/300 Write block 3569 len 12288 data: 2 2 2 211/300 Read block 2325 len 20480 212/300 Read block 4743 len 36864 213/300 Flush 214/300 Read block 1402 len 8192 215/300 Write block 341 len 4096 data: 4 216/300 Write block 4128 len 40960 data: 3 3 3 2 2 2 4 3 3 3 217/300 Read block 4077 len 8192 218/300 Read block 420 len 28672 219/300 Read block 3962 len 16384 220/300 Read block 1901 len 16384 221/300 Read block 4001 len 4096 222/300 Read block 2911 len 8192 223/300 Read block 4242 len 4096 224/300 Read block 2242 len 8192 225/300 Write block 3813 len 12288 data: 3 3 3 226/300 Read block 3144 len 24576 227/300 Write block 3418 len 8192 data: 2 2 228/300 Write block 454 len 8192 data: 2 2 229/300 Read block 1186 len 24576 230/300 Flush 231/300 Read block 3056 len 32768 232/300 Read block 4408 len 16384 233/300 Write block 4745 len 24576 data: 4 4 4 4 3 3 234/300 Read block 1195 len 16384 235/300 Write block 3205 len 28672 data: 2 2 3 3 3 2 2 236/300 Write block 2384 len 28672 data: 3 3 3 3 4 4 4 237/300 Read block 594 len 20480 238/300 Read block 4042 len 36864 239/300 Read block 4128 len 20480 240/300 Write block 2565 len 28672 data: 5 6 6 5 5 4 3 241/300 Write block 2888 len 40960 data: 3 3 3 3 2 3 3 3 3 3 242/300 Read block 2066 len 12288 243/300 Read block 1480 len 36864 244/300 Read block 2451 len 12288 245/300 Read block 4595 len 12288 246/300 Read block 3404 len 36864 247/300 Write block 3149 len 4096 data: 2 248/300 Read block 4824 len 28672 249/300 Write block 4589 len 28672 data: 2 2 2 2 2 3 3 250/300 Flush 251/300 Write block 3667 len 20480 data: 3 3 3 3 3 252/300 Flush 253/300 Write block 1668 len 4096 data: 2 254/300 Read block 463 len 36864 255/300 Read block 1803 len 12288 256/300 Read block 1057 len 40960 257/300 Read block 2343 len 40960 258/300 Read block 472 len 24576 259/300 Write block 2354 len 12288 data: 4 3 2 260/300 Write block 1863 len 28672 data: 3 4 4 4 4 4 4 261/300 Read block 2910 len 12288 262/300 Write block 3741 len 12288 data: 4 3 3 263/300 Read block 2535 len 24576 264/300 Write block 2901 len 12288 data: 3 2 2 265/300 Read block 3045 len 36864 266/300 Read block 126 len 16384 267/300 Read block 984 len 16384 268/300 Write block 1645 len 40960 data: 3 3 3 3 3 3 3 3 2 2 269/300 Write block 1981 len 4096 data: 2 270/300 Read block 2128 len 24576 271/300 Flush 272/300 Read block 3392 len 12288 273/300 Read block 3069 len 4096 274/300 Write block 1366 len 24576 data: 3 3 4 4 4 3 275/300 Read block 1739 len 36864 276/300 Read block 1370 len 8192 277/300 Read block 1937 len 24576 278/300 Write block 261 len 8192 data: 3 3 279/300 Read block 1707 len 16384 280/300 Flush 281/300 Write block 4262 len 16384 data: 3 3 3 3 282/300 Flush 283/300 Write block 1832 len 4096 data: 4 284/300 Read block 2340 len 4096 285/300 Write block 2556 len 8192 data: 3 3 286/300 Read block 4033 len 32768 287/300 Write block 1085 len 12288 data: 6 5 4 288/300 Flush 289/300 Read block 941 len 12288 290/300 Read block 1668 len 4096 291/300 Write block 858 len 24576 data: 2 2 2 2 2 2 292/300 Write block 3476 len 36864 data: 2 2 2 2 2 2 2 2 2 293/300 Read block 2777 len 36864 294/300 Read block 4908 len 16384 295/300 Write block 4654 len 20480 data: 2 3 3 4 5 296/300 Write block 4518 len 36864 data: 3 2 2 2 3 3 3 3 3 297/300 Read block 3151 len 16384 298/300 Write block 4929 len 4096 data: 3 299/300 Write block 2775 len 4096 data: 2 300/300 Write block 407 len 12288 data: 3 3 3 301/300 Read block 188 len 16384 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:42:53.150 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/1, method: POST, req_id: 63ff5d08-087a-4596-8adc-19cb88ca30a2, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2508 Acked Read 9 Done Sent Done false 2509 Acked Write 8 Done Sent Done false 2510 Acked Read 3 Done Sent Done false 2511 Acked Read 2 Done Sent Done false 2512 Acked Write 9 Done Sent Done false 2513 Acked Write 10 Done Sent Done false 2514 Acked Read 4 Done Sent Done false 2515 Acked Read 5 Done Sent Done false 2516 Acked Write 3 Done Sent Done false 2517 Acked Write 2 Done Sent Done false 2518 Acked Read 8 Done Sent Done false 2519 Acked Flush 0 Done Sent Done false 2520 Acked Read 3 Done Sent Done false 2521 Acked Read 3 Done Sent Done false 2522 Acked Read 9 Done Sent Done false 2523 Acked Read 2 Done Sent Done false 2524 Acked Write 9 Done Sent Done false 2525 Acked Write 10 Done Sent Done false 2526 Acked Write 1 Done Sent Done false 2527 Acked Read 10 Done Sent Done false 2528 Acked Write 1 Done Sent Done false 2529 Acked Write 6 Done Sent Done false 2530 Acked Read 1 Done Sent Done false 2531 Acked Flush 0 Done Sent Done false 2532 Acked Read 2 Done Sent Done false 2533 Acked Read 2 Done Sent Done false 2534 Acked Write 8 Done Sent Done false 2535 Acked Write 10 Done Sent Done false 2536 Acked Write 7 Done Sent Done false 2537 Acked Read 2 Done Sent Done false 2538 Acked Write 10 Done Sent Done false 2539 Acked Read 5 Done Sent Done false 2540 Acked Write 4 Done Sent Done false 2541 Acked Read 8 Done Sent Done false 2542 Acked Read 1 Done Sent Done false 2543 Acked Write 2 Done Sent Done false 2544 Acked Read 8 Done Sent Done false 2545 Acked Write 6 Done Sent Done false 2546 Acked Write 6 Done Sent Done false 2547 Acked Flush 0 Done Sent Done false 2548 Acked Read 8 Done Sent Done false 2549 Acked Write 3 Done Sent Done false 2550 Acked Flush 0 Done Sent Done false 2551 Acked Write 2 Done Sent Done false 2552 Acked Write 9 Done Sent Done false 2553 Acked Read 4 Done Sent Done false 2554 Acked Write 3 Done Sent Done false 2555 Acked Write 5 Done Sent Done false 2556 Acked Read 8 Done Sent Done false 2557 Acked Read 8 Done Sent Done false 2558 Acked Write 3 Done Sent Done false 2559 Acked Write 8 Done Sent Done false 2560 Acked Read 8 Done Sent Done false 2561 Acked Write 7 Done Sent Done false 2562 Acked Read 7 Done Sent Done false 2563 Acked Read 10 Done Sent Done false 2564 Acked Write 3 Done Sent Done false 2565 Acked Read 1 Done Sent Done false 2566 Acked Flush 0 Done Sent Done false 2567 Acked Read 9 Done Sent Done false 2568 Acked Read 9 Done Sent Done false 2569 Acked Write 8 Done Sent Done false 2570 Acked Write 6 Done Sent Done false 2571 Acked Write 8 Done Sent Done false 2572 Acked Write 1 Done Sent Done false 2573 Acked Write 10 Done Sent Done false 2574 Acked Read 1 Done Sent Done false 2575 Acked Read 5 Done Sent Done false 2576 Acked Read 8 Done Sent Done false 2577 Acked Read 3 Done Sent Done false 2578 Acked Read 2 Done Sent Done false 2579 Acked Read 7 Done Sent Done false 2580 Acked Read 10 Done Sent Done false 2581 Acked Read 2 Done Sent Done false 2582 Acked Write 3 Done Sent Done false 2583 Acked Read 1 Done Sent Done false 2584 Acked Write 10 Done Sent Done false 2585 Acked Read 9 Done Sent Done false 2586 Acked Write 10 Done Sent Done false 2587 Acked Write 3 Done Sent Done false 2588 Acked Read 7 Done Sent Done false 2589 Acked Read 1 Done Sent Done false 2590 Acked Read 9 Done Sent Done false 2591 Acked Write 9 Done Sent Done false 2592 Acked Write 6 Done Sent Done false 2593 Acked Read 7 Done Sent Done false 2594 Acked Read 9 Done Sent Done false 2595 Acked Write 5 Done Sent Done false 2596 Acked Flush 0 Done Sent Done false 2597 Acked Write 9 Done Sent Done false 2598 Acked Read 9 Done Sent Done false 2599 Acked Read 1 Done Sent Done false 2600 Acked Flush 0 Done Sent Done false 2601 Acked Flush 0 Done Sent Done false 2602 Acked Flush 0 Done Sent Done false 2603 Acked Read 2 Done Sent Done false 2604 Acked Write 8 Done Sent Done false 2605 Acked Write 6 Done Sent Done false 2606 Acked Write 7 Done Sent Done false 2607 Acked Read 2 Done Sent Done false 2608 Acked Read 10 Done Sent Done false 2609 Acked Flush 0 Done Sent Done false 2610 Acked Write 1 Done Sent Done false 2611 Acked Read 1 Done Sent Done false 2612 Acked Read 9 Done Sent Done false 2613 Acked Write 4 Done Sent Done false 2614 Acked Flush 0 Done Sent Done false 2615 Acked Write 9 Done Sent Done false 2616 Acked Write 8 Done Sent Done false 2617 Acked Read 3 Done Sent Done false 2618 Acked Write 8 Done Sent Done false 2619 Acked Read 4 Done Sent Done false 2620 Acked Read 7 Done Sent Done false 2621 Acked Write 6 Done Sent Done false 2622 Acked Read 4 Done Sent Done false 2623 Acked Read 5 Done Sent Done false 2624 Acked Read 6 Done Sent Done false 2625 Acked Flush 0 Done Sent Done false 2626 Acked Read 2 Done Sent Done false 2627 Acked Read 4 Done Sent Done false 2628 Acked Read 2 Done Sent Done false 2629 Acked Read 5 Done Sent Done false 2630 Acked Write 3 Done Sent Done false 2631 Acked Read 2 Done Sent Done false 2632 Acked Read 8 Done Sent Done false 2633 Acked Read 2 Done Sent Done false 2634 Acked Write 3 Done Sent Done false 2635 Acked Flush 0 Done Sent Done false 2636 Acked Write 3 Done Sent Done false 2637 Acked Write 3 Done Sent Done false 2638 Acked Read 2 Done Sent Done false 2639 Acked Write 6 Done Sent Done false 2640 Acked Write 1 Done Sent Done false 2641 Acked Read 9 Done Sent Done false 2642 Acked Write 10 Done Sent Done false 2643 Acked Flush 0 Done Sent Done false 2644 Acked Write 10 Done Sent Done false 2645 Acked Write 5 Done Sent Done false 2646 Acked Read 9 Done Sent Done false 2647 Acked Read 6 Done Sent Done false 2648 Acked Write 6 Done Sent Done false 2649 Acked Read 7 Done Sent Done false 2650 Acked Write 4 Done Sent Done false 2651 Acked Read 9 Done Sent Done false 2652 Acked Read 5 Done Sent Done false 2653 Acked Write 8 Done Sent Done false 2654 Acked Read 2 Done Sent Done false 2655 Acked Read 7 Done Sent Done false 2656 Acked Read 6 Done Sent Done false 2657 Acked Read 3 Done Sent Done false 2658 Acked Read 4 Done Sent Done false 2659 Acked Read 8 Done Sent Done false 2660 Acked Flush 0 Done Sent Done false 2661 Acked Write 7 Done Sent Done false 2662 Acked Flush 0 Done Sent Done false 2663 Acked Write 7 Done Sent Done false 2664 Acked Flush 0 Done Sent Done false 2665 Acked Read 9 Done Sent Done false 2666 Acked Read 1 Done Sent Done false 2667 Acked Read 4 Done Sent Done false 2668 Acked Read 4 Done Sent Done false 2669 Acked Read 9 Done Sent Done false 2670 Acked Write 7 Done Sent Done false 2671 Acked Read 6 Done Sent Done false 2672 Acked Read 10 Done Sent Done false 2673 Acked Write 4 Done Sent Done false 2674 Acked Read 7 Done Sent Done false 2675 Acked Read 4 Done Sent Done false 2676 Acked Read 6 Done Sent Done false 2677 Acked Read 6 Done Sent Done false 2678 Acked Read 6 Done Sent Done false 2679 Acked Read 6 Done Sent Done false 2680 Acked Read 8 Done Sent Done false 2681 Acked Write 1 Done Sent Done false 2682 Acked Read 1 Done Sent Done false 2683 Acked Read 2 Done Sent Done false 2684 Acked Write 4 Done Sent Done false 2685 Acked Write 5 Done Sent Done false 2686 Acked Read 9 Done Sent Done false 2687 Acked Read 2 Done Sent Done false 2688 Acked Read 7 Done Sent Done false 2689 Acked Read 5 Done Sent Done false 2690 Acked Read 1 Done Sent Done false 2691 Acked Read 10 Done Sent Done false 2692 Acked Read 5 Done Sent Done false 2693 Acked Write 4 Done Sent Done false 2694 Acked Write 8 Done Sent Done false 2695 Acked Flush 0 Done Sent Done false 2696 Acked Read 2 Done Sent Done false 2697 Acked Read 4 Done Sent Done false 2698 Acked Write 1 Done Sent Done false 2699 Acked Read 3 Done Sent Done false 2700 Acked Write 4 Done Sent Done false 2701 Acked Read 10 Done Sent Done false 2702 Acked Read 4 Done Sent Done false 2703 Acked Read 3 Done Sent Done false 2704 Acked Write 3 Done Sent Done false 2705 Acked Write 5 Done Sent Done false 2706 Acked Write 4 Done Sent Done false 2707 Acked Read 5 Done Sent Done false 2708 Acked Read 2 Done Sent Done false 2709 Acked Read 7 Done Sent Done false 2710 Acked Write 1 Done Sent Done false 2711 Acked Write 2 Done Sent Done false 2712 Acked Read 10 Done Sent Done false 2713 Acked Write 3 Done Sent Done false 2714 Acked Write 1 Done Sent Done false 2715 Acked Read 6 Done Sent Done false 2716 Acked Read 4 Done Sent Done false 2717 Acked Write 3 Done Sent Done false 2718 Acked Read 5 Done Sent Done false 2719 Acked Read 9 Done Sent Done false 2720 Acked Flush 0 Done Sent Done false 2721 Acked Read 2 Done Sent Done false 2722 Acked Write 1 Done Sent Done false 2723 Acked Write 10 Done Sent Done false 2724 Acked Read 2 Done Sent Done false 2725 Acked Read 7 Done Sent Done false 2726 Acked Read 4 Done Sent Done false 2727 Acked Read 4 Done Sent Done false 2728 Acked Read 1 Done Sent Done false 2729 Acked Read 2 Done Sent Done false 2730 Acked Read 1 Done Sent Done false 2731 Acked Read 2 Done Sent Done false 2732 Acked Write 3 Done Sent Done false 2733 Acked Read 6 Done Sent Done false 2734 Acked Write 2 Done Sent Done false 2735 Acked Write 2 Done Sent Done false 2736 Acked Read 6 Done Sent Done false 2737 Acked Flush 0 Done Sent Done false 2738 Acked Read 8 Done Sent Done false 2739 Acked Read 4 Done Sent Done false 2740 Acked Write 6 Done Sent Done false 2741 Acked Read 4 Done Sent Done false 2742 Acked Write 7 Done Sent Done false 2743 Acked Write 7 Done Sent Done false 2744 Acked Read 5 Done Sent Done false 2745 Acked Read 9 Done Sent Done false 2746 Acked Read 5 Done Sent Done false 2747 Acked Write 7 Done Sent Done false 2748 Acked Write 10 Done Sent Done false 2749 Acked Read 3 Done Sent Done false 2750 Acked Read 9 Done Sent Done false 2751 Acked Read 3 Done Sent Done false 2752 Acked Read 3 Done Sent Done false 2753 Acked Read 9 Done Sent Done false 2754 Acked Write 1 Done Sent Done false 2755 Acked Read 7 Done Sent Done false 2756 Acked Write 7 Done Sent Done false 2757 Acked Flush 0 Done Sent Done false 2758 Acked Write 5 Done Sent Done false 2759 Acked Flush 0 Done Sent Done false 2760 Acked Write 1 Done Sent Done false 2761 Acked Read 9 Done Sent Done false 2762 Acked Read 3 Done Sent Done false 2763 Acked Read 10 Done Sent Done false 2764 Acked Read 10 Done Sent Done false 2765 Acked Read 6 Done Sent Done false 2766 Acked Write 3 Done Sent Done false 2767 Acked Write 7 Done Sent Done false 2768 Acked Read 3 Done Sent Done false 2769 Acked Write 3 Done Sent Done false 2770 Acked Read 6 Done Sent Done false 2771 Acked Write 3 Done Sent Done false 2772 Acked Read 9 Done Sent Done false 2773 Acked Read 4 Done Sent Done false 2774 Acked Read 4 Done Sent Done false 2775 Acked Write 10 Done Sent Done false 2776 Acked Write 1 Done Sent Done false 2777 Acked Read 6 Done Sent Done false 2778 Acked Flush 0 Done Sent Done false 2779 Acked Read 3 Done Sent Done false 2780 Acked Read 1 Done Sent Done false 2781 Acked Write 6 Done Sent Done false 2782 Acked Read 9 Done Sent Done false 2783 Acked Read 2 Done Sent Done false 2784 Acked Read 6 Done Sent Done false 2785 Acked Write 2 Done Sent Done false 2786 Acked Read 4 Done Sent Done false 2787 Acked Flush 0 Done Sent Done false 2788 Acked Write 4 Done Sent Done false 2789 Acked Flush 0 Done Sent Done false 2790 Acked Write 1 Done Sent Done false 2791 Acked Read 1 Done Sent Done false 2792 Acked Write 2 Done Sent Done false 2793 Acked Read 8 Done Sent Done false 2794 Acked Write 3 Done Sent Done false 2795 Acked Flush 0 Done Sent Done false 2796 Acked Read 3 Done Sent Done false 2797 Acked Read 1 Done Sent Done false 2798 Acked Write 6 Done Sent Done false 2799 Acked Write 9 Done Sent Done false 2800 Acked Read 9 Done Sent Done false 2801 Acked Read 4 Done Sent Done false 2802 Acked Write 5 Done Sent Done false 2803 Acked Write 9 Done Sent Done false 2804 Acked Read 4 Done Sent Done false 2805 Acked Write 1 Done Sent Done false 2806 Acked Write 1 Done Sent Done false 2807 Acked Write 3 Done Sent Done false 2808 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2795 2507 2795 Downstairs last five completed: 2507 2506 2505 2504 2503 Upstairs last five completed: 2808 2807 2806 2805 2804 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1208) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2508 Acked Read 9 Done Sent Done false 2509 Acked Write 8 Done Sent Done false 2510 Acked Read 3 Done Sent Done false 2511 Acked Read 2 Done Sent Done false 2512 Acked Write 9 Done Sent Done false 2513 Acked Write 10 Done Sent Done false 2514 Acked Read 4 Done Sent Done false 2515 Acked Read 5 Done Sent Done false 2516 Acked Write 3 Done Sent Done false 2517 Acked Write 2 Done Sent Done false 2518 Acked Read 8 Done Sent Done false 2519 Acked Flush 0 Done Sent Done false 2520 Acked Read 3 Done Sent Done false 2521 Acked Read 3 Done Sent Done false 2522 Acked Read 9 Done Sent Done false 2523 Acked Read 2 Done Sent Done false 2524 Acked Write 9 Done Sent Done false 2525 Acked Write 10 Done Sent Done false 2526 Acked Write 1 Done Sent Done false 2527 Acked Read 10 Done Sent Done false 2528 Acked Write 1 Done Sent Done false 2529 Acked Write 6 Done Sent Done false 2530 Acked Read 1 Done Sent Done false 2531 Acked Flush 0 Done Sent Done false 2532 Acked Read 2 Done Sent Done false 2533 Acked Read 2 Done Sent Done false 2534 Acked Write 8 Done Sent Done false 2535 Acked Write 10 Done Sent Done false 2536 Acked Write 7 Done Sent Done false 2537 Acked Read 2 Done Sent Done false 2538 Acked Write 10 Done Sent Done false 2539 Acked Read 5 Done Sent Done false 2540 Acked Write 4 Done Sent Done false 2541 Acked Read 8 Done Sent Done false 2542 Acked Read 1 Done Sent Done false 2543 Acked Write 2 Done Sent Done false 2544 Acked Read 8 Done Sent Done false 2545 Acked Write 6 Done Sent Done false 2546 Acked Write 6 Done Sent Done false 2547 Acked Flush 0 Done Sent Done false 2548 Acked Read 8 Done Sent Done false 2549 Acked Write 3 Done Sent Done false 2550 Acked Flush 0 Done Sent Done false 2551 Acked Write 2 Done Sent Done false 2552 Acked Write 9 Done Sent Done false 2553 Acked Read 4 Done Sent Done false 2554 Acked Write 3 Done Sent Done false 2555 Acked Write 5 Done Sent Done false 2556 Acked Read 8 Done Sent Done false 2557 Acked Read 8 Done Sent Done false 2558 Acked Write 3 Done Sent Done false 2559 Acked Write 8 Done Sent Done false 2560 Acked Read 8 Done Sent Done false 2561 Acked Write 7 Done Sent Done false 2562 Acked Read 7 Done Sent Done false 2563 Acked Read 10 Done Sent Done false 2564 Acked Write 3 Done Sent Done false 2565 Acked Read 1 Done Sent Done false 2566 Acked Flush 0 Done Sent Done false 2567 Acked Read 9 Done Sent Done false 2568 Acked Read 9 Done Sent Done false 2569 Acked Write 8 Done Sent Done false 2570 Acked Write 6 Done Sent Done false 2571 Acked Write 8 Done Sent Done false 2572 Acked Write 1 Done Sent Done false 2573 Acked Write 10 Done Sent Done false 2574 Acked Read 1 Done Sent Done false 2575 Acked Read 5 Done Sent Done false 2576 Acked Read 8 Done Sent Done false 2577 Acked Read 3 Done Sent Done false 2578 Acked Read 2 Done Sent Done false 2579 Acked Read 7 Done Sent Done false 2580 Acked Read 10 Done Sent Done false 2581 Acked Read 2 Done Sent Done false 2582 Acked Write 3 Done Sent Done false 2583 Acked Read 1 Done Sent Done false 2584 Acked Write 10 Done Sent Done false 2585 Acked Read 9 Done Sent Done false 2586 Acked Write 10 Done Sent Done false 2587 Acked Write 3 Done Sent Done false 2588 Acked Read 7 Done Sent Done false 2589 Acked Read 1 Done Sent Done false 2590 Acked Read 9 Done Sent Done false 2591 Acked Write 9 Done Sent Done false 2592 Acked Write 6 Done Sent Done false 2593 Acked Read 7 Done Sent Done false 2594 Acked Read 9 Done Sent Done false 2595 Acked Write 5 Done Sent Done false 2596 Acked Flush 0 Done Sent Done false 2597 Acked Write 9 Done Sent Done false 2598 Acked Read 9 Done Sent Done false 2599 Acked Read 1 Done Sent Done false 2600 Acked Flush 0 Done Sent Done false 2601 Acked Flush 0 Done Sent Done false 2602 Acked Flush 0 Done Sent Done false 2603 Acked Read 2 Done Sent Done false 2604 Acked Write 8 Done Sent Done false 2605 Acked Write 6 Done Sent Done false 2606 Acked Write 7 Done Sent Done false 2607 Acked Read 2 Done Sent Done false 2608 Acked Read 10 Done Sent Done false 2609 Acked Flush 0 Done Sent Done false 2610 Acked Write 1 Done Sent Done false 2611 Acked Read 1 Done Sent Done false 2612 Acked Read 9 Done Sent Done false 2613 Acked Write 4 Done Sent Done false 2614 Acked Flush 0 Done Sent Done false 2615 Acked Write 9 Done Sent Done false 2616 Acked Write 8 Done Sent Done false 2617 Acked Read 3 Done Sent Done false 2618 Acked Write 8 Done Sent Done false 2619 Acked Read 4 Done Sent Done false 2620 Acked Read 7 Done Sent Done false 2621 Acked Write 6 Done Sent Done false 2622 Acked Read 4 Done Sent Done false 2623 Acked Read 5 Done Sent Done false 2624 Acked Read 6 Done Sent Done false 2625 Acked Flush 0 Done Sent Done false 2626 Acked Read 2 Done Sent Done false 2627 Acked Read 4 Done Sent Done false 2628 Acked Read 2 Done Sent Done false 2629 Acked Read 5 Done Sent Done false 2630 Acked Write 3 Done Sent Done false 2631 Acked Read 2 Done Sent Done false 2632 Acked Read 8 Done Sent Done false 2633 Acked Read 2 Done Sent Done false 2634 Acked Write 3 Done Sent Done false 2635 Acked Flush 0 Done Sent Done false 2636 Acked Write 3 Done Sent Done false 2637 Acked Write 3 Done Sent Done false 2638 Acked Read 2 Done Sent Done false 2639 Acked Write 6 Done Sent Done false 2640 Acked Write 1 Done Sent Done false 2641 Acked Read 9 Done Sent Done false 2642 Acked Write 10 Done Sent Done false 2643 Acked Flush 0 Done Sent Done false 2644 Acked Write 10 Done Sent Done false 2645 Acked Write 5 Done Sent Done false 2646 Acked Read 9 Done Sent Done false 2647 Acked Read 6 Done Sent Done false 2648 Acked Write 6 Done Sent Done false 2649 Acked Read 7 Done Sent Done false 2650 Acked Write 4 Done Sent Done false 2651 Acked Read 9 Done Sent Done false 2652 Acked Read 5 Done Sent Done false 2653 Acked Write 8 Done Sent Done false 2654 Acked Read 2 Done Sent Done false 2655 Acked Read 7 Done Sent Done false 2656 Acked Read 6 Done Sent Done false 2657 Acked Read 3 Done Sent Done false 2658 Acked Read 4 Done Sent Done false 2659 Acked Read 8 Done Sent Done false 2660 Acked Flush 0 Done Sent Done false 2661 Acked Write 7 Done Sent Done false 2662 Acked Flush 0 Done Sent Done false 2663 Acked Write 7 Done Sent Done false 2664 Acked Flush 0 Done Sent Done false 2665 Acked Read 9 Done Sent Done false 2666 Acked Read 1 Done Sent Done false 2667 Acked Read 4 Done Sent Done false 2668 Acked Read 4 Done Sent Done false 2669 Acked Read 9 Done Sent Done false 2670 Acked Write 7 Done Sent Done false 2671 Acked Read 6 Done Sent Done false 2672 Acked Read 10 Done Sent Done false 2673 Acked Write 4 Done Sent Done false 2674 Acked Read 7 Done Sent Done false 2675 Acked Read 4 Done Sent Done false 2676 Acked Read 6 Done Sent Done false 2677 Acked Read 6 Done Sent Done false 2678 Acked Read 6 Done Sent Done false 2679 Acked Read 6 Done Sent Done false 2680 Acked Read 8 Done Sent Done false 2681 Acked Write 1 Done Sent Done false 2682 Acked Read 1 Done Sent Done false 2683 Acked Read 2 Done Sent Done false 2684 Acked Write 4 Done Sent Done false 2685 Acked Write 5 Done Sent Done false 2686 Acked Read 9 Done Sent Done false 2687 Acked Read 2 Done Sent Done false 2688 Acked Read 7 Done Sent Done false 2689 Acked Read 5 Done Sent Done false 2690 Acked Read 1 Done Sent Done false 2691 Acked Read 10 Done Sent Done false 2692 Acked Read 5 Done Sent Done false 2693 Acked Write 4 Done Sent Done false 2694 Acked Write 8 Done Sent Done false 2695 Acked Flush 0 Done Sent Done false 2696 Acked Read 2 Done Sent Done false 2697 Acked Read 4 Done Sent Done false 2698 Acked Write 1 Done Sent Done false 2699 Acked Read 3 Done Sent Done false 2700 Acked Write 4 Done Sent Done false 2701 Acked Read 10 Done Sent Done false 2702 Acked Read 4 Done Sent Done false 2703 Acked Read 3 Done Sent Done false 2704 Acked Write 3 Done Sent Done false 2705 Acked Write 5 Done Sent Done false 2706 Acked Write 4 Done Sent Done false 2707 Acked Read 5 Done Sent Done false 2708 Acked Read 2 Done Sent Done false 2709 Acked Read 7 Done Sent Done false 2710 Acked Write 1 Done Sent Done false 2711 Acked Write 2 Done Sent Done false 2712 Acked Read 10 Done Sent Done false 2713 Acked Write 3 Done Sent Done false 2714 Acked Write 1 Done Sent Done false 2715 Acked Read 6 Done Sent Done false 2716 Acked Read 4 Done Sent Done false 2717 Acked Write 3 Done Sent Done false 2718 Acked Read 5 Done Sent Done false 2719 Acked Read 9 Done Sent Done false 2720 Acked Flush 0 Done Sent Done false 2721 Acked Read 2 Done Sent Done false 2722 Acked Write 1 Done Sent Done false 2723 Acked Write 10 Done Sent Done false 2724 Acked Read 2 Done Sent Done false 2725 Acked Read 7 Done Sent Done false 2726 Acked Read 4 Done Sent Done false 2727 Acked Read 4 Done Sent Done false 2728 Acked Read 1 Done Sent Done false 2729 Acked Read 2 Done Sent Done false 2730 Acked Read 1 Done Sent Done false 2731 Acked Read 2 Done Sent Done false 2732 Acked Write 3 Done Sent Done false 2733 Acked Read 6 Done Sent Done false 2734 Acked Write 2 Done Sent Done false 2735 Acked Write 2 Done Sent Done false 2736 Acked Read 6 Done Sent Done false 2737 Acked Flush 0 Done Sent Done false 2738 Acked Read 8 Done Sent Done false 2739 Acked Read 4 Done Sent Done false 2740 Acked Write 6 Done Sent Done false 2741 Acked Read 4 Done Sent Done false 2742 Acked Write 7 Done Sent Done false 2743 Acked Write 7 Done Sent Done false 2744 Acked Read 5 Done Sent Done false 2745 Acked Read 9 Done Sent Done false 2746 Acked Read 5 Done Sent Done false 2747 Acked Write 7 Done Sent Done false 2748 Acked Write 10 Done Sent Done false 2749 Acked Read 3 Done Sent Done false 2750 Acked Read 9 Done Sent Done false 2751 Acked Read 3 Done Sent Done false 2752 Acked Read 3 Done Sent Done false 2753 Acked Read 9 Done Sent Done false 2754 Acked Write 1 Done Sent Done false 2755 Acked Read 7 Done Sent Done false 2756 Acked Write 7 Done Sent Done false 2757 Acked Flush 0 Done Sent Done false 2758 Acked Write 5 Done Sent Done false 2759 Acked Flush 0 Done Sent Done false 2760 Acked Write 1 Done Sent Done false 2761 Acked Read 9 Done Sent Done false 2762 Acked Read 3 Done Sent Done false 2763 Acked Read 10 Done Sent Done false 2764 Acked Read 10 Done Sent Done false 2765 Acked Read 6 Done Sent Done false 2766 Acked Write 3 Done Sent Done false 2767 Acked Write 7 Done Sent Done false 2768 Acked Read 3 Done Sent Done false 2769 Acked Write 3 Done Sent Done false 2770 Acked Read 6 Done Sent Done false 2771 Acked Write 3 Done Sent Done false 2772 Acked Read 9 Done Sent Done false 2773 Acked Read 4 Done Sent Done false 2774 Acked Read 4 Done Sent Done false 2775 Acked Write 10 Done Sent Done false 2776 Acked Write 1 Done Sent Done false 2777 Acked Read 6 Done Sent Done false 2778 Acked Flush 0 Done Sent Done false 2779 Acked Read 3 Done Sent Done false 2780 Acked Read 1 Done Sent Done false 2781 Acked Write 6 Done Sent Done false 2782 Acked Read 9 Done Sent Done false 2783 Acked Read 2 Done Sent Done false 2784 Acked Read 6 Done Sent Done false 2785 Acked Write 2 Done Sent Done false 2786 Acked Read 4 Done Sent Done false 2787 Acked Flush 0 Done Sent Done false 2788 Acked Write 4 Done Sent Done false 2789 Acked Flush 0 Done Sent Done false 2790 Acked Write 1 Done Sent Done false 2791 Acked Read 1 Done Sent Done false 2792 Acked Write 2 Done Sent Done false 2793 Acked Read 8 Done Sent Done false 2794 Acked Write 3 Done Sent Done false 2795 Acked Flush 0 Done Sent Done false 2796 Acked Read 3 Done Sent Done false 2797 Acked Read 1 Done Sent Done false 2798 Acked Write 6 Done Sent Done false 2799 Acked Write 9 Done Sent Done false 2800 Acked Read 9 Done Sent Done false 2801 Acked Read 4 Done Sent Done false 2802 Acked Write 5 Done Sent Done false 2803 Acked Write 9 Done Sent Done false 2804 Acked Read 4 Done Sent Done false 2805 Acked Write 1 Done Sent Done false 2806 Acked Write 1 Done Sent Done false 2807 Acked Write 3 Done Sent Done false 2808 Acked Read 4 Done Sent Done false 2809 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2809 2507 2809 Downstairs last five completed: 2507 2506 2505 2504 2503 Upstairs last five completed: 2809 2808 2807 2806 2805 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.917673803Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:46942)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.917816895Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:42:58.918834432Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.91940105Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919425983Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 2507","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919441099Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 2507","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919665119Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919689572Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919704148Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 2507","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919718464Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 2507","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:42:58.919942364Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2809 2808 2807 2806 2805 Upstairs last five completed: 2809 2808 2807 2806 2805 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:43:01.155 INFO request completed[8810] Got stop action so:false kr:false , latency_us: 335, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4bc017f3-4cb6-4239-91cc-3fb6e277f9ec, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:01.156014772Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:01.156087033Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:01.156114765Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:01.1561356Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:01.156157774Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:01.156223856Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:43:01.156 INFO request completed, latency_us: 151, response_code: 200, uri: /state/cid/0, method: GET, req_id: a32939a7-0143-441a-901d-d2dbc113fe81, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:43:05.158 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/0, method: GET, req_id: bffe3a2a-2e14-4320-848b-d73f5ccd24bf, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 4809 len 32768 002/300 Read block 4452 len 32768 003/300 Read block 4339 len 4096 004/300 Write block 1849 len 24576 data: 2 2 2 2 2 2 005/300 Read block 1215 len 16384 006/300 Flush 007/300 Flush 008/300 Read block 2730 len 40960 009/300 Read block 1097 len 16384 010/300 Read block 3509 len 36864 011/300 Write block 4459 len 40960 data: 2 2 2 2 2 2 2 3 3 3 012/300 Write block 2255 len 28672 data: 2 3 3 3 3 3 2 013/300 Write block 3990 len 20480 data: 3 3 3 2 2 014/300 Read block 3678 len 36864 015/300 Read block 978 len 20480 016/300 Read block 1968 len 20480 017/300 Write block 3511 len 8192 data: 2 2 018/300 Flush 019/300 Write block 2051 len 32768 data: 2 2 2 2 2 4 4 4 020/300 Flush 021/300 Read block 1173 len 32768 022/300 Flush 023/300 Write block 3615 len 8192 data: 3 3 024/300 Read block 2224 len 4096 025/300 Read block 773 len 36864 026/300 Write block 4050 len 4096 data: 3 027/300 Read block 3091 len 16384 028/300 Read block 2446 len 16384 029/300 Write block 2005 len 12288 data: 3 2 2 030/300 Read block 3487 len 4096 031/300 Write block 1105 len 40960 data: 2 2 2 2 3 3 3 3 3 3 032/300 Read block 4875 len 40960 033/300 Read block 817 len 4096 034/300 Write block 2850 len 20480 data: 3 4 4 5 5 035/300 Write block 2364 len 12288 data: 3 3 3 036/300 Flush 037/300 Write block 2456 len 24576 data: 3 4 3 3 3 3 038/300 Write block 1871 len 32768 data: 4 3 3 2 2 2 2 2 039/300 Flush 040/300 Read block 4282 len 28672 041/300 Read block 2753 len 16384 042/300 Write block 2471 len 36864 data: 4 2 2 2 3 3 3 4 5 043/300 Write block 3871 len 4096 data: 2 044/300 Write block 707 len 24576 data: 2 2 2 2 3 3 045/300 Flush 046/300 Read block 975 len 28672 047/300 Write block 4314 len 24576 data: 2 2 2 2 2 2 048/300 Write block 3287 len 20480 data: 3 3 3 3 3 049/300 Write block 3558 len 8192 data: 3 3 050/300 Read block 339 len 32768 051/300 Read block 2759 len 16384 052/300 Write block 1527 len 16384 data: 2 2 2 2 053/300 Write block 1032 len 8192 data: 3 4 054/300 Read block 4017 len 28672 055/300 Read block 1774 len 24576 056/300 Read block 4672 len 36864 057/300 Read block 3488 len 40960 058/300 Write block 2918 len 4096 data: 2 059/300 Write block 4768 len 8192 data: 3 2 060/300 Write block 3897 len 16384 data: 3 3 3 3 061/300 Read block 1966 len 4096 062/300 Read block 2076 len 20480 063/300 Read block 3341 len 24576 064/300 Write block 348 len 28672 data: 2 2 2 2 2 2 3 065/300 Read block 1365 len 16384 066/300 Read block 596 len 32768 067/300 Read block 3414 len 24576 068/300 Read block 3565 len 12288 069/300 Read block 3651 len 20480 070/300 Read block 1344 len 24576 071/300 Write block 2347 len 4096 data: 5 072/300 Flush 073/300 Write block 4968 len 16384 data: 3 3 3 2 074/300 Flush 075/300 Read block 1390 len 12288 076/300 Write block 3761 len 20480 data: 4 4 3 3 3 077/300 Write block 4279 len 28672 data: 3 3 2 2 2 2 2 078/300 Read block 4952 len 32768 079/300 Write block 1011 len 28672 data: 3 3 2 2 2 2 2 080/300 Read block 2883 len 28672 081/300 Read block 2392 len 40960 082/300 Read block 435 len 4096 083/300 Write block 1991 len 36864 data: 6 6 5 3 3 4 5 5 5 084/300 Write block 2237 len 40960 data: 2 2 2 2 3 3 3 5 5 5 085/300 Read block 3299 len 28672 086/300 Write block 2154 len 4096 data: 2 087/300 Read block 3178 len 20480 088/300 Write block 2792 len 40960 data: 3 3 3 4 4 4 3 3 3 4 089/300 Read block 2730 len 8192 090/300 Read block 4868 len 28672 091/300 Read block 4261 len 28672 092/300 Read block 4898 len 8192 093/300 Read block 4705 len 8192 094/300 Write block 601 len 4096 data: 2 095/300 Read block 2304 len 8192 096/300 Write block 550 len 8192 data: 2 2 097/300 Read block 3972 len 32768 098/300 Write block 3846 len 4096 data: 2 099/300 Read block 2504 len 28672 100/300 Read block 1714 len 4096 101/300 Write block 1253 len 8192 data: 4 4 102/300 Write block 1604 len 16384 data: 3 2 2 3 103/300 Read block 3637 len 24576 104/300 Read block 2882 len 8192 105/300 Read block 2803 len 40960 106/300 Write block 4433 len 40960 data: 2 2 2 3 3 3 4 4 4 4 107/300 Read block 3607 len 40960 108/300 Read block 4204 len 12288 109/300 Write block 1467 len 24576 data: 3 4 3 3 3 3 110/300 Read block 612 len 4096 111/300 Write block 1157 len 12288 data: 2 2 2 112/300 Read block 1861 len 36864 113/300 Read block 3914 len 36864 114/300 Write block 3757 len 16384 data: 2 2 3 4 115/300 Read block 4325 len 12288 116/300 Write block 2555 len 24576 data: 3 4 4 4 4 5 117/300 Write block 1196 len 28672 data: 3 3 3 3 3 2 2 118/300 Write block 1928 len 28672 data: 3 3 3 2 2 2 2 119/300 Read block 2576 len 32768 120/300 Read block 4750 len 24576 121/300 Read block 4950 len 28672 122/300 Read block 4511 len 8192 123/300 Write block 100 len 12288 data: 2 2 2 124/300 Read block 3086 len 20480 125/300 Write block 917 len 32768 data: 2 2 2 2 2 2 2 2 126/300 Read block 4166 len 28672 127/300 Write block 3995 len 40960 data: 2 2 2 2 2 2 2 2 2 2 128/300 Read block 2573 len 36864 129/300 Read block 2952 len 40960 130/300 Write block 1429 len 8192 data: 2 2 131/300 Write block 3045 len 12288 data: 2 2 2 132/300 Read block 2699 len 8192 133/300 Flush 134/300 Write block 3103 len 28672 data: 3 3 3 3 3 3 3 135/300 Read block 1533 len 4096 136/300 Write block 678 len 16384 data: 2 2 2 2 137/300 Write block 1696 len 24576 data: 3 3 3 3 3 2 138/300 Read block 1100 len 20480 139/300 Read block 1242 len 28672 140/300 Read block 2660 len 16384 141/300 Read block 2221 len 12288 142/300 Write block 4985 len 40960 data: 5 3 3 3 2 2 2 2 2 3 143/300 Read block 1569 len 16384 144/300 Write block 1371 len 32768 data: 4 3 3 2 2 2 2 2 145/300 Write block 3541 len 36864 data: 3 3 3 3 3 2 2 2 3 146/300 Write block 3589 len 20480 data: 2 2 2 2 2 147/300 Read block 2407 len 28672 148/300 Read block 4485 len 20480 149/300 Read block 1040 len 36864 150/300 Read block 4844 len 32768 151/300 Write block 3249 len 36864 data: 2 2 2 2 2 2 2 2 3 152/300 Read block 4578 len 20480 153/300 Write block 1525 len 24576 data: 3 2 3 3 3 3 154/300 Write block 4395 len 32768 data: 2 2 2 2 3 3 3 3 155/300 Write block 1232 len 12288 data: 2 2 2 156/300 Read block 1548 len 28672 157/300 Read block 2506 len 32768 158/300 Write block 4602 len 36864 data: 2 2 2 2 2 3 2 3 3 159/300 Read block 4360 len 12288 160/300 Read block 1889 len 24576 161/300 Read block 4778 len 20480 162/300 Read block 4593 len 20480 163/300 Flush 164/300 Read block 2795 len 4096 165/300 Write block 2289 len 12288 data: 3 2 2 166/300 Flush 167/300 Read block 103 len 8192 168/300 Write block 2726 len 32768 data: 2 2 2 2 2 2 2 2 169/300 Write block 2878 len 20480 data: 2 2 2 2 2 170/300 Read block 4664 len 8192 171/300 Write block 2695 len 16384 data: 3 3 2 2 172/300 Read block 3850 len 24576 173/300 Write block 1890 len 40960 data: 4 4 4 4 4 2 2 2 2 3 174/300 Read block 2234 len 4096 175/300 Read block 4033 len 28672 176/300 Write block 4022 len 4096 data: 2 177/300 Write block 4049 len 36864 data: 3 4 3 3 3 2 2 2 2 178/300 Write block 2208 len 12288 data: 2 2 2 179/300 Write block 365 len 32768 data: 2 2 2 2 2 2 2 2 180/300 Read block 4729 len 32768 181/300 Write block 1785 len 40960 data: 2 2 2 2 2 2 2 3 3 3 182/300 Read block 4350 len 36864 183/300 Write block 2377 len 20480 data: 4 5 5 5 3 184/300 Write block 484 len 40960 data: 3 2 2 2 2 3 3 4 4 3 185/300 Write block 622 len 4096 data: 2 186/300 Read block 4282 len 20480 187/300 Flush 188/300 Write block 3107 len 24576 data: 4 4 4 3 3 3 189/300 Read block 3844 len 28672 190/300 Write block 1551 len 36864 data: 2 2 2 2 2 2 3 3 4 191/300 Read block 355 len 36864 192/300 Write block 3187 len 12288 data: 3 3 3 193/300 Write block 3208 len 24576 data: 4 4 3 3 2 3 194/300 Read block 2599 len 24576 195/300 Flush 196/300 Flush 197/300 Read block 3064 len 24576 198/300 Read block 2342 len 4096 199/300 Read block 1640 len 36864 200/300 Write block 2156 len 40960 data: 3 3 3 3 3 3 3 3 3 2 201/300 Read block 785 len 36864 202/300 Write block 1419 len 36864 data: 2 2 2 2 3 3 3 3 3 203/300 Read block 2324 len 32768 204/300 Read block 1105 len 4096 205/300 Read block 1643 len 40960 206/300 Write block 3013 len 16384 data: 4 4 4 4 207/300 Write block 1408 len 8192 data: 4 3 208/300 Read block 1737 len 12288 209/300 Read block 4818 len 4096 210/300 Read block 419 len 24576 211/300 Read block 2052 len 8192 212/300 Read block 2893 len 24576 213/300 Read block 4997 len 12288 214/300 Read block 4388 len 16384 215/300 Read block 1571 len 32768 216/300 Read block 3526 len 28672 217/300 Read block 2030 len 28672 218/300 Read block 4645 len 40960 219/300 Read block 3592 len 32768 220/300 Flush 221/300 Read block 519 len 36864 222/300 Read block 2021 len 4096 223/300 Write block 1587 len 16384 data: 2 2 2 2 224/300 Write block 476 len 12288 data: 2 2 2 225/300 Read block 4228 len 28672 226/300 Write block 3095 len 32768 data: 3 2 2 2 2 2 2 3 227/300 Read block 1657 len 28672 228/300 Read block 3944 len 12288 229/300 Read block 4636 len 16384 230/300 Write block 3807 len 24576 data: 2 2 2 2 3 3 231/300 Read block 1156 len 12288 232/300 Write block 939 len 40960 data: 3 3 3 3 2 2 2 3 3 3 233/300 Read block 3490 len 32768 234/300 Write block 862 len 8192 data: 3 3 235/300 Write block 682 len 36864 data: 2 2 2 2 2 2 2 2 2 236/300 Read block 1151 len 8192 237/300 Read block 4475 len 32768 238/300 Read block 3855 len 4096 239/300 Read block 426 len 12288 240/300 Write block 824 len 36864 data: 4 4 4 3 3 3 3 2 2 241/300 Write block 4659 len 4096 data: 4 242/300 Write block 2577 len 16384 data: 2 2 2 2 243/300 Write block 825 len 40960 data: 5 5 4 4 4 4 3 3 2 2 244/300 Write block 1200 len 32768 data: 4 3 3 2 2 2 2 2 245/300 Write block 1522 len 12288 data: 2 2 2 246/300 Read block 2515 len 36864 247/300 Write block 4300 len 12288 data: 2 2 2 248/300 Flush 249/300 Write block 1586 len 12288 data: 2 3 3 250/300 Write block 485 len 4096 data: 3 251/300 Read block 4721 len 36864 252/300 Read block 3599 len 32768 253/300 Read block 1817 len 8192 254/300 Read block 536 len 28672 255/300 Read block 2379 len 8192 256/300 Read block 3627 len 4096 257/300 Read block 4027 len 20480 258/300 Read block 1201 len 16384 259/300 Write block 3471 len 16384 data: 3 2 2 2 260/300 Write block 524 len 12288 data: 3 4 3 261/300 Write block 4237 len 8192 data: 2 2 262/300 Read block 8 len 40960 263/300 Write block 1168 len 28672 data: 3 3 4 5 4 4 3 264/300 Flush 265/300 Write block 917 len 32768 data: 3 3 3 3 3 3 3 3 266/300 Read block 3262 len 36864 267/300 Write block 2088 len 36864 data: 2 2 2 2 2 2 2 2 2 268/300 Write block 1801 len 12288 data: 6 5 5 269/300 Write block 4979 len 4096 data: 4 270/300 Write block 4563 len 32768 data: 3 4 4 3 3 3 3 3 271/300 Read block 745 len 12288 272/300 Write block 1466 len 28672 data: 2 4 5 4 4 4 4 273/300 Write block 3273 len 40960 data: 2 2 3 3 3 2 2 2 2 2 274/300 Read block 4549 len 28672 275/300 Flush 276/300 Read block 3315 len 32768 277/300 Write block 15 len 20480 data: 3 3 3 3 3 278/300 Read block 1658 len 20480 279/300 Read block 2810 len 4096 280/300 Write block 1769 len 4096 data: 2 281/300 Read block 1855 len 20480 282/300 Read block 4780 len 24576 283/300 Read block 839 len 40960 284/300 Flush 285/300 Read block 2410 len 36864 286/300 Write block 2771 len 40960 data: 3 3 3 2 3 2 2 3 3 2 287/300 Write block 2745 len 12288 data: 3 3 2 288/300 Flush 289/300 Read block 4590 len 8192 290/300 Write block 2740 len 4096 data: 4 291/300 Write block 4581 len 12288 data: 3 3 4 292/300 Read block 2241 len 16384 293/300 Write block 3805 len 20480 data: 2 2 3 3 3 294/300 Flush 295/300 Write block 294 len 20480 data: 2 2 2 2 2 296/300 Read block 866 len 24576 297/300 Read block 2194 len 4096 298/300 Write block 2542 len 28672 data: 5 5 5 4 3 4 3 299/300 Write block 2358 len 24576 data: 2 2 2 3 4 3 300/300 Read block 2778 len 32768 301/300 Read block 2361 len 40960 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:43:05.393 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/0, method: POST, req_id: d199a11c-f83d-4cd9-9f65-850b09ec19be, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2810 Acked Read 8 Sent Done Done false 2811 Acked Read 8 Sent Done Done false 2812 Acked Read 1 Sent Done Done false 2813 Acked Write 6 Sent Done Done false 2814 Acked Read 4 Sent Done Done false 2815 Acked Flush 0 Sent Done Done false 2816 Acked Flush 0 Sent Done Done false 2817 Acked Read 10 Sent Done Done false 2818 Acked Read 4 Sent Done Done false 2819 Acked Read 9 Sent Done Done false 2820 Acked Write 10 Sent Done Done false 2821 Acked Write 7 Sent Done Done false 2822 Acked Write 5 Sent Done Done false 2823 Acked Read 9 Sent Done Done false 2824 Acked Read 5 Sent Done Done false 2825 Acked Read 5 Sent Done Done false 2826 Acked Write 2 Sent Done Done false 2827 Acked Flush 0 Sent Done Done false 2828 Acked Write 8 Sent Done Done false 2829 Acked Flush 0 Sent Done Done false 2830 Acked Read 8 Sent Done Done false 2831 Acked Flush 0 Sent Done Done false 2832 Acked Write 2 Sent Done Done false 2833 Acked Read 1 Sent Done Done false 2834 Acked Read 9 Sent Done Done false 2835 Acked Write 1 Sent Done Done false 2836 Acked Read 4 Sent Done Done false 2837 Acked Read 4 Sent Done Done false 2838 Acked Write 3 Sent Done Done false 2839 Acked Read 1 Sent Done Done false 2840 Acked Write 10 Sent Done Done false 2841 Acked Read 10 Sent Done Done false 2842 Acked Read 1 Sent Done Done false 2843 Acked Write 5 Sent Done Done false 2844 Acked Write 3 Sent Done Done false 2845 Acked Flush 0 Sent Done Done false 2846 Acked Write 6 Sent Done Done false 2847 Acked Write 8 Sent Done Done false 2848 Acked Flush 0 Sent Done Done false 2849 Acked Read 7 Sent Done Done false 2850 Acked Read 4 Sent Done Done false 2851 Acked Write 9 Sent Done Done false 2852 Acked Write 1 Sent Done Done false 2853 Acked Write 6 Sent Done Done false 2854 Acked Flush 0 Sent Done Done false 2855 Acked Read 7 Sent Done Done false 2856 Acked Write 6 Sent Done Done false 2857 Acked Write 5 Sent Done Done false 2858 Acked Write 2 Sent Done Done false 2859 Acked Read 8 Sent Done Done false 2860 Acked Read 4 Sent Done Done false 2861 Acked Write 4 Sent Done Done false 2862 Acked Write 2 Sent Done Done false 2863 Acked Read 7 Sent Done Done false 2864 Acked Read 6 Sent Done Done false 2865 Acked Read 9 Sent Done Done false 2866 Acked Read 10 Sent Done Done false 2867 Acked Write 1 Sent Done Done false 2868 Acked Write 2 Sent Done Done false 2869 Acked Write 4 Sent Done Done false 2870 Acked Read 1 Sent Done Done false 2871 Acked Read 5 Sent Done Done false 2872 Acked Read 6 Sent Done Done false 2873 Acked Write 7 Sent Done Done false 2874 Acked Read 4 Sent Done Done false 2875 Acked Read 8 Sent Done Done false 2876 Acked Read 6 Sent Done Done false 2877 Acked Read 3 Sent Done Done false 2878 Acked Read 5 Sent Done Done false 2879 Acked Read 6 Sent Done Done false 2880 Acked Write 1 Sent Done Done false 2881 Acked Flush 0 Sent Done Done false 2882 Acked Write 4 Sent Done Done false 2883 Acked Flush 0 Sent Done Done false 2884 Acked Read 3 Sent Done Done false 2885 Acked Write 5 Sent Done Done false 2886 Acked Write 7 Sent Done Done false 2887 Acked Read 8 Sent Done Done false 2888 Acked Write 7 Sent Done Done false 2889 Acked Read 7 Sent Done Done false 2890 Acked Read 10 Sent Done Done false 2891 Acked Read 1 Sent Done Done false 2892 Acked Write 9 Sent Done Done false 2893 Acked Write 10 Sent Done Done false 2894 Acked Read 7 Sent Done Done false 2895 Acked Write 1 Sent Done Done false 2896 Acked Read 5 Sent Done Done false 2897 Acked Write 10 Sent Done Done false 2898 Acked Read 2 Sent Done Done false 2899 Acked Read 7 Sent Done Done false 2900 Acked Read 7 Sent Done Done false 2901 Acked Read 2 Sent Done Done false 2902 Acked Read 2 Sent Done Done false 2903 Acked Write 1 Sent Done Done false 2904 Acked Read 2 Sent Done Done false 2905 Acked Write 2 Sent Done Done false 2906 Acked Read 8 Sent Done Done false 2907 Acked Write 1 Sent Done Done false 2908 Acked Read 7 Sent Done Done false 2909 Acked Read 1 Sent Done Done false 2910 Acked Write 2 Sent Done Done false 2911 Acked Write 4 Sent Done Done false 2912 Acked Read 6 Sent Done Done false 2913 Acked Read 2 Sent Done Done false 2914 Acked Read 10 Sent Done Done false 2915 Acked Write 10 Sent Done Done false 2916 Acked Read 10 Sent Done Done false 2917 Acked Read 3 Sent Done Done false 2918 Acked Write 6 Sent Done Done false 2919 Acked Read 1 Sent Done Done false 2920 Acked Write 3 Sent Done Done false 2921 Acked Read 9 Sent Done Done false 2922 Acked Read 9 Sent Done Done false 2923 Acked Write 4 Sent Done Done false 2924 Acked Read 3 Sent Done Done false 2925 Acked Write 6 Sent Done Done false 2926 Acked Write 7 Sent Done Done false 2927 Acked Write 7 Sent Done Done false 2928 Acked Read 8 Sent Done Done false 2929 Acked Read 6 Sent Done Done false 2930 Acked Read 7 Sent Done Done false 2931 Acked Read 2 Sent Done Done false 2932 Acked Write 3 Sent Done Done false 2933 Acked Read 5 Sent Done Done false 2934 Acked Write 8 Sent Done Done false 2935 Acked Read 7 Sent Done Done false 2936 Acked Write 10 Sent Done Done false 2937 Acked Read 9 Sent Done Done false 2938 Acked Read 10 Sent Done Done false 2939 Acked Write 2 Sent Done Done false 2940 Acked Write 3 Sent Done Done false 2941 Acked Read 2 Sent Done Done false 2942 Acked Flush 0 Sent Done Done false 2943 Acked Write 7 Sent Done Done false 2944 Acked Read 1 Sent Done Done false 2945 Acked Write 4 Sent Done Done false 2946 Acked Write 6 Sent Done Done false 2947 Acked Read 5 Sent Done Done false 2948 Acked Read 7 Sent Done Done false 2949 Acked Read 4 Sent Done Done false 2950 Acked Read 3 Sent Done Done false 2951 Acked Write 10 Sent Done Done false 2952 Acked Read 4 Sent Done Done false 2953 Acked Write 8 Sent Done Done false 2954 Acked Write 9 Sent Done Done false 2955 Acked Write 5 Sent Done Done false 2956 Acked Read 7 Sent Done Done false 2957 Acked Read 5 Sent Done Done false 2958 Acked Read 9 Sent Done Done false 2959 Acked Read 8 Sent Done Done false 2960 Acked Write 9 Sent Done Done false 2961 Acked Read 5 Sent Done Done false 2962 Acked Write 6 Sent Done Done false 2963 Acked Write 8 Sent Done Done false 2964 Acked Write 3 Sent Done Done false 2965 Acked Read 7 Sent Done Done false 2966 Acked Read 8 Sent Done Done false 2967 Acked Write 9 Sent Done Done false 2968 Acked Read 3 Sent Done Done false 2969 Acked Read 6 Sent Done Done false 2970 Acked Read 5 Sent Done Done false 2971 Acked Read 5 Sent Done Done false 2972 Acked Flush 0 Sent Done Done false 2973 Acked Read 1 Sent Done Done false 2974 Acked Write 3 Sent Done Done false 2975 Acked Flush 0 Sent Done Done false 2976 Acked Read 2 Sent Done Done false 2977 Acked Write 8 Sent Done Done false 2978 Acked Write 5 Sent Done Done false 2979 Acked Read 2 Sent Done Done false 2980 Acked Write 4 Sent Done Done false 2981 Acked Read 6 Sent Done Done false 2982 Acked Write 10 Sent Done Done false 2983 Acked Read 1 Sent Done Done false 2984 Acked Read 7 Sent Done Done false 2985 Acked Write 1 Sent Done Done false 2986 Acked Write 9 Sent Done Done false 2987 Acked Write 3 Sent Done Done false 2988 Acked Write 8 Sent Done Done false 2989 Acked Read 8 Sent Done Done false 2990 Acked Write 10 Sent Done Done false 2991 Acked Read 9 Sent Done Done false 2992 Acked Write 5 Sent Done Done false 2993 Acked Write 10 Sent Done Done false 2994 Acked Write 1 Sent Done Done false 2995 Acked Read 5 Sent Done Done false 2996 Acked Flush 0 Sent Done Done false 2997 Acked Write 6 Sent Done Done false 2998 Acked Read 7 Sent Done Done false 2999 Acked Write 9 Sent Done Done false 3000 Acked Read 9 Sent Done Done false 3001 Acked Write 3 Sent Done Done false 3002 Acked Write 6 Sent Done Done false 3003 Acked Read 6 Sent Done Done false 3004 Acked Flush 0 Sent Done Done false 3005 Acked Flush 0 Sent Done Done false 3006 Acked Read 6 Sent Done Done false 3007 Acked Read 1 Sent Done Done false 3008 Acked Read 9 Sent Done Done false 3009 Acked Write 10 Sent Done Done false 3010 Acked Read 9 Sent Done Done false 3011 Acked Write 9 Sent Done Done false 3012 Acked Read 8 Sent Done Done false 3013 Acked Read 1 Sent Done Done false 3014 Acked Read 10 Sent Done Done false 3015 Acked Write 4 Sent Done Done false 3016 Acked Write 2 Sent Done Done false 3017 Acked Read 3 Sent Done Done false 3018 Acked Read 1 Sent Done Done false 3019 Acked Read 6 Sent Done Done false 3020 Acked Read 2 Sent Done Done false 3021 Acked Read 6 Sent Done Done false 3022 Acked Read 3 Sent Done Done false 3023 Acked Read 4 Sent Done Done false 3024 Acked Read 8 Sent Done Done false 3025 Acked Read 7 Sent Done Done false 3026 Acked Read 7 Sent Done Done false 3027 Acked Read 10 Sent Done Done false 3028 Acked Read 8 Sent Done Done false 3029 Acked Flush 0 Sent Done Done false 3030 Acked Read 9 Sent Done Done false 3031 Acked Read 1 Sent Done Done false 3032 Acked Write 4 Sent Done Done false 3033 Acked Write 3 Sent Done Done false 3034 Acked Read 7 Sent Done Done false 3035 Acked Write 8 Sent Done Done false 3036 Acked Read 7 Sent Done Done false 3037 Acked Read 3 Sent Done Done false 3038 Acked Read 4 Sent Done Done false 3039 Acked Write 6 Sent Done Done false 3040 Acked Read 3 Sent Done Done false 3041 Acked Write 10 Sent Done Done false 3042 Acked Read 8 Sent Done Done false 3043 Acked Write 2 Sent Done Done false 3044 Acked Write 9 Sent Done Done false 3045 Acked Read 2 Sent Done Done false 3046 Acked Read 8 Sent Done Done false 3047 Acked Read 1 Sent Done Done false 3048 Acked Read 3 Sent Done Done false 3049 Acked Write 9 Sent Done Done false 3050 Acked Write 1 Sent Done Done false 3051 Acked Write 4 Sent Done Done false 3052 Acked Write 10 Sent Done Done false 3053 Acked Write 8 Sent Done Done false 3054 Acked Write 3 Sent Done Done false 3055 Acked Read 9 Sent Done Done false 3056 Acked Write 3 Sent Done Done false 3057 Acked Flush 0 Sent Done Done false 3058 Acked Write 3 Sent Done Done false 3059 Acked Write 1 Sent Done Done false 3060 Acked Read 9 Sent Done Done false 3061 Acked Read 8 Sent Done Done false 3062 Acked Read 2 Sent Done Done false 3063 Acked Read 7 Sent Done Done false 3064 Acked Read 2 Sent Done Done false 3065 Acked Read 1 Sent Done Done false 3066 Acked Read 5 Sent Done Done false 3067 Acked Read 4 Sent Done Done false 3068 Acked Write 4 Sent Done Done false 3069 Acked Write 3 Sent Done Done false 3070 Acked Write 2 Sent Done Done false 3071 Acked Read 10 Sent Done Done false 3072 Acked Write 7 Sent Done Done false 3073 Acked Flush 0 Sent Done Done false 3074 Acked Write 8 Sent Done Done false 3075 Acked Read 9 Sent Done Done false 3076 Acked Write 9 Sent Done Done false 3077 Acked Write 3 Sent Done Done false 3078 Acked Write 1 Sent Done Done false 3079 Acked Write 8 Sent Done Done false 3080 Acked Read 3 Sent Done Done false 3081 Acked Write 7 Sent Done Done false 3082 Acked Write 10 Sent Done Done false 3083 Acked Read 7 Sent Done Done false 3084 Acked Flush 0 Sent Done Done false 3085 Acked Read 8 Sent Done Done false 3086 Acked Write 5 Sent Done Done false 3087 Acked Read 5 Sent Done Done false 3088 Acked Read 1 Sent Done Done false 3089 Acked Write 1 Sent Done Done false 3090 Acked Read 5 Sent Done Done false 3091 Acked Read 6 Sent Done Done false 3092 Acked Read 10 Sent Done Done false 3093 Acked Flush 0 Sent Done Done false 3094 Acked Read 9 Sent Done Done false 3095 Acked Write 10 Sent Done Done false 3096 Acked Write 3 Sent Done Done false 3097 Acked Flush 0 Sent Done Done false 3098 Acked Read 2 Sent Done Done false 3099 Acked Write 1 Sent Done Done false 3100 Acked Write 3 Sent Done Done false 3101 Acked Read 4 Sent Done Done false 3102 Acked Write 5 Sent Done Done false 3103 Acked Flush 0 Sent Done Done false 3104 Acked Write 5 Sent Done Done false 3105 Acked Read 6 Sent Done Done false 3106 Acked Read 1 Sent Done Done false 3107 Acked Write 7 Sent Done Done false 3108 Acked Write 6 Sent Done Done false 3109 Acked Read 8 Sent Done Done false 3110 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2809 3103 3103 Downstairs last five completed: 2809 2808 2807 2806 2805 Upstairs last five completed: 3110 3109 3108 3107 3106 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2810 Acked Read 8 Sent Done Done false 2811 Acked Read 8 Sent Done Done false 2812 Acked Read 1 Sent Done Done false 2813 Acked Write 6 Sent Done Done false 2814 Acked Read 4 Sent Done Done false 2815 Acked Flush 0 Sent Done Done false 2816 Acked Flush 0 Sent Done Done false 2817 Acked Read 10 Sent Done Done false 2818 Acked Read 4 Sent Done Done false 2819 Acked Read 9 Sent Done Done false 2820 Acked Write 10 Sent Done Done false 2821 Acked Write 7 Sent Done Done false 2822 Acked Write 5 Sent Done Done false 2823 Acked Read 9 Sent Done Done false 2824 Acked Read 5 Sent Done Done false 2825 Acked Read 5 Sent Done Done false 2826 Acked Write 2 Sent Done Done false 2827 Acked Flush 0 Sent Done Done false 2828 Acked Write 8 Sent Done Done false 2829 Acked Flush 0 Sent Done Done false 2830 Acked Read 8 Sent Done Done false 2831 Acked Flush 0 Sent Done Done false 2832 Acked Write 2 Sent Done Done false 2833 Acked Read 1 Sent Done Done false 2834 Acked Read 9 Sent Done Done false 2835 Acked Write 1 Sent Done Done false 2836 Acked Read 4 Sent Done Done false 2837 Acked Read 4 Sent Done Done false 2838 Acked Write 3 Sent Done Done false 2839 Acked Read 1 Sent Done Done false 2840 Acked Write 10 Sent Done Done false 2841 Acked Read 10 Sent Done Done false 2842 Acked Read 1 Sent Done Done false 2843 Acked Write 5 Sent Done Done false 2844 Acked Write 3 Sent Done Done false 2845 Acked Flush 0 Sent Done Done false 2846 Acked Write 6 Sent Done Done false 2847 Acked Write 8 Sent Done Done false 2848 Acked Flush 0 Sent Done Done false 2849 Acked Read 7 Sent Done Done false 2850 Acked Read 4 Sent Done Done false 2851 Acked Write 9 Sent Done Done false 2852 Acked Write 1 Sent Done Done false 2853 Acked Write 6 Sent Done Done false 2854 Acked Flush 0 Sent Done Done false 2855 Acked Read 7 Sent Done Done false 2856 Acked Write 6 Sent Done Done false 2857 Acked Write 5 Sent Done Done false 2858 Acked Write 2 Sent Done Done false 2859 Acked Read 8 Sent Done Done false 2860 Acked Read 4 Sent Done Done false 2861 Acked Write 4 Sent Done Done false 2862 Acked Write 2 Sent Done Done false 2863 Acked Read 7 Sent Done Done false 2864 Acked Read 6 Sent Done Done false 2865 Acked Read 9 Sent Done Done false 2866 Acked Read 10 Sent Done Done false 2867 Acked Write 1 Sent Done Done false 2868 Acked Write 2 Sent Done Done false 2869 Acked Write 4 Sent Done Done false 2870 Acked Read 1 Sent Done Done false 2871 Acked Read 5 Sent Done Done false 2872 Acked Read 6 Sent Done Done false 2873 Acked Write 7 Sent Done Done false 2874 Acked Read 4 Sent Done Done false 2875 Acked Read 8 Sent Done Done false 2876 Acked Read 6 Sent Done Done false 2877 Acked Read 3 Sent Done Done false 2878 Acked Read 5 Sent Done Done false 2879 Acked Read 6 Sent Done Done false 2880 Acked Write 1 Sent Done Done false 2881 Acked Flush 0 Sent Done Done false 2882 Acked Write 4 Sent Done Done false 2883 Acked Flush 0 Sent Done Done false 2884 Acked Read 3 Sent Done Done false 2885 Acked Write 5 Sent Done Done false 2886 Acked Write 7 Sent Done Done false 2887 Acked Read 8 Sent Done Done false 2888 Acked Write 7 Sent Done Done false 2889 Acked Read 7 Sent Done Done false 2890 Acked Read 10 Sent Done Done false 2891 Acked Read 1 Sent Done Done false 2892 Acked Write 9 Sent Done Done false 2893 Acked Write 10 Sent Done Done false 2894 Acked Read 7 Sent Done Done false 2895 Acked Write 1 Sent Done Done false 2896 Acked Read 5 Sent Done Done false 2897 Acked Write 10 Sent Done Done false 2898 Acked Read 2 Sent Done Done false 2899 Acked Read 7 Sent Done Done false 2900 Acked Read 7 Sent Done Done false 2901 Acked Read 2 Sent Done Done false 2902 Acked Read 2 Sent Done Done false 2903 Acked Write 1 Sent Done Done false 2904 Acked Read 2 Sent Done Done false 2905 Acked Write 2 Sent Done Done false 2906 Acked Read 8 Sent Done Done false 2907 Acked Write 1 Sent Done Done false 2908 Acked Read 7 Sent Done Done false 2909 Acked Read 1 Sent Done Done false 2910 Acked Write 2 Sent Done Done false 2911 Acked Write 4 Sent Done Done false 2912 Acked Read 6 Sent Done Done false 2913 Acked Read 2 Sent Done Done false 2914 Acked Read 10 Sent Done Done false 2915 Acked Write 10 Sent Done Done false 2916 Acked Read 10 Sent Done Done false 2917 Acked Read 3 Sent Done Done false 2918 Acked Write 6 Sent Done Done false 2919 Acked Read 1 Sent Done Done false 2920 Acked Write 3 Sent Done Done false 2921 Acked Read 9 Sent Done Done false 2922 Acked Read 9 Sent Done Done false 2923 Acked Write 4 Sent Done Done false 2924 Acked Read 3 Sent Done Done false 2925 Acked Write 6 Sent Done Done false 2926 Acked Write 7 Sent Done Done false 2927 Acked Write 7 Sent Done Done false 2928 Acked Read 8 Sent Done Done false 2929 Acked Read 6 Sent Done Done false 2930 Acked Read 7 Sent Done Done false 2931 Acked Read 2 Sent Done Done false 2932 Acked Write 3 Sent Done Done false 2933 Acked Read 5 Sent Done Done false 2934 Acked Write 8 Sent Done Done false 2935 Acked Read 7 Sent Done Done false 2936 Acked Write 10 Sent Done Done false 2937 Acked Read 9 Sent Done Done false 2938 Acked Read 10 Sent Done Done false 2939 Acked Write 2 Sent Done Done false 2940 Acked Write 3 Sent Done Done false 2941 Acked Read 2 Sent Done Done false 2942 Acked Flush 0 Sent Done Done false 2943 Acked Write 7 Sent Done Done false 2944 Acked Read 1 Sent Done Done false 2945 Acked Write 4 Sent Done Done false 2946 Acked Write 6 Sent Done Done false 2947 Acked Read 5 Sent Done Done false 2948 Acked Read 7 Sent Done Done false 2949 Acked Read 4 Sent Done Done false 2950 Acked Read 3 Sent Done Done false 2951 Acked Write 10 Sent Done Done false 2952 Acked Read 4 Sent Done Done false 2953 Acked Write 8 Sent Done Done false 2954 Acked Write 9 Sent Done Done false 2955 Acked Write 5 Sent Done Done false 2956 Acked Read 7 Sent Done Done false 2957 Acked Read 5 Sent Done Done false 2958 Acked Read 9 Sent Done Done false 2959 Acked Read 8 Sent Done Done false 2960 Acked Write 9 Sent Done Done false 2961 Acked Read 5 Sent Done Done false 2962 Acked Write 6 Sent Done Done false 2963 Acked Write 8 Sent Done Done false 2964 Acked Write 3 Sent Done Done false 2965 Acked Read 7 Sent Done Done false 2966 Acked Read 8 Sent Done Done false 2967 Acked Write 9 Sent Done Done false 2968 Acked Read 3 Sent Done Done false 2969 Acked Read 6 Sent Done Done false 2970 Acked Read 5 Sent Done Done false 2971 Acked Read 5 Sent Done Done false 2972 Acked Flush 0 Sent Done Done false 2973 Acked Read 1 Sent Done Done false 2974 Acked Write 3 Sent Done Done false 2975 Acked Flush 0 Sent Done Done false 2976 Acked Read 2 Sent Done Done false 2977 Acked Write 8 Sent Done Done false 2978 Acked Write 5 Sent Done Done false 2979 Acked Read 2 Sent Done Done false 2980 Acked Write 4 Sent Done Done false 2981 Acked Read 6 Sent Done Done false 2982 Acked Write 10 Sent Done Done false 2983 Acked Read 1 Sent Done Done false 2984 Acked Read 7 Sent Done Done false 2985 Acked Write 1 Sent Done Done false 2986 Acked Write 9 Sent Done Done false 2987 Acked Write 3 Sent Done Done false 2988 Acked Write 8 Sent Done Done false 2989 Acked Read 8 Sent Done Done false 2990 Acked Write 10 Sent Done Done false 2991 Acked Read 9 Sent Done Done false 2992 Acked Write 5 Sent Done Done false 2993 Acked Write 10 Sent Done Done false 2994 Acked Write 1 Sent Done Done false 2995 Acked Read 5 Sent Done Done false 2996 Acked Flush 0 Sent Done Done false 2997 Acked Write 6 Sent Done Done false 2998 Acked Read 7 Sent Done Done false 2999 Acked Write 9 Sent Done Done false 3000 Acked Read 9 Sent Done Done false 3001 Acked Write 3 Sent Done Done false 3002 Acked Write 6 Sent Done Done false 3003 Acked Read 6 Sent Done Done false 3004 Acked Flush 0 Sent Done Done false 3005 Acked Flush 0 Sent Done Done false 3006 Acked Read 6 Sent Done Done false 3007 Acked Read 1 Sent Done Done false 3008 Acked Read 9 Sent Done Done false 3009 Acked Write 10 Sent Done Done false 3010 Acked Read 9 Sent Done Done false 3011 Acked Write 9 Sent Done Done false 3012 Acked Read 8 Sent Done Done false 3013 Acked Read 1 Sent Done Done false 3014 Acked Read 10 Sent Done Done false 3015 Acked Write 4 Sent Done Done false 3016 Acked Write 2 Sent Done Done false 3017 Acked Read 3 Sent Done Done false 3018 Acked Read 1 Sent Done Done false 3019 Acked Read 6 Sent Done Done false 3020 Acked Read 2 Sent Done Done false 3021 Acked Read 6 Sent Done Done false 3022 Acked Read 3 Sent Done Done false 3023 Acked Read 4 Sent Done Done false 3024 Acked Read 8 Sent Done Done false 3025 Acked Read 7 Sent Done Done false 3026 Acked Read 7 Sent Done Done false 3027 Acked Read 10 Sent Done Done false 3028 Acked Read 8 Sent Done Done false 3029 Acked Flush 0 Sent Done Done false 3030 Acked Read 9 Sent Done Done false 3031 Acked Read 1 Sent Done Done false 3032 Acked Write 4 Sent Done Done false 3033 Acked Write 3 Sent Done Done false 3034 Acked Read 7 Sent Done Done false 3035 Acked Write 8 Sent Done Done false 3036 Acked Read 7 Sent Done Done false 3037 Acked Read 3 Sent Done Done false 3038 Acked Read 4 Sent Done Done false 3039 Acked Write 6 Sent Done Done false 3040 Acked Read 3 Sent Done Done false 3041 Acked Write 10 Sent Done Done false 3042 Acked Read 8 Sent Done Done false 3043 Acked Write 2 Sent Done Done false 3044 Acked Write 9 Sent Done Done false 3045 Acked Read 2 Sent Done Done false 3046 Acked Read 8 Sent Done Done false 3047 Acked Read 1 Sent Done Done false 3048 Acked Read 3 Sent Done Done false 3049 Acked Write 9 Sent Done Done false 3050 Acked Write 1 Sent Done Done false 3051 Acked Write 4 Sent Done Done false 3052 Acked Write 10 Sent Done Done false 3053 Acked Write 8 Sent Done Done false 3054 Acked Write 3 Sent Done Done false 3055 Acked Read 9 Sent Done Done false 3056 Acked Write 3 Sent Done Done false 3057 Acked Flush 0 Sent Done Done false 3058 Acked Write 3 Sent Done Done false 3059 Acked Write 1 Sent Done Done false 3060 Acked Read 9 Sent Done Done false 3061 Acked Read 8 Sent Done Done false 3062 Acked Read 2 Sent Done Done false 3063 Acked Read 7 Sent Done Done false 3064 Acked Read 2 Sent Done Done false 3065 Acked Read 1 Sent Done Done false 3066 Acked Read 5 Sent Done Done false 3067 Acked Read 4 Sent Done Done false 3068 Acked Write 4 Sent Done Done false 3069 Acked Write 3 Sent Done Done false 3070 Acked Write 2 Sent Done Done false 3071 Acked Read 10 Sent Done Done false 3072 Acked Write 7 Sent Done Done false 3073 Acked Flush 0 Sent Done Done false 3074 Acked Write 8 Sent Done Done false 3075 Acked Read 9 Sent Done Done false 3076 Acked Write 9 Sent Done Done false 3077 Acked Write 3 Sent Done Done false 3078 Acked Write 1 Sent Done Done false 3079 Acked Write 8 Sent Done Done false 3080 Acked Read 3 Sent Done Done false 3081 Acked Write 7 Sent Done Done false 3082 Acked Write 10 Sent Done Done false 3083 Acked Read 7 Sent Done Done false 3084 Acked Flush 0 Sent Done Done false 3085 Acked Read 8 Sent Done Done false 3086 Acked Write 5 Sent Done Done false 3087 Acked Read 5 Sent Done Done false 3088 Acked Read 1 Sent Done Done false 3089 Acked Write 1 Sent Done Done false 3090 Acked Read 5 Sent Done Done false 3091 Acked Read 6 Sent Done Done false 3092 Acked Read 10 Sent Done Done false 3093 Acked Flush 0 Sent Done Done false 3094 Acked Read 9 Sent Done Done false 3095 Acked Write 10 Sent Done Done false 3096 Acked Write 3 Sent Done Done false 3097 Acked Flush 0 Sent Done Done false 3098 Acked Read 2 Sent Done Done false 3099 Acked Write 1 Sent Done Done false 3100 Acked Write 3 Sent Done Done false 3101 Acked Read 4 Sent Done Done false 3102 Acked Write 5 Sent Done Done false 3103 Acked Flush 0 Sent Done Done false 3104 Acked Write 5 Sent Done Done false 3105 Acked Read 6 Sent Done Done false 3106 Acked Read 1 Sent Done Done false 3107 Acked Write 7 Sent Done Done false 3108 Acked Write 6 Sent Done Done false 3109 Acked Read 8 Sent Done Done false 3110 Acked Read 10 Sent Done Done false 3111 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2809 3111 3111 Downstairs last five completed: 2809 2808 2807 2806 2805 Upstairs last five completed: 3111 3110 3109 3108 3107 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:11.157560017Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:11.157628668Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:11.157651413Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:11.157671477Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:11.157692482Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2810 Acked Read 8 Sent Done Done false 2811 Acked Read 8 Sent Done Done false 2812 Acked Read 1 Sent Done Done false 2813 Acked Write 6 Sent Done Done false 2814 Acked Read 4 Sent Done Done false 2815 Acked Flush 0 Sent Done Done false 2816 Acked Flush 0 Sent Done Done false 2817 Acked Read 10 Sent Done Done false 2818 Acked Read 4 Sent Done Done false 2819 Acked Read 9 Sent Done Done false 2820 Acked Write 10 Sent Done Done false 2821 Acked Write 7 Sent Done Done false 2822 Acked Write 5 Sent Done Done false 2823 Acked Read 9 Sent Done Done false 2824 Acked Read 5 Sent Done Done false 2825 Acked Read 5 Sent Done Done false 2826 Acked Write 2 Sent Done Done false 2827 Acked Flush 0 Sent Done Done false 2828 Acked Write 8 Sent Done Done false 2829 Acked Flush 0 Sent Done Done false 2830 Acked Read 8 Sent Done Done false 2831 Acked Flush 0 Sent Done Done false 2832 Acked Write 2 Sent Done Done false 2833 Acked Read 1 Sent Done Done false 2834 Acked Read 9 Sent Done Done false 2835 Acked Write 1 Sent Done Done false 2836 Acked Read 4 Sent Done Done false 2837 Acked Read 4 Sent Done Done false 2838 Acked Write 3 Sent Done Done false 2839 Acked Read 1 Sent Done Done false 2840 Acked Write 10 Sent Done Done false 2841 Acked Read 10 Sent Done Done false 2842 Acked Read 1 Sent Done Done false 2843 Acked Write 5 Sent Done Done false 2844 Acked Write 3 Sent Done Done false 2845 Acked Flush 0 Sent Done Done false 2846 Acked Write 6 Sent Done Done false 2847 Acked Write 8 Sent Done Done false 2848 Acked Flush 0 Sent Done Done false 2849 Acked Read 7 Sent Done Done false 2850 Acked Read 4 Sent Done Done false 2851 Acked Write 9 Sent Done Done false 2852 Acked Write 1 Sent Done Done false 2853 Acked Write 6 Sent Done Done false 2854 Acked Flush 0 Sent Done Done false 2855 Acked Read 7 Sent Done Done false 2856 Acked Write 6 Sent Done Done false 2857 Acked Write 5 Sent Done Done false 2858 Acked Write 2 Sent Done Done false 2859 Acked Read 8 Sent Done Done false 2860 Acked Read 4 Sent Done Done false 2861 Acked Write 4 Sent Done Done false 2862 Acked Write 2 Sent Done Done false 2863 Acked Read 7 Sent Done Done false 2864 Acked Read 6 Sent Done Done false 2865 Acked Read 9 Sent Done Done false 2866 Acked Read 10 Sent Done Done false 2867 Acked Write 1 Sent Done Done false 2868 Acked Write 2 Sent Done Done false 2869 Acked Write 4 Sent Done Done false 2870 Acked Read 1 Sent Done Done false 2871 Acked Read 5 Sent Done Done false 2872 Acked Read 6 Sent Done Done false 2873 Acked Write 7 Sent Done Done false 2874 Acked Read 4 Sent Done Done false 2875 Acked Read 8 Sent Done Done false 2876 Acked Read 6 Sent Done Done false 2877 Acked Read 3 Sent Done Done false 2878 Acked Read 5 Sent Done Done false 2879 Acked Read 6 Sent Done Done false 2880 Acked Write 1 Sent Done Done false 2881 Acked Flush 0 Sent Done Done false 2882 Acked Write 4 Sent Done Done false 2883 Acked Flush 0 Sent Done Done false 2884 Acked Read 3 Sent Done Done false 2885 Acked Write 5 Sent Done Done false 2886 Acked Write 7 Sent Done Done false 2887 Acked Read 8 Sent Done Done false 2888 Acked Write 7 Sent Done Done false 2889 Acked Read 7 Sent Done Done false 2890 Acked Read 10 Sent Done Done false 2891 Acked Read 1 Sent Done Done false 2892 Acked Write 9 Sent Done Done false 2893 Acked Write 10 Sent Done Done false 2894 Acked Read 7 Sent Done Done false 2895 Acked Write 1 Sent Done Done false 2896 Acked Read 5 Sent Done Done false 2897 Acked Write 10 Sent Done Done false 2898 Acked Read 2 Sent Done Done false 2899 Acked Read 7 Sent Done Done false 2900 Acked Read 7 Sent Done Done false 2901 Acked Read 2 Sent Done Done false 2902 Acked Read 2 Sent Done Done false 2903 Acked Write 1 Sent Done Done false 2904 Acked Read 2 Sent Done Done false 2905 Acked Write 2 Sent Done Done false 2906 Acked Read 8 Sent Done Done false 2907 Acked Write 1 Sent Done Done false 2908 Acked Read 7 Sent Done Done false 2909 Acked Read 1 Sent Done Done false 2910 Acked Write 2 Sent Done Done false 2911 Acked Write 4 Sent Done Done false 2912 Acked Read 6 Sent Done Done false 2913 Acked Read 2 Sent Done Done false 2914 Acked Read 10 Sent Done Done false 2915 Acked Write 10 Sent Done Done false 2916 Acked Read 10 Sent Done Done false 2917 Acked Read 3 Sent Done Done false 2918 Acked Write 6 Sent Done Done false 2919 Acked Read 1 Sent Done Done false 2920 Acked Write 3 Sent Done Done false 2921 Acked Read 9 Sent Done Done false 2922 Acked Read 9 Sent Done Done false 2923 Acked Write 4 Sent Done Done false 2924 Acked Read 3 Sent Done Done false 2925 Acked Write 6 Sent Done Done false 2926 Acked Write 7 Sent Done Done false 2927 Acked Write 7 Sent Done Done false 2928 Acked Read 8 Sent Done Done false 2929 Acked Read 6 Sent Done Done false 2930 Acked Read 7 Sent Done Done false 2931 Acked Read 2 Sent Done Done false 2932 Acked Write 3 Sent Done Done false 2933 Acked Read 5 Sent Done Done false 2934 Acked Write 8 Sent Done Done false 2935 Acked Read 7 Sent Done Done false 2936 Acked Write 10 Sent Done Done false 2937 Acked Read 9 Sent Done Done false 2938 Acked Read 10 Sent Done Done false 2939 Acked Write 2 Sent Done Done false 2940 Acked Write 3 Sent Done Done false 2941 Acked Read 2 Sent Done Done false 2942 Acked Flush 0 Sent Done Done false 2943 Acked Write 7 Sent Done Done false 2944 Acked Read 1 Sent Done Done false 2945 Acked Write 4 Sent Done Done false 2946 Acked Write 6 Sent Done Done false 2947 Acked Read 5 Sent Done Done false 2948 Acked Read 7 Sent Done Done false 2949 Acked Read 4 Sent Done Done false 2950 Acked Read 3 Sent Done Done false 2951 Acked Write 10 Sent Done Done false 2952 Acked Read 4 Sent Done Done false 2953 Acked Write 8 Sent Done Done false 2954 Acked Write 9 Sent Done Done false 2955 Acked Write 5 Sent Done Done false 2956 Acked Read 7 Sent Done Done false 2957 Acked Read 5 Sent Done Done false 2958 Acked Read 9 Sent Done Done false 2959 Acked Read 8 Sent Done Done false 2960 Acked Write 9 Sent Done Done false 2961 Acked Read 5 Sent Done Done false 2962 Acked Write 6 Sent Done Done false 2963 Acked Write 8 Sent Done Done false 2964 Acked Write 3 Sent Done Done false 2965 Acked Read 7 Sent Done Done false 2966 Acked Read 8 Sent Done Done false 2967 Acked Write 9 Sent Done Done false 2968 Acked Read 3 Sent Done Done false 2969 Acked Read 6 Sent Done Done false 2970 Acked Read 5 Sent Done Done false 2971 Acked Read 5 Sent Done Done false 2972 Acked Flush 0 Sent Done Done false 2973 Acked Read 1 Sent Done Done false 2974 Acked Write 3 Sent Done Done false 2975 Acked Flush 0 Sent Done Done false 2976 Acked Read 2 Sent Done Done false 2977 Acked Write 8 Sent Done Done false 2978 Acked Write 5 Sent Done Done false 2979 Acked Read 2 Sent Done Done false 2980 Acked Write 4 Sent Done Done false 2981 Acked Read 6 Sent Done Done false 2982 Acked Write 10 Sent Done Done false 2983 Acked Read 1 Sent Done Done false 2984 Acked Read 7 Sent Done Done false 2985 Acked Write 1 Sent Done Done false 2986 Acked Write 9 Sent Done Done false 2987 Acked Write 3 Sent Done Done false 2988 Acked Write 8 Sent Done Done false 2989 Acked Read 8 Sent Done Done false 2990 Acked Write 10 Sent Done Done false 2991 Acked Read 9 Sent Done Done false 2992 Acked Write 5 Sent Done Done false 2993 Acked Write 10 Sent Done Done false 2994 Acked Write 1 Sent Done Done false 2995 Acked Read 5 Sent Done Done false 2996 Acked Flush 0 Sent Done Done false 2997 Acked Write 6 Sent Done Done false 2998 Acked Read 7 Sent Done Done false 2999 Acked Write 9 Sent Done Done false 3000 Acked Read 9 Sent Done Done false 3001 Acked Write 3 Sent Done Done false 3002 Acked Write 6 Sent Done Done false 3003 Acked Read 6 Sent Done Done false 3004 Acked Flush 0 Sent Done Done false 3005 Acked Flush 0 Sent Done Done false 3006 Acked Read 6 Sent Done Done false 3007 Acked Read 1 Sent Done Done false 3008 Acked Read 9 Sent Done Done false 3009 Acked Write 10 Sent Done Done false 3010 Acked Read 9 Sent Done Done false 3011 Acked Write 9 Sent Done Done false 3012 Acked Read 8 Sent Done Done false 3013 Acked Read 1 Sent Done Done false 3014 Acked Read 10 Sent Done Done false 3015 Acked Write 4 Sent Done Done false 3016 Acked Write 2 Sent Done Done false 3017 Acked Read 3 Sent Done Done false 3018 Acked Read 1 Sent Done Done false 3019 Acked Read 6 Sent Done Done false 3020 Acked Read 2 Sent Done Done false 3021 Acked Read 6 Sent Done Done false 3022 Acked Read 3 Sent Done Done false 3023 Acked Read 4 Sent Done Done false 3024 Acked Read 8 Sent Done Done false 3025 Acked Read 7 Sent Done Done false 3026 Acked Read 7 Sent Done Done false 3027 Acked Read 10 Sent Done Done false 3028 Acked Read 8 Sent Done Done false 3029 Acked Flush 0 Sent Done Done false 3030 Acked Read 9 Sent Done Done false 3031 Acked Read 1 Sent Done Done false 3032 Acked Write 4 Sent Done Done false 3033 Acked Write 3 Sent Done Done false 3034 Acked Read 7 Sent Done Done false 3035 Acked Write 8 Sent Done Done false 3036 Acked Read 7 Sent Done Done false 3037 Acked Read 3 Sent Done Done false 3038 Acked Read 4 Sent Done Done false 3039 Acked Write 6 Sent Done Done false 3040 Acked Read 3 Sent Done Done false 3041 Acked Write 10 Sent Done Done false 3042 Acked Read 8 Sent Done Done false 3043 Acked Write 2 Sent Done Done false 3044 Acked Write 9 Sent Done Done false 3045 Acked Read 2 Sent Done Done false 3046 Acked Read 8 Sent Done Done false 3047 Acked Read 1 Sent Done Done false 3048 Acked Read 3 Sent Done Done false 3049 Acked Write 9 Sent Done Done false 3050 Acked Write 1 Sent Done Done false 3051 Acked Write 4 Sent Done Done false 3052 Acked Write 10 Sent Done Done false 3053 Acked Write 8 Sent Done Done false 3054 Acked Write 3 Sent Done Done false 3055 Acked Read 9 Sent Done Done false 3056 Acked Write 3 Sent Done Done false 3057 Acked Flush 0 Sent Done Done false 3058 Acked Write 3 Sent Done Done false 3059 Acked Write 1 Sent Done Done false 3060 Acked Read 9 Sent Done Done false 3061 Acked Read 8 Sent Done Done false 3062 Acked Read 2 Sent Done Done false 3063 Acked Read 7 Sent Done Done false 3064 Acked Read 2 Sent Done Done false 3065 Acked Read 1 Sent Done Done false 3066 Acked Read 5 Sent Done Done false 3067 Acked Read 4 Sent Done Done false 3068 Acked Write 4 Sent Done Done false 3069 Acked Write 3 Sent Done Done false 3070 Acked Write 2 Sent Done Done false 3071 Acked Read 10 Sent Done Done false 3072 Acked Write 7 Sent Done Done false 3073 Acked Flush 0 Sent Done Done false 3074 Acked Write 8 Sent Done Done false 3075 Acked Read 9 Sent Done Done false 3076 Acked Write 9 Sent Done Done false 3077 Acked Write 3 Sent Done Done false 3078 Acked Write 1 Sent Done Done false 3079 Acked Write 8 Sent Done Done false 3080 Acked Read 3 Sent Done Done false 3081 Acked Write 7 Sent Done Done false 3082 Acked Write 10 Sent Done Done false 3083 Acked Read 7 Sent Done Done false 3084 Acked Flush 0 Sent Done Done false 3085 Acked Read 8 Sent Done Done false 3086 Acked Write 5 Sent Done Done false 3087 Acked Read 5 Sent Done Done false 3088 Acked Read 1 Sent Done Done false 3089 Acked Write 1 Sent Done Done false 3090 Acked Read 5 Sent Done Done false 3091 Acked Read 6 Sent Done Done false 3092 Acked Read 10 Sent Done Done false 3093 Acked Flush 0 Sent Done Done false 3094 Acked Read 9 Sent Done Done false 3095 Acked Write 10 Sent Done Done false 3096 Acked Write 3 Sent Done Done false 3097 Acked Flush 0 Sent Done Done false 3098 Acked Read 2 Sent Done Done false 3099 Acked Write 1 Sent Done Done false 3100 Acked Write 3 Sent Done Done false 3101 Acked Read 4 Sent Done Done false 3102 Acked Write 5 Sent Done Done false 3103 Acked Flush 0 Sent Done Done false 3104 Acked Write 5 Sent Done Done false 3105 Acked Read 6 Sent Done Done false 3106 Acked Read 1 Sent Done Done false 3107 Acked Write 7 Sent Done Done false 3108 Acked Write 6 Sent Done Done false 3109 Acked Read 8 Sent Done Done false 3110 Acked Read 10 Sent Done Done false 3111 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2809 3111 3111 Downstairs last five completed: 2809 2808 2807 2806 2805 Upstairs last five completed: 3111 3110 3109 3108 3107 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 2810 Acked Read 8 Sent Done Done false 2811 Acked Read 8 Sent Done Done false 2812 Acked Read 1 Sent Done Done false 2813 Acked Write 6 Sent Done Done false 2814 Acked Read 4 Sent Done Done false 2815 Acked Flush 0 Sent Done Done false 2816 Acked Flush 0 Sent Done Done false 2817 Acked Read 10 Sent Done Done false 2818 Acked Read 4 Sent Done Done false 2819 Acked Read 9 Sent Done Done false 2820 Acked Write 10 Sent Done Done false 2821 Acked Write 7 Sent Done Done false 2822 Acked Write 5 Sent Done Done false 2823 Acked Read 9 Sent Done Done false 2824 Acked Read 5 Sent Done Done false 2825 Acked Read 5 Sent Done Done false 2826 Acked Write 2 Sent Done Done false 2827 Acked Flush 0 Sent Done Done false 2828 Acked Write 8 Sent Done Done false 2829 Acked Flush 0 Sent Done Done false 2830 Acked Read 8 Sent Done Done false 2831 Acked Flush 0 Sent Done Done false 2832 Acked Write 2 Sent Done Done false 2833 Acked Read 1 Sent Done Done false 2834 Acked Read 9 Sent Done Done false 2835 Acked Write 1 Sent Done Done false 2836 Acked Read 4 Sent Done Done false 2837 Acked Read 4 Sent Done Done false 2838 Acked Write 3 Sent Done Done false 2839 Acked Read 1 Sent Done Done false 2840 Acked Write 10 Sent Done Done false 2841 Acked Read 10 Sent Done Done false 2842 Acked Read 1 Sent Done Done false 2843 Acked Write 5 Sent Done Done false 2844 Acked Write 3 Sent Done Done false 2845 Acked Flush 0 Sent Done Done false 2846 Acked Write 6 Sent Done Done false 2847 Acked Write 8 Sent Done Done false 2848 Acked Flush 0 Sent Done Done false 2849 Acked Read 7 Sent Done Done false 2850 Acked Read 4 Sent Done Done false 2851 Acked Write 9 Sent Done Done false 2852 Acked Write 1 Sent Done Done false 2853 Acked Write 6 Sent Done Done false 2854 Acked Flush 0 Sent Done Done false 2855 Acked Read 7 Sent Done Done false 2856 Acked Write 6 Sent Done Done false 2857 Acked Write 5 Sent Done Done false 2858 Acked Write 2 Sent Done Done false 2859 Acked Read 8 Sent Done Done false 2860 Acked Read 4 Sent Done Done false 2861 Acked Write 4 Sent Done Done false 2862 Acked Write 2 Sent Done Done false 2863 Acked Read 7 Sent Done Done false 2864 Acked Read 6 Sent Done Done false 2865 Acked Read 9 Sent Done Done false 2866 Acked Read 10 Sent Done Done false 2867 Acked Write 1 Sent Done Done false 2868 Acked Write 2 Sent Done Done false 2869 Acked Write 4 Sent Done Done false 2870 Acked Read 1 Sent Done Done false 2871 Acked Read 5 Sent Done Done false 2872 Acked Read 6 Sent Done Done false 2873 Acked Write 7 Sent Done Done false 2874 Acked Read 4 Sent Done Done false 2875 Acked Read 8 Sent Done Done false 2876 Acked Read 6 Sent Done Done false 2877 Acked Read 3 Sent Done Done false 2878 Acked Read 5 Sent Done Done false 2879 Acked Read 6 Sent Done Done false 2880 Acked Write 1 Sent Done Done false 2881 Acked Flush 0 Sent Done Done false 2882 Acked Write 4 Sent Done Done false 2883 Acked Flush 0 Sent Done Done false 2884 Acked Read 3 Sent Done Done false 2885 Acked Write 5 Sent Done Done false 2886 Acked Write 7 Sent Done Done false 2887 Acked Read 8 Sent Done Done false 2888 Acked Write 7 Sent Done Done false 2889 Acked Read 7 Sent Done Done false 2890 Acked Read 10 Sent Done Done false 2891 Acked Read 1 Sent Done Done false 2892 Acked Write 9 Sent Done Done false 2893 Acked Write 10 Sent Done Done false 2894 Acked Read 7 Sent Done Done false 2895 Acked Write 1 Sent Done Done false 2896 Acked Read 5 Sent Done Done false 2897 Acked Write 10 Sent Done Done false 2898 Acked Read 2 Sent Done Done false 2899 Acked Read 7 Sent Done Done false 2900 Acked Read 7 Sent Done Done false 2901 Acked Read 2 Sent Done Done false 2902 Acked Read 2 Sent Done Done false 2903 Acked Write 1 Sent Done Done false 2904 Acked Read 2 Sent Done Done false 2905 Acked Write 2 Sent Done Done false 2906 Acked Read 8 Sent Done Done false 2907 Acked Write 1 Sent Done Done false 2908 Acked Read 7 Sent Done Done false 2909 Acked Read 1 Sent Done Done false 2910 Acked Write 2 Sent Done Done false 2911 Acked Write 4 Sent Done Done false 2912 Acked Read 6 Sent Done Done false 2913 Acked Read 2 Sent Done Done false 2914 Acked Read 10 Sent Done Done false 2915 Acked Write 10 Sent Done Done false 2916 Acked Read 10 Sent Done Done false 2917 Acked Read 3 Sent Done Done false 2918 Acked Write 6 Sent Done Done false 2919 Acked Read 1 Sent Done Done false 2920 Acked Write 3 Sent Done Done false 2921 Acked Read 9 Sent Done Done false 2922 Acked Read 9 Sent Done Done false 2923 Acked Write 4 Sent Done Done false 2924 Acked Read 3 Sent Done Done false 2925 Acked Write 6 Sent Done Done false 2926 Acked Write 7 Sent Done Done false 2927 Acked Write 7 Sent Done Done false 2928 Acked Read 8 Sent Done Done false 2929 Acked Read 6 Sent Done Done false 2930 Acked Read 7 Sent Done Done false 2931 Acked Read 2 Sent Done Done false 2932 Acked Write 3 Sent Done Done false 2933 Acked Read 5 Sent Done Done false 2934 Acked Write 8 Sent Done Done false 2935 Acked Read 7 Sent Done Done false 2936 Acked Write 10 Sent Done Done false 2937 Acked Read 9 Sent Done Done false 2938 Acked Read 10 Sent Done Done false 2939 Acked Write 2 Sent Done Done false 2940 Acked Write 3 Sent Done Done false 2941 Acked Read 2 Sent Done Done false 2942 Acked Flush 0 Sent Done Done false 2943 Acked Write 7 Sent Done Done false 2944 Acked Read 1 Sent Done Done false 2945 Acked Write 4 Sent Done Done false 2946 Acked Write 6 Sent Done Done false 2947 Acked Read 5 Sent Done Done false 2948 Acked Read 7 Sent Done Done false 2949 Acked Read 4 Sent Done Done false 2950 Acked Read 3 Sent Done Done false 2951 Acked Write 10 Sent Done Done false 2952 Acked Read 4 Sent Done Done false 2953 Acked Write 8 Sent Done Done false 2954 Acked Write 9 Sent Done Done false 2955 Acked Write 5 Sent Done Done false 2956 Acked Read 7 Sent Done Done false 2957 Acked Read 5 Sent Done Done false 2958 Acked Read 9 Sent Done Done false 2959 Acked Read 8 Sent Done Done false 2960 Acked Write 9 Sent Done Done false 2961 Acked Read 5 Sent Done Done false 2962 Acked Write 6 Sent Done Done false 2963 Acked Write 8 Sent Done Done false 2964 Acked Write 3 Sent Done Done false 2965 Acked Read 7 Sent Done Done false 2966 Acked Read 8 Sent Done Done false 2967 Acked Write 9 Sent Done Done false 2968 Acked Read 3 Sent Done Done false 2969 Acked Read 6 Sent Done Done false 2970 Acked Read 5 Sent Done Done false 2971 Acked Read 5 Sent Done Done false 2972 Acked Flush 0 Sent Done Done false 2973 Acked Read 1 Sent Done Done false 2974 Acked Write 3 Sent Done Done false 2975 Acked Flush 0 Sent Done Done false 2976 Acked Read 2 Sent Done Done false 2977 Acked Write 8 Sent Done Done false 2978 Acked Write 5 Sent Done Done false 2979 Acked Read 2 Sent Done Done false 2980 Acked Write 4 Sent Done Done false 2981 Acked Read 6 Sent Done Done false 2982 Acked Write 10 Sent Done Done false 2983 Acked Read 1 Sent Done Done false 2984 Acked Read 7 Sent Done Done false 2985 Acked Write 1 Sent Done Done false 2986 Acked Write 9 Sent Done Done false 2987 Acked Write 3 Sent Done Done false 2988 Acked Write 8 Sent Done Done false 2989 Acked Read 8 Sent Done Done false 2990 Acked Write 10 Sent Done Done false 2991 Acked Read 9 Sent Done Done false 2992 Acked Write 5 Sent Done Done false 2993 Acked Write 10 Sent Done Done false 2994 Acked Write 1 Sent Done Done false 2995 Acked Read 5 Sent Done Done false 2996 Acked Flush 0 Sent Done Done false 2997 Acked Write 6 Sent Done Done false 2998 Acked Read 7 Sent Done Done false 2999 Acked Write 9 Sent Done Done false 3000 Acked Read 9 Sent Done Done false 3001 Acked Write 3 Sent Done Done false 3002 Acked Write 6 Sent Done Done false 3003 Acked Read 6 Sent Done Done false 3004 Acked Flush 0 Sent Done Done false 3005 Acked Flush 0 Sent Done Done false 3006 Acked Read 6 Sent Done Done false 3007 Acked Read 1 Sent Done Done false 3008 Acked Read 9 Sent Done Done false 3009 Acked Write 10 Sent Done Done false 3010 Acked Read 9 Sent Done Done false 3011 Acked Write 9 Sent Done Done false 3012 Acked Read 8 Sent Done Done false 3013 Acked Read 1 Sent Done Done false 3014 Acked Read 10 Sent Done Done false 3015 Acked Write 4 Sent Done Done false 3016 Acked Write 2 Sent Done Done false 3017 Acked Read 3 Sent Done Done false 3018 Acked Read 1 Sent Done Done false 3019 Acked Read 6 Sent Done Done false 3020 Acked Read 2 Sent Done Done false 3021 Acked Read 6 Sent Done Done false 3022 Acked Read 3 Sent Done Done false 3023 Acked Read 4 Sent Done Done false 3024 Acked Read 8 Sent Done Done false 3025 Acked Read 7 Sent Done Done false 3026 Acked Read 7 Sent Done Done false 3027 Acked Read 10 Sent Done Done false 3028 Acked Read 8 Sent Done Done false 3029 Acked Flush 0 Sent Done Done false 3030 Acked Read 9 Sent Done Done false 3031 Acked Read 1 Sent Done Done false 3032 Acked Write 4 Sent Done Done false 3033 Acked Write 3 Sent Done Done false 3034 Acked Read 7 Sent Done Done false 3035 Acked Write 8 Sent Done Done false 3036 Acked Read 7 Sent Done Done false 3037 Acked Read 3 Sent Done Done false 3038 Acked Read 4 Sent Done Done false 3039 Acked Write 6 Sent Done Done false 3040 Acked Read 3 Sent Done Done false 3041 Acked Write 10 Sent Done Done false 3042 Acked Read 8 Sent Done Done false 3043 Acked Write 2 Sent Done Done false 3044 Acked Write 9 Sent Done Done false 3045 Acked Read 2 Sent Done Done false 3046 Acked Read 8 Sent Done Done false 3047 Acked Read 1 Sent Done Done false 3048 Acked Read 3 Sent Done Done false 3049 Acked Write 9 Sent Done Done false 3050 Acked Write 1 Sent Done Done false 3051 Acked Write 4 Sent Done Done false 3052 Acked Write 10 Sent Done Done false 3053 Acked Write 8 Sent Done Done false 3054 Acked Write 3 Sent Done Done false 3055 Acked Read 9 Sent Done Done false 3056 Acked Write 3 Sent Done Done false 3057 Acked Flush 0 Sent Done Done false 3058 Acked Write 3 Sent Done Done false 3059 Acked Write 1 Sent Done Done false 3060 Acked Read 9 Sent Done Done false 3061 Acked Read 8 Sent Done Done false 3062 Acked Read 2 Sent Done Done false 3063 Acked Read 7 Sent Done Done false 3064 Acked Read 2 Sent Done Done false 3065 Acked Read 1 Sent Done Done false 3066 Acked Read 5 Sent Done Done false 3067 Acked Read 4 Sent Done Done false 3068 Acked Write 4 Sent Done Done false 3069 Acked Write 3 Sent Done Done false 3070 Acked Write 2 Sent Done Done false 3071 Acked Read 10 Sent Done Done false 3072 Acked Write 7 Sent Done Done false 3073 Acked Flush 0 Sent Done Done false 3074 Acked Write 8 Sent Done Done false 3075 Acked Read 9 Sent Done Done false 3076 Acked Write 9 Sent Done Done false 3077 Acked Write 3 Sent Done Done false 3078 Acked Write 1 Sent Done Done false 3079 Acked Write 8 Sent Done Done false 3080 Acked Read 3 Sent Done Done false 3081 Acked Write 7 Sent Done Done false 3082 Acked Write 10 Sent Done Done false 3083 Acked Read 7 Sent Done Done false 3084 Acked Flush 0 Sent Done Done false 3085 Acked Read 8 Sent Done Done false 3086 Acked Write 5 Sent Done Done false 3087 Acked Read 5 Sent Done Done false 3088 Acked Read 1 Sent Done Done false 3089 Acked Write 1 Sent Done Done false 3090 Acked Read 5 Sent Done Done false 3091 Acked Read 6 Sent Done Done false 3092 Acked Read 10 Sent Done Done false 3093 Acked Flush 0 Sent Done Done false 3094 Acked Read 9 Sent Done Done false 3095 Acked Write 10 Sent Done Done false 3096 Acked Write 3 Sent Done Done false 3097 Acked Flush 0 Sent Done Done false 3098 Acked Read 2 Sent Done Done false 3099 Acked Write 1 Sent Done Done false 3100 Acked Write 3 Sent Done Done false 3101 Acked Read 4 Sent Done Done false 3102 Acked Write 5 Sent Done Done false 3103 Acked Flush 0 Sent Done Done false 3104 Acked Write 5 Sent Done Done false 3105 Acked Read 6 Sent Done Done false 3106 Acked Read 1 Sent Done Done false 3107 Acked Write 7 Sent Done Done false 3108 Acked Write 6 Sent Done Done false 3109 Acked Read 8 Sent Done Done false 3110 Acked Read 10 Sent Done Done false 3111 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2809 3111 3111 Downstairs last five completed: 2809 2808 2807 2806 2805 Upstairs last five completed: 3111 3110 3109 3108 3107 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.158449908Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:62961)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.158524608Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:21.159684966Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160203007Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.16022896Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 2809","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160245096Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 2809","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160473904Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160499338Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160514924Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 2809","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.16052919Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 2809","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.160767306Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3111 3110 3109 3108 3107 Upstairs last five completed: 3111 3110 3109 3108 3107 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:43:21.402 INFO request completed, latency_us: 329[8810] Got stop action so:false kr:false , response_code: 204, uri: /stop/cid/0, method: POST, req_id: d30d805f-2823-4f34-8111-a9093b71caef, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:21.403076776Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:21.403131211Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:21.403158014Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.403266035Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.403307394Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:21.403332837Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:43:21.403 INFO request completed, latency_us: 161, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6846a097-1307-4536-82df-4f64ecd63151, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:43:25.405 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: ccb1801f-267a-47dc-933f-32109786763f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 2967 len 32768 002/300 Read block 3770 len 32768 003/300 Read block 367 len 20480 004/300 Write block 3648 len 8192 data: 2 2 005/300 Write block 2271 len 28672 data: 2 2 3 3 3 3 3 006/300 Write block 2600 len 28672 data: 2 3 2 3 3 3 3 007/300 Read block 2114 len 12288 008/300 Write block 2113 len 8192 data: 4 3 009/300 Write block 3001 len 12288 data: 2 2 2 010/300 Read block 3442 len 16384 011/300 Write block 1153 len 8192 data: 2 2 012/300 Flush 013/300 Read block 1774 len 12288 014/300 Read block 1474 len 36864 015/300 Read block 239 len 8192 016/300 Write block 1292 len 28672 data: 4 4 4 4 4 3 2 017/300 Read block 4811 len 24576 018/300 Read block 145 len 28672 019/300 Flush 020/300 Read block 538 len 8192 021/300 Write block 4930 len 24576 data: 3 3 3 2 2 2 022/300 Write block 2222 len 28672 data: 3 3 3 3 3 3 3 023/300 Write block 4424 len 20480 data: 4 4 4 3 3 024/300 Read block 3852 len 20480 025/300 Read block 866 len 8192 026/300 Read block 2376 len 8192 027/300 Read block 4039 len 20480 028/300 Write block 3685 len 28672 data: 4 4 3 3 2 3 3 029/300 Read block 1883 len 8192 030/300 Write block 3405 len 28672 data: 2 3 3 3 3 3 3 031/300 Read block 105 len 4096 032/300 Write block 258 len 20480 data: 2 2 2 4 4 033/300 Read block 3424 len 12288 034/300 Read block 4165 len 24576 035/300 Flush 036/300 Read block 4110 len 40960 037/300 Read block 880 len 40960 038/300 Read block 2884 len 40960 039/300 Read block 2164 len 4096 040/300 Read block 2927 len 28672 041/300 Read block 4318 len 8192 042/300 Write block 4352 len 20480 data: 3 3 4 3 4 043/300 Flush 044/300 Write block 3851 len 8192 data: 2 2 045/300 Flush 046/300 Read block 417 len 36864 047/300 Read block 502 len 36864 048/300 Read block 1081 len 8192 049/300 Write block 2180 len 28672 data: 2 2 2 2 2 2 3 050/300 Read block 1194 len 8192 051/300 Flush 052/300 Write block 745 len 40960 data: 3 4 4 4 5 5 5 6 5 4 053/300 Write block 208 len 32768 data: 4 4 4 4 3 3 3 2 054/300 Write block 4272 len 8192 data: 2 2 055/300 Read block 1921 len 12288 056/300 Read block 4911 len 16384 057/300 Read block 714 len 16384 058/300 Read block 541 len 12288 059/300 Read block 4793 len 16384 060/300 Read block 2128 len 16384 061/300 Write block 648 len 32768 data: 2 2 3 3 3 3 3 4 062/300 Write block 1334 len 36864 data: 2 2 2 2 2 2 2 2 2 063/300 Write block 4823 len 8192 data: 2 2 064/300 Read block 1114 len 28672 065/300 Flush 066/300 Write block 1508 len 8192 data: 3 3 067/300 Write block 2067 len 4096 data: 2 068/300 Write block 3391 len 24576 data: 2 2 2 2 2 2 069/300 Write block 4095 len 16384 data: 5 4 3 3 070/300 Flush 071/300 Write block 620 len 40960 data: 2 2 3 2 2 2 3 3 3 3 072/300 Read block 3829 len 28672 073/300 Read block 455 len 24576 074/300 Flush 075/300 Flush 076/300 Read block 4779 len 40960 077/300 Write block 3332 len 12288 data: 3 2 2 078/300 Write block 3655 len 36864 data: 3 3 3 3 2 2 2 2 2 079/300 Read block 3399 len 12288 080/300 Read block 4334 len 36864 081/300 Read block 209 len 8192 082/300 Read block 297 len 12288 083/300 Read block 3855 len 16384 084/300 Flush 085/300 Read block 2849 len 32768 086/300 Write block 2454 len 28672 data: 3 3 4 5 4 4 4 087/300 Write block 2948 len 4096 data: 2 088/300 Read block 4708 len 8192 089/300 Read block 1180 len 24576 090/300 Flush 091/300 Write block 4705 len 4096 data: 2 092/300 Flush 093/300 Read block 4076 len 8192 094/300 Read block 3084 len 36864 095/300 Write block 1632 len 12288 data: 3 3 3 096/300 Read block 3911 len 20480 097/300 Write block 3008 len 12288 data: 2 2 2 098/300 Read block 2646 len 8192 099/300 Read block 2206 len 24576 100/300 Read block 4091 len 4096 101/300 Read block 3806 len 8192 102/300 Read block 4913 len 40960 103/300 Read block 4298 len 24576 104/300 Write block 2738 len 24576 data: 3 3 5 4 4 4 105/300 Read block 3551 len 20480 106/300 Write block 1435 len 4096 data: 3 107/300 Read block 3742 len 24576 108/300 Read block 2879 len 24576 109/300 Write block 4443 len 12288 data: 5 4 3 110/300 Write block 2247 len 8192 data: 5 5 111/300 Write block 3068 len 8192 data: 2 2 112/300 Flush 113/300 Write block 4091 len 32768 data: 3 3 4 5 6 5 4 4 114/300 Read block 4417 len 16384 115/300 Flush 116/300 Read block 4111 len 24576 117/300 Write block 211 len 12288 data: 5 4 4 118/300 Read block 1008 len 28672 119/300 Read block 2226 len 8192 120/300 Read block 735 len 4096 121/300 Write block 1985 len 16384 data: 2 2 3 3 122/300 Read block 4426 len 8192 123/300 Write block 1597 len 16384 data: 2 3 3 3 124/300 Flush 125/300 Flush 126/300 Read block 2122 len 40960 127/300 Write block 2969 len 32768 data: 2 3 3 3 3 3 3 2 128/300 Read block 3289 len 40960 129/300 Write block 4433 len 20480 data: 3 3 3 4 4 130/300 Read block 3881 len 32768 131/300 Write block 1012 len 28672 data: 4 3 3 3 3 3 2 132/300 Read block 4651 len 36864 133/300 Write block 1387 len 12288 data: 3 3 3 134/300 Write block 876 len 24576 data: 3 3 3 3 3 3 135/300 Write block 3469 len 12288 data: 4 4 4 136/300 Flush 137/300 Read block 257 len 40960 138/300 Read block 203 len 20480 139/300 Write block 931 len 32768 data: 2 2 3 3 2 2 2 2 140/300 Read block 1328 len 40960 141/300 Write block 3819 len 12288 data: 3 3 3 142/300 Read block 3849 len 4096 143/300 Write block 3447 len 12288 data: 2 2 2 144/300 Read block 2563 len 28672 145/300 Read block 3350 len 36864 146/300 Write block 4275 len 28672 data: 2 2 2 3 4 4 3 147/300 Write block 368 len 32768 data: 3 3 3 3 3 2 2 2 148/300 Write block 4237 len 40960 data: 3 3 2 2 3 3 3 3 3 3 149/300 Write block 1247 len 16384 data: 3 4 4 4 150/300 Write block 2005 len 20480 data: 4 3 3 2 2 151/300 Read block 2438 len 12288 152/300 Write block 4691 len 4096 data: 4 153/300 Write block 4354 len 28672 data: 5 4 5 4 4 3 2 154/300 Flush 155/300 Write block 1088 len 24576 data: 4 4 2 3 4 4 156/300 Write block 4796 len 40960 data: 3 3 3 3 3 3 3 3 3 3 157/300 Read block 4560 len 40960 158/300 Read block 4579 len 20480 159/300 Read block 1993 len 12288 160/300 Write block 2142 len 12288 data: 4 4 4 161/300 Flush 162/300 Flush 163/300 Read block 3998 len 12288 164/300 Flush 165/300 Read block 2752 len 12288 166/300 Write block 2604 len 20480 data: 4 4 4 3 3 167/300 Write block 3359 len 28672 data: 3 3 4 4 3 2 2 168/300 Write block 4161 len 24576 data: 2 2 2 2 2 3 169/300 Read block 4649 len 28672 170/300 Flush 171/300 Read block 3040 len 4096 172/300 Write block 3246 len 8192 data: 2 2 173/300 Read block 292 len 24576 174/300 Read block 3794 len 20480 175/300 Read block 3726 len 24576 176/300 Write block 1253 len 8192 data: 5 5 177/300 Write block 1148 len 4096 data: 2 178/300 Flush 179/300 Write block 1312 len 40960 data: 3 3 4 4 3 3 3 2 2 2 180/300 Read block 1027 len 4096 181/300 Read block 1960 len 12288 182/300 Write block 4878 len 12288 data: 2 2 2 183/300 Read block 3939 len 12288 184/300 Write block 1733 len 36864 data: 2 2 2 2 2 2 3 3 3 185/300 Write block 1144 len 24576 data: 2 2 2 2 3 2 186/300 Write block 3237 len 4096 data: 3 187/300 Read block 1139 len 12288 188/300 Write block 2256 len 4096 data: 4 189/300 Read block 3417 len 8192 190/300 Read block 557 len 16384 191/300 Read block 3337 len 16384 192/300 Write block 2558 len 36864 data: 5 5 6 5 5 5 5 6 7 193/300 Read block 197 len 28672 194/300 Write block 1535 len 36864 data: 2 2 2 2 3 5 5 5 5 195/300 Flush 196/300 Write block 962 len 32768 data: 2 2 2 2 2 2 2 2 197/300 Read block 1032 len 28672 198/300 Read block 635 len 28672 199/300 Write block 2092 len 4096 data: 3 200/300 Write block 688 len 20480 data: 3 3 3 2 2 201/300 Write block 1599 len 36864 data: 4 4 3 3 3 4 3 3 4 202/300 Read block 108 len 40960 203/300 Write block 3620 len 4096 data: 4 204/300 Read block 656 len 16384 205/300 Flush 206/300 Read block 4794 len 16384 207/300 Write block 3873 len 4096 data: 2 208/300 Write block 3082 len 32768 data: 2 2 2 3 3 3 3 3 209/300 Write block 3754 len 4096 data: 3 210/300 Read block 4153 len 12288 211/300 Read block 78 len 32768 212/300 Write block 3030 len 16384 data: 2 2 2 2 213/300 Write block 760 len 20480 data: 3 3 2 2 2 214/300 Write block 854 len 8192 data: 3 3 215/300 Write block 2128 len 28672 data: 2 2 2 2 2 2 2 216/300 Write block 4356 len 36864 data: 6 5 5 4 3 2 3 3 3 217/300 Write block 2263 len 36864 data: 2 2 2 2 2 2 2 2 3 218/300 Read block 4669 len 36864 219/300 Read block 1555 len 20480 220/300 Read block 4652 len 16384 221/300 Write block 3889 len 28672 data: 3 4 4 3 3 3 3 222/300 Flush 223/300 Read block 3914 len 36864 224/300 Read block 539 len 4096 225/300 Read block 2966 len 32768 226/300 Write block 3923 len 12288 data: 2 2 2 227/300 Read block 4827 len 36864 228/300 Read block 4175 len 32768 229/300 Write block 3150 len 12288 data: 2 2 2 230/300 Read block 4745 len 8192 231/300 Read block 1753 len 40960 232/300 Read block 92 len 28672 233/300 Read block 2708 len 36864 234/300 Read block 4805 len 20480 235/300 Read block 1285 len 28672 236/300 Flush 237/300 Read block 1029 len 40960 238/300 Write block 391 len 16384 data: 2 2 2 2 239/300 Read block 2952 len 36864 240/300 Flush 241/300 Flush 242/300 Write block 4878 len 8192 data: 3 3 243/300 Write block 3042 len 12288 data: 2 2 2 244/300 Read block 1703 len 16384 245/300 Read block 2031 len 20480 246/300 Read block 1676 len 36864 247/300 Write block 1495 len 24576 data: 2 2 2 2 2 3 248/300 Flush 249/300 Write block 340 len 28672 data: 3 5 4 4 3 3 3 250/300 Write block 4270 len 28672 data: 2 2 3 3 2 3 3 251/300 Write block 656 len 16384 data: 3 3 4 4 252/300 Write block 421 len 24576 data: 2 2 2 2 2 2 253/300 Write block 829 len 24576 data: 5 5 4 4 3 3 254/300 Read block 248 len 28672 255/300 Read block 3470 len 40960 256/300 Write block 346 len 4096 data: 4 257/300 Read block 2685 len 12288 258/300 Write block 1268 len 36864 data: 3 3 3 3 3 2 2 2 2 259/300 Write block 388 len 16384 data: 2 2 2 3 260/300 Flush 261/300 Flush 262/300 Write block 370 len 4096 data: 4 263/300 Read block 784 len 16384 264/300 Write block 446 len 16384 data: 3 3 2 2 265/300 Read block 3777 len 28672 266/300 Flush 267/300 Flush 268/300 Read block 4328 len 32768 269/300 Write block 2384 len 24576 data: 4 4 4 4 5 5 270/300 Read block 3408 len 20480 271/300 Read block 981 len 20480 272/300 Read block 3863 len 20480 273/300 Flush 274/300 Flush 275/300 Write block 2523 len 28672 data: 3 3 3 3 4 4 4 276/300 Write block 523 len 40960 data: 3 4 5 4 3 4 4 4 4 4 277/300 Flush 278/300 Flush 279/300 Read block 721 len 40960 280/300 Read block 1128 len 4096 281/300 Read block 3633 len 32768 282/300 Write block 2788 len 8192 data: 3 3 283/300 Flush 284/300 Write block 4561 len 32768 data: 3 3 4 5 5 4 4 4 285/300 Read block 1903 len 40960 286/300 Read block 3944 len 8192 287/300 Read block 1905 len 36864 288/300 Read block 4062 len 8192 289/300 Write block 469 len 16384 data: 3 3 3 3 290/300 Read block 4439 len 12288 291/300 Write block 2238 len 8192 data: 3 3 292/300 Read block 2789 len 8192 293/300 Write block 4313 len 20480 data: 2 3 3 3 3 294/300 Read block 2228 len 4096 295/300 Read block 0 len 28672 296/300 Read block 3902 len 40960 297/300 Write block 3569 len 12288 data: 3 3 3 298/300 Read block 1148 len 28672 299/300 Write block 2615 len 12288 data: 2 2 2 300/300 Read block 2309 len 16384 301/300 Flush Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:43:25.628 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/0, method: POST, req_id: 030f80f8-c3df-40ca-9c9c-9607ffa04700, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3112 Acked Read 8 Sent Done Done false 3113 Acked Read 8 Sent Done Done false 3114 Acked Read 5 Sent Done Done false 3115 Acked Write 2 Sent Done Done false 3116 Acked Write 7 Sent Done Done false 3117 Acked Write 7 Sent Done Done false 3118 Acked Read 3 Sent Done Done false 3119 Acked Write 2 Sent Done Done false 3120 Acked Write 3 Sent Done Done false 3121 Acked Read 4 Sent Done Done false 3122 Acked Write 2 Sent Done Done false 3123 Acked Flush 0 Sent Done Done false 3124 Acked Read 3 Sent Done Done false 3125 Acked Read 9 Sent Done Done false 3126 Acked Read 2 Sent Done Done false 3127 Acked Write 7 Sent Done Done false 3128 Acked Read 6 Sent Done Done false 3129 Acked Read 7 Sent Done Done false 3130 Acked Flush 0 Sent Done Done false 3131 Acked Read 2 Sent Done Done false 3132 Acked Write 6 Sent Done Done false 3133 Acked Write 7 Sent Done Done false 3134 Acked Write 5 Sent Done Done false 3135 Acked Read 5 Sent Done Done false 3136 Acked Read 2 Sent Done Done false 3137 Acked Read 2 Sent Done Done false 3138 Acked Read 5 Sent Done Done false 3139 Acked Write 7 Sent Done Done false 3140 Acked Read 2 Sent Done Done false 3141 Acked Write 7 Sent Done Done false 3142 Acked Read 1 Sent Done Done false 3143 Acked Write 5 Sent Done Done false 3144 Acked Read 3 Sent Done Done false 3145 Acked Read 6 Sent Done Done false 3146 Acked Flush 0 Sent Done Done false 3147 Acked Read 10 Sent Done Done false 3148 Acked Read 10 Sent Done Done false 3149 Acked Read 10 Sent Done Done false 3150 Acked Read 1 Sent Done Done false 3151 Acked Read 7 Sent Done Done false 3152 Acked Read 2 Sent Done Done false 3153 Acked Write 5 Sent Done Done false 3154 Acked Flush 0 Sent Done Done false 3155 Acked Write 2 Sent Done Done false 3156 Acked Flush 0 Sent Done Done false 3157 Acked Read 9 Sent Done Done false 3158 Acked Read 9 Sent Done Done false 3159 Acked Read 2 Sent Done Done false 3160 Acked Write 7 Sent Done Done false 3161 Acked Read 2 Sent Done Done false 3162 Acked Flush 0 Sent Done Done false 3163 Acked Write 10 Sent Done Done false 3164 Acked Write 8 Sent Done Done false 3165 Acked Write 2 Sent Done Done false 3166 Acked Read 3 Sent Done Done false 3167 Acked Read 4 Sent Done Done false 3168 Acked Read 4 Sent Done Done false 3169 Acked Read 3 Sent Done Done false 3170 Acked Read 4 Sent Done Done false 3171 Acked Read 4 Sent Done Done false 3172 Acked Write 8 Sent Done Done false 3173 Acked Write 9 Sent Done Done false 3174 Acked Write 2 Sent Done Done false 3175 Acked Read 7 Sent Done Done false 3176 Acked Flush 0 Sent Done Done false 3177 Acked Write 2 Sent Done Done false 3178 Acked Write 1 Sent Done Done false 3179 Acked Write 6 Sent Done Done false 3180 Acked Write 4 Sent Done Done false 3181 Acked Flush 0 Sent Done Done false 3182 Acked Write 10 Sent Done Done false 3183 Acked Read 7 Sent Done Done false 3184 Acked Read 6 Sent Done Done false 3185 Acked Flush 0 Sent Done Done false 3186 Acked Flush 0 Sent Done Done false 3187 Acked Read 10 Sent Done Done false 3188 Acked Write 3 Sent Done Done false 3189 Acked Write 9 Sent Done Done false 3190 Acked Read 3 Sent Done Done false 3191 Acked Read 9 Sent Done Done false 3192 Acked Read 2 Sent Done Done false 3193 Acked Read 3 Sent Done Done false 3194 Acked Read 4 Sent Done Done false 3195 Acked Flush 0 Sent Done Done false 3196 Acked Read 8 Sent Done Done false 3197 Acked Write 7 Sent Done Done false 3198 Acked Write 1 Sent Done Done false 3199 Acked Read 2 Sent Done Done false 3200 Acked Read 6 Sent Done Done false 3201 Acked Flush 0 Sent Done Done false 3202 Acked Write 1 Sent Done Done false 3203 Acked Flush 0 Sent Done Done false 3204 Acked Read 2 Sent Done Done false 3205 Acked Read 9 Sent Done Done false 3206 Acked Write 3 Sent Done Done false 3207 Acked Read 5 Sent Done Done false 3208 Acked Write 3 Sent Done Done false 3209 Acked Read 2 Sent Done Done false 3210 Acked Read 6 Sent Done Done false 3211 Acked Read 1 Sent Done Done false 3212 Acked Read 2 Sent Done Done false 3213 Acked Read 10 Sent Done Done false 3214 Acked Read 6 Sent Done Done false 3215 Acked Write 6 Sent Done Done false 3216 Acked Read 5 Sent Done Done false 3217 Acked Write 1 Sent Done Done false 3218 Acked Read 6 Sent Done Done false 3219 Acked Read 6 Sent Done Done false 3220 Acked Write 3 Sent Done Done false 3221 Acked Write 2 Sent Done Done false 3222 Acked Write 2 Sent Done Done false 3223 Acked Flush 0 Sent Done Done false 3224 Acked Write 8 Sent Done Done false 3225 Acked Read 4 Sent Done Done false 3226 Acked Flush 0 Sent Done Done false 3227 Acked Read 6 Sent Done Done false 3228 Acked Write 3 Sent Done Done false 3229 Acked Read 7 Sent Done Done false 3230 Acked Read 2 Sent Done Done false 3231 Acked Read 1 Sent Done Done false 3232 Acked Write 4 Sent Done Done false 3233 Acked Read 2 Sent Done Done false 3234 Acked Write 4 Sent Done Done false 3235 Acked Flush 0 Sent Done Done false 3236 Acked Flush 0 Sent Done Done false 3237 Acked Read 10 Sent Done Done false 3238 Acked Write 8 Sent Done Done false 3239 Acked Read 10 Sent Done Done false 3240 Acked Write 5 Sent Done Done false 3241 Acked Read 8 Sent Done Done false 3242 Acked Write 7 Sent Done Done false 3243 Acked Read 9 Sent Done Done false 3244 Acked Write 3 Sent Done Done false 3245 Acked Write 6 Sent Done Done false 3246 Acked Write 3 Sent Done Done false 3247 Acked Flush 0 Sent Done Done false 3248 Acked Read 10 Sent Done Done false 3249 Acked Read 5 Sent Done Done false 3250 Acked Write 8 Sent Done Done false 3251 Acked Read 10 Sent Done Done false 3252 Acked Write 3 Sent Done Done false 3253 Acked Read 1 Sent Done Done false 3254 Acked Write 3 Sent Done Done false 3255 Acked Read 7 Sent Done Done false 3256 Acked Read 9 Sent Done Done false 3257 Acked Write 7 Sent Done Done false 3258 Acked Write 8 Sent Done Done false 3259 Acked Write 10 Sent Done Done false 3260 Acked Write 4 Sent Done Done false 3261 Acked Write 5 Sent Done Done false 3262 Acked Read 3 Sent Done Done false 3263 Acked Write 1 Sent Done Done false 3264 Acked Write 7 Sent Done Done false 3265 Acked Flush 0 Sent Done Done false 3266 Acked Write 6 Sent Done Done false 3267 Acked Write 10 Sent Done Done false 3268 Acked Read 10 Sent Done Done false 3269 Acked Read 5 Sent Done Done false 3270 Acked Read 3 Sent Done Done false 3271 Acked Write 3 Sent Done Done false 3272 Acked Flush 0 Sent Done Done false 3273 Acked Flush 0 Sent Done Done false 3274 Acked Read 3 Sent Done Done false 3275 Acked Flush 0 Sent Done Done false 3276 Acked Read 3 Sent Done Done false 3277 Acked Write 5 Sent Done Done false 3278 Acked Write 7 Sent Done Done false 3279 Acked Write 6 Sent Done Done false 3280 Acked Read 7 Sent Done Done false 3281 Acked Flush 0 Sent Done Done false 3282 Acked Read 1 Sent Done Done false 3283 Acked Write 2 Sent Done Done false 3284 Acked Read 6 Sent Done Done false 3285 Acked Read 5 Sent Done Done false 3286 Acked Read 6 Sent Done Done false 3287 Acked Write 2 Sent Done Done false 3288 Acked Write 1 Sent Done Done false 3289 Acked Flush 0 Sent Done Done false 3290 Acked Write 10 Sent Done Done false 3291 Acked Read 1 Sent Done Done false 3292 Acked Read 3 Sent Done Done false 3293 Acked Write 3 Sent Done Done false 3294 Acked Read 3 Sent Done Done false 3295 Acked Write 9 Sent Done Done false 3296 Acked Write 6 Sent Done Done false 3297 Acked Write 1 Sent Done Done false 3298 Acked Read 3 Sent Done Done false 3299 Acked Write 1 Sent Done Done false 3300 Acked Read 2 Sent Done Done false 3301 Acked Read 4 Sent Done Done false 3302 Acked Read 4 Sent Done Done false 3303 Acked Write 9 Sent Done Done false 3304 Acked Read 7 Sent Done Done false 3305 Acked Write 9 Sent Done Done false 3306 Acked Flush 0 Sent Done Done false 3307 Acked Write 8 Sent Done Done false 3308 Acked Read 7 Sent Done Done false 3309 Acked Read 7 Sent Done Done false 3310 Acked Write 1 Sent Done Done false 3311 Acked Write 5 Sent Done Done false 3312 Acked Write 9 Sent Done Done false 3313 Acked Read 10 Sent Done Done false 3314 Acked Write 1 Sent Done Done false 3315 Acked Read 4 Sent Done Done false 3316 Acked Flush 0 Sent Done Done false 3317 Acked Read 4 Sent Done Done false 3318 Acked Write 1 Sent Done Done false 3319 Acked Write 8 Sent Done Done false 3320 Acked Write 1 Sent Done Done false 3321 Acked Read 3 Sent Done Done false 3322 Acked Read 8 Sent Done Done false 3323 Acked Write 4 Sent Done Done false 3324 Acked Write 5 Sent Done Done false 3325 Acked Write 2 Sent Done Done false 3326 Acked Write 7 Sent Done Done false 3327 Acked Write 9 Sent Done Done false 3328 Acked Write 9 Sent Done Done false 3329 Acked Read 9 Sent Done Done false 3330 Acked Read 5 Sent Done Done false 3331 Acked Read 4 Sent Done Done false 3332 Acked Write 7 Sent Done Done false 3333 Acked Flush 0 Sent Done Done false 3334 Acked Read 9 Sent Done Done false 3335 Acked Read 1 Sent Done Done false 3336 Acked Read 8 Sent Done Done false 3337 Acked Write 3 Sent Done Done false 3338 Acked Read 9 Sent Done Done false 3339 Acked Read 8 Sent Done Done false 3340 Acked Write 3 Sent Done Done false 3341 Acked Read 2 Sent Done Done false 3342 Acked Read 10 Sent Done Done false 3343 Acked Read 7 Sent Done Done false 3344 Acked Read 9 Sent Done Done false 3345 Acked Read 5 Sent Done Done false 3346 Acked Read 7 Sent Done Done false 3347 Acked Flush 0 Sent Done Done false 3348 Acked Read 10 Sent Done Done false 3349 Acked Write 4 Sent Done Done false 3350 Acked Read 9 Sent Done Done false 3351 Acked Flush 0 Sent Done Done false 3352 Acked Flush 0 Sent Done Done false 3353 Acked Write 2 Sent Done Done false 3354 Acked Write 3 Sent Done Done false 3355 Acked Read 4 Sent Done Done false 3356 Acked Read 5 Sent Done Done false 3357 Acked Read 9 Sent Done Done false 3358 Acked Write 6 Sent Done Done false 3359 Acked Flush 0 Sent Done Done false 3360 Acked Write 7 Sent Done Done false 3361 Acked Write 7 Sent Done Done false 3362 Acked Write 4 Sent Done Done false 3363 Acked Write 6 Sent Done Done false 3364 Acked Write 6 Sent Done Done false 3365 Acked Read 7 Sent Done Done false 3366 Acked Read 10 Sent Done Done false 3367 Acked Write 1 Sent Done Done false 3368 Acked Read 3 Sent Done Done false 3369 Acked Write 9 Sent Done Done false 3370 Acked Write 4 Sent Done Done false 3371 Acked Flush 0 Sent Done Done false 3372 Acked Flush 0 Sent Done Done false 3373 Acked Write 1 Sent Done Done false 3374 Acked Read 4 Sent Done Done false 3375 Acked Write 4 Sent Done Done false 3376 Acked Read 7 Sent Done Done false 3377 Acked Flush 0 Sent Done Done false 3378 Acked Flush 0 Sent Done Done false 3379 Acked Read 8 Sent Done Done false 3380 Acked Write 6 Sent Done Done false 3381 Acked Read 5 Sent Done Done false 3382 Acked Read 5 Sent Done Done false 3383 Acked Read 5 Sent Done Done false 3384 Acked Flush 0 Sent Done Done false 3385 Acked Flush 0 Sent Done Done false 3386 Acked Write 7 Sent Done Done false 3387 Acked Write 10 Sent Done Done false 3388 Acked Flush 0 Sent Done Done false 3389 Acked Flush 0 Sent Done Done false 3390 Acked Read 10 Sent Done Done false 3391 Acked Read 1 Sent Done Done false 3392 Acked Read 8 Sent Done Done false 3393 Acked Write 2 Sent Done Done false 3394 Acked Flush 0 Sent Done Done false 3395 Acked Write 8 Sent Done Done false 3396 Acked Read 10 Sent Done Done false 3397 Acked Read 2 Sent Done Done false 3398 Acked Read 9 Sent Done Done false 3399 Acked Read 2 Sent Done Done false 3400 Acked Write 4 Sent Done Done false 3401 Acked Read 3 Sent Done Done false 3402 Acked Write 2 Sent Done Done false 3403 Acked Read 2 Sent Done Done false 3404 Acked Write 5 Sent Done Done false 3405 Acked Read 1 Sent Done Done false 3406 Acked Read 7 Sent Done Done false 3407 Acked Read 10 Sent Done Done false 3408 Acked Write 3 Sent Done Done false 3409 Acked Read 7 Sent Done Done false 3410 Acked Write 3 Sent Done Done false 3411 Acked Read 4 Sent Done Done false 3412 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3111 3412 3412 Downstairs last five completed: 3111 3110 3109 3108 3107 Upstairs last five completed: 3412 3411 3410 3409 3408 CLIENT: Up:0 ds:301 act:2 [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3112 Acked Read 8 Sent Done Done false 3113 Acked Read 8 Sent Done Done false 3114 Acked Read 5 Sent Done Done false 3115 Acked Write 2 Sent Done Done false 3116 Acked Write 7 Sent Done Done false 3117 Acked Write 7 Sent Done Done false 3118 Acked Read 3 Sent Done Done false 3119 Acked Write 2 Sent Done Done false 3120 Acked Write 3 Sent Done Done false 3121 Acked Read 4 Sent Done Done false 3122 Acked Write 2 Sent Done Done false 3123 Acked Flush 0 Sent Done Done false 3124 Acked Read 3 Sent Done Done false 3125 Acked Read 9 Sent Done Done false 3126 Acked Read 2 Sent Done Done false 3127 Acked Write 7 Sent Done Done false 3128 Acked Read 6 Sent Done Done false 3129 Acked Read 7 Sent Done Done false 3130 Acked Flush 0 Sent Done Done false 3131 Acked Read 2 Sent Done Done false 3132 Acked Write 6 Sent Done Done false 3133 Acked Write 7 Sent Done Done false 3134 Acked Write 5 Sent Done Done false 3135 Acked Read 5 Sent Done Done false 3136 Acked Read 2 Sent Done Done false 3137 Acked Read 2 Sent Done Done false 3138 Acked Read 5 Sent Done Done false 3139 Acked Write 7 Sent Done Done false 3140 Acked Read 2 Sent Done Done false 3141 Acked Write 7 Sent Done Done false 3142 Acked Read 1 Sent Done Done false 3143 Acked Write 5 Sent Done Done false 3144 Acked Read 3 Sent Done Done false 3145 Acked Read 6 Sent Done Done false 3146 Acked Flush 0 Sent Done Done false 3147 Acked Read 10 Sent Done Done false 3148 Acked Read 10 Sent Done Done false 3149 Acked Read 10 Sent Done Done false 3150 Acked Read 1 Sent Done Done false 3151 Acked Read 7 Sent Done Done false 3152 Acked Read 2 Sent Done Done false 3153 Acked Write 5 Sent Done Done false 3154 Acked Flush 0 Sent Done Done false 3155 Acked Write 2 Sent Done Done false 3156 Acked Flush 0 Sent Done Done false 3157 Acked Read 9 Sent Done Done false 3158 Acked Read 9 Sent Done Done false 3159 Acked Read 2 Sent Done Done false 3160 Acked Write 7 Sent Done Done false 3161 Acked Read 2 Sent Done Done false 3162 Acked Flush 0 Sent Done Done false 3163 Acked Write 10 Sent Done Done false 3164 Acked Write 8 Sent Done Done false 3165 Acked Write 2 Sent Done Done false 3166 Acked Read 3 Sent Done Done false 3167 Acked Read 4 Sent Done Done false 3168 Acked Read 4 Sent Done Done false 3169 Acked Read 3 Sent Done Done false 3170 Acked Read 4 Sent Done Done false 3171 Acked Read 4 Sent Done Done false 3172 Acked Write 8 Sent Done Done false 3173 Acked Write 9 Sent Done Done false 3174 Acked Write 2 Sent Done Done false 3175 Acked Read 7 Sent Done Done false 3176 Acked Flush 0 Sent Done Done false 3177 Acked Write 2 Sent Done Done false 3178 Acked Write 1 Sent Done Done false 3179 Acked Write 6 Sent Done Done false 3180 Acked Write 4 Sent Done Done false 3181 Acked Flush 0 Sent Done Done false 3182 Acked Write 10 Sent Done Done false 3183 Acked Read 7 Sent Done Done false 3184 Acked Read 6 Sent Done Done false 3185 Acked Flush 0 Sent Done Done false 3186 Acked Flush 0 Sent Done Done false 3187 Acked Read 10 Sent Done Done false 3188 Acked Write 3 Sent Done Done false 3189 Acked Write 9 Sent Done Done false 3190 Acked Read 3 Sent Done Done false 3191 Acked Read 9 Sent Done Done false 3192 Acked Read 2 Sent Done Done false 3193 Acked Read 3 Sent Done Done false 3194 Acked Read 4 Sent Done Done false 3195 Acked Flush 0 Sent Done Done false 3196 Acked Read 8 Sent Done Done false 3197 Acked Write 7 Sent Done Done false 3198 Acked Write 1 Sent Done Done false 3199 Acked Read 2 Sent Done Done false 3200 Acked Read 6 Sent Done Done false 3201 Acked Flush 0 Sent Done Done false 3202 Acked Write 1 Sent Done Done false 3203 Acked Flush 0 Sent Done Done false 3204 Acked Read 2 Sent Done Done false 3205 Acked Read 9 Sent Done Done false 3206 Acked Write 3 Sent Done Done false 3207 Acked Read 5 Sent Done Done false 3208 Acked Write 3 Sent Done Done false 3209 Acked Read 2 Sent Done Done false 3210 Acked Read 6 Sent Done Done false 3211 Acked Read 1 Sent Done Done false 3212 Acked Read 2 Sent Done Done false 3213 Acked Read 10 Sent Done Done false 3214 Acked Read 6 Sent Done Done false 3215 Acked Write 6 Sent Done Done false 3216 Acked Read 5 Sent Done Done false 3217 Acked Write 1 Sent Done Done false 3218 Acked Read 6 Sent Done Done false 3219 Acked Read 6 Sent Done Done false 3220 Acked Write 3 Sent Done Done false 3221 Acked Write 2 Sent Done Done false 3222 Acked Write 2 Sent Done Done false 3223 Acked Flush 0 Sent Done Done false 3224 Acked Write 8 Sent Done Done false 3225 Acked Read 4 Sent Done Done false 3226 Acked Flush 0 Sent Done Done false 3227 Acked Read 6 Sent Done Done false 3228 Acked Write 3 Sent Done Done false 3229 Acked Read 7 Sent Done Done false 3230 Acked Read 2 Sent Done Done false 3231 Acked Read 1 Sent Done Done false 3232 Acked Write 4 Sent Done Done false 3233 Acked Read 2 Sent Done Done false 3234 Acked Write 4 Sent Done Done false 3235 Acked Flush 0 Sent Done Done false 3236 Acked Flush 0 Sent Done Done false 3237 Acked Read 10 Sent Done Done false 3238 Acked Write 8 Sent Done Done false 3239 Acked Read 10 Sent Done Done false 3240 Acked Write 5 Sent Done Done false 3241 Acked Read 8 Sent Done Done false 3242 Acked Write 7 Sent Done Done false 3243 Acked Read 9 Sent Done Done false 3244 Acked Write 3 Sent Done Done false 3245 Acked Write 6 Sent Done Done false 3246 Acked Write 3 Sent Done Done false 3247 Acked Flush 0 Sent Done Done false 3248 Acked Read 10 Sent Done Done false 3249 Acked Read 5 Sent Done Done false 3250 Acked Write 8 Sent Done Done false 3251 Acked Read 10 Sent Done Done false 3252 Acked Write 3 Sent Done Done false 3253 Acked Read 1 Sent Done Done false 3254 Acked Write 3 Sent Done Done false 3255 Acked Read 7 Sent Done Done false 3256 Acked Read 9 Sent Done Done false 3257 Acked Write 7 Sent Done Done false 3258 Acked Write 8 Sent Done Done false 3259 Acked Write 10 Sent Done Done false 3260 Acked Write 4 Sent Done Done false 3261 Acked Write 5 Sent Done Done false 3262 Acked Read 3 Sent Done Done false 3263 Acked Write 1 Sent Done Done false 3264 Acked Write 7 Sent Done Done false 3265 Acked Flush 0 Sent Done Done false 3266 Acked Write 6 Sent Done Done false 3267 Acked Write 10 Sent Done Done false 3268 Acked Read 10 Sent Done Done false 3269 Acked Read 5 Sent Done Done false 3270 Acked Read 3 Sent Done Done false 3271 Acked Write 3 Sent Done Done false 3272 Acked Flush 0 Sent Done Done false 3273 Acked Flush 0 Sent Done Done false 3274 Acked Read 3 Sent Done Done false 3275 Acked Flush 0 Sent Done Done false 3276 Acked Read 3 Sent Done Done false 3277 Acked Write 5 Sent Done Done false 3278 Acked Write 7 Sent Done Done false 3279 Acked Write 6 Sent Done Done false 3280 Acked Read 7 Sent Done Done false 3281 Acked Flush 0 Sent Done Done false 3282 Acked Read 1 Sent Done Done false 3283 Acked Write 2 Sent Done Done false 3284 Acked Read 6 Sent Done Done false 3285 Acked Read 5 Sent Done Done false 3286 Acked Read 6 Sent Done Done false 3287 Acked Write 2 Sent Done Done false 3288 Acked Write 1 Sent Done Done false 3289 Acked Flush 0 Sent Done Done false 3290 Acked Write 10 Sent Done Done false 3291 Acked Read 1 Sent Done Done false 3292 Acked Read 3 Sent Done Done false 3293 Acked Write 3 Sent Done Done false 3294 Acked Read 3 Sent Done Done false 3295 Acked Write 9 Sent Done Done false 3296 Acked Write 6 Sent Done Done false 3297 Acked Write 1 Sent Done Done false 3298 Acked Read 3 Sent Done Done false 3299 Acked Write 1 Sent Done Done false 3300 Acked Read 2 Sent Done Done false 3301 Acked Read 4 Sent Done Done false 3302 Acked Read 4 Sent Done Done false 3303 Acked Write 9 Sent Done Done false 3304 Acked Read 7 Sent Done Done false 3305 Acked Write 9 Sent Done Done false 3306 Acked Flush 0 Sent Done Done false 3307 Acked Write 8 Sent Done Done false 3308 Acked Read 7 Sent Done Done false 3309 Acked Read 7 Sent Done Done false 3310 Acked Write 1 Sent Done Done false 3311 Acked Write 5 Sent Done Done false 3312 Acked Write 9 Sent Done Done false 3313 Acked Read 10 Sent Done Done false 3314 Acked Write 1 Sent Done Done false 3315 Acked Read 4 Sent Done Done false 3316 Acked Flush 0 Sent Done Done false 3317 Acked Read 4 Sent Done Done false 3318 Acked Write 1 Sent Done Done false 3319 Acked Write 8 Sent Done Done false 3320 Acked Write 1 Sent Done Done false 3321 Acked Read 3 Sent Done Done false 3322 Acked Read 8 Sent Done Done false 3323 Acked Write 4 Sent Done Done false 3324 Acked Write 5 Sent Done Done false 3325 Acked Write 2 Sent Done Done false 3326 Acked Write 7 Sent Done Done false 3327 Acked Write 9 Sent Done Done false 3328 Acked Write 9 Sent Done Done false 3329 Acked Read 9 Sent Done Done false 3330 Acked Read 5 Sent Done Done false 3331 Acked Read 4 Sent Done Done false 3332 Acked Write 7 Sent Done Done false 3333 Acked Flush 0 Sent Done Done false 3334 Acked Read 9 Sent Done Done false 3335 Acked Read 1 Sent Done Done false 3336 Acked Read 8 Sent Done Done false 3337 Acked Write 3 Sent Done Done false 3338 Acked Read 9 Sent Done Done false 3339 Acked Read 8 Sent Done Done false 3340 Acked Write 3 Sent Done Done false 3341 Acked Read 2 Sent Done Done false 3342 Acked Read 10 Sent Done Done false 3343 Acked Read 7 Sent Done Done false 3344 Acked Read 9 Sent Done Done false 3345 Acked Read 5 Sent Done Done false 3346 Acked Read 7 Sent Done Done false 3347 Acked Flush 0 Sent Done Done false 3348 Acked Read 10 Sent Done Done false 3349 Acked Write 4 Sent Done Done false 3350 Acked Read 9 Sent Done Done false 3351 Acked Flush 0 Sent Done Done false 3352 Acked Flush 0 Sent Done Done false 3353 Acked Write 2 Sent Done Done false 3354 Acked Write 3 Sent Done Done false 3355 Acked Read 4 Sent Done Done false 3356 Acked Read 5 Sent Done Done false 3357 Acked Read 9 Sent Done Done false 3358 Acked Write 6 Sent Done Done false 3359 Acked Flush 0 Sent Done Done false 3360 Acked Write 7 Sent Done Done false 3361 Acked Write 7 Sent Done Done false 3362 Acked Write 4 Sent Done Done false 3363 Acked Write 6 Sent Done Done false 3364 Acked Write 6 Sent Done Done false 3365 Acked Read 7 Sent Done Done false 3366 Acked Read 10 Sent Done Done false 3367 Acked Write 1 Sent Done Done false 3368 Acked Read 3 Sent Done Done false 3369 Acked Write 9 Sent Done Done false 3370 Acked Write 4 Sent Done Done false 3371 Acked Flush 0 Sent Done Done false 3372 Acked Flush 0 Sent Done Done false 3373 Acked Write 1 Sent Done Done false 3374 Acked Read 4 Sent Done Done false 3375 Acked Write 4 Sent Done Done false 3376 Acked Read 7 Sent Done Done false 3377 Acked Flush 0 Sent Done Done false 3378 Acked Flush 0 Sent Done Done false 3379 Acked Read 8 Sent Done Done false 3380 Acked Write 6 Sent Done Done false 3381 Acked Read 5 Sent Done Done false 3382 Acked Read 5 Sent Done Done false 3383 Acked Read 5 Sent Done Done false 3384 Acked Flush 0 Sent Done Done false 3385 Acked Flush 0 Sent Done Done false 3386 Acked Write 7 Sent Done Done false 3387 Acked Write 10 Sent Done Done false 3388 Acked Flush 0 Sent Done Done false 3389 Acked Flush 0 Sent Done Done false 3390 Acked Read 10 Sent Done Done false 3391 Acked Read 1 Sent Done Done false 3392 Acked Read 8 Sent Done Done false 3393 Acked Write 2 Sent Done Done false 3394 Acked Flush 0 Sent Done Done false 3395 Acked Write 8 Sent Done Done false 3396 Acked Read 10 Sent Done Done false 3397 Acked Read 2 Sent Done Done false 3398 Acked Read 9 Sent Done Done false 3399 Acked Read 2 Sent Done Done false 3400 Acked Write 4 Sent Done Done false 3401 Acked Read 3 Sent Done Done false 3402 Acked Write 2 Sent Done Done false 3403 Acked Read 2 Sent Done Done false 3404 Acked Write 5 Sent Done Done false 3405 Acked Read 1 Sent Done Done false 3406 Acked Read 7 Sent Done Done false 3407 Acked Read 10 Sent Done Done false 3408 Acked Write 3 Sent Done Done false 3409 Acked Read 7 Sent Done Done false 3410 Acked Write 3 Sent Done Done false 3411 Acked Read 4 Sent Done Done false 3412 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3111 3412 3412 Downstairs last five completed: 3111 3110 3109 3108 3107 Upstairs last five completed: 3412 3411 3410 3409 3408 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.404532415Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:64075)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.404599357Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:31.40551932Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406073991Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406103674Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 3111","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406120579Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 3111","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406329553Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406354126Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406369152Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 301 jobs since flush 3111","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406383059Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 301 jobs since flush: 3111","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:31.406598441Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3412 3411 3410 3409 3408 Upstairs last five completed: 3412 3411 3410 3409 3408 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:43:33.633 INFO request completed, latency_us: 380, response_code: [8810] Got stop action so:false kr:false 204, uri: /stop/cid/0, method: POST, req_id: 4dba9a70-cc82-4495-b04a-c11024943ce5, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:43:33.634 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3a755622-149d-458d-bafb-526640d29caf, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:33.634229075Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:33.634289579Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:33.634311063Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:33.634327999Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:33.634351772Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:33.634385063Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:43:37.635 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/0, method: GET, req_id: 27bd1abb-d47d-47f5-9651-1aaf2d7484e6, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 4096 len 12288 002/300 Read block 1592 len 36864 003/300 Read block 4556 len 40960 004/300 Read block 4867 len 28672 005/300 Write block 3085 len 20480 data: 4 4 4 4 4 006/300 Write block 2810 len 20480 data: 3 2 2 2 4 007/300 Write block 263 len 32768 data: 4 4 4 4 3 3 3 2 008/300 Write block 1628 len 4096 data: 4 009/300 Read block 1806 len 4096 010/300 Flush 011/300 Read block 440 len 28672 012/300 Read block 550 len 28672 013/300 Read block 1146 len 32768 014/300 Write block 4855 len 4096 data: 4 015/300 Read block 387 len 8192 016/300 Flush 017/300 Read block 4367 len 32768 018/300 Write block 1237 len 28672 data: 2 2 2 2 2 2 3 019/300 Read block 4313 len 8192 020/300 Read block 4340 len 4096 021/300 Write block 3455 len 24576 data: 2 2 2 2 3 3 022/300 Read block 3195 len 24576 023/300 Flush 024/300 Read block 1404 len 36864 025/300 Read block 4208 len 12288 026/300 Write block 1517 len 12288 data: 2 2 2 027/300 Read block 2533 len 16384 028/300 Read block 1671 len 12288 029/300 Write block 486 len 36864 data: 3 3 3 4 4 5 5 4 3 030/300 Read block 386 len 24576 031/300 Write block 3719 len 36864 data: 2 2 2 2 2 2 2 3 3 032/300 Read block 3498 len 20480 033/300 Flush 034/300 Read block 2071 len 8192 035/300 Read block 4162 len 16384 036/300 Write block 3658 len 16384 data: 4 3 3 3 037/300 Write block 4790 len 40960 data: 2 2 2 2 3 3 4 4 4 4 038/300 Write block 2969 len 32768 data: 3 4 4 4 4 4 4 3 039/300 Read block 4778 len 28672 040/300 Write block 4560 len 20480 data: 3 4 4 5 6 041/300 Read block 1674 len 20480 042/300 Read block 2415 len 16384 043/300 Write block 4317 len 32768 data: 4 3 3 2 2 2 2 2 044/300 Write block 4468 len 28672 data: 4 2 3 3 3 3 3 045/300 Write block 3917 len 8192 data: 2 2 046/300 Read block 2242 len 40960 047/300 Read block 2243 len 12288 048/300 Write block 4232 len 28672 data: 3 3 2 2 2 4 4 049/300 Write block 1537 len 32768 data: 3 3 4 6 6 6 6 4 050/300 Read block 4676 len 8192 051/300 Write block 3653 len 40960 data: 3 3 4 4 4 5 4 4 4 3 052/300 Read block 4700 len 16384 053/300 Flush 054/300 Flush 055/300 Read block 3433 len 36864 056/300 Read block 537 len 40960 057/300 Read block 3769 len 28672 058/300 Read block 2715 len 16384 059/300 Write block 2550 len 32768 data: 3 4 4 4 3 4 5 5 060/300 Write block 3356 len 36864 data: 2 3 3 4 4 5 5 4 3 061/300 Read block 2141 len 20480 062/300 Write block 4943 len 28672 data: 4 3 3 2 2 2 2 063/300 Read block 3907 len 24576 064/300 Read block 1644 len 16384 065/300 Write block 1313 len 16384 data: 4 5 5 4 066/300 Write block 3600 len 24576 data: 2 2 2 2 2 2 067/300 Read block 4618 len 40960 068/300 Read block 4541 len 20480 069/300 Write block 767 len 40960 data: 2 2 2 2 2 2 2 2 2 2 070/300 Flush 071/300 Write block 2099 len 16384 data: 3 4 4 4 072/300 Read block 1506 len 8192 073/300 Write block 4148 len 28672 data: 2 2 2 2 3 3 4 074/300 Read block 2678 len 32768 075/300 Write block 1645 len 32768 data: 4 4 4 4 4 4 4 4 076/300 Read block 1556 len 16384 077/300 Read block 4505 len 20480 078/300 Read block 1582 len 8192 079/300 Read block 4498 len 4096 080/300 Read block 2466 len 12288 081/300 Flush 082/300 Read block 1765 len 4096 083/300 Read block 3922 len 20480 084/300 Read block 3604 len 12288 085/300 Write block 3778 len 36864 data: 3 2 2 3 3 3 3 3 3 086/300 Write block 21 len 24576 data: 4 3 3 3 3 3 087/300 Read block 4803 len 28672 088/300 Flush 089/300 Write block 770 len 4096 data: 3 090/300 Write block 2109 len 28672 data: 4 4 3 4 5 4 4 091/300 Write block 1852 len 40960 data: 3 3 3 2 2 2 2 2 2 2 092/300 Write block 1123 len 4096 data: 2 093/300 Write block 2480 len 28672 data: 5 4 3 2 2 2 2 094/300 Read block 1173 len 32768 095/300 Read block 2026 len 12288 096/300 Read block 1017 len 16384 097/300 Read block 1509 len 28672 098/300 Write block 2739 len 4096 data: 4 099/300 Flush 100/300 Read block 3537 len 12288 101/300 Write block 3596 len 8192 data: 2 2 102/300 Write block 1263 len 24576 data: 2 2 3 3 3 4 103/300 Write block 814 len 16384 data: 2 2 2 2 104/300 Read block 2442 len 4096 105/300 Write block 240 len 32768 data: 4 4 3 3 2 2 2 2 106/300 Read block 1822 len 36864 107/300 Read block 4903 len 28672 108/300 Write block 3127 len 40960 data: 2 3 3 4 4 4 4 4 5 5 109/300 Flush 110/300 Write block 3676 len 16384 data: 3 4 4 4 111/300 Flush 112/300 Read block 4452 len 24576 113/300 Read block 920 len 40960 114/300 Read block 1308 len 16384 115/300 Write block 1320 len 4096 data: 3 116/300 Read block 1208 len 12288 117/300 Write block 3383 len 20480 data: 4 4 3 3 3 118/300 Read block 3087 len 4096 119/300 Write block 439 len 20480 data: 4 4 4 3 3 120/300 Read block 1901 len 36864 121/300 Write block 2950 len 16384 data: 2 2 2 2 122/300 Read block 3830 len 16384 123/300 Read block 4136 len 32768 124/300 Read block 373 len 20480 125/300 Read block 360 len 28672 126/300 Flush 127/300 Write block 3076 len 4096 data: 2 128/300 Read block 3640 len 4096 129/300 Read block 2587 len 40960 130/300 Read block 2182 len 24576 131/300 Read block 4145 len 4096 132/300 Write block 1971 len 4096 data: 2 133/300 Write block 3590 len 20480 data: 3 3 3 3 2 134/300 Write block 692 len 4096 data: 3 135/300 Read block 1910 len 20480 136/300 Read block 2947 len 28672 137/300 Read block 4571 len 36864 138/300 Flush 139/300 Read block 2204 len 20480 140/300 Write block 1905 len 12288 data: 4 4 5 141/300 Write block 134 len 36864 data: 2 2 2 2 2 2 2 2 2 142/300 Read block 3455 len 12288 143/300 Write block 1999 len 36864 data: 6 5 5 4 4 4 5 4 4 144/300 Write block 218 len 4096 data: 2 145/300 Write block 1497 len 20480 data: 3 3 3 4 3 146/300 Read block 3352 len 36864 147/300 Read block 4836 len 36864 148/300 Write block 1481 len 36864 data: 2 2 3 3 2 2 2 2 2 149/300 Write block 2132 len 40960 data: 3 3 3 2 2 2 3 3 3 3 150/300 Write block 1817 len 28672 data: 4 4 4 5 5 6 6 151/300 Read block 668 len 20480 152/300 Read block 1094 len 16384 153/300 Read block 3580 len 28672 154/300 Write block 3689 len 8192 data: 3 4 155/300 Write block 1617 len 28672 data: 2 2 3 3 3 3 3 156/300 Flush 157/300 Write block 153 len 28672 data: 3 3 3 3 3 2 2 158/300 Write block 3326 len 40960 data: 2 3 3 3 3 3 4 3 3 2 159/300 Read block 2148 len 32768 160/300 Read block 1822 len 32768 161/300 Read block 4418 len 20480 162/300 Read block 962 len 16384 163/300 Read block 2487 len 16384 164/300 Read block 4643 len 8192 165/300 Read block 691 len 8192 166/300 Read block 736 len 8192 167/300 Read block 4176 len 40960 168/300 Read block 3652 len 28672 169/300 Write block 1430 len 20480 data: 3 2 2 2 3 170/300 Read block 4447 len 32768 171/300 Read block 1959 len 12288 172/300 Write block 4 len 16384 data: 2 2 2 2 173/300 Write block 4937 len 36864 data: 3 5 5 5 5 5 5 4 4 174/300 Read block 1699 len 16384 175/300 Write block 580 len 12288 data: 3 3 3 176/300 Read block 2156 len 20480 177/300 Read block 507 len 24576 178/300 Write block 224 len 20480 data: 3 3 3 3 2 179/300 Flush 180/300 Read block 791 len 28672 181/300 Read block 563 len 40960 182/300 Write block 917 len 28672 data: 4 4 4 4 4 4 4 183/300 Write block 4212 len 16384 data: 3 3 2 2 184/300 Read block 717 len 20480 185/300 Read block 4421 len 8192 186/300 Write block 692 len 20480 data: 4 2 2 3 3 187/300 Write block 898 len 40960 data: 3 3 2 2 2 3 3 4 3 3 188/300 Read block 4013 len 24576 189/300 Read block 4282 len 12288 190/300 Read block 2373 len 36864 191/300 Write block 864 len 32768 data: 2 3 3 3 3 3 3 2 192/300 Write block 773 len 32768 data: 3 3 3 3 2 2 2 2 193/300 Write block 261 len 16384 data: 5 5 5 5 194/300 Read block 2515 len 12288 195/300 Read block 4800 len 28672 196/300 Read block 1350 len 36864 197/300 Write block 377 len 20480 data: 2 2 2 2 2 198/300 Write block 2842 len 28672 data: 3 3 2 2 2 2 2 199/300 Read block 2540 len 16384 200/300 Write block 3045 len 28672 data: 3 3 3 2 2 2 2 201/300 Read block 733 len 8192 202/300 Read block 2037 len 36864 203/300 Write block 207 len 8192 data: 4 5 204/300 Read block 4624 len 20480 205/300 Read block 2581 len 32768 206/300 Write block 87 len 36864 data: 2 2 2 2 2 2 2 2 2 207/300 Write block 465 len 24576 data: 2 2 3 3 4 4 208/300 Write block 3570 len 28672 data: 4 4 2 2 3 3 3 209/300 Read block 2758 len 16384 210/300 Write block 4419 len 28672 data: 4 4 4 5 5 5 5 211/300 Read block 4391 len 28672 212/300 Read block 1494 len 8192 213/300 Flush 214/300 Write block 4327 len 36864 data: 2 2 3 3 4 5 5 4 4 215/300 Flush 216/300 Read block 4615 len 4096 217/300 Read block 4280 len 12288 218/300 Read block 2616 len 28672 219/300 Read block 4417 len 20480 220/300 Read block 807 len 28672 221/300 Write block 2370 len 28672 data: 3 3 3 5 5 5 4 222/300 Read block 3503 len 24576 223/300 Write block 3163 len 12288 data: 4 4 5 224/300 Read block 1732 len 8192 225/300 Read block 3557 len 40960 226/300 Read block 2994 len 16384 227/300 Read block 2771 len 36864 228/300 Read block 1102 len 28672 229/300 Read block 1602 len 12288 230/300 Read block 4430 len 32768 231/300 Read block 284 len 40960 232/300 Write block 2303 len 16384 data: 2 2 3 4 233/300 Write block 3699 len 8192 data: 5 6 234/300 Flush 235/300 Write block 811 len 20480 data: 3 3 2 3 3 236/300 Read block 3730 len 16384 237/300 Read block 4089 len 8192 238/300 Write block 1385 len 32768 data: 3 3 4 4 4 4 4 4 239/300 Write block 3001 len 20480 data: 3 3 3 2 3 240/300 Read block 4507 len 40960 241/300 Read block 3078 len 20480 242/300 Write block 868 len 28672 data: 4 4 4 3 2 3 3 243/300 Read block 4190 len 8192 244/300 Write block 4471 len 16384 data: 4 4 4 4 245/300 Write block 3542 len 4096 data: 4 246/300 Write block 2955 len 40960 data: 2 2 4 3 3 3 4 3 3 3 247/300 Write block 3598 len 28672 data: 2 2 3 3 3 3 3 248/300 Write block 4214 len 12288 data: 3 3 2 249/300 Write block 2022 len 16384 data: 3 3 3 3 250/300 Write block 866 len 8192 data: 4 4 251/300 Read block 3649 len 8192 252/300 Write block 2975 len 32768 data: 5 4 2 2 2 2 2 2 253/300 Read block 2235 len 32768 254/300 Write block 2956 len 20480 data: 3 5 4 4 4 255/300 Read block 1065 len 12288 256/300 Read block 3434 len 12288 257/300 Read block 2562 len 8192 258/300 Write block 2990 len 20480 data: 3 3 3 2 2 259/300 Read block 293 len 36864 260/300 Write block 2723 len 32768 data: 2 2 2 3 3 3 3 3 261/300 Read block 3130 len 12288 262/300 Write block 1815 len 12288 data: 4 5 5 263/300 Write block 524 len 12288 data: 5 6 5 264/300 Read block 3234 len 36864 265/300 Write block 4959 len 28672 data: 3 3 3 4 4 4 3 266/300 Read block 2542 len 24576 267/300 Write block 886 len 8192 data: 3 3 268/300 Write block 1401 len 20480 data: 2 2 3 4 4 269/300 Write block 2114 len 28672 data: 5 5 4 4 4 4 4 270/300 Write block 114 len 24576 data: 2 2 3 3 3 3 271/300 Write block 4699 len 16384 data: 2 2 2 2 272/300 Read block 4623 len 36864 273/300 Read block 3013 len 4096 274/300 Write block 1402 len 8192 data: 3 4 275/300 Read block 2614 len 12288 276/300 Read block 4893 len 32768 277/300 Read block 2562 len 36864 278/300 Write block 2245 len 16384 data: 6 6 6 6 279/300 Read block 3873 len 20480 280/300 Read block 1178 len 4096 281/300 Read block 1257 len 20480 282/300 Write block 3766 len 20480 data: 3 2 2 2 3 283/300 Read block 3644 len 32768 284/300 Read block 1208 len 8192 285/300 Flush 286/300 Write block 637 len 12288 data: 2 2 2 287/300 Read block 2600 len 16384 288/300 Read block 4562 len 4096 289/300 Write block 2571 len 8192 data: 4 4 290/300 Read block 1196 len 40960 291/300 Write block 228 len 28672 data: 3 2 3 3 4 4 4 292/300 Flush 293/300 Read block 3550 len 24576 294/300 Write block 1177 len 24576 data: 5 5 6 4 4 3 295/300 Read block 1159 len 36864 296/300 Read block 1672 len 8192 297/300 Write block 3403 len 28672 data: 2 2 3 4 4 4 4 298/300 Read block 835 len 12288 299/300 Write block 870 len 16384 data: 5 4 3 4 300/300 Read block 4085 len 16384 301/300 Read block 1408 len 24576 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:43:37.870 INFO request completed, latency_us: 187, response_code: 204, uri: /start/cid/0, method: POST, req_id: c74e1a72-ce08-481c-9c72-b4fe812e2829, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3413 Acked Read 3 Sent Done Done false 3414 Acked Read 9 Sent Done Done false 3415 Acked Read 10 Sent Done Done false 3416 Acked Read 7 Sent Done Done false 3417 Acked Write 5 Sent Done Done false 3418 Acked Write 5 Sent Done Done false 3419 Acked Write 8 Sent Done Done false 3420 Acked Write 1 Sent Done Done false 3421 Acked Read 1 Sent Done Done false 3422 Acked Flush 0 Sent Done Done false 3423 Acked Read 7 Sent Done Done false 3424 Acked Read 7 Sent Done Done false 3425 Acked Read 8 Sent Done Done false 3426 Acked Write 1 Sent Done Done false 3427 Acked Read 2 Sent Done Done false 3428 Acked Flush 0 Sent Done Done false 3429 Acked Read 8 Sent Done Done false 3430 Acked Write 7 Sent Done Done false 3431 Acked Read 2 Sent Done Done false 3432 Acked Read 1 Sent Done Done false 3433 Acked Write 6 Sent Done Done false 3434 Acked Read 6 Sent Done Done false 3435 Acked Flush 0 Sent Done Done false 3436 Acked Read 9 Sent Done Done false 3437 Acked Read 3 Sent Done Done false 3438 Acked Write 3 Sent Done Done false 3439 Acked Read 4 Sent Done Done false 3440 Acked Read 3 Sent Done Done false 3441 Acked Write 9 Sent Done Done false 3442 Acked Read 6 Sent Done Done false 3443 Acked Write 9 Sent Done Done false 3444 Acked Read 5 Sent Done Done false 3445 Acked Flush 0 Sent Done Done false 3446 Acked Read 2 Sent Done Done false 3447 Acked Read 4 Sent Done Done false 3448 Acked Write 4 Sent Done Done false 3449 Acked Write 10 Sent Done Done false 3450 Acked Write 8 Sent Done Done false 3451 Acked Read 7 Sent Done Done false 3452 Acked Write 5 Sent Done Done false 3453 Acked Read 5 Sent Done Done false 3454 Acked Read 4 Sent Done Done false 3455 Acked Write 8 Sent Done Done false 3456 Acked Write 7 Sent Done Done false 3457 Acked Write 2 Sent Done Done false 3458 Acked Read 10 Sent Done Done false 3459 Acked Read 3 Sent Done Done false 3460 Acked Write 7 Sent Done Done false 3461 Acked Write 8 Sent Done Done false 3462 Acked Read 2 Sent Done Done false 3463 Acked Write 10 Sent Done Done false 3464 Acked Read 4 Sent Done Done false 3465 Acked Flush 0 Sent Done Done false 3466 Acked Flush 0 Sent Done Done false 3467 Acked Read 9 Sent Done Done false 3468 Acked Read 10 Sent Done Done false 3469 Acked Read 7 Sent Done Done false 3470 Acked Read 4 Sent Done Done false 3471 Acked Write 8 Sent Done Done false 3472 Acked Write 9 Sent Done Done false 3473 Acked Read 5 Sent Done Done false 3474 Acked Write 7 Sent Done Done false 3475 Acked Read 6 Sent Done Done false 3476 Acked Read 4 Sent Done Done false 3477 Acked Write 4 Sent Done Done false 3478 Acked Write 6 Sent Done Done false 3479 Acked Read 10 Sent Done Done false 3480 Acked Read 5 Sent Done Done false 3481 Acked Write 10 Sent Done Done false 3482 Acked Flush 0 Sent Done Done false 3483 Acked Write 4 Sent Done Done false 3484 Acked Read 2 Sent Done Done false 3485 Acked Write 7 Sent Done Done false 3486 Acked Read 8 Sent Done Done false 3487 Acked Write 8 Sent Done Done false 3488 Acked Read 4 Sent Done Done false 3489 Acked Read 5 Sent Done Done false 3490 Acked Read 2 Sent Done Done false 3491 Acked Read 1 Sent Done Done false 3492 Acked Read 3 Sent Done Done false 3493 Acked Flush 0 Sent Done Done false 3494 Acked Read 1 Sent Done Done false 3495 Acked Read 5 Sent Done Done false 3496 Acked Read 3 Sent Done Done false 3497 Acked Write 9 Sent Done Done false 3498 Acked Write 6 Sent Done Done false 3499 Acked Read 7 Sent Done Done false 3500 Acked Flush 0 Sent Done Done false 3501 Acked Write 1 Sent Done Done false 3502 Acked Write 7 Sent Done Done false 3503 Acked Write 10 Sent Done Done false 3504 Acked Write 1 Sent Done Done false 3505 Acked Write 7 Sent Done Done false 3506 Acked Read 8 Sent Done Done false 3507 Acked Read 3 Sent Done Done false 3508 Acked Read 4 Sent Done Done false 3509 Acked Read 7 Sent Done Done false 3510 Acked Write 1 Sent Done Done false 3511 Acked Flush 0 Sent Done Done false 3512 Acked Read 3 Sent Done Done false 3513 Acked Write 2 Sent Done Done false 3514 Acked Write 6 Sent Done Done false 3515 Acked Write 4 Sent Done Done false 3516 Acked Read 1 Sent Done Done false 3517 Acked Write 8 Sent Done Done false 3518 Acked Read 9 Sent Done Done false 3519 Acked Read 7 Sent Done Done false 3520 Acked Write 10 Sent Done Done false 3521 Acked Flush 0 Sent Done Done false 3522 Acked Write 4 Sent Done Done false 3523 Acked Flush 0 Sent Done Done false 3524 Acked Read 6 Sent Done Done false 3525 Acked Read 10 Sent Done Done false 3526 Acked Read 4 Sent Done Done false 3527 Acked Write 1 Sent Done Done false 3528 Acked Read 3 Sent Done Done false 3529 Acked Write 5 Sent Done Done false 3530 Acked Read 1 Sent Done Done false 3531 Acked Write 5 Sent Done Done false 3532 Acked Read 9 Sent Done Done false 3533 Acked Write 4 Sent Done Done false 3534 Acked Read 4 Sent Done Done false 3535 Acked Read 8 Sent Done Done false 3536 Acked Read 5 Sent Done Done false 3537 Acked Read 7 Sent Done Done false 3538 Acked Flush 0 Sent Done Done false 3539 Acked Write 1 Sent Done Done false 3540 Acked Read 1 Sent Done Done false 3541 Acked Read 10 Sent Done Done false 3542 Acked Read 6 Sent Done Done false 3543 Acked Read 1 Sent Done Done false 3544 Acked Write 1 Sent Done Done false 3545 Acked Write 5 Sent Done Done false 3546 Acked Write 1 Sent Done Done false 3547 Acked Read 5 Sent Done Done false 3548 Acked Read 7 Sent Done Done false 3549 Acked Read 9 Sent Done Done false 3550 Acked Flush 0 Sent Done Done false 3551 Acked Read 5 Sent Done Done false 3552 Acked Write 3 Sent Done Done false 3553 Acked Write 9 Sent Done Done false 3554 Acked Read 3 Sent Done Done false 3555 Acked Write 9 Sent Done Done false 3556 Acked Write 1 Sent Done Done false 3557 Acked Write 5 Sent Done Done false 3558 Acked Read 9 Sent Done Done false 3559 Acked Read 9 Sent Done Done false 3560 Acked Write 9 Sent Done Done false 3561 Acked Write 10 Sent Done Done false 3562 Acked Write 7 Sent Done Done false 3563 Acked Read 5 Sent Done Done false 3564 Acked Read 4 Sent Done Done false 3565 Acked Read 7 Sent Done Done false 3566 Acked Write 2 Sent Done Done false 3567 Acked Write 7 Sent Done Done false 3568 Acked Flush 0 Sent Done Done false 3569 Acked Write 7 Sent Done Done false 3570 Acked Write 10 Sent Done Done false 3571 Acked Read 8 Sent Done Done false 3572 Acked Read 8 Sent Done Done false 3573 Acked Read 5 Sent Done Done false 3574 Acked Read 4 Sent Done Done false 3575 Acked Read 4 Sent Done Done false 3576 Acked Read 2 Sent Done Done false 3577 Acked Read 2 Sent Done Done false 3578 Acked Read 2 Sent Done Done false 3579 Acked Read 10 Sent Done Done false 3580 Acked Read 7 Sent Done Done false 3581 Acked Write 5 Sent Done Done false 3582 Acked Read 8 Sent Done Done false 3583 Acked Read 3 Sent Done Done false 3584 Acked Write 4 Sent Done Done false 3585 Acked Write 9 Sent Done Done false 3586 Acked Read 4 Sent Done Done false 3587 Acked Write 3 Sent Done Done false 3588 Acked Read 5 Sent Done Done false 3589 Acked Read 6 Sent Done Done false 3590 Acked Write 5 Sent Done Done false 3591 Acked Flush 0 Sent Done Done false 3592 Acked Read 7 Sent Done Done false 3593 Acked Read 10 Sent Done Done false 3594 Acked Write 7 Sent Done Done false 3595 Acked Write 4 Sent Done Done false 3596 Acked Read 5 Sent Done Done false 3597 Acked Read 2 Sent Done Done false 3598 Acked Write 5 Sent Done Done false 3599 Acked Write 10 Sent Done Done false 3600 Acked Read 6 Sent Done Done false 3601 Acked Read 3 Sent Done Done false 3602 Acked Read 9 Sent Done Done false 3603 Acked Write 8 Sent Done Done false 3604 Acked Write 8 Sent Done Done false 3605 Acked Write 4 Sent Done Done false 3606 Acked Read 3 Sent Done Done false 3607 Acked Read 7 Sent Done Done false 3608 Acked Read 9 Sent Done Done false 3609 Acked Write 5 Sent Done Done false 3610 Acked Write 7 Sent Done Done false 3611 Acked Read 4 Sent Done Done false 3612 Acked Write 7 Sent Done Done false 3613 Acked Read 2 Sent Done Done false 3614 Acked Read 9 Sent Done Done false 3615 Acked Write 2 Sent Done Done false 3616 Acked Read 5 Sent Done Done false 3617 Acked Read 8 Sent Done Done false 3618 Acked Write 9 Sent Done Done false 3619 Acked Write 6 Sent Done Done false 3620 Acked Write 7 Sent Done Done false 3621 Acked Read 4 Sent Done Done false 3622 Acked Write 7 Sent Done Done false 3623 Acked Read 7 Sent Done Done false 3624 Acked Read 2 Sent Done Done false 3625 Acked Flush 0 Sent Done Done false 3626 Acked Write 9 Sent Done Done false 3627 Acked Flush 0 Sent Done Done false 3628 Acked Read 1 Sent Done Done false 3629 Acked Read 3 Sent Done Done false 3630 Acked Read 7 Sent Done Done false 3631 Acked Read 5 Sent Done Done false 3632 Acked Read 7 Sent Done Done false 3633 Acked Write 7 Sent Done Done false 3634 Acked Read 6 Sent Done Done false 3635 Acked Write 3 Sent Done Done false 3636 Acked Read 2 Sent Done Done false 3637 Acked Read 10 Sent Done Done false 3638 Acked Read 4 Sent Done Done false 3639 Acked Read 9 Sent Done Done false 3640 Acked Read 7 Sent Done Done false 3641 Acked Read 3 Sent Done Done false 3642 Acked Read 8 Sent Done Done false 3643 Acked Read 10 Sent Done Done false 3644 Acked Write 4 Sent Done Done false 3645 Acked Write 2 Sent Done Done false 3646 Acked Flush 0 Sent Done Done false 3647 Acked Write 5 Sent Done Done false 3648 Acked Read 4 Sent Done Done false 3649 Acked Read 2 Sent Done Done false 3650 Acked Write 8 Sent Done Done false 3651 Acked Write 5 Sent Done Done false 3652 Acked Read 10 Sent Done Done false 3653 Acked Read 5 Sent Done Done false 3654 Acked Write 7 Sent Done Done false 3655 Acked Read 2 Sent Done Done false 3656 Acked Write 4 Sent Done Done false 3657 Acked Write 1 Sent Done Done false 3658 Acked Write 10 Sent Done Done false 3659 Acked Write 7 Sent Done Done false 3660 Acked Write 3 Sent Done Done false 3661 Acked Write 4 Sent Done Done false 3662 Acked Write 2 Sent Done Done false 3663 Acked Read 2 Sent Done Done false 3664 Acked Write 8 Sent Done Done false 3665 Acked Read 8 Sent Done Done false 3666 Acked Write 5 Sent Done Done false 3667 Acked Read 3 Sent Done Done false 3668 Acked Read 3 Sent Done Done false 3669 Acked Read 2 Sent Done Done false 3670 Acked Write 5 Sent Done Done false 3671 Acked Read 9 Sent Done Done false 3672 Acked Write 8 Sent Done Done false 3673 Acked Read 3 Sent Done Done false 3674 Acked Write 3 Sent Done Done false 3675 Acked Write 3 Sent Done Done false 3676 Acked Read 9 Sent Done Done false 3677 Acked Write 7 Sent Done Done false 3678 Acked Read 6 Sent Done Done false 3679 Acked Write 2 Sent Done Done false 3680 Acked Write 5 Sent Done Done false 3681 Acked Write 7 Sent Done Done false 3682 Acked Write 6 Sent Done Done false 3683 Acked Write 4 Sent Done Done false 3684 Acked Read 9 Sent Done Done false 3685 Acked Read 1 Sent Done Done false 3686 Acked Write 2 Sent Done Done false 3687 Acked Read 3 Sent Done Done false 3688 Acked Read 8 Sent Done Done false 3689 Acked Read 9 Sent Done Done false 3690 Acked Write 4 Sent Done Done false 3691 Acked Read 5 Sent Done Done false 3692 Acked Read 1 Sent Done Done false 3693 Acked Read 5 Sent Done Done false 3694 Acked Write 5 Sent Done Done false 3695 Acked Read 8 Sent Done Done false 3696 Acked Read 2 Sent Done Done false 3697 Acked Flush 0 Sent Done Done false 3698 Acked Write 3 Sent Done Done false 3699 Acked Read 4 Sent Done Done false 3700 Acked Read 1 Sent Done Done false 3701 Acked Write 2 Sent Done Done false 3702 Acked Read 10 Sent Done Done false 3703 Acked Write 7 Sent Done Done false 3704 Acked Flush 0 Sent Done Done false 3705 Acked Read 6 Sent Done Done false 3706 Acked Write 6 Sent Done Done false 3707 Acked Read 9 Sent Done Done false 3708 Acked Read 2 Sent Done Done false 3709 Acked Write 7 Sent Done Done false 3710 Acked Read 3 Sent Done Done false 3711 Acked Write 4 Sent Done Done false 3712 Acked Read 4 Sent Done Done false 3713 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3412 3704 3704 Downstairs last five completed: 3412 3411 3410 3409 3408 Upstairs last five completed: 3713 3712 3711 3710 3709 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3413 Acked Read 3 Sent Done Done false 3414 Acked Read 9 Sent Done Done false 3415 Acked Read 10 Sent Done Done false 3416 Acked Read 7 Sent Done Done false 3417 Acked Write 5 Sent Done Done false 3418 Acked Write 5 Sent Done Done false 3419 Acked Write 8 Sent Done Done false 3420 Acked Write 1 Sent Done Done false 3421 Acked Read 1 Sent Done Done false 3422 Acked Flush 0 Sent Done Done false 3423 Acked Read 7 Sent Done Done false 3424 Acked Read 7 Sent Done Done false 3425 Acked Read 8 Sent Done Done false 3426 Acked Write 1 Sent Done Done false 3427 Acked Read 2 Sent Done Done false 3428 Acked Flush 0 Sent Done Done false 3429 Acked Read 8 Sent Done Done false 3430 Acked Write 7 Sent Done Done false 3431 Acked Read 2 Sent Done Done false 3432 Acked Read 1 Sent Done Done false 3433 Acked Write 6 Sent Done Done false 3434 Acked Read 6 Sent Done Done false 3435 Acked Flush 0 Sent Done Done false 3436 Acked Read 9 Sent Done Done false 3437 Acked Read 3 Sent Done Done false 3438 Acked Write 3 Sent Done Done false 3439 Acked Read 4 Sent Done Done false 3440 Acked Read 3 Sent Done Done false 3441 Acked Write 9 Sent Done Done false 3442 Acked Read 6 Sent Done Done false 3443 Acked Write 9 Sent Done Done false 3444 Acked Read 5 Sent Done Done false 3445 Acked Flush 0 Sent Done Done false 3446 Acked Read 2 Sent Done Done false 3447 Acked Read 4 Sent Done Done false 3448 Acked Write 4 Sent Done Done false 3449 Acked Write 10 Sent Done Done false 3450 Acked Write 8 Sent Done Done false 3451 Acked Read 7 Sent Done Done false 3452 Acked Write 5 Sent Done Done false 3453 Acked Read 5 Sent Done Done false 3454 Acked Read 4 Sent Done Done false 3455 Acked Write 8 Sent Done Done false 3456 Acked Write 7 Sent Done Done false 3457 Acked Write 2 Sent Done Done false 3458 Acked Read 10 Sent Done Done false 3459 Acked Read 3 Sent Done Done false 3460 Acked Write 7 Sent Done Done false 3461 Acked Write 8 Sent Done Done false 3462 Acked Read 2 Sent Done Done false 3463 Acked Write 10 Sent Done Done false 3464 Acked Read 4 Sent Done Done false 3465 Acked Flush 0 Sent Done Done false 3466 Acked Flush 0 Sent Done Done false 3467 Acked Read 9 Sent Done Done false 3468 Acked Read 10 Sent Done Done false 3469 Acked Read 7 Sent Done Done false 3470 Acked Read 4 Sent Done Done false 3471 Acked Write 8 Sent Done Done false 3472 Acked Write 9 Sent Done Done false 3473 Acked Read 5 Sent Done Done false 3474 Acked Write 7 Sent Done Done false 3475 Acked Read 6 Sent Done Done false 3476 Acked Read 4 Sent Done Done false 3477 Acked Write 4 Sent Done Done false 3478 Acked Write 6 Sent Done Done false 3479 Acked Read 10 Sent Done Done false 3480 Acked Read 5 Sent Done Done false 3481 Acked Write 10 Sent Done Done false 3482 Acked Flush 0 Sent Done Done false 3483 Acked Write 4 Sent Done Done false 3484 Acked Read 2 Sent Done Done false 3485 Acked Write 7 Sent Done Done false 3486 Acked Read 8 Sent Done Done false 3487 Acked Write 8 Sent Done Done false 3488 Acked Read 4 Sent Done Done false 3489 Acked Read 5 Sent Done Done false 3490 Acked Read 2 Sent Done Done false 3491 Acked Read 1 Sent Done Done false 3492 Acked Read 3 Sent Done Done false 3493 Acked Flush 0 Sent Done Done false 3494 Acked Read 1 Sent Done Done false 3495 Acked Read 5 Sent Done Done false 3496 Acked Read 3 Sent Done Done false 3497 Acked Write 9 Sent Done Done false 3498 Acked Write 6 Sent Done Done false 3499 Acked Read 7 Sent Done Done false 3500 Acked Flush 0 Sent Done Done false 3501 Acked Write 1 Sent Done Done false 3502 Acked Write 7 Sent Done Done false 3503 Acked Write 10 Sent Done Done false 3504 Acked Write 1 Sent Done Done false 3505 Acked Write 7 Sent Done Done false 3506 Acked Read 8 Sent Done Done false 3507 Acked Read 3 Sent Done Done false 3508 Acked Read 4 Sent Done Done false 3509 Acked Read 7 Sent Done Done false 3510 Acked Write 1 Sent Done Done false 3511 Acked Flush 0 Sent Done Done false 3512 Acked Read 3 Sent Done Done false 3513 Acked Write 2 Sent Done Done false 3514 Acked Write 6 Sent Done Done false 3515 Acked Write 4 Sent Done Done false 3516 Acked Read 1 Sent Done Done false 3517 Acked Write 8 Sent Done Done false 3518 Acked Read 9 Sent Done Done false 3519 Acked Read 7 Sent Done Done false 3520 Acked Write 10 Sent Done Done false 3521 Acked Flush 0 Sent Done Done false 3522 Acked Write 4 Sent Done Done false 3523 Acked Flush 0 Sent Done Done false 3524 Acked Read 6 Sent Done Done false 3525 Acked Read 10 Sent Done Done false 3526 Acked Read 4 Sent Done Done false 3527 Acked Write 1 Sent Done Done false 3528 Acked Read 3 Sent Done Done false 3529 Acked Write 5 Sent Done Done false 3530 Acked Read 1 Sent Done Done false 3531 Acked Write 5 Sent Done Done false 3532 Acked Read 9 Sent Done Done false 3533 Acked Write 4 Sent Done Done false 3534 Acked Read 4 Sent Done Done false 3535 Acked Read 8 Sent Done Done false 3536 Acked Read 5 Sent Done Done false 3537 Acked Read 7 Sent Done Done false 3538 Acked Flush 0 Sent Done Done false 3539 Acked Write 1 Sent Done Done false 3540 Acked Read 1 Sent Done Done false 3541 Acked Read 10 Sent Done Done false 3542 Acked Read 6 Sent Done Done false 3543 Acked Read 1 Sent Done Done false 3544 Acked Write 1 Sent Done Done false 3545 Acked Write 5 Sent Done Done false 3546 Acked Write 1 Sent Done Done false 3547 Acked Read 5 Sent Done Done false 3548 Acked Read 7 Sent Done Done false 3549 Acked Read 9 Sent Done Done false 3550 Acked Flush 0 Sent Done Done false 3551 Acked Read 5 Sent Done Done false 3552 Acked Write 3 Sent Done Done false 3553 Acked Write 9 Sent Done Done false 3554 Acked Read 3 Sent Done Done false 3555 Acked Write 9 Sent Done Done false 3556 Acked Write 1 Sent Done Done false 3557 Acked Write 5 Sent Done Done false 3558 Acked Read 9 Sent Done Done false 3559 Acked Read 9 Sent Done Done false 3560 Acked Write 9 Sent Done Done false 3561 Acked Write 10 Sent Done Done false 3562 Acked Write 7 Sent Done Done false 3563 Acked Read 5 Sent Done Done false 3564 Acked Read 4 Sent Done Done false 3565 Acked Read 7 Sent Done Done false 3566 Acked Write 2 Sent Done Done false 3567 Acked Write 7 Sent Done Done false 3568 Acked Flush 0 Sent Done Done false 3569 Acked Write 7 Sent Done Done false 3570 Acked Write 10 Sent Done Done false 3571 Acked Read 8 Sent Done Done false 3572 Acked Read 8 Sent Done Done false 3573 Acked Read 5 Sent Done Done false 3574 Acked Read 4 Sent Done Done false 3575 Acked Read 4 Sent Done Done false 3576 Acked Read 2 Sent Done Done false 3577 Acked Read 2 Sent Done Done false 3578 Acked Read 2 Sent Done Done false 3579 Acked Read 10 Sent Done Done false 3580 Acked Read 7 Sent Done Done false 3581 Acked Write 5 Sent Done Done false 3582 Acked Read 8 Sent Done Done false 3583 Acked Read 3 Sent Done Done false 3584 Acked Write 4 Sent Done Done false 3585 Acked Write 9 Sent Done Done false 3586 Acked Read 4 Sent Done Done false 3587 Acked Write 3 Sent Done Done false 3588 Acked Read 5 Sent Done Done false 3589 Acked Read 6 Sent Done Done false 3590 Acked Write 5 Sent Done Done false 3591 Acked Flush 0 Sent Done Done false 3592 Acked Read 7 Sent Done Done false 3593 Acked Read 10 Sent Done Done false 3594 Acked Write 7 Sent Done Done false 3595 Acked Write 4 Sent Done Done false 3596 Acked Read 5 Sent Done Done false 3597 Acked Read 2 Sent Done Done false 3598 Acked Write 5 Sent Done Done false 3599 Acked Write 10 Sent Done Done false 3600 Acked Read 6 Sent Done Done false 3601 Acked Read 3 Sent Done Done false 3602 Acked Read 9 Sent Done Done false 3603 Acked Write 8 Sent Done Done false 3604 Acked Write 8 Sent Done Done false 3605 Acked Write 4 Sent Done Done false 3606 Acked Read 3 Sent Done Done false 3607 Acked Read 7 Sent Done Done false 3608 Acked Read 9 Sent Done Done false 3609 Acked Write 5 Sent Done Done false 3610 Acked Write 7 Sent Done Done false 3611 Acked Read 4 Sent Done Done false 3612 Acked Write 7 Sent Done Done false 3613 Acked Read 2 Sent Done Done false 3614 Acked Read 9 Sent Done Done false 3615 Acked Write 2 Sent Done Done false 3616 Acked Read 5 Sent Done Done false 3617 Acked Read 8 Sent Done Done false 3618 Acked Write 9 Sent Done Done false 3619 Acked Write 6 Sent Done Done false 3620 Acked Write 7 Sent Done Done false 3621 Acked Read 4 Sent Done Done false 3622 Acked Write 7 Sent Done Done false 3623 Acked Read 7 Sent Done Done false 3624 Acked Read 2 Sent Done Done false 3625 Acked Flush 0 Sent Done Done false 3626 Acked Write 9 Sent Done Done false 3627 Acked Flush 0 Sent Done Done false 3628 Acked Read 1 Sent Done Done false 3629 Acked Read 3 Sent Done Done false 3630 Acked Read 7 Sent Done Done false 3631 Acked Read 5 Sent Done Done false 3632 Acked Read 7 Sent Done Done false 3633 Acked Write 7 Sent Done Done false 3634 Acked Read 6 Sent Done Done false 3635 Acked Write 3 Sent Done Done false 3636 Acked Read 2 Sent Done Done false 3637 Acked Read 10 Sent Done Done false 3638 Acked Read 4 Sent Done Done false 3639 Acked Read 9 Sent Done Done false 3640 Acked Read 7 Sent Done Done false 3641 Acked Read 3 Sent Done Done false 3642 Acked Read 8 Sent Done Done false 3643 Acked Read 10 Sent Done Done false 3644 Acked Write 4 Sent Done Done false 3645 Acked Write 2 Sent Done Done false 3646 Acked Flush 0 Sent Done Done false 3647 Acked Write 5 Sent Done Done false 3648 Acked Read 4 Sent Done Done false 3649 Acked Read 2 Sent Done Done false 3650 Acked Write 8 Sent Done Done false 3651 Acked Write 5 Sent Done Done false 3652 Acked Read 10 Sent Done Done false 3653 Acked Read 5 Sent Done Done false 3654 Acked Write 7 Sent Done Done false 3655 Acked Read 2 Sent Done Done false 3656 Acked Write 4 Sent Done Done false 3657 Acked Write 1 Sent Done Done false 3658 Acked Write 10 Sent Done Done false 3659 Acked Write 7 Sent Done Done false 3660 Acked Write 3 Sent Done Done false 3661 Acked Write 4 Sent Done Done false 3662 Acked Write 2 Sent Done Done false 3663 Acked Read 2 Sent Done Done false 3664 Acked Write 8 Sent Done Done false 3665 Acked Read 8 Sent Done Done false 3666 Acked Write 5 Sent Done Done false 3667 Acked Read 3 Sent Done Done false 3668 Acked Read 3 Sent Done Done false 3669 Acked Read 2 Sent Done Done false 3670 Acked Write 5 Sent Done Done false 3671 Acked Read 9 Sent Done Done false 3672 Acked Write 8 Sent Done Done false 3673 Acked Read 3 Sent Done Done false 3674 Acked Write 3 Sent Done Done false 3675 Acked Write 3 Sent Done Done false 3676 Acked Read 9 Sent Done Done false 3677 Acked Write 7 Sent Done Done false 3678 Acked Read 6 Sent Done Done false 3679 Acked Write 2 Sent Done Done false 3680 Acked Write 5 Sent Done Done false 3681 Acked Write 7 Sent Done Done false 3682 Acked Write 6 Sent Done Done false 3683 Acked Write 4 Sent Done Done false 3684 Acked Read 9 Sent Done Done false 3685 Acked Read 1 Sent Done Done false 3686 Acked Write 2 Sent Done Done false 3687 Acked Read 3 Sent Done Done false 3688 Acked Read 8 Sent Done Done false 3689 Acked Read 9 Sent Done Done false 3690 Acked Write 4 Sent Done Done false 3691 Acked Read 5 Sent Done Done false 3692 Acked Read 1 Sent Done Done false 3693 Acked Read 5 Sent Done Done false 3694 Acked Write 5 Sent Done Done false 3695 Acked Read 8 Sent Done Done false 3696 Acked Read 2 Sent Done Done false 3697 Acked Flush 0 Sent Done Done false 3698 Acked Write 3 Sent Done Done false 3699 Acked Read 4 Sent Done Done false 3700 Acked Read 1 Sent Done Done false 3701 Acked Write 2 Sent Done Done false 3702 Acked Read 10 Sent Done Done false 3703 Acked Write 7 Sent Done Done false 3704 Acked Flush 0 Sent Done Done false 3705 Acked Read 6 Sent Done Done false 3706 Acked Write 6 Sent Done Done false 3707 Acked Read 9 Sent Done Done false 3708 Acked Read 2 Sent Done Done false 3709 Acked Write 7 Sent Done Done false 3710 Acked Read 3 Sent Done Done false 3711 Acked Write 4 Sent Done Done false 3712 Acked Read 4 Sent Done Done false 3713 Acked Read 6 Sent Done Done false 3714 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3412 3714 3714 Downstairs last five completed: 3412 3411 3410 3409 3408 Upstairs last five completed: 3714 3713 3712 3711 3710 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:43.634936584Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:43.635001637Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:43.63502589Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:43.635047955Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:43.63510216Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3413 Acked Read 3 Sent Done Done false 3414 Acked Read 9 Sent Done Done false 3415 Acked Read 10 Sent Done Done false 3416 Acked Read 7 Sent Done Done false 3417 Acked Write 5 Sent Done Done false 3418 Acked Write 5 Sent Done Done false 3419 Acked Write 8 Sent Done Done false 3420 Acked Write 1 Sent Done Done false 3421 Acked Read 1 Sent Done Done false 3422 Acked Flush 0 Sent Done Done false 3423 Acked Read 7 Sent Done Done false 3424 Acked Read 7 Sent Done Done false 3425 Acked Read 8 Sent Done Done false 3426 Acked Write 1 Sent Done Done false 3427 Acked Read 2 Sent Done Done false 3428 Acked Flush 0 Sent Done Done false 3429 Acked Read 8 Sent Done Done false 3430 Acked Write 7 Sent Done Done false 3431 Acked Read 2 Sent Done Done false 3432 Acked Read 1 Sent Done Done false 3433 Acked Write 6 Sent Done Done false 3434 Acked Read 6 Sent Done Done false 3435 Acked Flush 0 Sent Done Done false 3436 Acked Read 9 Sent Done Done false 3437 Acked Read 3 Sent Done Done false 3438 Acked Write 3 Sent Done Done false 3439 Acked Read 4 Sent Done Done false 3440 Acked Read 3 Sent Done Done false 3441 Acked Write 9 Sent Done Done false 3442 Acked Read 6 Sent Done Done false 3443 Acked Write 9 Sent Done Done false 3444 Acked Read 5 Sent Done Done false 3445 Acked Flush 0 Sent Done Done false 3446 Acked Read 2 Sent Done Done false 3447 Acked Read 4 Sent Done Done false 3448 Acked Write 4 Sent Done Done false 3449 Acked Write 10 Sent Done Done false 3450 Acked Write 8 Sent Done Done false 3451 Acked Read 7 Sent Done Done false 3452 Acked Write 5 Sent Done Done false 3453 Acked Read 5 Sent Done Done false 3454 Acked Read 4 Sent Done Done false 3455 Acked Write 8 Sent Done Done false 3456 Acked Write 7 Sent Done Done false 3457 Acked Write 2 Sent Done Done false 3458 Acked Read 10 Sent Done Done false 3459 Acked Read 3 Sent Done Done false 3460 Acked Write 7 Sent Done Done false 3461 Acked Write 8 Sent Done Done false 3462 Acked Read 2 Sent Done Done false 3463 Acked Write 10 Sent Done Done false 3464 Acked Read 4 Sent Done Done false 3465 Acked Flush 0 Sent Done Done false 3466 Acked Flush 0 Sent Done Done false 3467 Acked Read 9 Sent Done Done false 3468 Acked Read 10 Sent Done Done false 3469 Acked Read 7 Sent Done Done false 3470 Acked Read 4 Sent Done Done false 3471 Acked Write 8 Sent Done Done false 3472 Acked Write 9 Sent Done Done false 3473 Acked Read 5 Sent Done Done false 3474 Acked Write 7 Sent Done Done false 3475 Acked Read 6 Sent Done Done false 3476 Acked Read 4 Sent Done Done false 3477 Acked Write 4 Sent Done Done false 3478 Acked Write 6 Sent Done Done false 3479 Acked Read 10 Sent Done Done false 3480 Acked Read 5 Sent Done Done false 3481 Acked Write 10 Sent Done Done false 3482 Acked Flush 0 Sent Done Done false 3483 Acked Write 4 Sent Done Done false 3484 Acked Read 2 Sent Done Done false 3485 Acked Write 7 Sent Done Done false 3486 Acked Read 8 Sent Done Done false 3487 Acked Write 8 Sent Done Done false 3488 Acked Read 4 Sent Done Done false 3489 Acked Read 5 Sent Done Done false 3490 Acked Read 2 Sent Done Done false 3491 Acked Read 1 Sent Done Done false 3492 Acked Read 3 Sent Done Done false 3493 Acked Flush 0 Sent Done Done false 3494 Acked Read 1 Sent Done Done false 3495 Acked Read 5 Sent Done Done false 3496 Acked Read 3 Sent Done Done false 3497 Acked Write 9 Sent Done Done false 3498 Acked Write 6 Sent Done Done false 3499 Acked Read 7 Sent Done Done false 3500 Acked Flush 0 Sent Done Done false 3501 Acked Write 1 Sent Done Done false 3502 Acked Write 7 Sent Done Done false 3503 Acked Write 10 Sent Done Done false 3504 Acked Write 1 Sent Done Done false 3505 Acked Write 7 Sent Done Done false 3506 Acked Read 8 Sent Done Done false 3507 Acked Read 3 Sent Done Done false 3508 Acked Read 4 Sent Done Done false 3509 Acked Read 7 Sent Done Done false 3510 Acked Write 1 Sent Done Done false 3511 Acked Flush 0 Sent Done Done false 3512 Acked Read 3 Sent Done Done false 3513 Acked Write 2 Sent Done Done false 3514 Acked Write 6 Sent Done Done false 3515 Acked Write 4 Sent Done Done false 3516 Acked Read 1 Sent Done Done false 3517 Acked Write 8 Sent Done Done false 3518 Acked Read 9 Sent Done Done false 3519 Acked Read 7 Sent Done Done false 3520 Acked Write 10 Sent Done Done false 3521 Acked Flush 0 Sent Done Done false 3522 Acked Write 4 Sent Done Done false 3523 Acked Flush 0 Sent Done Done false 3524 Acked Read 6 Sent Done Done false 3525 Acked Read 10 Sent Done Done false 3526 Acked Read 4 Sent Done Done false 3527 Acked Write 1 Sent Done Done false 3528 Acked Read 3 Sent Done Done false 3529 Acked Write 5 Sent Done Done false 3530 Acked Read 1 Sent Done Done false 3531 Acked Write 5 Sent Done Done false 3532 Acked Read 9 Sent Done Done false 3533 Acked Write 4 Sent Done Done false 3534 Acked Read 4 Sent Done Done false 3535 Acked Read 8 Sent Done Done false 3536 Acked Read 5 Sent Done Done false 3537 Acked Read 7 Sent Done Done false 3538 Acked Flush 0 Sent Done Done false 3539 Acked Write 1 Sent Done Done false 3540 Acked Read 1 Sent Done Done false 3541 Acked Read 10 Sent Done Done false 3542 Acked Read 6 Sent Done Done false 3543 Acked Read 1 Sent Done Done false 3544 Acked Write 1 Sent Done Done false 3545 Acked Write 5 Sent Done Done false 3546 Acked Write 1 Sent Done Done false 3547 Acked Read 5 Sent Done Done false 3548 Acked Read 7 Sent Done Done false 3549 Acked Read 9 Sent Done Done false 3550 Acked Flush 0 Sent Done Done false 3551 Acked Read 5 Sent Done Done false 3552 Acked Write 3 Sent Done Done false 3553 Acked Write 9 Sent Done Done false 3554 Acked Read 3 Sent Done Done false 3555 Acked Write 9 Sent Done Done false 3556 Acked Write 1 Sent Done Done false 3557 Acked Write 5 Sent Done Done false 3558 Acked Read 9 Sent Done Done false 3559 Acked Read 9 Sent Done Done false 3560 Acked Write 9 Sent Done Done false 3561 Acked Write 10 Sent Done Done false 3562 Acked Write 7 Sent Done Done false 3563 Acked Read 5 Sent Done Done false 3564 Acked Read 4 Sent Done Done false 3565 Acked Read 7 Sent Done Done false 3566 Acked Write 2 Sent Done Done false 3567 Acked Write 7 Sent Done Done false 3568 Acked Flush 0 Sent Done Done false 3569 Acked Write 7 Sent Done Done false 3570 Acked Write 10 Sent Done Done false 3571 Acked Read 8 Sent Done Done false 3572 Acked Read 8 Sent Done Done false 3573 Acked Read 5 Sent Done Done false 3574 Acked Read 4 Sent Done Done false 3575 Acked Read 4 Sent Done Done false 3576 Acked Read 2 Sent Done Done false 3577 Acked Read 2 Sent Done Done false 3578 Acked Read 2 Sent Done Done false 3579 Acked Read 10 Sent Done Done false 3580 Acked Read 7 Sent Done Done false 3581 Acked Write 5 Sent Done Done false 3582 Acked Read 8 Sent Done Done false 3583 Acked Read 3 Sent Done Done false 3584 Acked Write 4 Sent Done Done false 3585 Acked Write 9 Sent Done Done false 3586 Acked Read 4 Sent Done Done false 3587 Acked Write 3 Sent Done Done false 3588 Acked Read 5 Sent Done Done false 3589 Acked Read 6 Sent Done Done false 3590 Acked Write 5 Sent Done Done false 3591 Acked Flush 0 Sent Done Done false 3592 Acked Read 7 Sent Done Done false 3593 Acked Read 10 Sent Done Done false 3594 Acked Write 7 Sent Done Done false 3595 Acked Write 4 Sent Done Done false 3596 Acked Read 5 Sent Done Done false 3597 Acked Read 2 Sent Done Done false 3598 Acked Write 5 Sent Done Done false 3599 Acked Write 10 Sent Done Done false 3600 Acked Read 6 Sent Done Done false 3601 Acked Read 3 Sent Done Done false 3602 Acked Read 9 Sent Done Done false 3603 Acked Write 8 Sent Done Done false 3604 Acked Write 8 Sent Done Done false 3605 Acked Write 4 Sent Done Done false 3606 Acked Read 3 Sent Done Done false 3607 Acked Read 7 Sent Done Done false 3608 Acked Read 9 Sent Done Done false 3609 Acked Write 5 Sent Done Done false 3610 Acked Write 7 Sent Done Done false 3611 Acked Read 4 Sent Done Done false 3612 Acked Write 7 Sent Done Done false 3613 Acked Read 2 Sent Done Done false 3614 Acked Read 9 Sent Done Done false 3615 Acked Write 2 Sent Done Done false 3616 Acked Read 5 Sent Done Done false 3617 Acked Read 8 Sent Done Done false 3618 Acked Write 9 Sent Done Done false 3619 Acked Write 6 Sent Done Done false 3620 Acked Write 7 Sent Done Done false 3621 Acked Read 4 Sent Done Done false 3622 Acked Write 7 Sent Done Done false 3623 Acked Read 7 Sent Done Done false 3624 Acked Read 2 Sent Done Done false 3625 Acked Flush 0 Sent Done Done false 3626 Acked Write 9 Sent Done Done false 3627 Acked Flush 0 Sent Done Done false 3628 Acked Read 1 Sent Done Done false 3629 Acked Read 3 Sent Done Done false 3630 Acked Read 7 Sent Done Done false 3631 Acked Read 5 Sent Done Done false 3632 Acked Read 7 Sent Done Done false 3633 Acked Write 7 Sent Done Done false 3634 Acked Read 6 Sent Done Done false 3635 Acked Write 3 Sent Done Done false 3636 Acked Read 2 Sent Done Done false 3637 Acked Read 10 Sent Done Done false 3638 Acked Read 4 Sent Done Done false 3639 Acked Read 9 Sent Done Done false 3640 Acked Read 7 Sent Done Done false 3641 Acked Read 3 Sent Done Done false 3642 Acked Read 8 Sent Done Done false 3643 Acked Read 10 Sent Done Done false 3644 Acked Write 4 Sent Done Done false 3645 Acked Write 2 Sent Done Done false 3646 Acked Flush 0 Sent Done Done false 3647 Acked Write 5 Sent Done Done false 3648 Acked Read 4 Sent Done Done false 3649 Acked Read 2 Sent Done Done false 3650 Acked Write 8 Sent Done Done false 3651 Acked Write 5 Sent Done Done false 3652 Acked Read 10 Sent Done Done false 3653 Acked Read 5 Sent Done Done false 3654 Acked Write 7 Sent Done Done false 3655 Acked Read 2 Sent Done Done false 3656 Acked Write 4 Sent Done Done false 3657 Acked Write 1 Sent Done Done false 3658 Acked Write 10 Sent Done Done false 3659 Acked Write 7 Sent Done Done false 3660 Acked Write 3 Sent Done Done false 3661 Acked Write 4 Sent Done Done false 3662 Acked Write 2 Sent Done Done false 3663 Acked Read 2 Sent Done Done false 3664 Acked Write 8 Sent Done Done false 3665 Acked Read 8 Sent Done Done false 3666 Acked Write 5 Sent Done Done false 3667 Acked Read 3 Sent Done Done false 3668 Acked Read 3 Sent Done Done false 3669 Acked Read 2 Sent Done Done false 3670 Acked Write 5 Sent Done Done false 3671 Acked Read 9 Sent Done Done false 3672 Acked Write 8 Sent Done Done false 3673 Acked Read 3 Sent Done Done false 3674 Acked Write 3 Sent Done Done false 3675 Acked Write 3 Sent Done Done false 3676 Acked Read 9 Sent Done Done false 3677 Acked Write 7 Sent Done Done false 3678 Acked Read 6 Sent Done Done false 3679 Acked Write 2 Sent Done Done false 3680 Acked Write 5 Sent Done Done false 3681 Acked Write 7 Sent Done Done false 3682 Acked Write 6 Sent Done Done false 3683 Acked Write 4 Sent Done Done false 3684 Acked Read 9 Sent Done Done false 3685 Acked Read 1 Sent Done Done false 3686 Acked Write 2 Sent Done Done false 3687 Acked Read 3 Sent Done Done false 3688 Acked Read 8 Sent Done Done false 3689 Acked Read 9 Sent Done Done false 3690 Acked Write 4 Sent Done Done false 3691 Acked Read 5 Sent Done Done false 3692 Acked Read 1 Sent Done Done false 3693 Acked Read 5 Sent Done Done false 3694 Acked Write 5 Sent Done Done false 3695 Acked Read 8 Sent Done Done false 3696 Acked Read 2 Sent Done Done false 3697 Acked Flush 0 Sent Done Done false 3698 Acked Write 3 Sent Done Done false 3699 Acked Read 4 Sent Done Done false 3700 Acked Read 1 Sent Done Done false 3701 Acked Write 2 Sent Done Done false 3702 Acked Read 10 Sent Done Done false 3703 Acked Write 7 Sent Done Done false 3704 Acked Flush 0 Sent Done Done false 3705 Acked Read 6 Sent Done Done false 3706 Acked Write 6 Sent Done Done false 3707 Acked Read 9 Sent Done Done false 3708 Acked Read 2 Sent Done Done false 3709 Acked Write 7 Sent Done Done false 3710 Acked Read 3 Sent Done Done false 3711 Acked Write 4 Sent Done Done false 3712 Acked Read 4 Sent Done Done false 3713 Acked Read 6 Sent Done Done false 3714 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3412 3714 3714 Downstairs last five completed: 3412 3411 3410 3409 3408 Upstairs last five completed: 3714 3713 3712 3711 3710 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3413 Acked Read 3 Sent Done Done false 3414 Acked Read 9 Sent Done Done false 3415 Acked Read 10 Sent Done Done false 3416 Acked Read 7 Sent Done Done false 3417 Acked Write 5 Sent Done Done false 3418 Acked Write 5 Sent Done Done false 3419 Acked Write 8 Sent Done Done false 3420 Acked Write 1 Sent Done Done false 3421 Acked Read 1 Sent Done Done false 3422 Acked Flush 0 Sent Done Done false 3423 Acked Read 7 Sent Done Done false 3424 Acked Read 7 Sent Done Done false 3425 Acked Read 8 Sent Done Done false 3426 Acked Write 1 Sent Done Done false 3427 Acked Read 2 Sent Done Done false 3428 Acked Flush 0 Sent Done Done false 3429 Acked Read 8 Sent Done Done false 3430 Acked Write 7 Sent Done Done false 3431 Acked Read 2 Sent Done Done false 3432 Acked Read 1 Sent Done Done false 3433 Acked Write 6 Sent Done Done false 3434 Acked Read 6 Sent Done Done false 3435 Acked Flush 0 Sent Done Done false 3436 Acked Read 9 Sent Done Done false 3437 Acked Read 3 Sent Done Done false 3438 Acked Write 3 Sent Done Done false 3439 Acked Read 4 Sent Done Done false 3440 Acked Read 3 Sent Done Done false 3441 Acked Write 9 Sent Done Done false 3442 Acked Read 6 Sent Done Done false 3443 Acked Write 9 Sent Done Done false 3444 Acked Read 5 Sent Done Done false 3445 Acked Flush 0 Sent Done Done false 3446 Acked Read 2 Sent Done Done false 3447 Acked Read 4 Sent Done Done false 3448 Acked Write 4 Sent Done Done false 3449 Acked Write 10 Sent Done Done false 3450 Acked Write 8 Sent Done Done false 3451 Acked Read 7 Sent Done Done false 3452 Acked Write 5 Sent Done Done false 3453 Acked Read 5 Sent Done Done false 3454 Acked Read 4 Sent Done Done false 3455 Acked Write 8 Sent Done Done false 3456 Acked Write 7 Sent Done Done false 3457 Acked Write 2 Sent Done Done false 3458 Acked Read 10 Sent Done Done false 3459 Acked Read 3 Sent Done Done false 3460 Acked Write 7 Sent Done Done false 3461 Acked Write 8 Sent Done Done false 3462 Acked Read 2 Sent Done Done false 3463 Acked Write 10 Sent Done Done false 3464 Acked Read 4 Sent Done Done false 3465 Acked Flush 0 Sent Done Done false 3466 Acked Flush 0 Sent Done Done false 3467 Acked Read 9 Sent Done Done false 3468 Acked Read 10 Sent Done Done false 3469 Acked Read 7 Sent Done Done false 3470 Acked Read 4 Sent Done Done false 3471 Acked Write 8 Sent Done Done false 3472 Acked Write 9 Sent Done Done false 3473 Acked Read 5 Sent Done Done false 3474 Acked Write 7 Sent Done Done false 3475 Acked Read 6 Sent Done Done false 3476 Acked Read 4 Sent Done Done false 3477 Acked Write 4 Sent Done Done false 3478 Acked Write 6 Sent Done Done false 3479 Acked Read 10 Sent Done Done false 3480 Acked Read 5 Sent Done Done false 3481 Acked Write 10 Sent Done Done false 3482 Acked Flush 0 Sent Done Done false 3483 Acked Write 4 Sent Done Done false 3484 Acked Read 2 Sent Done Done false 3485 Acked Write 7 Sent Done Done false 3486 Acked Read 8 Sent Done Done false 3487 Acked Write 8 Sent Done Done false 3488 Acked Read 4 Sent Done Done false 3489 Acked Read 5 Sent Done Done false 3490 Acked Read 2 Sent Done Done false 3491 Acked Read 1 Sent Done Done false 3492 Acked Read 3 Sent Done Done false 3493 Acked Flush 0 Sent Done Done false 3494 Acked Read 1 Sent Done Done false 3495 Acked Read 5 Sent Done Done false 3496 Acked Read 3 Sent Done Done false 3497 Acked Write 9 Sent Done Done false 3498 Acked Write 6 Sent Done Done false 3499 Acked Read 7 Sent Done Done false 3500 Acked Flush 0 Sent Done Done false 3501 Acked Write 1 Sent Done Done false 3502 Acked Write 7 Sent Done Done false 3503 Acked Write 10 Sent Done Done false 3504 Acked Write 1 Sent Done Done false 3505 Acked Write 7 Sent Done Done false 3506 Acked Read 8 Sent Done Done false 3507 Acked Read 3 Sent Done Done false 3508 Acked Read 4 Sent Done Done false 3509 Acked Read 7 Sent Done Done false 3510 Acked Write 1 Sent Done Done false 3511 Acked Flush 0 Sent Done Done false 3512 Acked Read 3 Sent Done Done false 3513 Acked Write 2 Sent Done Done false 3514 Acked Write 6 Sent Done Done false 3515 Acked Write 4 Sent Done Done false 3516 Acked Read 1 Sent Done Done false 3517 Acked Write 8 Sent Done Done false 3518 Acked Read 9 Sent Done Done false 3519 Acked Read 7 Sent Done Done false 3520 Acked Write 10 Sent Done Done false 3521 Acked Flush 0 Sent Done Done false 3522 Acked Write 4 Sent Done Done false 3523 Acked Flush 0 Sent Done Done false 3524 Acked Read 6 Sent Done Done false 3525 Acked Read 10 Sent Done Done false 3526 Acked Read 4 Sent Done Done false 3527 Acked Write 1 Sent Done Done false 3528 Acked Read 3 Sent Done Done false 3529 Acked Write 5 Sent Done Done false 3530 Acked Read 1 Sent Done Done false 3531 Acked Write 5 Sent Done Done false 3532 Acked Read 9 Sent Done Done false 3533 Acked Write 4 Sent Done Done false 3534 Acked Read 4 Sent Done Done false 3535 Acked Read 8 Sent Done Done false 3536 Acked Read 5 Sent Done Done false 3537 Acked Read 7 Sent Done Done false 3538 Acked Flush 0 Sent Done Done false 3539 Acked Write 1 Sent Done Done false 3540 Acked Read 1 Sent Done Done false 3541 Acked Read 10 Sent Done Done false 3542 Acked Read 6 Sent Done Done false 3543 Acked Read 1 Sent Done Done false 3544 Acked Write 1 Sent Done Done false 3545 Acked Write 5 Sent Done Done false 3546 Acked Write 1 Sent Done Done false 3547 Acked Read 5 Sent Done Done false 3548 Acked Read 7 Sent Done Done false 3549 Acked Read 9 Sent Done Done false 3550 Acked Flush 0 Sent Done Done false 3551 Acked Read 5 Sent Done Done false 3552 Acked Write 3 Sent Done Done false 3553 Acked Write 9 Sent Done Done false 3554 Acked Read 3 Sent Done Done false 3555 Acked Write 9 Sent Done Done false 3556 Acked Write 1 Sent Done Done false 3557 Acked Write 5 Sent Done Done false 3558 Acked Read 9 Sent Done Done false 3559 Acked Read 9 Sent Done Done false 3560 Acked Write 9 Sent Done Done false 3561 Acked Write 10 Sent Done Done false 3562 Acked Write 7 Sent Done Done false 3563 Acked Read 5 Sent Done Done false 3564 Acked Read 4 Sent Done Done false 3565 Acked Read 7 Sent Done Done false 3566 Acked Write 2 Sent Done Done false 3567 Acked Write 7 Sent Done Done false 3568 Acked Flush 0 Sent Done Done false 3569 Acked Write 7 Sent Done Done false 3570 Acked Write 10 Sent Done Done false 3571 Acked Read 8 Sent Done Done false 3572 Acked Read 8 Sent Done Done false 3573 Acked Read 5 Sent Done Done false 3574 Acked Read 4 Sent Done Done false 3575 Acked Read 4 Sent Done Done false 3576 Acked Read 2 Sent Done Done false 3577 Acked Read 2 Sent Done Done false 3578 Acked Read 2 Sent Done Done false 3579 Acked Read 10 Sent Done Done false 3580 Acked Read 7 Sent Done Done false 3581 Acked Write 5 Sent Done Done false 3582 Acked Read 8 Sent Done Done false 3583 Acked Read 3 Sent Done Done false 3584 Acked Write 4 Sent Done Done false 3585 Acked Write 9 Sent Done Done false 3586 Acked Read 4 Sent Done Done false 3587 Acked Write 3 Sent Done Done false 3588 Acked Read 5 Sent Done Done false 3589 Acked Read 6 Sent Done Done false 3590 Acked Write 5 Sent Done Done false 3591 Acked Flush 0 Sent Done Done false 3592 Acked Read 7 Sent Done Done false 3593 Acked Read 10 Sent Done Done false 3594 Acked Write 7 Sent Done Done false 3595 Acked Write 4 Sent Done Done false 3596 Acked Read 5 Sent Done Done false 3597 Acked Read 2 Sent Done Done false 3598 Acked Write 5 Sent Done Done false 3599 Acked Write 10 Sent Done Done false 3600 Acked Read 6 Sent Done Done false 3601 Acked Read 3 Sent Done Done false 3602 Acked Read 9 Sent Done Done false 3603 Acked Write 8 Sent Done Done false 3604 Acked Write 8 Sent Done Done false 3605 Acked Write 4 Sent Done Done false 3606 Acked Read 3 Sent Done Done false 3607 Acked Read 7 Sent Done Done false 3608 Acked Read 9 Sent Done Done false 3609 Acked Write 5 Sent Done Done false 3610 Acked Write 7 Sent Done Done false 3611 Acked Read 4 Sent Done Done false 3612 Acked Write 7 Sent Done Done false 3613 Acked Read 2 Sent Done Done false 3614 Acked Read 9 Sent Done Done false 3615 Acked Write 2 Sent Done Done false 3616 Acked Read 5 Sent Done Done false 3617 Acked Read 8 Sent Done Done false 3618 Acked Write 9 Sent Done Done false 3619 Acked Write 6 Sent Done Done false 3620 Acked Write 7 Sent Done Done false 3621 Acked Read 4 Sent Done Done false 3622 Acked Write 7 Sent Done Done false 3623 Acked Read 7 Sent Done Done false 3624 Acked Read 2 Sent Done Done false 3625 Acked Flush 0 Sent Done Done false 3626 Acked Write 9 Sent Done Done false 3627 Acked Flush 0 Sent Done Done false 3628 Acked Read 1 Sent Done Done false 3629 Acked Read 3 Sent Done Done false 3630 Acked Read 7 Sent Done Done false 3631 Acked Read 5 Sent Done Done false 3632 Acked Read 7 Sent Done Done false 3633 Acked Write 7 Sent Done Done false 3634 Acked Read 6 Sent Done Done false 3635 Acked Write 3 Sent Done Done false 3636 Acked Read 2 Sent Done Done false 3637 Acked Read 10 Sent Done Done false 3638 Acked Read 4 Sent Done Done false 3639 Acked Read 9 Sent Done Done false 3640 Acked Read 7 Sent Done Done false 3641 Acked Read 3 Sent Done Done false 3642 Acked Read 8 Sent Done Done false 3643 Acked Read 10 Sent Done Done false 3644 Acked Write 4 Sent Done Done false 3645 Acked Write 2 Sent Done Done false 3646 Acked Flush 0 Sent Done Done false 3647 Acked Write 5 Sent Done Done false 3648 Acked Read 4 Sent Done Done false 3649 Acked Read 2 Sent Done Done false 3650 Acked Write 8 Sent Done Done false 3651 Acked Write 5 Sent Done Done false 3652 Acked Read 10 Sent Done Done false 3653 Acked Read 5 Sent Done Done false 3654 Acked Write 7 Sent Done Done false 3655 Acked Read 2 Sent Done Done false 3656 Acked Write 4 Sent Done Done false 3657 Acked Write 1 Sent Done Done false 3658 Acked Write 10 Sent Done Done false 3659 Acked Write 7 Sent Done Done false 3660 Acked Write 3 Sent Done Done false 3661 Acked Write 4 Sent Done Done false 3662 Acked Write 2 Sent Done Done false 3663 Acked Read 2 Sent Done Done false 3664 Acked Write 8 Sent Done Done false 3665 Acked Read 8 Sent Done Done false 3666 Acked Write 5 Sent Done Done false 3667 Acked Read 3 Sent Done Done false 3668 Acked Read 3 Sent Done Done false 3669 Acked Read 2 Sent Done Done false 3670 Acked Write 5 Sent Done Done false 3671 Acked Read 9 Sent Done Done false 3672 Acked Write 8 Sent Done Done false 3673 Acked Read 3 Sent Done Done false 3674 Acked Write 3 Sent Done Done false 3675 Acked Write 3 Sent Done Done false 3676 Acked Read 9 Sent Done Done false 3677 Acked Write 7 Sent Done Done false 3678 Acked Read 6 Sent Done Done false 3679 Acked Write 2 Sent Done Done false 3680 Acked Write 5 Sent Done Done false 3681 Acked Write 7 Sent Done Done false 3682 Acked Write 6 Sent Done Done false 3683 Acked Write 4 Sent Done Done false 3684 Acked Read 9 Sent Done Done false 3685 Acked Read 1 Sent Done Done false 3686 Acked Write 2 Sent Done Done false 3687 Acked Read 3 Sent Done Done false 3688 Acked Read 8 Sent Done Done false 3689 Acked Read 9 Sent Done Done false 3690 Acked Write 4 Sent Done Done false 3691 Acked Read 5 Sent Done Done false 3692 Acked Read 1 Sent Done Done false 3693 Acked Read 5 Sent Done Done false 3694 Acked Write 5 Sent Done Done false 3695 Acked Read 8 Sent Done Done false 3696 Acked Read 2 Sent Done Done false 3697 Acked Flush 0 Sent Done Done false 3698 Acked Write 3 Sent Done Done false 3699 Acked Read 4 Sent Done Done false 3700 Acked Read 1 Sent Done Done false 3701 Acked Write 2 Sent Done Done false 3702 Acked Read 10 Sent Done Done false 3703 Acked Write 7 Sent Done Done false 3704 Acked Flush 0 Sent Done Done false 3705 Acked Read 6 Sent Done Done false 3706 Acked Write 6 Sent Done Done false 3707 Acked Read 9 Sent Done Done false 3708 Acked Read 2 Sent Done Done false 3709 Acked Write 7 Sent Done Done false 3710 Acked Read 3 Sent Done Done false 3711 Acked Write 4 Sent Done Done false 3712 Acked Read 4 Sent Done Done false 3713 Acked Read 6 Sent Done Done false 3714 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3412 3714 3714 Downstairs last five completed: 3412 3411 3410 3409 3408 Upstairs last five completed: 3714 3713 3712 3711 3710 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.636158796Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:56999)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.636349695Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:53.637315476Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.63782303Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.637845284Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 3412","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.637860209Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 3412","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.638085879Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.638110682Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.638124978Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 3412","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.638138515Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 3412","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.638356646Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3714 3713 3712 3711 3710 Upstairs last five completed: 3714 3713 3712 3711 3710 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:43:53.878 INFO request completed, latency_us: [8820] Got stop action so:false kr:false 359, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 17eeabdf-1063-4a0e-859b-5009f13a52b2, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:53.879585006Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:53.879625125Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:43:53.8796421Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.879661635Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.879677611Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:43:53.879 INFO request completed, latency_us:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:43:53.879757789Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 122, response_code: 200, uri: /state/cid/1, method: GET, req_id: 290f12b1-a160-4a49-8e2c-17fccab4d444, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:43:57.882 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: da2336da-d4c4-45ed-8087-7f4c27a6652c, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 3086 len 36864 002/300 Read block 2104 len 20480 003/300 Write block 1318 len 12288 data: 4 3 4 004/300 Read block 3125 len 20480 005/300 Read block 261 len 12288 006/300 Write block 3825 len 20480 data: 2 2 2 2 2 007/300 Write block 673 len 36864 data: 2 2 2 2 2 3 3 3 3 008/300 Read block 3180 len 40960 009/300 Write block 3774 len 12288 data: 4 4 4 010/300 Write block 3032 len 40960 data: 3 3 3 3 3 3 2 2 2 2 011/300 Write block 3803 len 32768 data: 2 2 3 3 4 4 4 3 012/300 Read block 1089 len 32768 013/300 Read block 3521 len 8192 014/300 Write block 493 len 12288 data: 5 4 3 015/300 Write block 506 len 20480 data: 2 3 3 3 3 016/300 Read block 2010 len 12288 017/300 Read block 4205 len 16384 018/300 Read block 99 len 24576 019/300 Write block 3307 len 12288 data: 2 2 2 020/300 Write block 1396 len 32768 data: 2 2 2 2 2 3 4 5 021/300 Write block 1040 len 4096 data: 3 022/300 Read block 436 len 28672 023/300 Read block 2475 len 40960 024/300 Read block 4870 len 16384 025/300 Read block 1826 len 32768 026/300 Read block 2594 len 24576 027/300 Write block 1207 len 36864 data: 3 2 2 2 2 2 2 2 2 028/300 Write block 740 len 36864 data: 3 3 3 3 3 4 5 5 5 029/300 Read block 3169 len 40960 030/300 Read block 1191 len 40960 031/300 Read block 3573 len 24576 032/300 Flush 033/300 Write block 1752 len 4096 data: 4 034/300 Write block 3712 len 4096 data: 2 035/300 Read block 2698 len 28672 036/300 Read block 3370 len 16384 037/300 Read block 2030 len 4096 038/300 Flush 039/300 Read block 3615 len 16384 040/300 Write block 769 len 16384 data: 3 4 3 3 041/300 Read block 4225 len 40960 042/300 Read block 891 len 8192 043/300 Read block 2623 len 24576 044/300 Read block 4601 len 16384 045/300 Write block 4791 len 20480 data: 3 3 3 4 4 046/300 Write block 3859 len 12288 data: 3 3 3 047/300 Write block 2620 len 36864 data: 3 3 3 2 2 2 2 2 2 048/300 Write block 435 len 20480 data: 4 3 4 4 5 049/300 Write block 2459 len 12288 data: 5 5 4 050/300 Write block 4898 len 8192 data: 3 4 051/300 Write block 680 len 36864 data: 4 4 3 3 3 3 3 3 4 052/300 Write block 1887 len 28672 data: 4 4 4 5 5 5 5 053/300 Read block 39 len 4096 054/300 Read block 4033 len 28672 055/300 Read block 4007 len 40960 056/300 Read block 369 len 4096 057/300 Read block 4293 len 12288 058/300 Write block 3662 len 16384 data: 4 3 2 2 059/300 Read block 3493 len 28672 060/300 Write block 996 len 16384 data: 3 3 3 3 061/300 Read block 3419 len 4096 062/300 Write block 220 len 8192 data: 4 4 063/300 Write block 1486 len 8192 data: 3 3 064/300 Write block 4355 len 40960 data: 5 7 6 6 5 4 3 4 4 4 065/300 Read block 1482 len 24576 066/300 Read block 4625 len 20480 067/300 Write block 1476 len 8192 data: 3 3 068/300 Write block 3045 len 40960 data: 4 4 4 3 3 3 3 2 2 2 069/300 Read block 684 len 20480 070/300 Read block 2655 len 16384 071/300 Write block 4989 len 28672 data: 3 3 3 3 3 4 3 072/300 Write block 4110 len 16384 data: 2 2 2 2 073/300 Write block 4035 len 28672 data: 3 2 2 2 2 2 2 074/300 Read block 2761 len 40960 075/300 Write block 2768 len 28672 data: 3 2 2 4 4 4 3 076/300 Read block 3168 len 40960 077/300 Read block 804 len 24576 078/300 Read block 3161 len 32768 079/300 Read block 4376 len 28672 080/300 Write block 1148 len 16384 data: 4 3 3 3 081/300 Read block 1535 len 20480 082/300 Write block 4435 len 4096 data: 4 083/300 Read block 637 len 16384 084/300 Read block 847 len 4096 085/300 Read block 477 len 8192 086/300 Read block 210 len 40960 087/300 Write block 4375 len 16384 data: 2 2 2 2 088/300 Write block 616 len 24576 data: 2 2 2 2 3 3 089/300 Write block 2938 len 24576 data: 4 4 4 4 4 3 090/300 Read block 218 len 24576 091/300 Write block 1459 len 20480 data: 3 3 3 2 3 092/300 Read block 2037 len 12288 093/300 Write block 3734 len 16384 data: 4 4 4 4 094/300 Write block 641 len 12288 data: 2 2 2 095/300 Read block 2960 len 4096 096/300 Write block 711 len 32768 data: 4 4 2 3 3 3 3 3 097/300 Read block 3643 len 4096 098/300 Read block 4051 len 24576 099/300 Read block 3995 len 16384 100/300 Flush 101/300 Read block 3339 len 12288 102/300 Write block 4992 len 4096 data: 4 103/300 Write block 1220 len 16384 data: 3 3 5 5 104/300 Write block 4828 len 24576 data: 4 3 3 3 3 3 105/300 Write block 3812 len 16384 data: 4 4 4 4 106/300 Write block 1870 len 12288 data: 4 5 4 107/300 Write block 2545 len 12288 data: 5 4 5 108/300 Read block 4113 len 16384 109/300 Write block 2018 len 36864 data: 3 2 3 3 4 4 4 4 3 110/300 Write block 3721 len 32768 data: 3 3 3 3 3 4 4 3 111/300 Read block 1195 len 32768 112/300 Read block 27 len 8192 113/300 Flush 114/300 Read block 1369 len 4096 115/300 Write block 1931 len 40960 data: 3 3 3 3 2 2 2 3 3 3 116/300 Write block 1467 len 32768 data: 5 6 5 5 5 5 3 3 117/300 Read block 4113 len 20480 118/300 Read block 141 len 20480 119/300 Read block 2604 len 32768 120/300 Write block 418 len 36864 data: 3 3 3 3 3 3 3 3 3 121/300 Write block 1945 len 24576 data: 2 3 3 3 3 3 122/300 Read block 1981 len 16384 123/300 Flush 124/300 Write block 4938 len 28672 data: 6 6 6 6 6 6 5 125/300 Read block 129 len 36864 126/300 Read block 3914 len 20480 127/300 Write block 1460 len 24576 data: 4 4 3 4 3 3 128/300 Read block 1716 len 4096 129/300 Write block 1097 len 28672 data: 4 3 3 2 2 2 2 130/300 Read block 785 len 24576 131/300 Write block 1683 len 32768 data: 3 3 3 3 3 3 3 3 132/300 Write block 4800 len 28672 data: 4 4 4 4 4 4 3 133/300 Read block 919 len 36864 134/300 Read block 4920 len 8192 135/300 Read block 2564 len 20480 136/300 Read block 4021 len 12288 137/300 Read block 3947 len 32768 138/300 Write block 3174 len 12288 data: 2 2 2 139/300 Flush 140/300 Read block 708 len 8192 141/300 Write block 4178 len 40960 data: 2 3 3 3 4 4 4 4 5 4 142/300 Read block 4734 len 8192 143/300 Flush 144/300 Write block 3291 len 16384 data: 4 3 3 2 145/300 Flush 146/300 Write block 3547 len 40960 data: 3 3 4 3 3 4 3 3 4 4 147/300 Write block 2902 len 4096 data: 3 148/300 Read block 1585 len 12288 149/300 Flush 150/300 Write block 1217 len 8192 data: 3 3 151/300 Write block 703 len 40960 data: 2 2 3 2 3 3 3 3 5 5 152/300 Write block 1536 len 40960 data: 3 4 4 5 7 7 7 7 5 4 153/300 Flush 154/300 Flush 155/300 Write block 4428 len 32768 data: 4 3 2 2 2 4 4 5 156/300 Read block 951 len 4096 157/300 Read block 3249 len 4096 158/300 Write block 3204 len 12288 data: 2 3 3 159/300 Write block 2107 len 8192 data: 4 4 160/300 Read block 1441 len 40960 161/300 Write block 4941 len 28672 data: 7 7 7 6 5 3 3 162/300 Flush 163/300 Write block 4283 len 40960 data: 3 3 3 3 3 3 3 3 3 3 164/300 Write block 64 len 28672 data: 3 4 4 3 4 4 4 165/300 Write block 4269 len 40960 data: 2 3 3 4 4 3 4 4 3 4 166/300 Write block 1663 len 16384 data: 2 2 2 2 167/300 Write block 3630 len 16384 data: 2 2 2 2 168/300 Flush 169/300 Read block 2034 len 4096 170/300 Write block 540 len 8192 data: 3 3 171/300 Read block 3810 len 28672 172/300 Read block 59 len 24576 173/300 Write block 4123 len 32768 data: 2 3 3 3 3 4 4 4 174/300 Write block 1611 len 28672 data: 2 2 2 3 3 2 3 175/300 Read block 4738 len 20480 176/300 Read block 439 len 36864 177/300 Write block 2247 len 24576 data: 7 7 5 3 3 3 178/300 Read block 3448 len 16384 179/300 Read block 1884 len 28672 180/300 Flush 181/300 Write block 4662 len 8192 data: 3 3 182/300 Flush 183/300 Read block 4414 len 4096 184/300 Read block 947 len 8192 185/300 Read block 438 len 40960 186/300 Read block 4725 len 16384 187/300 Write block 1219 len 24576 data: 3 4 4 6 6 5 188/300 Write block 191 len 20480 data: 3 3 2 2 2 189/300 Write block 2716 len 24576 data: 2 2 2 2 2 2 190/300 Read block 3228 len 40960 191/300 Read block 1313 len 32768 192/300 Read block 1016 len 16384 193/300 Read block 2919 len 16384 194/300 Read block 2719 len 16384 195/300 Read block 4451 len 12288 196/300 Write block 976 len 4096 data: 2 197/300 Write block 3633 len 12288 data: 3 2 3 198/300 Read block 1967 len 24576 199/300 Read block 1530 len 28672 200/300 Read block 3379 len 32768 201/300 Read block 1930 len 24576 202/300 Write block 4144 len 20480 data: 2 2 2 2 3 203/300 Read block 4888 len 36864 204/300 Write block 3753 len 32768 data: 4 4 3 3 3 3 4 5 205/300 Read block 693 len 24576 206/300 Read block 822 len 12288 207/300 Write block 3668 len 28672 data: 4 4 4 4 2 3 3 208/300 Read block 1197 len 40960 209/300 Write block 3225 len 32768 data: 2 2 2 2 2 2 2 2 210/300 Write block 1910 len 4096 data: 4 211/300 Write block 4209 len 32768 data: 4 4 4 4 4 4 4 3 212/300 Write block 31 len 32768 data: 4 4 4 4 3 2 2 2 213/300 Write block 1499 len 16384 data: 4 5 4 3 214/300 Read block 3115 len 32768 215/300 Read block 2374 len 24576 216/300 Read block 463 len 36864 217/300 Write block 1343 len 20480 data: 2 2 2 2 2 218/300 Flush 219/300 Write block 1155 len 4096 data: 2 220/300 Write block 3949 len 32768 data: 3 3 3 3 3 4 3 3 221/300 Read block 59 len 8192 222/300 Read block 2306 len 8192 223/300 Read block 4106 len 32768 224/300 Write block 4400 len 4096 data: 4 225/300 Read block 2878 len 12288 226/300 Read block 1228 len 36864 227/300 Read block 3720 len 8192 228/300 Read block 3136 len 24576 229/300 Read block 2853 len 4096 230/300 Read block 2862 len 16384 231/300 Read block 4757 len 4096 232/300 Read block 2443 len 28672 233/300 Write block 1910 len 20480 data: 5 4 4 4 3 234/300 Write block 223 len 40960 data: 3 4 4 4 4 4 3 4 4 5 235/300 Write block 744 len 4096 data: 4 236/300 Flush 237/300 Read block 3385 len 4096 238/300 Read block 3800 len 4096 239/300 Flush 240/300 Write block 3508 len 40960 data: 2 2 2 3 3 2 2 2 2 2 241/300 Flush 242/300 Write block 1279 len 24576 data: 3 3 3 3 3 4 243/300 Read block 3847 len 20480 244/300 Flush 245/300 Write block 2976 len 40960 data: 5 3 3 3 3 3 3 3 3 3 246/300 Read block 1187 len 32768 247/300 Read block 2298 len 24576 248/300 Read block 4019 len 4096 249/300 Write block 2805 len 36864 data: 3 4 3 3 3 4 3 3 3 250/300 Write block 3030 len 40960 data: 3 3 4 4 4 4 4 4 3 3 251/300 Write block 4611 len 12288 data: 2 2 2 252/300 Flush 253/300 Flush 254/300 Read block 1052 len 32768 255/300 Read block 4849 len 32768 256/300 Read block 3482 len 28672 257/300 Write block 1849 len 16384 data: 3 3 3 4 258/300 Read block 1302 len 16384 259/300 Write block 1751 len 40960 data: 4 5 4 3 2 2 2 2 2 2 260/300 Read block 3947 len 16384 261/300 Write block 555 len 8192 data: 2 2 262/300 Write block 322 len 28672 data: 2 2 2 2 2 2 2 263/300 Read block 2000 len 16384 264/300 Read block 552 len 8192 265/300 Write block 3230 len 20480 data: 3 3 3 3 3 266/300 Read block 1147 len 28672 267/300 Read block 2635 len 12288 268/300 Read block 2391 len 12288 269/300 Flush 270/300 Read block 628 len 24576 271/300 Read block 1045 len 40960 272/300 Read block 4039 len 36864 273/300 Flush 274/300 Write block 2043 len 16384 data: 4 5 5 4 275/300 Read block 478 len 12288 276/300 Read block 2080 len 16384 277/300 Write block 473 len 24576 data: 3 2 2 3 3 3 278/300 Flush 279/300 Read block 2693 len 32768 280/300 Read block 2987 len 28672 281/300 Write block 1361 len 40960 data: 4 4 4 4 3 4 4 5 5 5 282/300 Write block 1469 len 20480 data: 6 6 6 6 4 283/300 Read block 2224 len 16384 284/300 Read block 3759 len 4096 285/300 Write block 2217 len 4096 data: 3 286/300 Read block 2415 len 24576 287/300 Write block 3857 len 28672 data: 2 3 4 4 4 2 2 288/300 Flush 289/300 Write block 168 len 32768 data: 3 3 3 3 2 2 2 2 290/300 Read block 4018 len 32768 291/300 Write block 4554 len 8192 data: 3 3 292/300 Read block 3616 len 24576 293/300 Write block 1372 len 40960 data: 4 4 3 3 3 3 3 2 2 2 294/300 Read block 4320 len 8192 295/300 Write block 4414 len 24576 data: 2 2 2 3 3 5 296/300 Read block 1993 len 40960 297/300 Read block 2918 len 20480 298/300 Write block 3707 len 28672 data: 3 2 2 2 2 3 2 299/300 Write block 1590 len 8192 data: 3 2 300/300 Write block 124 len 24576 data: 3 3 3 3 4 4 301/300 Read block 214 len 40960 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:43:58.109 INFO request completed, latency_us: 190, response_code: 204, uri: /start/cid/1, method: POST, req_id: 77038622-606a-4c6b-8bae-ca0b8c1cf90f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3715 Acked Read 9 Done Sent Done false 3716 Acked Read 5 Done Sent Done false 3717 Acked Write 3 Done Sent Done false 3718 Acked Read 5 Done Sent Done false 3719 Acked Read 3 Done Sent Done false 3720 Acked Write 5 Done Sent Done false 3721 Acked Write 9 Done Sent Done false 3722 Acked Read 10 Done Sent Done false 3723 Acked Write 3 Done Sent Done false 3724 Acked Write 10 Done Sent Done false 3725 Acked Write 8 Done Sent Done false 3726 Acked Read 8 Done Sent Done false 3727 Acked Read 2 Done Sent Done false 3728 Acked Write 3 Done Sent Done false 3729 Acked Write 5 Done Sent Done false 3730 Acked Read 3 Done Sent Done false 3731 Acked Read 4 Done Sent Done false 3732 Acked Read 6 Done Sent Done false 3733 Acked Write 3 Done Sent Done false 3734 Acked Write 8 Done Sent Done false 3735 Acked Write 1 Done Sent Done false 3736 Acked Read 7 Done Sent Done false 3737 Acked Read 10 Done Sent Done false 3738 Acked Read 4 Done Sent Done false 3739 Acked Read 8 Done Sent Done false 3740 Acked Read 6 Done Sent Done false 3741 Acked Write 9 Done Sent Done false 3742 Acked Write 9 Done Sent Done false 3743 Acked Read 10 Done Sent Done false 3744 Acked Read 10 Done Sent Done false 3745 Acked Read 6 Done Sent Done false 3746 Acked Flush 0 Done Sent Done false 3747 Acked Write 1 Done Sent Done false 3748 Acked Write 1 Done Sent Done false 3749 Acked Read 7 Done Sent Done false 3750 Acked Read 4 Done Sent Done false 3751 Acked Read 1 Done Sent Done false 3752 Acked Flush 0 Done Sent Done false 3753 Acked Read 4 Done Sent Done false 3754 Acked Write 4 Done Sent Done false 3755 Acked Read 10 Done Sent Done false 3756 Acked Read 2 Done Sent Done false 3757 Acked Read 6 Done Sent Done false 3758 Acked Read 4 Done Sent Done false 3759 Acked Write 5 Done Sent Done false 3760 Acked Write 3 Done Sent Done false 3761 Acked Write 9 Done Sent Done false 3762 Acked Write 5 Done Sent Done false 3763 Acked Write 3 Done Sent Done false 3764 Acked Write 2 Done Sent Done false 3765 Acked Write 9 Done Sent Done false 3766 Acked Write 7 Done Sent Done false 3767 Acked Read 1 Done Sent Done false 3768 Acked Read 7 Done Sent Done false 3769 Acked Read 10 Done Sent Done false 3770 Acked Read 1 Done Sent Done false 3771 Acked Read 3 Done Sent Done false 3772 Acked Write 4 Done Sent Done false 3773 Acked Read 7 Done Sent Done false 3774 Acked Write 4 Done Sent Done false 3775 Acked Read 1 Done Sent Done false 3776 Acked Write 2 Done Sent Done false 3777 Acked Write 2 Done Sent Done false 3778 Acked Write 10 Done Sent Done false 3779 Acked Read 6 Done Sent Done false 3780 Acked Read 5 Done Sent Done false 3781 Acked Write 2 Done Sent Done false 3782 Acked Write 10 Done Sent Done false 3783 Acked Read 5 Done Sent Done false 3784 Acked Read 4 Done Sent Done false 3785 Acked Write 7 Done Sent Done false 3786 Acked Write 4 Done Sent Done false 3787 Acked Write 7 Done Sent Done false 3788 Acked Read 10 Done Sent Done false 3789 Acked Write 7 Done Sent Done false 3790 Acked Read 10 Done Sent Done false 3791 Acked Read 6 Done Sent Done false 3792 Acked Read 8 Done Sent Done false 3793 Acked Read 7 Done Sent Done false 3794 Acked Write 4 Done Sent Done false 3795 Acked Read 5 Done Sent Done false 3796 Acked Write 1 Done Sent Done false 3797 Acked Read 4 Done Sent Done false 3798 Acked Read 1 Done Sent Done false 3799 Acked Read 2 Done Sent Done false 3800 Acked Read 10 Done Sent Done false 3801 Acked Write 4 Done Sent Done false 3802 Acked Write 6 Done Sent Done false 3803 Acked Write 6 Done Sent Done false 3804 Acked Read 6 Done Sent Done false 3805 Acked Write 5 Done Sent Done false 3806 Acked Read 3 Done Sent Done false 3807 Acked Write 4 Done Sent Done false 3808 Acked Write 3 Done Sent Done false 3809 Acked Read 1 Done Sent Done false 3810 Acked Write 8 Done Sent Done false 3811 Acked Read 1 Done Sent Done false 3812 Acked Read 6 Done Sent Done false 3813 Acked Read 4 Done Sent Done false 3814 Acked Flush 0 Done Sent Done false 3815 Acked Read 3 Done Sent Done false 3816 Acked Write 1 Done Sent Done false 3817 Acked Write 4 Done Sent Done false 3818 Acked Write 6 Done Sent Done false 3819 Acked Write 4 Done Sent Done false 3820 Acked Write 3 Done Sent Done false 3821 Acked Write 3 Done Sent Done false 3822 Acked Read 4 Done Sent Done false 3823 Acked Write 9 Done Sent Done false 3824 Acked Write 8 Done Sent Done false 3825 Acked Read 8 Done Sent Done false 3826 Acked Read 2 Done Sent Done false 3827 Acked Flush 0 Done Sent Done false 3828 Acked Read 1 Done Sent Done false 3829 Acked Write 10 Done Sent Done false 3830 Acked Write 8 Done Sent Done false 3831 Acked Read 5 Done Sent Done false 3832 Acked Read 5 Done Sent Done false 3833 Acked Read 8 Done Sent Done false 3834 Acked Write 9 Done Sent Done false 3835 Acked Write 6 Done Sent Done false 3836 Acked Read 4 Done Sent Done false 3837 Acked Flush 0 Done Sent Done false 3838 Acked Write 7 Done Sent Done false 3839 Acked Read 9 Done Sent Done false 3840 Acked Read 5 Done Sent Done false 3841 Acked Write 6 Done Sent Done false 3842 Acked Read 1 Done Sent Done false 3843 Acked Write 7 Done Sent Done false 3844 Acked Read 6 Done Sent Done false 3845 Acked Write 8 Done Sent Done false 3846 Acked Write 7 Done Sent Done false 3847 Acked Read 9 Done Sent Done false 3848 Acked Read 2 Done Sent Done false 3849 Acked Read 5 Done Sent Done false 3850 Acked Read 3 Done Sent Done false 3851 Acked Read 8 Done Sent Done false 3852 Acked Write 3 Done Sent Done false 3853 Acked Flush 0 Done Sent Done false 3854 Acked Read 2 Done Sent Done false 3855 Acked Write 10 Done Sent Done false 3856 Acked Read 2 Done Sent Done false 3857 Acked Flush 0 Done Sent Done false 3858 Acked Write 4 Done Sent Done false 3859 Acked Flush 0 Done Sent Done false 3860 Acked Write 10 Done Sent Done false 3861 Acked Write 1 Done Sent Done false 3862 Acked Read 3 Done Sent Done false 3863 Acked Flush 0 Done Sent Done false 3864 Acked Write 2 Done Sent Done false 3865 Acked Write 10 Done Sent Done false 3866 Acked Write 10 Done Sent Done false 3867 Acked Flush 0 Done Sent Done false 3868 Acked Flush 0 Done Sent Done false 3869 Acked Write 8 Done Sent Done false 3870 Acked Read 1 Done Sent Done false 3871 Acked Read 1 Done Sent Done false 3872 Acked Write 3 Done Sent Done false 3873 Acked Write 2 Done Sent Done false 3874 Acked Read 10 Done Sent Done false 3875 Acked Write 7 Done Sent Done false 3876 Acked Flush 0 Done Sent Done false 3877 Acked Write 10 Done Sent Done false 3878 Acked Write 7 Done Sent Done false 3879 Acked Write 10 Done Sent Done false 3880 Acked Write 4 Done Sent Done false 3881 Acked Write 4 Done Sent Done false 3882 Acked Flush 0 Done Sent Done false 3883 Acked Read 1 Done Sent Done false 3884 Acked Write 2 Done Sent Done false 3885 Acked Read 7 Done Sent Done false 3886 Acked Read 6 Done Sent Done false 3887 Acked Write 8 Done Sent Done false 3888 Acked Write 7 Done Sent Done false 3889 Acked Read 5 Done Sent Done false 3890 Acked Read 9 Done Sent Done false 3891 Acked Write 6 Done Sent Done false 3892 Acked Read 4 Done Sent Done false 3893 Acked Read 7 Done Sent Done false 3894 Acked Flush 0 Done Sent Done false 3895 Acked Write 2 Done Sent Done false 3896 Acked Flush 0 Done Sent Done false 3897 Acked Read 1 Done Sent Done false 3898 Acked Read 2 Done Sent Done false 3899 Acked Read 10 Done Sent Done false 3900 Acked Read 4 Done Sent Done false 3901 Acked Write 6 Done Sent Done false 3902 Acked Write 5 Done Sent Done false 3903 Acked Write 6 Done Sent Done false 3904 Acked Read 10 Done Sent Done false 3905 Acked Read 8 Done Sent Done false 3906 Acked Read 4 Done Sent Done false 3907 Acked Read 4 Done Sent Done false 3908 Acked Read 4 Done Sent Done false 3909 Acked Read 3 Done Sent Done false 3910 Acked Write 1 Done Sent Done false 3911 Acked Write 3 Done Sent Done false 3912 Acked Read 6 Done Sent Done false 3913 Acked Read 7 Done Sent Done false 3914 Acked Read 8 Done Sent Done false 3915 Acked Read 6 Done Sent Done false 3916 Acked Write 5 Done Sent Done false 3917 Acked Read 9 Done Sent Done false 3918 Acked Write 8 Done Sent Done false 3919 Acked Read 6 Done Sent Done false 3920 Acked Read 3 Done Sent Done false 3921 Acked Write 7 Done Sent Done false 3922 Acked Read 10 Done Sent Done false 3923 Acked Write 8 Done Sent Done false 3924 Acked Write 1 Done Sent Done false 3925 Acked Write 8 Done Sent Done false 3926 Acked Write 8 Done Sent Done false 3927 Acked Write 4 Done Sent Done false 3928 Acked Read 8 Done Sent Done false 3929 Acked Read 6 Done Sent Done false 3930 Acked Read 9 Done Sent Done false 3931 Acked Write 5 Done Sent Done false 3932 Acked Flush 0 Done Sent Done false 3933 Acked Write 1 Done Sent Done false 3934 Acked Write 8 Done Sent Done false 3935 Acked Read 2 Done Sent Done false 3936 Acked Read 2 Done Sent Done false 3937 Acked Read 8 Done Sent Done false 3938 Acked Write 1 Done Sent Done false 3939 Acked Read 3 Done Sent Done false 3940 Acked Read 9 Done Sent Done false 3941 Acked Read 2 Done Sent Done false 3942 Acked Read 6 Done Sent Done false 3943 Acked Read 1 Done Sent Done false 3944 Acked Read 4 Done Sent Done false 3945 Acked Read 1 Done Sent Done false 3946 Acked Read 7 Done Sent Done false 3947 Acked Write 5 Done Sent Done false 3948 Acked Write 10 Done Sent Done false 3949 Acked Write 1 Done Sent Done false 3950 Acked Flush 0 Done Sent Done false 3951 Acked Read 1 Done Sent Done false 3952 Acked Read 1 Done Sent Done false 3953 Acked Flush 0 Done Sent Done false 3954 Acked Write 10 Done Sent Done false 3955 Acked Flush 0 Done Sent Done false 3956 Acked Write 6 Done Sent Done false 3957 Acked Read 5 Done Sent Done false 3958 Acked Flush 0 Done Sent Done false 3959 Acked Write 10 Done Sent Done false 3960 Acked Read 8 Done Sent Done false 3961 Acked Read 6 Done Sent Done false 3962 Acked Read 1 Done Sent Done false 3963 Acked Write 9 Done Sent Done false 3964 Acked Write 10 Done Sent Done false 3965 Acked Write 3 Done Sent Done false 3966 Acked Flush 0 Done Sent Done false 3967 Acked Flush 0 Done Sent Done false 3968 Acked Read 8 Done Sent Done false 3969 Acked Read 8 Done Sent Done false 3970 Acked Read 7 Done Sent Done false 3971 Acked Write 4 Done Sent Done false 3972 Acked Read 4 Done Sent Done false 3973 Acked Write 10 Done Sent Done false 3974 Acked Read 4 Done Sent Done false 3975 Acked Write 2 Done Sent Done false 3976 Acked Write 7 Done Sent Done false 3977 Acked Read 4 Done Sent Done false 3978 Acked Read 2 Done Sent Done false 3979 Acked Write 5 Done Sent Done false 3980 Acked Read 7 Done Sent Done false 3981 Acked Read 3 Done Sent Done false 3982 Acked Read 3 Done Sent Done false 3983 Acked Flush 0 Done Sent Done false 3984 Acked Read 6 Done Sent Done false 3985 Acked Read 10 Done Sent Done false 3986 Acked Read 9 Done Sent Done false 3987 Acked Flush 0 Done Sent Done false 3988 Acked Write 4 Done Sent Done false 3989 Acked Read 3 Done Sent Done false 3990 Acked Read 4 Done Sent Done false 3991 Acked Write 6 Done Sent Done false 3992 Acked Flush 0 Done Sent Done false 3993 Acked Read 8 Done Sent Done false 3994 Acked Read 7 Done Sent Done false 3995 Acked Write 10 Done Sent Done false 3996 Acked Write 5 Done Sent Done false 3997 Acked Read 4 Done Sent Done false 3998 Acked Read 1 Done Sent Done false 3999 Acked Write 1 Done Sent Done false 4000 Acked Read 6 Done Sent Done false 4001 Acked Write 7 Done Sent Done false 4002 Acked Flush 0 Done Sent Done false 4003 Acked Write 8 Done Sent Done false 4004 Acked Read 8 Done Sent Done false 4005 Acked Write 2 Done Sent Done false 4006 Acked Read 6 Done Sent Done false 4007 Acked Write 10 Done Sent Done false 4008 Acked Read 2 Done Sent Done false 4009 Acked Write 6 Done Sent Done false 4010 Acked Read 10 Done Sent Done false 4011 Acked Read 5 Done Sent Done false 4012 Acked Write 7 Done Sent Done false 4013 Acked Write 2 Done Sent Done false 4014 Acked Write 6 Done Sent Done false 4015 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4002 3714 4002 Downstairs last five completed: 3714 3713 3712 3711 3710 Upstairs last five completed: 4015 4014 4013 4012 4011 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3715 Acked Read 9 Done Sent Done false 3716 Acked Read 5 Done Sent Done false 3717 Acked Write 3 Done Sent Done false 3718 Acked Read 5 Done Sent Done false 3719 Acked Read 3 Done Sent Done false 3720 Acked Write 5 Done Sent Done false 3721 Acked Write 9 Done Sent Done false 3722 Acked Read 10 Done Sent Done false 3723 Acked Write 3 Done Sent Done false 3724 Acked Write 10 Done Sent Done false 3725 Acked Write 8 Done Sent Done false 3726 Acked Read 8 Done Sent Done false 3727 Acked Read 2 Done Sent Done false 3728 Acked Write 3 Done Sent Done false 3729 Acked Write 5 Done Sent Done false 3730 Acked Read 3 Done Sent Done false 3731 Acked Read 4 Done Sent Done false 3732 Acked Read 6 Done Sent Done false 3733 Acked Write 3 Done Sent Done false 3734 Acked Write 8 Done Sent Done false 3735 Acked Write 1 Done Sent Done false 3736 Acked Read 7 Done Sent Done false 3737 Acked Read 10 Done Sent Done false 3738 Acked Read 4 Done Sent Done false 3739 Acked Read 8 Done Sent Done false 3740 Acked Read 6 Done Sent Done false 3741 Acked Write 9 Done Sent Done false 3742 Acked Write 9 Done Sent Done false 3743 Acked Read 10 Done Sent Done false 3744 Acked Read 10 Done Sent Done false 3745 Acked Read 6 Done Sent Done false 3746 Acked Flush 0 Done Sent Done false 3747 Acked Write 1 Done Sent Done false 3748 Acked Write 1 Done Sent Done false 3749 Acked Read 7 Done Sent Done false 3750 Acked Read 4 Done Sent Done false 3751 Acked Read 1 Done Sent Done false 3752 Acked Flush 0 Done Sent Done false 3753 Acked Read 4 Done Sent Done false 3754 Acked Write 4 Done Sent Done false 3755 Acked Read 10 Done Sent Done false 3756 Acked Read 2 Done Sent Done false 3757 Acked Read 6 Done Sent Done false 3758 Acked Read 4 Done Sent Done false 3759 Acked Write 5 Done Sent Done false 3760 Acked Write 3 Done Sent Done false 3761 Acked Write 9 Done Sent Done false 3762 Acked Write 5 Done Sent Done false 3763 Acked Write 3 Done Sent Done false 3764 Acked Write 2 Done Sent Done false 3765 Acked Write 9 Done Sent Done false 3766 Acked Write 7 Done Sent Done false 3767 Acked Read 1 Done Sent Done false 3768 Acked Read 7 Done Sent Done false 3769 Acked Read 10 Done Sent Done false 3770 Acked Read 1 Done Sent Done false 3771 Acked Read 3 Done Sent Done false 3772 Acked Write 4 Done Sent Done false 3773 Acked Read 7 Done Sent Done false 3774 Acked Write 4 Done Sent Done false 3775 Acked Read 1 Done Sent Done false 3776 Acked Write 2 Done Sent Done false 3777 Acked Write 2 Done Sent Done false 3778 Acked Write 10 Done Sent Done false 3779 Acked Read 6 Done Sent Done false 3780 Acked Read 5 Done Sent Done false 3781 Acked Write 2 Done Sent Done false 3782 Acked Write 10 Done Sent Done false 3783 Acked Read 5 Done Sent Done false 3784 Acked Read 4 Done Sent Done false 3785 Acked Write 7 Done Sent Done false 3786 Acked Write 4 Done Sent Done false 3787 Acked Write 7 Done Sent Done false 3788 Acked Read 10 Done Sent Done false 3789 Acked Write 7 Done Sent Done false 3790 Acked Read 10 Done Sent Done false 3791 Acked Read 6 Done Sent Done false 3792 Acked Read 8 Done Sent Done false 3793 Acked Read 7 Done Sent Done false 3794 Acked Write 4 Done Sent Done false 3795 Acked Read 5 Done Sent Done false 3796 Acked Write 1 Done Sent Done false 3797 Acked Read 4 Done Sent Done false 3798 Acked Read 1 Done Sent Done false 3799 Acked Read 2 Done Sent Done false 3800 Acked Read 10 Done Sent Done false 3801 Acked Write 4 Done Sent Done false 3802 Acked Write 6 Done Sent Done false 3803 Acked Write 6 Done Sent Done false 3804 Acked Read 6 Done Sent Done false 3805 Acked Write 5 Done Sent Done false 3806 Acked Read 3 Done Sent Done false 3807 Acked Write 4 Done Sent Done false 3808 Acked Write 3 Done Sent Done false 3809 Acked Read 1 Done Sent Done false 3810 Acked Write 8 Done Sent Done false 3811 Acked Read 1 Done Sent Done false 3812 Acked Read 6 Done Sent Done false 3813 Acked Read 4 Done Sent Done false 3814 Acked Flush 0 Done Sent Done false 3815 Acked Read 3 Done Sent Done false 3816 Acked Write 1 Done Sent Done false 3817 Acked Write 4 Done Sent Done false 3818 Acked Write 6 Done Sent Done false 3819 Acked Write 4 Done Sent Done false 3820 Acked Write 3 Done Sent Done false 3821 Acked Write 3 Done Sent Done false 3822 Acked Read 4 Done Sent Done false 3823 Acked Write 9 Done Sent Done false 3824 Acked Write 8 Done Sent Done false 3825 Acked Read 8 Done Sent Done false 3826 Acked Read 2 Done Sent Done false 3827 Acked Flush 0 Done Sent Done false 3828 Acked Read 1 Done Sent Done false 3829 Acked Write 10 Done Sent Done false 3830 Acked Write 8 Done Sent Done false 3831 Acked Read 5 Done Sent Done false 3832 Acked Read 5 Done Sent Done false 3833 Acked Read 8 Done Sent Done false 3834 Acked Write 9 Done Sent Done false 3835 Acked Write 6 Done Sent Done false 3836 Acked Read 4 Done Sent Done false 3837 Acked Flush 0 Done Sent Done false 3838 Acked Write 7 Done Sent Done false 3839 Acked Read 9 Done Sent Done false 3840 Acked Read 5 Done Sent Done false 3841 Acked Write 6 Done Sent Done false 3842 Acked Read 1 Done Sent Done false 3843 Acked Write 7 Done Sent Done false 3844 Acked Read 6 Done Sent Done false 3845 Acked Write 8 Done Sent Done false 3846 Acked Write 7 Done Sent Done false 3847 Acked Read 9 Done Sent Done false 3848 Acked Read 2 Done Sent Done false 3849 Acked Read 5 Done Sent Done false 3850 Acked Read 3 Done Sent Done false 3851 Acked Read 8 Done Sent Done false 3852 Acked Write 3 Done Sent Done false 3853 Acked Flush 0 Done Sent Done false 3854 Acked Read 2 Done Sent Done false 3855 Acked Write 10 Done Sent Done false 3856 Acked Read 2 Done Sent Done false 3857 Acked Flush 0 Done Sent Done false 3858 Acked Write 4 Done Sent Done false 3859 Acked Flush 0 Done Sent Done false 3860 Acked Write 10 Done Sent Done false 3861 Acked Write 1 Done Sent Done false 3862 Acked Read 3 Done Sent Done false 3863 Acked Flush 0 Done Sent Done false 3864 Acked Write 2 Done Sent Done false 3865 Acked Write 10 Done Sent Done false 3866 Acked Write 10 Done Sent Done false 3867 Acked Flush 0 Done Sent Done false 3868 Acked Flush 0 Done Sent Done false 3869 Acked Write 8 Done Sent Done false 3870 Acked Read 1 Done Sent Done false 3871 Acked Read 1 Done Sent Done false 3872 Acked Write 3 Done Sent Done false 3873 Acked Write 2 Done Sent Done false 3874 Acked Read 10 Done Sent Done false 3875 Acked Write 7 Done Sent Done false 3876 Acked Flush 0 Done Sent Done false 3877 Acked Write 10 Done Sent Done false 3878 Acked Write 7 Done Sent Done false 3879 Acked Write 10 Done Sent Done false 3880 Acked Write 4 Done Sent Done false 3881 Acked Write 4 Done Sent Done false 3882 Acked Flush 0 Done Sent Done false 3883 Acked Read 1 Done Sent Done false 3884 Acked Write 2 Done Sent Done false 3885 Acked Read 7 Done Sent Done false 3886 Acked Read 6 Done Sent Done false 3887 Acked Write 8 Done Sent Done false 3888 Acked Write 7 Done Sent Done false 3889 Acked Read 5 Done Sent Done false 3890 Acked Read 9 Done Sent Done false 3891 Acked Write 6 Done Sent Done false 3892 Acked Read 4 Done Sent Done false 3893 Acked Read 7 Done Sent Done false 3894 Acked Flush 0 Done Sent Done false 3895 Acked Write 2 Done Sent Done false 3896 Acked Flush 0 Done Sent Done false 3897 Acked Read 1 Done Sent Done false 3898 Acked Read 2 Done Sent Done false 3899 Acked Read 10 Done Sent Done false 3900 Acked Read 4 Done Sent Done false 3901 Acked Write 6 Done Sent Done false 3902 Acked Write 5 Done Sent Done false 3903 Acked Write 6 Done Sent Done false 3904 Acked Read 10 Done Sent Done false 3905 Acked Read 8 Done Sent Done false 3906 Acked Read 4 Done Sent Done false 3907 Acked Read 4 Done Sent Done false 3908 Acked Read 4 Done Sent Done false 3909 Acked Read 3 Done Sent Done false 3910 Acked Write 1 Done Sent Done false 3911 Acked Write 3 Done Sent Done false 3912 Acked Read 6 Done Sent Done false 3913 Acked Read 7 Done Sent Done false 3914 Acked Read 8 Done Sent Done false 3915 Acked Read 6 Done Sent Done false 3916 Acked Write 5 Done Sent Done false 3917 Acked Read 9 Done Sent Done false 3918 Acked Write 8 Done Sent Done false 3919 Acked Read 6 Done Sent Done false 3920 Acked Read 3 Done Sent Done false 3921 Acked Write 7 Done Sent Done false 3922 Acked Read 10 Done Sent Done false 3923 Acked Write 8 Done Sent Done false 3924 Acked Write 1 Done Sent Done false 3925 Acked Write 8 Done Sent Done false 3926 Acked Write 8 Done Sent Done false 3927 Acked Write 4 Done Sent Done false 3928 Acked Read 8 Done Sent Done false 3929 Acked Read 6 Done Sent Done false 3930 Acked Read 9 Done Sent Done false 3931 Acked Write 5 Done Sent Done false 3932 Acked Flush 0 Done Sent Done false 3933 Acked Write 1 Done Sent Done false 3934 Acked Write 8 Done Sent Done false 3935 Acked Read 2 Done Sent Done false 3936 Acked Read 2 Done Sent Done false 3937 Acked Read 8 Done Sent Done false 3938 Acked Write 1 Done Sent Done false 3939 Acked Read 3 Done Sent Done false 3940 Acked Read 9 Done Sent Done false 3941 Acked Read 2 Done Sent Done false 3942 Acked Read 6 Done Sent Done false 3943 Acked Read 1 Done Sent Done false 3944 Acked Read 4 Done Sent Done false 3945 Acked Read 1 Done Sent Done false 3946 Acked Read 7 Done Sent Done false 3947 Acked Write 5 Done Sent Done false 3948 Acked Write 10 Done Sent Done false 3949 Acked Write 1 Done Sent Done false 3950 Acked Flush 0 Done Sent Done false 3951 Acked Read 1 Done Sent Done false 3952 Acked Read 1 Done Sent Done false 3953 Acked Flush 0 Done Sent Done false 3954 Acked Write 10 Done Sent Done false 3955 Acked Flush 0 Done Sent Done false 3956 Acked Write 6 Done Sent Done false 3957 Acked Read 5 Done Sent Done false 3958 Acked Flush 0 Done Sent Done false 3959 Acked Write 10 Done Sent Done false 3960 Acked Read 8 Done Sent Done false 3961 Acked Read 6 Done Sent Done false 3962 Acked Read 1 Done Sent Done false 3963 Acked Write 9 Done Sent Done false 3964 Acked Write 10 Done Sent Done false 3965 Acked Write 3 Done Sent Done false 3966 Acked Flush 0 Done Sent Done false 3967 Acked Flush 0 Done Sent Done false 3968 Acked Read 8 Done Sent Done false 3969 Acked Read 8 Done Sent Done false 3970 Acked Read 7 Done Sent Done false 3971 Acked Write 4 Done Sent Done false 3972 Acked Read 4 Done Sent Done false 3973 Acked Write 10 Done Sent Done false 3974 Acked Read 4 Done Sent Done false 3975 Acked Write 2 Done Sent Done false 3976 Acked Write 7 Done Sent Done false 3977 Acked Read 4 Done Sent Done false 3978 Acked Read 2 Done Sent Done false 3979 Acked Write 5 Done Sent Done false 3980 Acked Read 7 Done Sent Done false 3981 Acked Read 3 Done Sent Done false 3982 Acked Read 3 Done Sent Done false 3983 Acked Flush 0 Done Sent Done false 3984 Acked Read 6 Done Sent Done false 3985 Acked Read 10 Done Sent Done false 3986 Acked Read 9 Done Sent Done false 3987 Acked Flush 0 Done Sent Done false 3988 Acked Write 4 Done Sent Done false 3989 Acked Read 3 Done Sent Done false 3990 Acked Read 4 Done Sent Done false 3991 Acked Write 6 Done Sent Done false 3992 Acked Flush 0 Done Sent Done false 3993 Acked Read 8 Done Sent Done false 3994 Acked Read 7 Done Sent Done false 3995 Acked Write 10 Done Sent Done false 3996 Acked Write 5 Done Sent Done false 3997 Acked Read 4 Done Sent Done false 3998 Acked Read 1 Done Sent Done false 3999 Acked Write 1 Done Sent Done false 4000 Acked Read 6 Done Sent Done false 4001 Acked Write 7 Done Sent Done false 4002 Acked Flush 0 Done Sent Done false 4003 Acked Write 8 Done Sent Done false 4004 Acked Read 8 Done Sent Done false 4005 Acked Write 2 Done Sent Done false 4006 Acked Read 6 Done Sent Done false 4007 Acked Write 10 Done Sent Done false 4008 Acked Read 2 Done Sent Done false 4009 Acked Write 6 Done Sent Done false 4010 Acked Read 10 Done Sent Done false 4011 Acked Read 5 Done Sent Done false 4012 Acked Write 7 Done Sent Done false 4013 Acked Write 2 Done Sent Done false 4014 Acked Write 6 Done Sent Done false 4015 Acked Read 10 Done Sent Done false 4016 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4016 3714 4016 Downstairs last five completed: 3714 3713 3712 3711 3710 Upstairs last five completed: 4016 4015 4014 4013 4012 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.88001653Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:46232)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.880086801Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:03.880956557Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881528574Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881549738Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 3714","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881564954Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 3714","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881817256Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881836741Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881851087Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 3714","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.881865053Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 3714","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:03.882086854Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4016 4015 4014 4013 4012 Upstairs last five completed: 4016 4015 4014 4013 4012 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:44:06.114 INFO request completed[8810] Got stop action so:false kr:false , latency_us: 335, response_code: 204, uri: /stop/cid/0, method: POST, req_id: da562389-7764-4258-9394-4a72e2228ee8, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:44:06.115 INFO request completed, latency_us:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:06.115004779Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 132, response_code: 200, uri: /state/cid/0, method: GET, req_id:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:06.115111661Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 23030203-e26f-4094-8919-afa69bcc1fa4, remote_addr: 127.0.0.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:06.115134244Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 1:51788, local_addr: 127.0.0{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:06.115165026Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .1:9998 {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:06.115180662Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:06.115199097Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:44:10.117 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/0, method: GET, req_id: 489717fc-e115-4534-9236-99ddea6ed96a, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 1532 len 4096 data: 2 002/300 Write block 1883 len 24576 data: 3 3 4 4 5 5 003/300 Write block 2095 len 8192 data: 3 3 004/300 Read block 882 len 20480 005/300 Read block 3810 len 12288 006/300 Write block 1413 len 32768 data: 2 2 2 2 2 2 3 3 007/300 Write block 4063 len 20480 data: 2 2 2 3 3 008/300 Write block 1180 len 16384 data: 5 5 4 3 009/300 Read block 1272 len 4096 010/300 Read block 1489 len 40960 011/300 Write block 2032 len 20480 data: 2 2 2 2 2 012/300 Write block 435 len 24576 data: 5 4 5 5 6 5 013/300 Flush 014/300 Write block 1569 len 20480 data: 2 2 2 2 4 015/300 Flush 016/300 Read block 749 len 24576 017/300 Write block 3192 len 24576 data: 2 2 2 2 2 3 018/300 Read block 1219 len 16384 019/300 Flush 020/300 Read block 2977 len 40960 021/300 Read block 644 len 36864 022/300 Flush 023/300 Read block 1294 len 32768 024/300 Write block 3804 len 32768 data: 3 4 4 5 5 5 4 4 025/300 Read block 3770 len 4096 026/300 Read block 4053 len 20480 027/300 Read block 130 len 40960 028/300 Write block 2246 len 24576 data: 7 8 8 6 4 4 029/300 Read block 1991 len 28672 030/300 Read block 4109 len 8192 031/300 Write block 4062 len 36864 data: 2 3 3 3 4 4 3 2 2 032/300 Read block 3591 len 12288 033/300 Write block 898 len 24576 data: 4 4 3 3 3 4 034/300 Read block 4237 len 36864 035/300 Read block 1354 len 20480 036/300 Read block 2505 len 4096 037/300 Read block 2003 len 36864 038/300 Read block 2364 len 28672 039/300 Write block 4889 len 4096 data: 4 040/300 Write block 1624 len 4096 data: 2 041/300 Read block 4145 len 32768 042/300 Write block 4603 len 40960 data: 3 3 3 3 4 3 4 4 3 3 043/300 Write block 883 len 32768 data: 2 2 2 4 4 3 4 4 044/300 Read block 709 len 12288 045/300 Read block 3922 len 24576 046/300 Read block 3751 len 40960 047/300 Write block 4827 len 16384 data: 3 5 4 4 048/300 Write block 295 len 20480 data: 3 3 3 3 2 049/300 Read block 1329 len 4096 050/300 Write block 4793 len 32768 data: 4 5 5 5 5 5 5 5 051/300 Write block 3313 len 28672 data: 3 3 3 3 3 3 4 052/300 Read block 3837 len 4096 053/300 Read block 929 len 12288 054/300 Write block 4077 len 36864 data: 3 3 3 2 2 3 3 3 3 055/300 Read block 288 len 36864 056/300 Read block 2480 len 40960 057/300 Read block 2776 len 32768 058/300 Write block 3085 len 20480 data: 5 5 5 5 5 059/300 Write block 1805 len 36864 data: 4 3 4 3 3 3 3 3 3 060/300 Read block 4681 len 16384 061/300 Read block 1847 len 32768 062/300 Flush 063/300 Read block 1626 len 12288 064/300 Write block 3986 len 24576 data: 2 2 2 2 4 4 065/300 Read block 1872 len 4096 066/300 Read block 1303 len 28672 067/300 Write block 2181 len 12288 data: 3 3 3 068/300 Read block 1434 len 16384 069/300 Read block 1847 len 16384 070/300 Read block 357 len 28672 071/300 Write block 3231 len 4096 data: 4 072/300 Flush 073/300 Flush 074/300 Read block 4870 len 4096 075/300 Write block 4273 len 40960 data: 5 4 5 5 4 5 5 5 4 3 076/300 Write block 4751 len 12288 data: 2 2 3 077/300 Read block 3079 len 32768 078/300 Write block 324 len 28672 data: 3 3 3 3 3 2 2 079/300 Write block 3534 len 28672 data: 2 2 3 3 3 3 3 080/300 Read block 3199 len 24576 081/300 Write block 3275 len 12288 data: 4 4 4 082/300 Read block 2568 len 24576 083/300 Read block 1540 len 24576 084/300 Write block 3347 len 36864 data: 2 2 2 2 2 2 2 2 2 085/300 Flush 086/300 Read block 1412 len 16384 087/300 Read block 3690 len 40960 088/300 Write block 2156 len 8192 data: 4 4 089/300 Read block 2902 len 28672 090/300 Read block 4656 len 16384 091/300 Write block 2502 len 40960 data: 4 3 3 3 3 3 2 2 2 2 092/300 Write block 4827 len 4096 data: 4 093/300 Read block 3306 len 24576 094/300 Read block 528 len 16384 095/300 Write block 1700 len 40960 data: 4 3 2 2 2 2 2 2 3 3 096/300 Read block 3922 len 20480 097/300 Write block 827 len 12288 data: 5 5 6 098/300 Write block 146 len 16384 data: 3 3 3 2 099/300 Read block 2313 len 4096 100/300 Read block 3984 len 36864 101/300 Flush 102/300 Read block 3025 len 12288 103/300 Write block 4699 len 40960 data: 3 3 3 3 2 2 3 2 3 4 104/300 Read block 2577 len 16384 105/300 Write block 504 len 36864 data: 3 2 3 4 4 4 4 3 4 106/300 Flush 107/300 Write block 3494 len 8192 data: 3 2 108/300 Write block 4067 len 36864 data: 5 4 3 3 2 3 3 3 3 109/300 Read block 3433 len 32768 110/300 Flush 111/300 Read block 3163 len 16384 112/300 Read block 2855 len 28672 113/300 Read block 985 len 20480 114/300 Read block 2987 len 16384 115/300 Write block 1308 len 40960 data: 2 2 2 3 4 5 6 6 5 4 116/300 Flush 117/300 Read block 1833 len 36864 118/300 Flush 119/300 Write block 1233 len 12288 data: 3 3 2 120/300 Read block 2595 len 40960 121/300 Write block 4150 len 20480 data: 3 3 4 4 5 122/300 Write block 2858 len 12288 data: 4 5 4 123/300 Write block 3946 len 32768 data: 2 2 3 4 4 4 4 4 124/300 Read block 2580 len 24576 125/300 Read block 3952 len 20480 126/300 Write block 3151 len 32768 data: 3 3 2 3 3 3 4 4 127/300 Write block 314 len 8192 data: 2 2 128/300 Read block 2469 len 28672 129/300 Write block 3927 len 8192 data: 2 2 130/300 Read block 625 len 24576 131/300 Write block 2267 len 20480 data: 3 3 3 3 4 132/300 Read block 3340 len 32768 133/300 Read block 4198 len 16384 134/300 Write block 2042 len 20480 data: 3 5 6 6 5 135/300 Write block 678 len 4096 data: 4 136/300 Write block 826 len 36864 data: 6 6 6 7 6 5 5 4 4 137/300 Write block 3487 len 16384 data: 3 3 4 4 138/300 Write block 3530 len 12288 data: 2 2 2 139/300 Write block 2078 len 8192 data: 2 2 140/300 Read block 2621 len 32768 141/300 Read block 3114 len 16384 142/300 Flush 143/300 Write block 2395 len 28672 data: 3 3 3 4 4 4 4 144/300 Read block 3630 len 28672 145/300 Read block 2537 len 16384 146/300 Read block 3640 len 16384 147/300 Flush 148/300 Write block 532 len 28672 data: 5 4 3 3 3 3 3 149/300 Flush 150/300 Read block 3386 len 40960 151/300 Read block 1913 len 28672 152/300 Read block 2137 len 8192 153/300 Write block 962 len 4096 data: 3 154/300 Read block 4298 len 20480 155/300 Read block 2900 len 20480 156/300 Write block 2918 len 36864 data: 3 2 2 2 2 3 3 2 2 157/300 Write block 1678 len 24576 data: 2 2 2 3 3 4 158/300 Read block 1262 len 8192 159/300 Read block 244 len 32768 160/300 Write block 3389 len 36864 data: 2 2 3 3 3 3 3 3 3 161/300 Read block 4536 len 4096 162/300 Read block 2465 len 4096 163/300 Write block 840 len 12288 data: 3 2 2 164/300 Write block 2486 len 40960 data: 3 2 2 2 3 3 3 4 3 3 165/300 Read block 3238 len 16384 166/300 Flush 167/300 Write block 1127 len 24576 data: 2 2 2 2 2 2 168/300 Write block 2218 len 20480 data: 2 2 3 3 4 169/300 Write block 1985 len 20480 data: 3 3 4 4 5 170/300 Write block 4090 len 20480 data: 3 4 4 5 6 171/300 Read block 2176 len 36864 172/300 Write block 2692 len 8192 data: 3 3 173/300 Read block 1960 len 32768 174/300 Read block 2805 len 8192 175/300 Read block 2606 len 36864 176/300 Write block 3124 len 24576 data: 3 3 3 3 4 4 177/300 Read block 4582 len 36864 178/300 Write block 2257 len 24576 data: 4 4 4 4 3 2 179/300 Read block 3328 len 40960 180/300 Read block 4350 len 32768 181/300 Read block 1725 len 28672 182/300 Write block 4441 len 20480 data: 5 5 6 5 4 183/300 Read block 4029 len 8192 184/300 Read block 1941 len 16384 185/300 Write block 1158 len 40960 data: 3 3 2 2 2 2 2 3 3 3 186/300 Write block 460 len 12288 data: 2 2 2 187/300 Read block 960 len 4096 188/300 Write block 4509 len 36864 data: 2 2 2 3 3 3 3 3 3 189/300 Read block 1283 len 28672 190/300 Read block 1351 len 20480 191/300 Write block 3402 len 32768 data: 3 3 3 4 5 5 5 5 192/300 Read block 3279 len 12288 193/300 Write block 1387 len 12288 data: 5 5 5 194/300 Read block 425 len 28672 195/300 Flush 196/300 Read block 1477 len 20480 197/300 Write block 3340 len 32768 data: 2 2 2 2 2 2 2 3 198/300 Read block 525 len 8192 199/300 Read block 2396 len 8192 200/300 Read block 3289 len 36864 201/300 Write block 1992 len 28672 data: 7 6 4 4 5 6 6 202/300 Write block 2114 len 4096 data: 6 203/300 Read block 1905 len 20480 204/300 Read block 3743 len 8192 205/300 Write block 2446 len 40960 data: 3 2 2 2 2 2 2 3 4 4 206/300 Flush 207/300 Read block 248 len 8192 208/300 Read block 1087 len 8192 209/300 Write block 797 len 20480 data: 3 4 3 3 3 210/300 Write block 4126 len 16384 data: 4 4 5 5 211/300 Read block 1930 len 28672 212/300 Flush 213/300 Read block 4859 len 36864 214/300 Write block 3021 len 4096 data: 4 215/300 Write block 4126 len 20480 data: 5 5 6 6 5 216/300 Read block 1499 len 4096 217/300 Write block 1957 len 32768 data: 3 3 3 3 2 3 3 3 218/300 Write block 2533 len 40960 data: 2 2 2 2 2 2 2 3 5 6 219/300 Write block 1300 len 8192 data: 2 2 220/300 Read block 2770 len 8192 221/300 Read block 2474 len 8192 222/300 Read block 267 len 40960 223/300 Flush 224/300 Flush 225/300 Read block 376 len 28672 226/300 Read block 4414 len 20480 227/300 Write block 4029 len 16384 data: 2 4 4 4 228/300 Write block 2541 len 24576 data: 6 7 6 6 6 5 229/300 Write block 3710 len 28672 data: 3 3 4 3 2 2 2 230/300 Write block 1337 len 8192 data: 3 3 231/300 Write block 531 len 8192 data: 5 6 232/300 Read block 3035 len 4096 233/300 Read block 124 len 24576 234/300 Flush 235/300 Write block 98 len 20480 data: 2 2 3 3 3 236/300 Flush 237/300 Read block 2165 len 4096 238/300 Write block 748 len 32768 data: 6 6 6 6 7 6 5 5 239/300 Flush 240/300 Read block 3663 len 28672 241/300 Read block 3519 len 36864 242/300 Read block 2236 len 12288 243/300 Read block 1404 len 16384 244/300 Flush 245/300 Read block 2936 len 24576 246/300 Read block 4875 len 40960 247/300 Write block 371 len 36864 data: 4 4 3 3 3 2 3 3 3 248/300 Write block 10 len 12288 data: 2 2 2 249/300 Write block 114 len 16384 data: 3 3 4 4 250/300 Write block 820 len 36864 data: 2 3 3 3 5 6 7 7 7 251/300 Write block 3326 len 16384 data: 3 4 4 4 252/300 Read block 736 len 8192 253/300 Write block 3858 len 36864 data: 4 5 5 5 3 3 2 2 2 254/300 Read block 838 len 32768 255/300 Read block 4833 len 20480 256/300 Flush 257/300 Read block 4559 len 12288 258/300 Read block 538 len 28672 259/300 Read block 2731 len 20480 260/300 Read block 3107 len 28672 261/300 Read block 357 len 24576 262/300 Read block 55 len 28672 263/300 Read block 2271 len 16384 264/300 Write block 2626 len 36864 data: 3 3 3 2 2 2 2 2 2 265/300 Read block 4730 len 8192 266/300 Read block 1512 len 36864 267/300 Read block 2859 len 32768 268/300 Read block 2630 len 12288 269/300 Read block 2318 len 12288 270/300 Read block 3000 len 32768 271/300 Write block 64 len 24576 data: 4 5 5 4 5 5 272/300 Flush 273/300 Read block 3574 len 12288 274/300 Read block 4431 len 40960 275/300 Read block 4600 len 12288 276/300 Read block 1720 len 28672 277/300 Write block 1438 len 20480 data: 3 4 3 3 3 278/300 Read block 2334 len 40960 279/300 Read block 78 len 24576 280/300 Read block 4261 len 32768 281/300 Read block 3816 len 4096 282/300 Flush 283/300 Flush 284/300 Write block 4430 len 20480 data: 3 3 3 5 5 285/300 Flush 286/300 Read block 4910 len 36864 287/300 Flush 288/300 Read block 3671 len 36864 289/300 Read block 3856 len 24576 290/300 Read block 2381 len 24576 291/300 Flush 292/300 Read block 1598 len 16384 293/300 Write block 4575 len 4096 data: 3 294/300 Write block 840 len 40960 data: 4 3 3 2 2 2 2 2 2 3 295/300 Flush 296/300 Write block 2101 len 28672 data: 5 5 4 4 4 4 5 297/300 Read block 1960 len 16384 298/300 Read block 2071 len 40960 299/300 Write block 2052 len 24576 data: 3 3 3 3 5 5 300/300 Flush 301/300 Read block 4212 len 40960 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:44:10.354 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/0, method: POST, req_id: e2e97cae-f370-4490-98d1-419a137aafbc, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4017 Acked Write 1 Sent Done Done false 4018 Acked Write 6 Sent Done Done false 4019 Acked Write 2 Sent Done Done false 4020 Acked Read 5 Sent Done Done false 4021 Acked Read 3 Sent Done Done false 4022 Acked Write 8 Sent Done Done false 4023 Acked Write 5 Sent Done Done false 4024 Acked Write 4 Sent Done Done false 4025 Acked Read 1 Sent Done Done false 4026 Acked Read 10 Sent Done Done false 4027 Acked Write 5 Sent Done Done false 4028 Acked Write 6 Sent Done Done false 4029 Acked Flush 0 Sent Done Done false 4030 Acked Write 5 Sent Done Done false 4031 Acked Flush 0 Sent Done Done false 4032 Acked Read 6 Sent Done Done false 4033 Acked Write 6 Sent Done Done false 4034 Acked Read 4 Sent Done Done false 4035 Acked Flush 0 Sent Done Done false 4036 Acked Read 10 Sent Done Done false 4037 Acked Read 9 Sent Done Done false 4038 Acked Flush 0 Sent Done Done false 4039 Acked Read 8 Sent Done Done false 4040 Acked Write 8 Sent Done Done false 4041 Acked Read 1 Sent Done Done false 4042 Acked Read 5 Sent Done Done false 4043 Acked Read 10 Sent Done Done false 4044 Acked Write 6 Sent Done Done false 4045 Acked Read 7 Sent Done Done false 4046 Acked Read 2 Sent Done Done false 4047 Acked Write 9 Sent Done Done false 4048 Acked Read 3 Sent Done Done false 4049 Acked Write 6 Sent Done Done false 4050 Acked Read 9 Sent Done Done false 4051 Acked Read 5 Sent Done Done false 4052 Acked Read 1 Sent Done Done false 4053 Acked Read 9 Sent Done Done false 4054 Acked Read 7 Sent Done Done false 4055 Acked Write 1 Sent Done Done false 4056 Acked Write 1 Sent Done Done false 4057 Acked Read 8 Sent Done Done false 4058 Acked Write 10 Sent Done Done false 4059 Acked Write 8 Sent Done Done false 4060 Acked Read 3 Sent Done Done false 4061 Acked Read 6 Sent Done Done false 4062 Acked Read 10 Sent Done Done false 4063 Acked Write 4 Sent Done Done false 4064 Acked Write 5 Sent Done Done false 4065 Acked Read 1 Sent Done Done false 4066 Acked Write 8 Sent Done Done false 4067 Acked Write 7 Sent Done Done false 4068 Acked Read 1 Sent Done Done false 4069 Acked Read 3 Sent Done Done false 4070 Acked Write 9 Sent Done Done false 4071 Acked Read 9 Sent Done Done false 4072 Acked Read 10 Sent Done Done false 4073 Acked Read 8 Sent Done Done false 4074 Acked Write 5 Sent Done Done false 4075 Acked Write 9 Sent Done Done false 4076 Acked Read 4 Sent Done Done false 4077 Acked Read 8 Sent Done Done false 4078 Acked Flush 0 Sent Done Done false 4079 Acked Read 3 Sent Done Done false 4080 Acked Write 6 Sent Done Done false 4081 Acked Read 1 Sent Done Done false 4082 Acked Read 7 Sent Done Done false 4083 Acked Write 3 Sent Done Done false 4084 Acked Read 4 Sent Done Done false 4085 Acked Read 4 Sent Done Done false 4086 Acked Read 7 Sent Done Done false 4087 Acked Write 1 Sent Done Done false 4088 Acked Flush 0 Sent Done Done false 4089 Acked Flush 0 Sent Done Done false 4090 Acked Read 1 Sent Done Done false 4091 Acked Write 10 Sent Done Done false 4092 Acked Write 3 Sent Done Done false 4093 Acked Read 8 Sent Done Done false 4094 Acked Write 7 Sent Done Done false 4095 Acked Write 7 Sent Done Done false 4096 Acked Read 6 Sent Done Done false 4097 Acked Write 3 Sent Done Done false 4098 Acked Read 6 Sent Done Done false 4099 Acked Read 6 Sent Done Done false 4100 Acked Write 9 Sent Done Done false 4101 Acked Flush 0 Sent Done Done false 4102 Acked Read 4 Sent Done Done false 4103 Acked Read 10 Sent Done Done false 4104 Acked Write 2 Sent Done Done false 4105 Acked Read 7 Sent Done Done false 4106 Acked Read 4 Sent Done Done false 4107 Acked Write 10 Sent Done Done false 4108 Acked Write 1 Sent Done Done false 4109 Acked Read 6 Sent Done Done false 4110 Acked Read 4 Sent Done Done false 4111 Acked Write 10 Sent Done Done false 4112 Acked Read 5 Sent Done Done false 4113 Acked Write 3 Sent Done Done false 4114 Acked Write 4 Sent Done Done false 4115 Acked Read 1 Sent Done Done false 4116 Acked Read 9 Sent Done Done false 4117 Acked Flush 0 Sent Done Done false 4118 Acked Read 3 Sent Done Done false 4119 Acked Write 10 Sent Done Done false 4120 Acked Read 4 Sent Done Done false 4121 Acked Write 9 Sent Done Done false 4122 Acked Flush 0 Sent Done Done false 4123 Acked Write 2 Sent Done Done false 4124 Acked Write 9 Sent Done Done false 4125 Acked Read 8 Sent Done Done false 4126 Acked Flush 0 Sent Done Done false 4127 Acked Read 4 Sent Done Done false 4128 Acked Read 7 Sent Done Done false 4129 Acked Read 5 Sent Done Done false 4130 Acked Read 4 Sent Done Done false 4131 Acked Write 10 Sent Done Done false 4132 Acked Flush 0 Sent Done Done false 4133 Acked Read 9 Sent Done Done false 4134 Acked Flush 0 Sent Done Done false 4135 Acked Write 3 Sent Done Done false 4136 Acked Read 10 Sent Done Done false 4137 Acked Write 5 Sent Done Done false 4138 Acked Write 3 Sent Done Done false 4139 Acked Write 8 Sent Done Done false 4140 Acked Read 6 Sent Done Done false 4141 Acked Read 5 Sent Done Done false 4142 Acked Write 8 Sent Done Done false 4143 Acked Write 2 Sent Done Done false 4144 Acked Read 7 Sent Done Done false 4145 Acked Write 2 Sent Done Done false 4146 Acked Read 6 Sent Done Done false 4147 Acked Write 5 Sent Done Done false 4148 Acked Read 8 Sent Done Done false 4149 Acked Read 4 Sent Done Done false 4150 Acked Write 5 Sent Done Done false 4151 Acked Write 1 Sent Done Done false 4152 Acked Write 9 Sent Done Done false 4153 Acked Write 4 Sent Done Done false 4154 Acked Write 3 Sent Done Done false 4155 Acked Write 2 Sent Done Done false 4156 Acked Read 8 Sent Done Done false 4157 Acked Read 4 Sent Done Done false 4158 Acked Flush 0 Sent Done Done false 4159 Acked Write 7 Sent Done Done false 4160 Acked Read 7 Sent Done Done false 4161 Acked Read 4 Sent Done Done false 4162 Acked Read 4 Sent Done Done false 4163 Acked Flush 0 Sent Done Done false 4164 Acked Write 7 Sent Done Done false 4165 Acked Flush 0 Sent Done Done false 4166 Acked Read 10 Sent Done Done false 4167 Acked Read 7 Sent Done Done false 4168 Acked Read 2 Sent Done Done false 4169 Acked Write 1 Sent Done Done false 4170 Acked Read 5 Sent Done Done false 4171 Acked Read 5 Sent Done Done false 4172 Acked Write 9 Sent Done Done false 4173 Acked Write 6 Sent Done Done false 4174 Acked Read 2 Sent Done Done false 4175 Acked Read 8 Sent Done Done false 4176 Acked Write 9 Sent Done Done false 4177 Acked Read 1 Sent Done Done false 4178 Acked Read 1 Sent Done Done false 4179 Acked Write 3 Sent Done Done false 4180 Acked Write 10 Sent Done Done false 4181 Acked Read 4 Sent Done Done false 4182 Acked Flush 0 Sent Done Done false 4183 Acked Write 6 Sent Done Done false 4184 Acked Write 5 Sent Done Done false 4185 Acked Write 5 Sent Done Done false 4186 Acked Write 5 Sent Done Done false 4187 Acked Read 9 Sent Done Done false 4188 Acked Write 2 Sent Done Done false 4189 Acked Read 8 Sent Done Done false 4190 Acked Read 2 Sent Done Done false 4191 Acked Read 9 Sent Done Done false 4192 Acked Write 6 Sent Done Done false 4193 Acked Read 9 Sent Done Done false 4194 Acked Write 6 Sent Done Done false 4195 Acked Read 10 Sent Done Done false 4196 Acked Read 8 Sent Done Done false 4197 Acked Read 7 Sent Done Done false 4198 Acked Write 5 Sent Done Done false 4199 Acked Read 2 Sent Done Done false 4200 Acked Read 4 Sent Done Done false 4201 Acked Write 10 Sent Done Done false 4202 Acked Write 3 Sent Done Done false 4203 Acked Read 1 Sent Done Done false 4204 Acked Write 9 Sent Done Done false 4205 Acked Read 7 Sent Done Done false 4206 Acked Read 5 Sent Done Done false 4207 Acked Write 8 Sent Done Done false 4208 Acked Read 3 Sent Done Done false 4209 Acked Write 3 Sent Done Done false 4210 Acked Read 7 Sent Done Done false 4211 Acked Flush 0 Sent Done Done false 4212 Acked Read 5 Sent Done Done false 4213 Acked Write 8 Sent Done Done false 4214 Acked Read 2 Sent Done Done false 4215 Acked Read 2 Sent Done Done false 4216 Acked Read 9 Sent Done Done false 4217 Acked Write 7 Sent Done Done false 4218 Acked Write 1 Sent Done Done false 4219 Acked Read 5 Sent Done Done false 4220 Acked Read 2 Sent Done Done false 4221 Acked Write 10 Sent Done Done false 4222 Acked Flush 0 Sent Done Done false 4223 Acked Read 2 Sent Done Done false 4224 Acked Read 2 Sent Done Done false 4225 Acked Write 5 Sent Done Done false 4226 Acked Write 4 Sent Done Done false 4227 Acked Read 7 Sent Done Done false 4228 Acked Flush 0 Sent Done Done false 4229 Acked Read 9 Sent Done Done false 4230 Acked Write 1 Sent Done Done false 4231 Acked Write 5 Sent Done Done false 4232 Acked Read 1 Sent Done Done false 4233 Acked Write 8 Sent Done Done false 4234 Acked Write 10 Sent Done Done false 4235 Acked Write 2 Sent Done Done false 4236 Acked Read 2 Sent Done Done false 4237 Acked Read 2 Sent Done Done false 4238 Acked Read 10 Sent Done Done false 4239 Acked Flush 0 Sent Done Done false 4240 Acked Flush 0 Sent Done Done false 4241 Acked Read 7 Sent Done Done false 4242 Acked Read 5 Sent Done Done false 4243 Acked Write 4 Sent Done Done false 4244 Acked Write 6 Sent Done Done false 4245 Acked Write 7 Sent Done Done false 4246 Acked Write 2 Sent Done Done false 4247 Acked Write 2 Sent Done Done false 4248 Acked Read 1 Sent Done Done false 4249 Acked Read 6 Sent Done Done false 4250 Acked Flush 0 Sent Done Done false 4251 Acked Write 5 Sent Done Done false 4252 Acked Flush 0 Sent Done Done false 4253 Acked Read 1 Sent Done Done false 4254 Acked Write 8 Sent Done Done false 4255 Acked Flush 0 Sent Done Done false 4256 Acked Read 7 Sent Done Done false 4257 Acked Read 9 Sent Done Done false 4258 Acked Read 3 Sent Done Done false 4259 Acked Read 4 Sent Done Done false 4260 Acked Flush 0 Sent Done Done false 4261 Acked Read 6 Sent Done Done false 4262 Acked Read 10 Sent Done Done false 4263 Acked Write 9 Sent Done Done false 4264 Acked Write 3 Sent Done Done false 4265 Acked Write 4 Sent Done Done false 4266 Acked Write 9 Sent Done Done false 4267 Acked Write 4 Sent Done Done false 4268 Acked Read 2 Sent Done Done false 4269 Acked Write 9 Sent Done Done false 4270 Acked Read 8 Sent Done Done false 4271 Acked Read 5 Sent Done Done false 4272 Acked Flush 0 Sent Done Done false 4273 Acked Read 3 Sent Done Done false 4274 Acked Read 7 Sent Done Done false 4275 Acked Read 5 Sent Done Done false 4276 Acked Read 7 Sent Done Done false 4277 Acked Read 6 Sent Done Done false 4278 Acked Read 7 Sent Done Done false 4279 Acked Read 4 Sent Done Done false 4280 Acked Write 9 Sent Done Done false 4281 Acked Read 2 Sent Done Done false 4282 Acked Read 9 Sent Done Done false 4283 Acked Read 8 Sent Done Done false 4284 Acked Read 3 Sent Done Done false 4285 Acked Read 3 Sent Done Done false 4286 Acked Read 8 Sent Done Done false 4287 Acked Write 6 Sent Done Done false 4288 Acked Flush 0 Sent Done Done false 4289 Acked Read 3 Sent Done Done false 4290 Acked Read 10 Sent Done Done false 4291 Acked Read 3 Sent Done Done false 4292 Acked Read 7 Sent Done Done false 4293 Acked Write 5 Sent Done Done false 4294 Acked Read 10 Sent Done Done false 4295 Acked Read 6 Sent Done Done false 4296 Acked Read 8 Sent Done Done false 4297 Acked Read 1 Sent Done Done false 4298 Acked Flush 0 Sent Done Done false 4299 Acked Flush 0 Sent Done Done false 4300 Acked Write 5 Sent Done Done false 4301 Acked Flush 0 Sent Done Done false 4302 Acked Read 9 Sent Done Done false 4303 Acked Flush 0 Sent Done Done false 4304 Acked Read 9 Sent Done Done false 4305 Acked Read 6 Sent Done Done false 4306 Acked Read 6 Sent Done Done false 4307 Acked Flush 0 Sent Done Done false 4308 Acked Read 4 Sent Done Done false 4309 Acked Write 1 Sent Done Done false 4310 Acked Write 10 Sent Done Done false 4311 Acked Flush 0 Sent Done Done false 4312 Acked Write 7 Sent Done Done false 4313 Acked Read 4 Sent Done Done false 4314 Acked Read 10 Sent Done Done false 4315 Acked Write 6 Sent Done Done false 4316 Acked Flush 0 Sent Done Done false 4317 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4016 4316 4316 Downstairs last five completed: 4016 4015 4014 4013 4012 Upstairs last five completed: 4317 4316 4315 4314 4313 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4017 Acked Write 1 Sent Done Done false 4018 Acked Write 6 Sent Done Done false 4019 Acked Write 2 Sent Done Done false 4020 Acked Read 5 Sent Done Done false 4021 Acked Read 3 Sent Done Done false 4022 Acked Write 8 Sent Done Done false 4023 Acked Write 5 Sent Done Done false 4024 Acked Write 4 Sent Done Done false 4025 Acked Read 1 Sent Done Done false 4026 Acked Read 10 Sent Done Done false 4027 Acked Write 5 Sent Done Done false 4028 Acked Write 6 Sent Done Done false 4029 Acked Flush 0 Sent Done Done false 4030 Acked Write 5 Sent Done Done false 4031 Acked Flush 0 Sent Done Done false 4032 Acked Read 6 Sent Done Done false 4033 Acked Write 6 Sent Done Done false 4034 Acked Read 4 Sent Done Done false 4035 Acked Flush 0 Sent Done Done false 4036 Acked Read 10 Sent Done Done false 4037 Acked Read 9 Sent Done Done false 4038 Acked Flush 0 Sent Done Done false 4039 Acked Read 8 Sent Done Done false 4040 Acked Write 8 Sent Done Done false 4041 Acked Read 1 Sent Done Done false 4042 Acked Read 5 Sent Done Done false 4043 Acked Read 10 Sent Done Done false 4044 Acked Write 6 Sent Done Done false 4045 Acked Read 7 Sent Done Done false 4046 Acked Read 2 Sent Done Done false 4047 Acked Write 9 Sent Done Done false 4048 Acked Read 3 Sent Done Done false 4049 Acked Write 6 Sent Done Done false 4050 Acked Read 9 Sent Done Done false 4051 Acked Read 5 Sent Done Done false 4052 Acked Read 1 Sent Done Done false 4053 Acked Read 9 Sent Done Done false 4054 Acked Read 7 Sent Done Done false 4055 Acked Write 1 Sent Done Done false 4056 Acked Write 1 Sent Done Done false 4057 Acked Read 8 Sent Done Done false 4058 Acked Write 10 Sent Done Done false 4059 Acked Write 8 Sent Done Done false 4060 Acked Read 3 Sent Done Done false 4061 Acked Read 6 Sent Done Done false 4062 Acked Read 10 Sent Done Done false 4063 Acked Write 4 Sent Done Done false 4064 Acked Write 5 Sent Done Done false 4065 Acked Read 1 Sent Done Done false 4066 Acked Write 8 Sent Done Done false 4067 Acked Write 7 Sent Done Done false 4068 Acked Read 1 Sent Done Done false 4069 Acked Read 3 Sent Done Done false 4070 Acked Write 9 Sent Done Done false 4071 Acked Read 9 Sent Done Done false 4072 Acked Read 10 Sent Done Done false 4073 Acked Read 8 Sent Done Done false 4074 Acked Write 5 Sent Done Done false 4075 Acked Write 9 Sent Done Done false 4076 Acked Read 4 Sent Done Done false 4077 Acked Read 8 Sent Done Done false 4078 Acked Flush 0 Sent Done Done false 4079 Acked Read 3 Sent Done Done false 4080 Acked Write 6 Sent Done Done false 4081 Acked Read 1 Sent Done Done false 4082 Acked Read 7 Sent Done Done false 4083 Acked Write 3 Sent Done Done false 4084 Acked Read 4 Sent Done Done false 4085 Acked Read 4 Sent Done Done false 4086 Acked Read 7 Sent Done Done false 4087 Acked Write 1 Sent Done Done false 4088 Acked Flush 0 Sent Done Done false 4089 Acked Flush 0 Sent Done Done false 4090 Acked Read 1 Sent Done Done false 4091 Acked Write 10 Sent Done Done false 4092 Acked Write 3 Sent Done Done false 4093 Acked Read 8 Sent Done Done false 4094 Acked Write 7 Sent Done Done false 4095 Acked Write 7 Sent Done Done false 4096 Acked Read 6 Sent Done Done false 4097 Acked Write 3 Sent Done Done false 4098 Acked Read 6 Sent Done Done false 4099 Acked Read 6 Sent Done Done false 4100 Acked Write 9 Sent Done Done false 4101 Acked Flush 0 Sent Done Done false 4102 Acked Read 4 Sent Done Done false 4103 Acked Read 10 Sent Done Done false 4104 Acked Write 2 Sent Done Done false 4105 Acked Read 7 Sent Done Done false 4106 Acked Read 4 Sent Done Done false 4107 Acked Write 10 Sent Done Done false 4108 Acked Write 1 Sent Done Done false 4109 Acked Read 6 Sent Done Done false 4110 Acked Read 4 Sent Done Done false 4111 Acked Write 10 Sent Done Done false 4112 Acked Read 5 Sent Done Done false 4113 Acked Write 3 Sent Done Done false 4114 Acked Write 4 Sent Done Done false 4115 Acked Read 1 Sent Done Done false 4116 Acked Read 9 Sent Done Done false 4117 Acked Flush 0 Sent Done Done false 4118 Acked Read 3 Sent Done Done false 4119 Acked Write 10 Sent Done Done false 4120 Acked Read 4 Sent Done Done false 4121 Acked Write 9 Sent Done Done false 4122 Acked Flush 0 Sent Done Done false 4123 Acked Write 2 Sent Done Done false 4124 Acked Write 9 Sent Done Done false 4125 Acked Read 8 Sent Done Done false 4126 Acked Flush 0 Sent Done Done false 4127 Acked Read 4 Sent Done Done false 4128 Acked Read 7 Sent Done Done false 4129 Acked Read 5 Sent Done Done false 4130 Acked Read 4 Sent Done Done false 4131 Acked Write 10 Sent Done Done false 4132 Acked Flush 0 Sent Done Done false 4133 Acked Read 9 Sent Done Done false 4134 Acked Flush 0 Sent Done Done false 4135 Acked Write 3 Sent Done Done false 4136 Acked Read 10 Sent Done Done false 4137 Acked Write 5 Sent Done Done false 4138 Acked Write 3 Sent Done Done false 4139 Acked Write 8 Sent Done Done false 4140 Acked Read 6 Sent Done Done false 4141 Acked Read 5 Sent Done Done false 4142 Acked Write 8 Sent Done Done false 4143 Acked Write 2 Sent Done Done false 4144 Acked Read 7 Sent Done Done false 4145 Acked Write 2 Sent Done Done false 4146 Acked Read 6 Sent Done Done false 4147 Acked Write 5 Sent Done Done false 4148 Acked Read 8 Sent Done Done false 4149 Acked Read 4 Sent Done Done false 4150 Acked Write 5 Sent Done Done false 4151 Acked Write 1 Sent Done Done false 4152 Acked Write 9 Sent Done Done false 4153 Acked Write 4 Sent Done Done false 4154 Acked Write 3 Sent Done Done false 4155 Acked Write 2 Sent Done Done false 4156 Acked Read 8 Sent Done Done false 4157 Acked Read 4 Sent Done Done false 4158 Acked Flush 0 Sent Done Done false 4159 Acked Write 7 Sent Done Done false 4160 Acked Read 7 Sent Done Done false 4161 Acked Read 4 Sent Done Done false 4162 Acked Read 4 Sent Done Done false 4163 Acked Flush 0 Sent Done Done false 4164 Acked Write 7 Sent Done Done false 4165 Acked Flush 0 Sent Done Done false 4166 Acked Read 10 Sent Done Done false 4167 Acked Read 7 Sent Done Done false 4168 Acked Read 2 Sent Done Done false 4169 Acked Write 1 Sent Done Done false 4170 Acked Read 5 Sent Done Done false 4171 Acked Read 5 Sent Done Done false 4172 Acked Write 9 Sent Done Done false 4173 Acked Write 6 Sent Done Done false 4174 Acked Read 2 Sent Done Done false 4175 Acked Read 8 Sent Done Done false 4176 Acked Write 9 Sent Done Done false 4177 Acked Read 1 Sent Done Done false 4178 Acked Read 1 Sent Done Done false 4179 Acked Write 3 Sent Done Done false 4180 Acked Write 10 Sent Done Done false 4181 Acked Read 4 Sent Done Done false 4182 Acked Flush 0 Sent Done Done false 4183 Acked Write 6 Sent Done Done false 4184 Acked Write 5 Sent Done Done false 4185 Acked Write 5 Sent Done Done false 4186 Acked Write 5 Sent Done Done false 4187 Acked Read 9 Sent Done Done false 4188 Acked Write 2 Sent Done Done false 4189 Acked Read 8 Sent Done Done false 4190 Acked Read 2 Sent Done Done false 4191 Acked Read 9 Sent Done Done false 4192 Acked Write 6 Sent Done Done false 4193 Acked Read 9 Sent Done Done false 4194 Acked Write 6 Sent Done Done false 4195 Acked Read 10 Sent Done Done false 4196 Acked Read 8 Sent Done Done false 4197 Acked Read 7 Sent Done Done false 4198 Acked Write 5 Sent Done Done false 4199 Acked Read 2 Sent Done Done false 4200 Acked Read 4 Sent Done Done false 4201 Acked Write 10 Sent Done Done false 4202 Acked Write 3 Sent Done Done false 4203 Acked Read 1 Sent Done Done false 4204 Acked Write 9 Sent Done Done false 4205 Acked Read 7 Sent Done Done false 4206 Acked Read 5 Sent Done Done false 4207 Acked Write 8 Sent Done Done false 4208 Acked Read 3 Sent Done Done false 4209 Acked Write 3 Sent Done Done false 4210 Acked Read 7 Sent Done Done false 4211 Acked Flush 0 Sent Done Done false 4212 Acked Read 5 Sent Done Done false 4213 Acked Write 8 Sent Done Done false 4214 Acked Read 2 Sent Done Done false 4215 Acked Read 2 Sent Done Done false 4216 Acked Read 9 Sent Done Done false 4217 Acked Write 7 Sent Done Done false 4218 Acked Write 1 Sent Done Done false 4219 Acked Read 5 Sent Done Done false 4220 Acked Read 2 Sent Done Done false 4221 Acked Write 10 Sent Done Done false 4222 Acked Flush 0 Sent Done Done false 4223 Acked Read 2 Sent Done Done false 4224 Acked Read 2 Sent Done Done false 4225 Acked Write 5 Sent Done Done false 4226 Acked Write 4 Sent Done Done false 4227 Acked Read 7 Sent Done Done false 4228 Acked Flush 0 Sent Done Done false 4229 Acked Read 9 Sent Done Done false 4230 Acked Write 1 Sent Done Done false 4231 Acked Write 5 Sent Done Done false 4232 Acked Read 1 Sent Done Done false 4233 Acked Write 8 Sent Done Done false 4234 Acked Write 10 Sent Done Done false 4235 Acked Write 2 Sent Done Done false 4236 Acked Read 2 Sent Done Done false 4237 Acked Read 2 Sent Done Done false 4238 Acked Read 10 Sent Done Done false 4239 Acked Flush 0 Sent Done Done false 4240 Acked Flush 0 Sent Done Done false 4241 Acked Read 7 Sent Done Done false 4242 Acked Read 5 Sent Done Done false 4243 Acked Write 4 Sent Done Done false 4244 Acked Write 6 Sent Done Done false 4245 Acked Write 7 Sent Done Done false 4246 Acked Write 2 Sent Done Done false 4247 Acked Write 2 Sent Done Done false 4248 Acked Read 1 Sent Done Done false 4249 Acked Read 6 Sent Done Done false 4250 Acked Flush 0 Sent Done Done false 4251 Acked Write 5 Sent Done Done false 4252 Acked Flush 0 Sent Done Done false 4253 Acked Read 1 Sent Done Done false 4254 Acked Write 8 Sent Done Done false 4255 Acked Flush 0 Sent Done Done false 4256 Acked Read 7 Sent Done Done false 4257 Acked Read 9 Sent Done Done false 4258 Acked Read 3 Sent Done Done false 4259 Acked Read 4 Sent Done Done false 4260 Acked Flush 0 Sent Done Done false 4261 Acked Read 6 Sent Done Done false 4262 Acked Read 10 Sent Done Done false 4263 Acked Write 9 Sent Done Done false 4264 Acked Write 3 Sent Done Done false 4265 Acked Write 4 Sent Done Done false 4266 Acked Write 9 Sent Done Done false 4267 Acked Write 4 Sent Done Done false 4268 Acked Read 2 Sent Done Done false 4269 Acked Write 9 Sent Done Done false 4270 Acked Read 8 Sent Done Done false 4271 Acked Read 5 Sent Done Done false 4272 Acked Flush 0 Sent Done Done false 4273 Acked Read 3 Sent Done Done false 4274 Acked Read 7 Sent Done Done false 4275 Acked Read 5 Sent Done Done false 4276 Acked Read 7 Sent Done Done false 4277 Acked Read 6 Sent Done Done false 4278 Acked Read 7 Sent Done Done false 4279 Acked Read 4 Sent Done Done false 4280 Acked Write 9 Sent Done Done false 4281 Acked Read 2 Sent Done Done false 4282 Acked Read 9 Sent Done Done false 4283 Acked Read 8 Sent Done Done false 4284 Acked Read 3 Sent Done Done false 4285 Acked Read 3 Sent Done Done false 4286 Acked Read 8 Sent Done Done false 4287 Acked Write 6 Sent Done Done false 4288 Acked Flush 0 Sent Done Done false 4289 Acked Read 3 Sent Done Done false 4290 Acked Read 10 Sent Done Done false 4291 Acked Read 3 Sent Done Done false 4292 Acked Read 7 Sent Done Done false 4293 Acked Write 5 Sent Done Done false 4294 Acked Read 10 Sent Done Done false 4295 Acked Read 6 Sent Done Done false 4296 Acked Read 8 Sent Done Done false 4297 Acked Read 1 Sent Done Done false 4298 Acked Flush 0 Sent Done Done false 4299 Acked Flush 0 Sent Done Done false 4300 Acked Write 5 Sent Done Done false 4301 Acked Flush 0 Sent Done Done false 4302 Acked Read 9 Sent Done Done false 4303 Acked Flush 0 Sent Done Done false 4304 Acked Read 9 Sent Done Done false 4305 Acked Read 6 Sent Done Done false 4306 Acked Read 6 Sent Done Done false 4307 Acked Flush 0 Sent Done Done false 4308 Acked Read 4 Sent Done Done false 4309 Acked Write 1 Sent Done Done false 4310 Acked Write 10 Sent Done Done false 4311 Acked Flush 0 Sent Done Done false 4312 Acked Write 7 Sent Done Done false 4313 Acked Read 4 Sent Done Done false 4314 Acked Read 10 Sent Done Done false 4315 Acked Write 6 Sent Done Done false 4316 Acked Flush 0 Sent Done Done false 4317 Acked Read 10 Sent Done Done false 4318 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4016 4318 4318 Downstairs last five completed: 4016 4015 4014 4013 4012 Upstairs last five completed: 4318 4317 4316 4315 4314 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:16.116405643Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:16.116482853Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:16.116516533Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:16.116538927Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:16.116554653Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4017 Acked Write 1 Sent Done Done false 4018 Acked Write 6 Sent Done Done false 4019 Acked Write 2 Sent Done Done false 4020 Acked Read 5 Sent Done Done false 4021 Acked Read 3 Sent Done Done false 4022 Acked Write 8 Sent Done Done false 4023 Acked Write 5 Sent Done Done false 4024 Acked Write 4 Sent Done Done false 4025 Acked Read 1 Sent Done Done false 4026 Acked Read 10 Sent Done Done false 4027 Acked Write 5 Sent Done Done false 4028 Acked Write 6 Sent Done Done false 4029 Acked Flush 0 Sent Done Done false 4030 Acked Write 5 Sent Done Done false 4031 Acked Flush 0 Sent Done Done false 4032 Acked Read 6 Sent Done Done false 4033 Acked Write 6 Sent Done Done false 4034 Acked Read 4 Sent Done Done false 4035 Acked Flush 0 Sent Done Done false 4036 Acked Read 10 Sent Done Done false 4037 Acked Read 9 Sent Done Done false 4038 Acked Flush 0 Sent Done Done false 4039 Acked Read 8 Sent Done Done false 4040 Acked Write 8 Sent Done Done false 4041 Acked Read 1 Sent Done Done false 4042 Acked Read 5 Sent Done Done false 4043 Acked Read 10 Sent Done Done false 4044 Acked Write 6 Sent Done Done false 4045 Acked Read 7 Sent Done Done false 4046 Acked Read 2 Sent Done Done false 4047 Acked Write 9 Sent Done Done false 4048 Acked Read 3 Sent Done Done false 4049 Acked Write 6 Sent Done Done false 4050 Acked Read 9 Sent Done Done false 4051 Acked Read 5 Sent Done Done false 4052 Acked Read 1 Sent Done Done false 4053 Acked Read 9 Sent Done Done false 4054 Acked Read 7 Sent Done Done false 4055 Acked Write 1 Sent Done Done false 4056 Acked Write 1 Sent Done Done false 4057 Acked Read 8 Sent Done Done false 4058 Acked Write 10 Sent Done Done false 4059 Acked Write 8 Sent Done Done false 4060 Acked Read 3 Sent Done Done false 4061 Acked Read 6 Sent Done Done false 4062 Acked Read 10 Sent Done Done false 4063 Acked Write 4 Sent Done Done false 4064 Acked Write 5 Sent Done Done false 4065 Acked Read 1 Sent Done Done false 4066 Acked Write 8 Sent Done Done false 4067 Acked Write 7 Sent Done Done false 4068 Acked Read 1 Sent Done Done false 4069 Acked Read 3 Sent Done Done false 4070 Acked Write 9 Sent Done Done false 4071 Acked Read 9 Sent Done Done false 4072 Acked Read 10 Sent Done Done false 4073 Acked Read 8 Sent Done Done false 4074 Acked Write 5 Sent Done Done false 4075 Acked Write 9 Sent Done Done false 4076 Acked Read 4 Sent Done Done false 4077 Acked Read 8 Sent Done Done false 4078 Acked Flush 0 Sent Done Done false 4079 Acked Read 3 Sent Done Done false 4080 Acked Write 6 Sent Done Done false 4081 Acked Read 1 Sent Done Done false 4082 Acked Read 7 Sent Done Done false 4083 Acked Write 3 Sent Done Done false 4084 Acked Read 4 Sent Done Done false 4085 Acked Read 4 Sent Done Done false 4086 Acked Read 7 Sent Done Done false 4087 Acked Write 1 Sent Done Done false 4088 Acked Flush 0 Sent Done Done false 4089 Acked Flush 0 Sent Done Done false 4090 Acked Read 1 Sent Done Done false 4091 Acked Write 10 Sent Done Done false 4092 Acked Write 3 Sent Done Done false 4093 Acked Read 8 Sent Done Done false 4094 Acked Write 7 Sent Done Done false 4095 Acked Write 7 Sent Done Done false 4096 Acked Read 6 Sent Done Done false 4097 Acked Write 3 Sent Done Done false 4098 Acked Read 6 Sent Done Done false 4099 Acked Read 6 Sent Done Done false 4100 Acked Write 9 Sent Done Done false 4101 Acked Flush 0 Sent Done Done false 4102 Acked Read 4 Sent Done Done false 4103 Acked Read 10 Sent Done Done false 4104 Acked Write 2 Sent Done Done false 4105 Acked Read 7 Sent Done Done false 4106 Acked Read 4 Sent Done Done false 4107 Acked Write 10 Sent Done Done false 4108 Acked Write 1 Sent Done Done false 4109 Acked Read 6 Sent Done Done false 4110 Acked Read 4 Sent Done Done false 4111 Acked Write 10 Sent Done Done false 4112 Acked Read 5 Sent Done Done false 4113 Acked Write 3 Sent Done Done false 4114 Acked Write 4 Sent Done Done false 4115 Acked Read 1 Sent Done Done false 4116 Acked Read 9 Sent Done Done false 4117 Acked Flush 0 Sent Done Done false 4118 Acked Read 3 Sent Done Done false 4119 Acked Write 10 Sent Done Done false 4120 Acked Read 4 Sent Done Done false 4121 Acked Write 9 Sent Done Done false 4122 Acked Flush 0 Sent Done Done false 4123 Acked Write 2 Sent Done Done false 4124 Acked Write 9 Sent Done Done false 4125 Acked Read 8 Sent Done Done false 4126 Acked Flush 0 Sent Done Done false 4127 Acked Read 4 Sent Done Done false 4128 Acked Read 7 Sent Done Done false 4129 Acked Read 5 Sent Done Done false 4130 Acked Read 4 Sent Done Done false 4131 Acked Write 10 Sent Done Done false 4132 Acked Flush 0 Sent Done Done false 4133 Acked Read 9 Sent Done Done false 4134 Acked Flush 0 Sent Done Done false 4135 Acked Write 3 Sent Done Done false 4136 Acked Read 10 Sent Done Done false 4137 Acked Write 5 Sent Done Done false 4138 Acked Write 3 Sent Done Done false 4139 Acked Write 8 Sent Done Done false 4140 Acked Read 6 Sent Done Done false 4141 Acked Read 5 Sent Done Done false 4142 Acked Write 8 Sent Done Done false 4143 Acked Write 2 Sent Done Done false 4144 Acked Read 7 Sent Done Done false 4145 Acked Write 2 Sent Done Done false 4146 Acked Read 6 Sent Done Done false 4147 Acked Write 5 Sent Done Done false 4148 Acked Read 8 Sent Done Done false 4149 Acked Read 4 Sent Done Done false 4150 Acked Write 5 Sent Done Done false 4151 Acked Write 1 Sent Done Done false 4152 Acked Write 9 Sent Done Done false 4153 Acked Write 4 Sent Done Done false 4154 Acked Write 3 Sent Done Done false 4155 Acked Write 2 Sent Done Done false 4156 Acked Read 8 Sent Done Done false 4157 Acked Read 4 Sent Done Done false 4158 Acked Flush 0 Sent Done Done false 4159 Acked Write 7 Sent Done Done false 4160 Acked Read 7 Sent Done Done false 4161 Acked Read 4 Sent Done Done false 4162 Acked Read 4 Sent Done Done false 4163 Acked Flush 0 Sent Done Done false 4164 Acked Write 7 Sent Done Done false 4165 Acked Flush 0 Sent Done Done false 4166 Acked Read 10 Sent Done Done false 4167 Acked Read 7 Sent Done Done false 4168 Acked Read 2 Sent Done Done false 4169 Acked Write 1 Sent Done Done false 4170 Acked Read 5 Sent Done Done false 4171 Acked Read 5 Sent Done Done false 4172 Acked Write 9 Sent Done Done false 4173 Acked Write 6 Sent Done Done false 4174 Acked Read 2 Sent Done Done false 4175 Acked Read 8 Sent Done Done false 4176 Acked Write 9 Sent Done Done false 4177 Acked Read 1 Sent Done Done false 4178 Acked Read 1 Sent Done Done false 4179 Acked Write 3 Sent Done Done false 4180 Acked Write 10 Sent Done Done false 4181 Acked Read 4 Sent Done Done false 4182 Acked Flush 0 Sent Done Done false 4183 Acked Write 6 Sent Done Done false 4184 Acked Write 5 Sent Done Done false 4185 Acked Write 5 Sent Done Done false 4186 Acked Write 5 Sent Done Done false 4187 Acked Read 9 Sent Done Done false 4188 Acked Write 2 Sent Done Done false 4189 Acked Read 8 Sent Done Done false 4190 Acked Read 2 Sent Done Done false 4191 Acked Read 9 Sent Done Done false 4192 Acked Write 6 Sent Done Done false 4193 Acked Read 9 Sent Done Done false 4194 Acked Write 6 Sent Done Done false 4195 Acked Read 10 Sent Done Done false 4196 Acked Read 8 Sent Done Done false 4197 Acked Read 7 Sent Done Done false 4198 Acked Write 5 Sent Done Done false 4199 Acked Read 2 Sent Done Done false 4200 Acked Read 4 Sent Done Done false 4201 Acked Write 10 Sent Done Done false 4202 Acked Write 3 Sent Done Done false 4203 Acked Read 1 Sent Done Done false 4204 Acked Write 9 Sent Done Done false 4205 Acked Read 7 Sent Done Done false 4206 Acked Read 5 Sent Done Done false 4207 Acked Write 8 Sent Done Done false 4208 Acked Read 3 Sent Done Done false 4209 Acked Write 3 Sent Done Done false 4210 Acked Read 7 Sent Done Done false 4211 Acked Flush 0 Sent Done Done false 4212 Acked Read 5 Sent Done Done false 4213 Acked Write 8 Sent Done Done false 4214 Acked Read 2 Sent Done Done false 4215 Acked Read 2 Sent Done Done false 4216 Acked Read 9 Sent Done Done false 4217 Acked Write 7 Sent Done Done false 4218 Acked Write 1 Sent Done Done false 4219 Acked Read 5 Sent Done Done false 4220 Acked Read 2 Sent Done Done false 4221 Acked Write 10 Sent Done Done false 4222 Acked Flush 0 Sent Done Done false 4223 Acked Read 2 Sent Done Done false 4224 Acked Read 2 Sent Done Done false 4225 Acked Write 5 Sent Done Done false 4226 Acked Write 4 Sent Done Done false 4227 Acked Read 7 Sent Done Done false 4228 Acked Flush 0 Sent Done Done false 4229 Acked Read 9 Sent Done Done false 4230 Acked Write 1 Sent Done Done false 4231 Acked Write 5 Sent Done Done false 4232 Acked Read 1 Sent Done Done false 4233 Acked Write 8 Sent Done Done false 4234 Acked Write 10 Sent Done Done false 4235 Acked Write 2 Sent Done Done false 4236 Acked Read 2 Sent Done Done false 4237 Acked Read 2 Sent Done Done false 4238 Acked Read 10 Sent Done Done false 4239 Acked Flush 0 Sent Done Done false 4240 Acked Flush 0 Sent Done Done false 4241 Acked Read 7 Sent Done Done false 4242 Acked Read 5 Sent Done Done false 4243 Acked Write 4 Sent Done Done false 4244 Acked Write 6 Sent Done Done false 4245 Acked Write 7 Sent Done Done false 4246 Acked Write 2 Sent Done Done false 4247 Acked Write 2 Sent Done Done false 4248 Acked Read 1 Sent Done Done false 4249 Acked Read 6 Sent Done Done false 4250 Acked Flush 0 Sent Done Done false 4251 Acked Write 5 Sent Done Done false 4252 Acked Flush 0 Sent Done Done false 4253 Acked Read 1 Sent Done Done false 4254 Acked Write 8 Sent Done Done false 4255 Acked Flush 0 Sent Done Done false 4256 Acked Read 7 Sent Done Done false 4257 Acked Read 9 Sent Done Done false 4258 Acked Read 3 Sent Done Done false 4259 Acked Read 4 Sent Done Done false 4260 Acked Flush 0 Sent Done Done false 4261 Acked Read 6 Sent Done Done false 4262 Acked Read 10 Sent Done Done false 4263 Acked Write 9 Sent Done Done false 4264 Acked Write 3 Sent Done Done false 4265 Acked Write 4 Sent Done Done false 4266 Acked Write 9 Sent Done Done false 4267 Acked Write 4 Sent Done Done false 4268 Acked Read 2 Sent Done Done false 4269 Acked Write 9 Sent Done Done false 4270 Acked Read 8 Sent Done Done false 4271 Acked Read 5 Sent Done Done false 4272 Acked Flush 0 Sent Done Done false 4273 Acked Read 3 Sent Done Done false 4274 Acked Read 7 Sent Done Done false 4275 Acked Read 5 Sent Done Done false 4276 Acked Read 7 Sent Done Done false 4277 Acked Read 6 Sent Done Done false 4278 Acked Read 7 Sent Done Done false 4279 Acked Read 4 Sent Done Done false 4280 Acked Write 9 Sent Done Done false 4281 Acked Read 2 Sent Done Done false 4282 Acked Read 9 Sent Done Done false 4283 Acked Read 8 Sent Done Done false 4284 Acked Read 3 Sent Done Done false 4285 Acked Read 3 Sent Done Done false 4286 Acked Read 8 Sent Done Done false 4287 Acked Write 6 Sent Done Done false 4288 Acked Flush 0 Sent Done Done false 4289 Acked Read 3 Sent Done Done false 4290 Acked Read 10 Sent Done Done false 4291 Acked Read 3 Sent Done Done false 4292 Acked Read 7 Sent Done Done false 4293 Acked Write 5 Sent Done Done false 4294 Acked Read 10 Sent Done Done false 4295 Acked Read 6 Sent Done Done false 4296 Acked Read 8 Sent Done Done false 4297 Acked Read 1 Sent Done Done false 4298 Acked Flush 0 Sent Done Done false 4299 Acked Flush 0 Sent Done Done false 4300 Acked Write 5 Sent Done Done false 4301 Acked Flush 0 Sent Done Done false 4302 Acked Read 9 Sent Done Done false 4303 Acked Flush 0 Sent Done Done false 4304 Acked Read 9 Sent Done Done false 4305 Acked Read 6 Sent Done Done false 4306 Acked Read 6 Sent Done Done false 4307 Acked Flush 0 Sent Done Done false 4308 Acked Read 4 Sent Done Done false 4309 Acked Write 1 Sent Done Done false 4310 Acked Write 10 Sent Done Done false 4311 Acked Flush 0 Sent Done Done false 4312 Acked Write 7 Sent Done Done false 4313 Acked Read 4 Sent Done Done false 4314 Acked Read 10 Sent Done Done false 4315 Acked Write 6 Sent Done Done false 4316 Acked Flush 0 Sent Done Done false 4317 Acked Read 10 Sent Done Done false 4318 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4016 4318 4318 Downstairs last five completed: 4016 4015 4014 4013 4012 Upstairs last five completed: 4318 4317 4316 4315 4314 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4017 Acked Write 1 Sent Done Done false 4018 Acked Write 6 Sent Done Done false 4019 Acked Write 2 Sent Done Done false 4020 Acked Read 5 Sent Done Done false 4021 Acked Read 3 Sent Done Done false 4022 Acked Write 8 Sent Done Done false 4023 Acked Write 5 Sent Done Done false 4024 Acked Write 4 Sent Done Done false 4025 Acked Read 1 Sent Done Done false 4026 Acked Read 10 Sent Done Done false 4027 Acked Write 5 Sent Done Done false 4028 Acked Write 6 Sent Done Done false 4029 Acked Flush 0 Sent Done Done false 4030 Acked Write 5 Sent Done Done false 4031 Acked Flush 0 Sent Done Done false 4032 Acked Read 6 Sent Done Done false 4033 Acked Write 6 Sent Done Done false 4034 Acked Read 4 Sent Done Done false 4035 Acked Flush 0 Sent Done Done false 4036 Acked Read 10 Sent Done Done false 4037 Acked Read 9 Sent Done Done false 4038 Acked Flush 0 Sent Done Done false 4039 Acked Read 8 Sent Done Done false 4040 Acked Write 8 Sent Done Done false 4041 Acked Read 1 Sent Done Done false 4042 Acked Read 5 Sent Done Done false 4043 Acked Read 10 Sent Done Done false 4044 Acked Write 6 Sent Done Done false 4045 Acked Read 7 Sent Done Done false 4046 Acked Read 2 Sent Done Done false 4047 Acked Write 9 Sent Done Done false 4048 Acked Read 3 Sent Done Done false 4049 Acked Write 6 Sent Done Done false 4050 Acked Read 9 Sent Done Done false 4051 Acked Read 5 Sent Done Done false 4052 Acked Read 1 Sent Done Done false 4053 Acked Read 9 Sent Done Done false 4054 Acked Read 7 Sent Done Done false 4055 Acked Write 1 Sent Done Done false 4056 Acked Write 1 Sent Done Done false 4057 Acked Read 8 Sent Done Done false 4058 Acked Write 10 Sent Done Done false 4059 Acked Write 8 Sent Done Done false 4060 Acked Read 3 Sent Done Done false 4061 Acked Read 6 Sent Done Done false 4062 Acked Read 10 Sent Done Done false 4063 Acked Write 4 Sent Done Done false 4064 Acked Write 5 Sent Done Done false 4065 Acked Read 1 Sent Done Done false 4066 Acked Write 8 Sent Done Done false 4067 Acked Write 7 Sent Done Done false 4068 Acked Read 1 Sent Done Done false 4069 Acked Read 3 Sent Done Done false 4070 Acked Write 9 Sent Done Done false 4071 Acked Read 9 Sent Done Done false 4072 Acked Read 10 Sent Done Done false 4073 Acked Read 8 Sent Done Done false 4074 Acked Write 5 Sent Done Done false 4075 Acked Write 9 Sent Done Done false 4076 Acked Read 4 Sent Done Done false 4077 Acked Read 8 Sent Done Done false 4078 Acked Flush 0 Sent Done Done false 4079 Acked Read 3 Sent Done Done false 4080 Acked Write 6 Sent Done Done false 4081 Acked Read 1 Sent Done Done false 4082 Acked Read 7 Sent Done Done false 4083 Acked Write 3 Sent Done Done false 4084 Acked Read 4 Sent Done Done false 4085 Acked Read 4 Sent Done Done false 4086 Acked Read 7 Sent Done Done false 4087 Acked Write 1 Sent Done Done false 4088 Acked Flush 0 Sent Done Done false 4089 Acked Flush 0 Sent Done Done false 4090 Acked Read 1 Sent Done Done false 4091 Acked Write 10 Sent Done Done false 4092 Acked Write 3 Sent Done Done false 4093 Acked Read 8 Sent Done Done false 4094 Acked Write 7 Sent Done Done false 4095 Acked Write 7 Sent Done Done false 4096 Acked Read 6 Sent Done Done false 4097 Acked Write 3 Sent Done Done false 4098 Acked Read 6 Sent Done Done false 4099 Acked Read 6 Sent Done Done false 4100 Acked Write 9 Sent Done Done false 4101 Acked Flush 0 Sent Done Done false 4102 Acked Read 4 Sent Done Done false 4103 Acked Read 10 Sent Done Done false 4104 Acked Write 2 Sent Done Done false 4105 Acked Read 7 Sent Done Done false 4106 Acked Read 4 Sent Done Done false 4107 Acked Write 10 Sent Done Done false 4108 Acked Write 1 Sent Done Done false 4109 Acked Read 6 Sent Done Done false 4110 Acked Read 4 Sent Done Done false 4111 Acked Write 10 Sent Done Done false 4112 Acked Read 5 Sent Done Done false 4113 Acked Write 3 Sent Done Done false 4114 Acked Write 4 Sent Done Done false 4115 Acked Read 1 Sent Done Done false 4116 Acked Read 9 Sent Done Done false 4117 Acked Flush 0 Sent Done Done false 4118 Acked Read 3 Sent Done Done false 4119 Acked Write 10 Sent Done Done false 4120 Acked Read 4 Sent Done Done false 4121 Acked Write 9 Sent Done Done false 4122 Acked Flush 0 Sent Done Done false 4123 Acked Write 2 Sent Done Done false 4124 Acked Write 9 Sent Done Done false 4125 Acked Read 8 Sent Done Done false 4126 Acked Flush 0 Sent Done Done false 4127 Acked Read 4 Sent Done Done false 4128 Acked Read 7 Sent Done Done false 4129 Acked Read 5 Sent Done Done false 4130 Acked Read 4 Sent Done Done false 4131 Acked Write 10 Sent Done Done false 4132 Acked Flush 0 Sent Done Done false 4133 Acked Read 9 Sent Done Done false 4134 Acked Flush 0 Sent Done Done false 4135 Acked Write 3 Sent Done Done false 4136 Acked Read 10 Sent Done Done false 4137 Acked Write 5 Sent Done Done false 4138 Acked Write 3 Sent Done Done false 4139 Acked Write 8 Sent Done Done false 4140 Acked Read 6 Sent Done Done false 4141 Acked Read 5 Sent Done Done false 4142 Acked Write 8 Sent Done Done false 4143 Acked Write 2 Sent Done Done false 4144 Acked Read 7 Sent Done Done false 4145 Acked Write 2 Sent Done Done false 4146 Acked Read 6 Sent Done Done false 4147 Acked Write 5 Sent Done Done false 4148 Acked Read 8 Sent Done Done false 4149 Acked Read 4 Sent Done Done false 4150 Acked Write 5 Sent Done Done false 4151 Acked Write 1 Sent Done Done false 4152 Acked Write 9 Sent Done Done false 4153 Acked Write 4 Sent Done Done false 4154 Acked Write 3 Sent Done Done false 4155 Acked Write 2 Sent Done Done false 4156 Acked Read 8 Sent Done Done false 4157 Acked Read 4 Sent Done Done false 4158 Acked Flush 0 Sent Done Done false 4159 Acked Write 7 Sent Done Done false 4160 Acked Read 7 Sent Done Done false 4161 Acked Read 4 Sent Done Done false 4162 Acked Read 4 Sent Done Done false 4163 Acked Flush 0 Sent Done Done false 4164 Acked Write 7 Sent Done Done false 4165 Acked Flush 0 Sent Done Done false 4166 Acked Read 10 Sent Done Done false 4167 Acked Read 7 Sent Done Done false 4168 Acked Read 2 Sent Done Done false 4169 Acked Write 1 Sent Done Done false 4170 Acked Read 5 Sent Done Done false 4171 Acked Read 5 Sent Done Done false 4172 Acked Write 9 Sent Done Done false 4173 Acked Write 6 Sent Done Done false 4174 Acked Read 2 Sent Done Done false 4175 Acked Read 8 Sent Done Done false 4176 Acked Write 9 Sent Done Done false 4177 Acked Read 1 Sent Done Done false 4178 Acked Read 1 Sent Done Done false 4179 Acked Write 3 Sent Done Done false 4180 Acked Write 10 Sent Done Done false 4181 Acked Read 4 Sent Done Done false 4182 Acked Flush 0 Sent Done Done false 4183 Acked Write 6 Sent Done Done false 4184 Acked Write 5 Sent Done Done false 4185 Acked Write 5 Sent Done Done false 4186 Acked Write 5 Sent Done Done false 4187 Acked Read 9 Sent Done Done false 4188 Acked Write 2 Sent Done Done false 4189 Acked Read 8 Sent Done Done false 4190 Acked Read 2 Sent Done Done false 4191 Acked Read 9 Sent Done Done false 4192 Acked Write 6 Sent Done Done false 4193 Acked Read 9 Sent Done Done false 4194 Acked Write 6 Sent Done Done false 4195 Acked Read 10 Sent Done Done false 4196 Acked Read 8 Sent Done Done false 4197 Acked Read 7 Sent Done Done false 4198 Acked Write 5 Sent Done Done false 4199 Acked Read 2 Sent Done Done false 4200 Acked Read 4 Sent Done Done false 4201 Acked Write 10 Sent Done Done false 4202 Acked Write 3 Sent Done Done false 4203 Acked Read 1 Sent Done Done false 4204 Acked Write 9 Sent Done Done false 4205 Acked Read 7 Sent Done Done false 4206 Acked Read 5 Sent Done Done false 4207 Acked Write 8 Sent Done Done false 4208 Acked Read 3 Sent Done Done false 4209 Acked Write 3 Sent Done Done false 4210 Acked Read 7 Sent Done Done false 4211 Acked Flush 0 Sent Done Done false 4212 Acked Read 5 Sent Done Done false 4213 Acked Write 8 Sent Done Done false 4214 Acked Read 2 Sent Done Done false 4215 Acked Read 2 Sent Done Done false 4216 Acked Read 9 Sent Done Done false 4217 Acked Write 7 Sent Done Done false 4218 Acked Write 1 Sent Done Done false 4219 Acked Read 5 Sent Done Done false 4220 Acked Read 2 Sent Done Done false 4221 Acked Write 10 Sent Done Done false 4222 Acked Flush 0 Sent Done Done false 4223 Acked Read 2 Sent Done Done false 4224 Acked Read 2 Sent Done Done false 4225 Acked Write 5 Sent Done Done false 4226 Acked Write 4 Sent Done Done false 4227 Acked Read 7 Sent Done Done false 4228 Acked Flush 0 Sent Done Done false 4229 Acked Read 9 Sent Done Done false 4230 Acked Write 1 Sent Done Done false 4231 Acked Write 5 Sent Done Done false 4232 Acked Read 1 Sent Done Done false 4233 Acked Write 8 Sent Done Done false 4234 Acked Write 10 Sent Done Done false 4235 Acked Write 2 Sent Done Done false 4236 Acked Read 2 Sent Done Done false 4237 Acked Read 2 Sent Done Done false 4238 Acked Read 10 Sent Done Done false 4239 Acked Flush 0 Sent Done Done false 4240 Acked Flush 0 Sent Done Done false 4241 Acked Read 7 Sent Done Done false 4242 Acked Read 5 Sent Done Done false 4243 Acked Write 4 Sent Done Done false 4244 Acked Write 6 Sent Done Done false 4245 Acked Write 7 Sent Done Done false 4246 Acked Write 2 Sent Done Done false 4247 Acked Write 2 Sent Done Done false 4248 Acked Read 1 Sent Done Done false 4249 Acked Read 6 Sent Done Done false 4250 Acked Flush 0 Sent Done Done false 4251 Acked Write 5 Sent Done Done false 4252 Acked Flush 0 Sent Done Done false 4253 Acked Read 1 Sent Done Done false 4254 Acked Write 8 Sent Done Done false 4255 Acked Flush 0 Sent Done Done false 4256 Acked Read 7 Sent Done Done false 4257 Acked Read 9 Sent Done Done false 4258 Acked Read 3 Sent Done Done false 4259 Acked Read 4 Sent Done Done false 4260 Acked Flush 0 Sent Done Done false 4261 Acked Read 6 Sent Done Done false 4262 Acked Read 10 Sent Done Done false 4263 Acked Write 9 Sent Done Done false 4264 Acked Write 3 Sent Done Done false 4265 Acked Write 4 Sent Done Done false 4266 Acked Write 9 Sent Done Done false 4267 Acked Write 4 Sent Done Done false 4268 Acked Read 2 Sent Done Done false 4269 Acked Write 9 Sent Done Done false 4270 Acked Read 8 Sent Done Done false 4271 Acked Read 5 Sent Done Done false 4272 Acked Flush 0 Sent Done Done false 4273 Acked Read 3 Sent Done Done false 4274 Acked Read 7 Sent Done Done false 4275 Acked Read 5 Sent Done Done false 4276 Acked Read 7 Sent Done Done false 4277 Acked Read 6 Sent Done Done false 4278 Acked Read 7 Sent Done Done false 4279 Acked Read 4 Sent Done Done false 4280 Acked Write 9 Sent Done Done false 4281 Acked Read 2 Sent Done Done false 4282 Acked Read 9 Sent Done Done false 4283 Acked Read 8 Sent Done Done false 4284 Acked Read 3 Sent Done Done false 4285 Acked Read 3 Sent Done Done false 4286 Acked Read 8 Sent Done Done false 4287 Acked Write 6 Sent Done Done false 4288 Acked Flush 0 Sent Done Done false 4289 Acked Read 3 Sent Done Done false 4290 Acked Read 10 Sent Done Done false 4291 Acked Read 3 Sent Done Done false 4292 Acked Read 7 Sent Done Done false 4293 Acked Write 5 Sent Done Done false 4294 Acked Read 10 Sent Done Done false 4295 Acked Read 6 Sent Done Done false 4296 Acked Read 8 Sent Done Done false 4297 Acked Read 1 Sent Done Done false 4298 Acked Flush 0 Sent Done Done false 4299 Acked Flush 0 Sent Done Done false 4300 Acked Write 5 Sent Done Done false 4301 Acked Flush 0 Sent Done Done false 4302 Acked Read 9 Sent Done Done false 4303 Acked Flush 0 Sent Done Done false 4304 Acked Read 9 Sent Done Done false 4305 Acked Read 6 Sent Done Done false 4306 Acked Read 6 Sent Done Done false 4307 Acked Flush 0 Sent Done Done false 4308 Acked Read 4 Sent Done Done false 4309 Acked Write 1 Sent Done Done false 4310 Acked Write 10 Sent Done Done false 4311 Acked Flush 0 Sent Done Done false 4312 Acked Write 7 Sent Done Done false 4313 Acked Read 4 Sent Done Done false 4314 Acked Read 10 Sent Done Done false 4315 Acked Write 6 Sent Done Done false 4316 Acked Flush 0 Sent Done Done false 4317 Acked Read 10 Sent Done Done false 4318 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4016 4318 4318 Downstairs last five completed: 4016 4015 4014 4013 4012 Upstairs last five completed: 4318 4317 4316 4315 4314 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.117892524Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:63797)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.117960736Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:26.119034977Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119499243Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119524896Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 4016","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119540121Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 4016","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119745816Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.1197695Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119784036Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 4016","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.119797752Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 4016","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.120010445Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4318 4317 4316 4315 4314 Upstairs last five completed: 4318 4317 4316 4315 4314 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:44:26.363 INFO request completed, latency_us: 371, [8820] Got stop action so:false kr:false response_code: 204, uri: /stop/cid/1, method: POST, req_id: f4f5ced1-7a9d-4b86-a02e-9a6a4f1927ab, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:26.364378293Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:26.364418252Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:26.364435568Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.364451593Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.364467009Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:44:26.364 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:26.364550257Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} INFO request completed, latency_us: 134, response_code: 200, uri: /state/cid/1, method: GET, req_id: 181fd0a6-3426-4a7a-811f-c0746b966d07, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:44:30.365 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/1, method: GET, req_id: 46b4687c-7094-47c6-af43-1b03bec28f67, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 2474 len 12288 data: 3 4 4 002/300 Read block 2553 len 4096 003/300 Flush 004/300 Write block 2611 len 20480 data: 3 2 2 2 3 005/300 Read block 681 len 16384 006/300 Write block 2763 len 32768 data: 4 4 4 3 3 4 3 3 007/300 Read block 4096 len 8192 008/300 Read block 299 len 36864 009/300 Read block 1022 len 20480 010/300 Read block 322 len 8192 011/300 Read block 138 len 36864 012/300 Read block 2305 len 4096 013/300 Read block 902 len 36864 014/300 Read block 2035 len 20480 015/300 Read block 1321 len 28672 016/300 Write block 1879 len 24576 data: 2 2 2 2 4 4 017/300 Read block 3018 len 8192 018/300 Write block 2539 len 28672 data: 3 4 7 8 7 7 7 019/300 Flush 020/300 Write block 207 len 40960 data: 5 6 5 5 6 5 5 4 3 2 021/300 Write block 3549 len 40960 data: 5 4 4 5 4 4 5 5 3 4 022/300 Write block 1601 len 8192 data: 4 4 023/300 Read block 3642 len 28672 024/300 Write block 3535 len 28672 data: 3 4 4 4 4 4 4 025/300 Read block 3434 len 28672 026/300 Flush 027/300 Read block 1282 len 28672 028/300 Read block 2423 len 40960 029/300 Read block 1244 len 36864 030/300 Read block 3267 len 32768 031/300 Write block 1233 len 16384 data: 4 4 3 2 032/300 Write block 3911 len 8192 data: 2 2 033/300 Read block 3523 len 4096 034/300 Write block 4599 len 12288 data: 2 2 2 035/300 Write block 3192 len 12288 data: 3 3 3 036/300 Write block 2019 len 16384 data: 3 4 4 5 037/300 Read block 4176 len 12288 038/300 Read block 1310 len 4096 039/300 Read block 3945 len 40960 040/300 Write block 1078 len 28672 data: 2 2 2 3 5 5 5 041/300 Read block 3564 len 24576 042/300 Write block 3618 len 28672 data: 3 3 5 4 4 3 3 043/300 Write block 3029 len 8192 data: 2 4 044/300 Write block 541 len 36864 data: 4 3 3 3 3 2 2 2 2 045/300 Flush 046/300 Read block 3060 len 36864 047/300 Write block 95 len 12288 data: 3 2 2 048/300 Flush 049/300 Write block 3355 len 20480 data: 3 3 4 4 5 050/300 Write block 678 len 20480 data: 5 4 5 5 4 051/300 Write block 3193 len 28672 data: 4 4 3 3 4 4 3 052/300 Write block 340 len 16384 data: 4 6 5 5 053/300 Write block 2372 len 24576 data: 4 6 6 6 5 5 054/300 Flush 055/300 Read block 2765 len 4096 056/300 Read block 2432 len 20480 057/300 Read block 730 len 16384 058/300 Read block 4007 len 8192 059/300 Write block 3799 len 24576 data: 2 2 2 2 3 4 060/300 Read block 19 len 24576 061/300 Write block 4780 len 32768 data: 3 2 2 2 2 2 2 2 062/300 Read block 3408 len 16384 063/300 Read block 3138 len 8192 064/300 Read block 1877 len 20480 065/300 Write block 1193 len 36864 data: 2 2 3 4 4 4 4 5 4 066/300 Read block 640 len 4096 067/300 Read block 2074 len 40960 068/300 Flush 069/300 Write block 1248 len 36864 data: 5 5 5 4 4 6 6 3 3 070/300 Write block 4226 len 16384 data: 2 3 3 3 071/300 Flush 072/300 Write block 1719 len 36864 data: 2 2 3 3 3 3 3 3 3 073/300 Read block 4107 len 12288 074/300 Read block 1967 len 24576 075/300 Write block 2702 len 20480 data: 3 3 3 3 2 076/300 Read block 4009 len 4096 077/300 Read block 4051 len 8192 078/300 Write block 1957 len 28672 data: 4 4 4 4 3 4 4 079/300 Write block 1575 len 12288 data: 4 4 3 080/300 Read block 521 len 40960 081/300 Read block 537 len 20480 082/300 Write block 1612 len 4096 data: 3 083/300 Read block 2527 len 4096 084/300 Write block 1258 len 36864 data: 2 2 2 2 2 3 3 4 4 085/300 Write block 1974 len 20480 data: 2 2 2 2 2 086/300 Write block 4743 len 36864 data: 5 5 5 5 5 5 4 4 3 087/300 Write block 3423 len 8192 data: 2 2 088/300 Read block 2219 len 28672 089/300 Write block 3765 len 16384 data: 4 4 3 3 090/300 Write block 1297 len 12288 data: 4 3 2 091/300 Write block 4410 len 36864 data: 2 2 2 2 3 3 3 4 4 092/300 Read block 3739 len 16384 093/300 Flush 094/300 Read block 18 len 4096 095/300 Write block 311 len 32768 data: 2 2 2 3 3 2 3 2 096/300 Read block 452 len 40960 097/300 Write block 3069 len 40960 data: 3 2 2 3 2 2 2 3 3 2 098/300 Write block 1553 len 36864 data: 3 3 3 3 4 4 5 4 4 099/300 Read block 3854 len 20480 100/300 Read block 179 len 16384 101/300 Read block 3369 len 28672 102/300 Read block 3992 len 32768 103/300 Flush 104/300 Write block 694 len 32768 data: 3 4 4 4 4 3 2 2 105/300 Write block 1102 len 40960 data: 3 3 2 3 3 3 3 4 4 4 106/300 Write block 206 len 28672 data: 3 6 7 6 6 7 6 107/300 Write block 2922 len 16384 data: 3 4 4 3 108/300 Flush 109/300 Read block 4059 len 28672 110/300 Write block 631 len 8192 data: 4 4 111/300 Flush 112/300 Write block 3377 len 36864 data: 4 4 5 4 4 4 5 5 4 113/300 Flush 114/300 Read block 1891 len 20480 115/300 Read block 338 len 40960 116/300 Read block 2389 len 28672 117/300 Write block 1666 len 20480 data: 3 2 3 2 2 118/300 Write block 1854 len 32768 data: 4 3 3 3 3 3 3 3 119/300 Read block 2963 len 20480 120/300 Write block 4031 len 36864 data: 5 5 4 3 4 3 3 3 3 121/300 Write block 166 len 40960 data: 3 3 4 4 4 4 3 3 3 3 122/300 Flush 123/300 Read block 2973 len 24576 124/300 Read block 2229 len 4096 125/300 Flush 126/300 Write block 178 len 32768 data: 3 3 4 4 4 5 5 4 127/300 Read block 4927 len 24576 128/300 Read block 531 len 12288 129/300 Flush 130/300 Write block 1609 len 24576 data: 3 2 3 4 3 4 131/300 Read block 3959 len 16384 132/300 Read block 3823 len 40960 133/300 Write block 1032 len 28672 data: 4 5 4 2 2 2 3 134/300 Read block 4457 len 36864 135/300 Read block 95 len 32768 136/300 Write block 1494 len 16384 data: 2 3 3 4 137/300 Write block 2839 len 28672 data: 3 3 3 4 4 3 3 138/300 Read block 4749 len 8192 139/300 Flush 140/300 Read block 367 len 4096 141/300 Write block 2027 len 20480 data: 3 3 3 2 2 142/300 Write block 3197 len 32768 data: 5 5 4 3 3 3 2 3 143/300 Read block 4232 len 20480 144/300 Write block 4876 len 28672 data: 2 2 4 4 3 2 2 145/300 Flush 146/300 Write block 952 len 40960 data: 3 3 3 3 2 2 2 2 2 2 147/300 Write block 2191 len 20480 data: 2 2 2 2 2 148/300 Write block 448 len 8192 data: 3 3 149/300 Read block 4831 len 36864 150/300 Write block 4633 len 24576 data: 2 2 2 3 3 2 151/300 Write block 3733 len 28672 data: 4 5 5 5 5 5 4 152/300 Write block 2660 len 8192 data: 2 2 153/300 Read block 4520 len 12288 154/300 Read block 700 len 32768 155/300 Write block 2298 len 40960 data: 3 3 2 2 2 3 3 4 5 4 156/300 Read block 4685 len 12288 157/300 Write block 718 len 20480 data: 4 3 3 2 3 158/300 Write block 2069 len 40960 data: 2 2 3 3 3 3 3 2 2 3 159/300 Write block 2814 len 4096 data: 5 160/300 Write block 3643 len 32768 data: 3 3 2 2 2 3 3 2 161/300 Read block 1952 len 24576 162/300 Write block 2096 len 36864 data: 4 2 3 4 5 6 6 5 5 163/300 Read block 4238 len 28672 164/300 Read block 1638 len 40960 165/300 Read block 162 len 40960 166/300 Read block 4431 len 8192 167/300 Read block 3544 len 40960 168/300 Read block 3663 len 40960 169/300 Write block 2375 len 28672 data: 7 6 6 6 6 6 4 170/300 Write block 571 len 40960 data: 3 3 3 3 3 3 3 2 3 4 171/300 Read block 3144 len 32768 172/300 Write block 2967 len 40960 data: 3 2 4 5 5 5 5 5 6 6 173/300 Write block 1588 len 8192 data: 4 3 174/300 Read block 2371 len 40960 175/300 Write block 4943 len 4096 data: 8 176/300 Read block 962 len 16384 177/300 Read block 4432 len 8192 178/300 Write block 1003 len 40960 data: 3 3 3 3 3 3 3 3 4 5 179/300 Flush 180/300 Read block 1660 len 4096 181/300 Read block 2911 len 36864 182/300 Write block 4472 len 20480 data: 5 5 5 2 3 183/300 Write block 1897 len 4096 data: 3 184/300 Write block 2694 len 28672 data: 3 4 4 3 3 3 3 185/300 Read block 2783 len 24576 186/300 Read block 733 len 16384 187/300 Write block 3269 len 16384 data: 2 2 2 2 188/300 Read block 3732 len 20480 189/300 Read block 971 len 28672 190/300 Write block 4632 len 16384 data: 2 3 3 3 191/300 Write block 1108 len 8192 data: 4 5 192/300 Write block 2038 len 32768 data: 2 4 4 4 4 6 7 7 193/300 Read block 3090 len 36864 194/300 Read block 551 len 36864 195/300 Read block 1184 len 8192 196/300 Write block 1545 len 36864 data: 5 2 2 2 2 2 3 3 4 197/300 Write block 908 len 8192 data: 2 2 198/300 Write block 674 len 24576 data: 3 3 3 3 6 5 199/300 Read block 3990 len 36864 200/300 Write block 276 len 8192 data: 3 3 201/300 Read block 1226 len 4096 202/300 Write block 3332 len 8192 data: 5 4 203/300 Read block 237 len 8192 204/300 Write block 3382 len 16384 data: 5 6 6 5 205/300 Read block 3934 len 12288 206/300 Write block 630 len 24576 data: 3 5 5 4 3 4 207/300 Read block 3133 len 24576 208/300 Write block 3209 len 20480 data: 5 4 4 3 4 209/300 Read block 1509 len 8192 210/300 Write block 1153 len 40960 data: 3 3 3 2 3 4 4 3 3 3 211/300 Write block 723 len 8192 data: 3 2 212/300 Flush 213/300 Read block 2088 len 12288 214/300 Write block 3256 len 16384 data: 3 4 2 2 215/300 Write block 2469 len 32768 data: 4 4 5 3 3 4 5 5 216/300 Write block 1157 len 4096 data: 4 217/300 Write block 1815 len 32768 data: 5 6 6 5 5 6 6 7 218/300 Read block 1478 len 32768 219/300 Read block 1803 len 24576 220/300 Read block 2489 len 28672 221/300 Read block 4719 len 4096 222/300 Write block 4069 len 32768 data: 4 4 3 4 4 4 4 3 223/300 Write block 3831 len 8192 data: 3 3 224/300 Read block 1696 len 28672 225/300 Read block 4921 len 12288 226/300 Write block 4413 len 40960 data: 3 4 4 4 5 5 6 5 5 6 227/300 Write block 4389 len 8192 data: 5 4 228/300 Flush 229/300 Write block 4008 len 16384 data: 2 2 2 2 230/300 Write block 3214 len 4096 data: 3 231/300 Read block 3584 len 32768 232/300 Read block 71 len 4096 233/300 Flush 234/300 Write block 522 len 36864 data: 2 4 6 7 6 4 5 5 5 235/300 Write block 1958 len 8192 data: 5 5 236/300 Read block 4145 len 16384 237/300 Read block 3463 len 28672 238/300 Write block 3551 len 28672 data: 5 6 5 5 6 6 4 239/300 Read block 888 len 16384 240/300 Write block 4131 len 12288 data: 3 3 3 241/300 Write block 2899 len 36864 data: 3 3 4 4 3 2 2 2 2 242/300 Read block 4243 len 8192 243/300 Read block 1030 len 24576 244/300 Write block 3061 len 24576 data: 2 2 2 2 2 2 245/300 Read block 4042 len 40960 246/300 Read block 1109 len 8192 247/300 Read block 1717 len 36864 248/300 Write block 144 len 32768 data: 2 2 4 4 4 3 2 2 249/300 Read block 4318 len 40960 250/300 Read block 3235 len 16384 251/300 Read block 195 len 40960 252/300 Read block 3887 len 20480 253/300 Write block 3611 len 28672 data: 2 2 3 3 4 4 3 254/300 Read block 501 len 24576 255/300 Read block 4106 len 40960 256/300 Read block 4895 len 28672 257/300 Read block 2694 len 12288 258/300 Write block 4758 len 24576 data: 3 3 3 3 3 3 259/300 Write block 2501 len 36864 data: 4 5 4 4 4 4 4 3 3 260/300 Write block 333 len 16384 data: 3 3 3 3 261/300 Read block 2310 len 36864 262/300 Read block 298 len 4096 263/300 Write block 1149 len 8192 data: 4 4 264/300 Write block 2144 len 36864 data: 5 3 2 2 2 2 2 2 2 265/300 Write block 2777 len 40960 data: 3 4 4 3 2 2 2 3 3 3 266/300 Read block 3751 len 20480 267/300 Write block 2715 len 36864 data: 2 3 3 3 3 3 3 2 3 268/300 Write block 2309 len 16384 data: 4 4 4 4 269/300 Read block 1611 len 16384 270/300 Read block 5 len 24576 271/300 Flush 272/300 Flush 273/300 Read block 4825 len 4096 274/300 Flush 275/300 Write block 4413 len 8192 data: 4 5 276/300 Read block 301 len 36864 277/300 Write block 389 len 20480 data: 3 3 4 3 3 278/300 Read block 3307 len 8192 279/300 Read block 2934 len 36864 280/300 Read block 136 len 28672 281/300 Write block 1540 len 4096 data: 8 282/300 Flush 283/300 Read block 1220 len 24576 284/300 Write block 802 len 12288 data: 3 3 4 285/300 Write block 202 len 32768 data: 2 2 2 2 4 7 8 7 286/300 Write block 4798 len 4096 data: 6 287/300 Read block 2445 len 4096 288/300 Read block 1329 len 32768 289/300 Write block 528 len 16384 data: 6 6 6 6 290/300 Read block 386 len 28672 291/300 Read block 4087 len 28672 292/300 Read block 4412 len 20480 293/300 Write block 1909 len 28672 data: 4 6 5 5 5 4 3 294/300 Read block 3757 len 16384 295/300 Read block 2411 len 28672 296/300 Read block 2016 len 32768 297/300 Read block 3099 len 40960 298/300 Write block 2399 len 40960 data: 5 5 5 5 5 5 5 5 4 4 299/300 Read block 1340 len 8192 300/300 Write block 1748 len 4096 data: 2 301/300 Read block 4206 len 24576 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:44:30.596 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 1adbf31f-adf9-441b-a48e-23b1ef933126, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4319 Acked Write 3 Done Sent Done false 4320 Acked Read 1 Done Sent Done false 4321 Acked Flush 0 Done Sent Done false 4322 Acked Write 5 Done Sent Done false 4323 Acked Read 4 Done Sent Done false 4324 Acked Write 8 Done Sent Done false 4325 Acked Read 2 Done Sent Done false 4326 Acked Read 9 Done Sent Done false 4327 Acked Read 5 Done Sent Done false 4328 Acked Read 2 Done Sent Done false 4329 Acked Read 9 Done Sent Done false 4330 Acked Read 1 Done Sent Done false 4331 Acked Read 9 Done Sent Done false 4332 Acked Read 5 Done Sent Done false 4333 Acked Read 7 Done Sent Done false 4334 Acked Write 6 Done Sent Done false 4335 Acked Read 2 Done Sent Done false 4336 Acked Write 7 Done Sent Done false 4337 Acked Flush 0 Done Sent Done false 4338 Acked Write 10 Done Sent Done false 4339 Acked Write 10 Done Sent Done false 4340 Acked Write 2 Done Sent Done false 4341 Acked Read 7 Done Sent Done false 4342 Acked Write 7 Done Sent Done false 4343 Acked Read 7 Done Sent Done false 4344 Acked Flush 0 Done Sent Done false 4345 Acked Read 7 Done Sent Done false 4346 Acked Read 10 Done Sent Done false 4347 Acked Read 9 Done Sent Done false 4348 Acked Read 8 Done Sent Done false 4349 Acked Write 4 Done Sent Done false 4350 Acked Write 2 Done Sent Done false 4351 Acked Read 1 Done Sent Done false 4352 Acked Write 3 Done Sent Done false 4353 Acked Write 3 Done Sent Done false 4354 Acked Write 4 Done Sent Done false 4355 Acked Read 3 Done Sent Done false 4356 Acked Read 1 Done Sent Done false 4357 Acked Read 10 Done Sent Done false 4358 Acked Write 7 Done Sent Done false 4359 Acked Read 6 Done Sent Done false 4360 Acked Write 7 Done Sent Done false 4361 Acked Write 2 Done Sent Done false 4362 Acked Write 9 Done Sent Done false 4363 Acked Flush 0 Done Sent Done false 4364 Acked Read 9 Done Sent Done false 4365 Acked Write 3 Done Sent Done false 4366 Acked Flush 0 Done Sent Done false 4367 Acked Write 5 Done Sent Done false 4368 Acked Write 5 Done Sent Done false 4369 Acked Write 7 Done Sent Done false 4370 Acked Write 4 Done Sent Done false 4371 Acked Write 6 Done Sent Done false 4372 Acked Flush 0 Done Sent Done false 4373 Acked Read 1 Done Sent Done false 4374 Acked Read 5 Done Sent Done false 4375 Acked Read 4 Done Sent Done false 4376 Acked Read 2 Done Sent Done false 4377 Acked Write 6 Done Sent Done false 4378 Acked Read 6 Done Sent Done false 4379 Acked Write 8 Done Sent Done false 4380 Acked Read 4 Done Sent Done false 4381 Acked Read 2 Done Sent Done false 4382 Acked Read 5 Done Sent Done false 4383 Acked Write 9 Done Sent Done false 4384 Acked Read 1 Done Sent Done false 4385 Acked Read 10 Done Sent Done false 4386 Acked Flush 0 Done Sent Done false 4387 Acked Write 9 Done Sent Done false 4388 Acked Write 4 Done Sent Done false 4389 Acked Flush 0 Done Sent Done false 4390 Acked Write 9 Done Sent Done false 4391 Acked Read 3 Done Sent Done false 4392 Acked Read 6 Done Sent Done false 4393 Acked Write 5 Done Sent Done false 4394 Acked Read 1 Done Sent Done false 4395 Acked Read 2 Done Sent Done false 4396 Acked Write 7 Done Sent Done false 4397 Acked Write 3 Done Sent Done false 4398 Acked Read 10 Done Sent Done false 4399 Acked Read 5 Done Sent Done false 4400 Acked Write 1 Done Sent Done false 4401 Acked Read 1 Done Sent Done false 4402 Acked Write 9 Done Sent Done false 4403 Acked Write 5 Done Sent Done false 4404 Acked Write 9 Done Sent Done false 4405 Acked Write 2 Done Sent Done false 4406 Acked Read 7 Done Sent Done false 4407 Acked Write 4 Done Sent Done false 4408 Acked Write 3 Done Sent Done false 4409 Acked Write 9 Done Sent Done false 4410 Acked Read 4 Done Sent Done false 4411 Acked Flush 0 Done Sent Done false 4412 Acked Read 1 Done Sent Done false 4413 Acked Write 8 Done Sent Done false 4414 Acked Read 10 Done Sent Done false 4415 Acked Write 10 Done Sent Done false 4416 Acked Write 9 Done Sent Done false 4417 Acked Read 5 Done Sent Done false 4418 Acked Read 4 Done Sent Done false 4419 Acked Read 7 Done Sent Done false 4420 Acked Read 8 Done Sent Done false 4421 Acked Flush 0 Done Sent Done false 4422 Acked Write 8 Done Sent Done false 4423 Acked Write 10 Done Sent Done false 4424 Acked Write 7 Done Sent Done false 4425 Acked Write 4 Done Sent Done false 4426 Acked Flush 0 Done Sent Done false 4427 Acked Read 7 Done Sent Done false 4428 Acked Write 2 Done Sent Done false 4429 Acked Flush 0 Done Sent Done false 4430 Acked Write 9 Done Sent Done false 4431 Acked Flush 0 Done Sent Done false 4432 Acked Read 5 Done Sent Done false 4433 Acked Read 10 Done Sent Done false 4434 Acked Read 7 Done Sent Done false 4435 Acked Write 5 Done Sent Done false 4436 Acked Write 8 Done Sent Done false 4437 Acked Read 5 Done Sent Done false 4438 Acked Write 9 Done Sent Done false 4439 Acked Write 10 Done Sent Done false 4440 Acked Flush 0 Done Sent Done false 4441 Acked Read 6 Done Sent Done false 4442 Acked Read 1 Done Sent Done false 4443 Acked Flush 0 Done Sent Done false 4444 Acked Write 8 Done Sent Done false 4445 Acked Read 6 Done Sent Done false 4446 Acked Read 3 Done Sent Done false 4447 Acked Flush 0 Done Sent Done false 4448 Acked Write 6 Done Sent Done false 4449 Acked Read 4 Done Sent Done false 4450 Acked Read 10 Done Sent Done false 4451 Acked Write 7 Done Sent Done false 4452 Acked Read 9 Done Sent Done false 4453 Acked Read 8 Done Sent Done false 4454 Acked Write 4 Done Sent Done false 4455 Acked Write 7 Done Sent Done false 4456 Acked Read 2 Done Sent Done false 4457 Acked Flush 0 Done Sent Done false 4458 Acked Read 1 Done Sent Done false 4459 Acked Write 5 Done Sent Done false 4460 Acked Write 8 Done Sent Done false 4461 Acked Read 5 Done Sent Done false 4462 Acked Write 7 Done Sent Done false 4463 Acked Flush 0 Done Sent Done false 4464 Acked Write 10 Done Sent Done false 4465 Acked Write 5 Done Sent Done false 4466 Acked Write 2 Done Sent Done false 4467 Acked Read 9 Done Sent Done false 4468 Acked Write 6 Done Sent Done false 4469 Acked Write 7 Done Sent Done false 4470 Acked Write 2 Done Sent Done false 4471 Acked Read 3 Done Sent Done false 4472 Acked Read 8 Done Sent Done false 4473 Acked Write 10 Done Sent Done false 4474 Acked Read 3 Done Sent Done false 4475 Acked Write 5 Done Sent Done false 4476 Acked Write 10 Done Sent Done false 4477 Acked Write 1 Done Sent Done false 4478 Acked Write 8 Done Sent Done false 4479 Acked Read 6 Done Sent Done false 4480 Acked Write 9 Done Sent Done false 4481 Acked Read 7 Done Sent Done false 4482 Acked Read 10 Done Sent Done false 4483 Acked Read 10 Done Sent Done false 4484 Acked Read 2 Done Sent Done false 4485 Acked Read 10 Done Sent Done false 4486 Acked Read 10 Done Sent Done false 4487 Acked Write 7 Done Sent Done false 4488 Acked Write 10 Done Sent Done false 4489 Acked Read 8 Done Sent Done false 4490 Acked Write 10 Done Sent Done false 4491 Acked Write 2 Done Sent Done false 4492 Acked Read 10 Done Sent Done false 4493 Acked Write 1 Done Sent Done false 4494 Acked Read 4 Done Sent Done false 4495 Acked Read 2 Done Sent Done false 4496 Acked Write 10 Done Sent Done false 4497 Acked Flush 0 Done Sent Done false 4498 Acked Read 1 Done Sent Done false 4499 Acked Read 9 Done Sent Done false 4500 Acked Write 5 Done Sent Done false 4501 Acked Write 1 Done Sent Done false 4502 Acked Write 7 Done Sent Done false 4503 Acked Read 6 Done Sent Done false 4504 Acked Read 4 Done Sent Done false 4505 Acked Write 4 Done Sent Done false 4506 Acked Read 5 Done Sent Done false 4507 Acked Read 7 Done Sent Done false 4508 Acked Write 4 Done Sent Done false 4509 Acked Write 2 Done Sent Done false 4510 Acked Write 8 Done Sent Done false 4511 Acked Read 9 Done Sent Done false 4512 Acked Read 9 Done Sent Done false 4513 Acked Read 2 Done Sent Done false 4514 Acked Write 9 Done Sent Done false 4515 Acked Write 2 Done Sent Done false 4516 Acked Write 6 Done Sent Done false 4517 Acked Read 9 Done Sent Done false 4518 Acked Write 2 Done Sent Done false 4519 Acked Read 1 Done Sent Done false 4520 Acked Write 2 Done Sent Done false 4521 Acked Read 2 Done Sent Done false 4522 Acked Write 4 Done Sent Done false 4523 Acked Read 3 Done Sent Done false 4524 Acked Write 6 Done Sent Done false 4525 Acked Read 6 Done Sent Done false 4526 Acked Write 5 Done Sent Done false 4527 Acked Read 2 Done Sent Done false 4528 Acked Write 10 Done Sent Done false 4529 Acked Write 2 Done Sent Done false 4530 Acked Flush 0 Done Sent Done false 4531 Acked Read 3 Done Sent Done false 4532 Acked Write 4 Done Sent Done false 4533 Acked Write 8 Done Sent Done false 4534 Acked Write 1 Done Sent Done false 4535 Acked Write 8 Done Sent Done false 4536 Acked Read 8 Done Sent Done false 4537 Acked Read 6 Done Sent Done false 4538 Acked Read 7 Done Sent Done false 4539 Acked Read 1 Done Sent Done false 4540 Acked Write 8 Done Sent Done false 4541 Acked Write 2 Done Sent Done false 4542 Acked Read 7 Done Sent Done false 4543 Acked Read 3 Done Sent Done false 4544 Acked Write 10 Done Sent Done false 4545 Acked Write 2 Done Sent Done false 4546 Acked Flush 0 Done Sent Done false 4547 Acked Write 4 Done Sent Done false 4548 Acked Write 1 Done Sent Done false 4549 Acked Read 8 Done Sent Done false 4550 Acked Read 1 Done Sent Done false 4551 Acked Flush 0 Done Sent Done false 4552 Acked Write 9 Done Sent Done false 4553 Acked Write 2 Done Sent Done false 4554 Acked Read 4 Done Sent Done false 4555 Acked Read 7 Done Sent Done false 4556 Acked Write 7 Done Sent Done false 4557 Acked Read 4 Done Sent Done false 4558 Acked Write 3 Done Sent Done false 4559 Acked Write 9 Done Sent Done false 4560 Acked Read 2 Done Sent Done false 4561 Acked Read 6 Done Sent Done false 4562 Acked Write 6 Done Sent Done false 4563 Acked Read 10 Done Sent Done false 4564 Acked Read 2 Done Sent Done false 4565 Acked Read 9 Done Sent Done false 4566 Acked Write 8 Done Sent Done false 4567 Acked Read 10 Done Sent Done false 4568 Acked Read 4 Done Sent Done false 4569 Acked Read 10 Done Sent Done false 4570 Acked Read 5 Done Sent Done false 4571 Acked Write 7 Done Sent Done false 4572 Acked Read 6 Done Sent Done false 4573 Acked Read 10 Done Sent Done false 4574 Acked Read 7 Done Sent Done false 4575 Acked Read 3 Done Sent Done false 4576 Acked Write 6 Done Sent Done false 4577 Acked Write 9 Done Sent Done false 4578 Acked Write 4 Done Sent Done false 4579 Acked Read 9 Done Sent Done false 4580 Acked Read 1 Done Sent Done false 4581 Acked Write 2 Done Sent Done false 4582 Acked Write 9 Done Sent Done false 4583 Acked Write 10 Done Sent Done false 4584 Acked Read 5 Done Sent Done false 4585 Acked Write 9 Done Sent Done false 4586 Acked Write 4 Done Sent Done false 4587 Acked Read 4 Done Sent Done false 4588 Acked Read 6 Done Sent Done false 4589 Acked Flush 0 Done Sent Done false 4590 Acked Flush 0 Done Sent Done false 4591 Acked Read 1 Done Sent Done false 4592 Acked Flush 0 Done Sent Done false 4593 Acked Write 2 Done Sent Done false 4594 Acked Read 9 Done Sent Done false 4595 Acked Write 5 Done Sent Done false 4596 Acked Read 2 Done Sent Done false 4597 Acked Read 9 Done Sent Done false 4598 Acked Read 7 Done Sent Done false 4599 Acked Write 1 Done Sent Done false 4600 Acked Flush 0 Done Sent Done false 4601 Acked Read 6 Done Sent Done false 4602 Acked Write 3 Done Sent Done false 4603 Acked Write 8 Done Sent Done false 4604 Acked Write 1 Done Sent Done false 4605 Acked Read 1 Done Sent Done false 4606 Acked Read 8 Done Sent Done false 4607 Acked Write 4 Done Sent Done false 4608 Acked Read 7 Done Sent Done false 4609 Acked Read 7 Done Sent Done false 4610 Acked Read 5 Done Sent Done false 4611 Acked Write 7 Done Sent Done false 4612 Acked Read 4 Done Sent Done false 4613 Acked Read 7 Done Sent Done false 4614 Acked Read 8 Done Sent Done false 4615 Acked Read 10 Done Sent Done false 4616 Acked Write 10 Done Sent Done false 4617 Acked Read 2 Done Sent Done false 4618 Acked Write 1 Done Sent Done false 4619 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4600 4318 4600 Downstairs last five completed: 4318 4317 4316 4315 4314 Upstairs last five completed: 4619 4618 4617 4616 4615 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4319 Acked Write 3 Done Sent Done false 4320 Acked Read 1 Done Sent Done false 4321 Acked Flush 0 Done Sent Done false 4322 Acked Write 5 Done Sent Done false 4323 Acked Read 4 Done Sent Done false 4324 Acked Write 8 Done Sent Done false 4325 Acked Read 2 Done Sent Done false 4326 Acked Read 9 Done Sent Done false 4327 Acked Read 5 Done Sent Done false 4328 Acked Read 2 Done Sent Done false 4329 Acked Read 9 Done Sent Done false 4330 Acked Read 1 Done Sent Done false 4331 Acked Read 9 Done Sent Done false 4332 Acked Read 5 Done Sent Done false 4333 Acked Read 7 Done Sent Done false 4334 Acked Write 6 Done Sent Done false 4335 Acked Read 2 Done Sent Done false 4336 Acked Write 7 Done Sent Done false 4337 Acked Flush 0 Done Sent Done false 4338 Acked Write 10 Done Sent Done false 4339 Acked Write 10 Done Sent Done false 4340 Acked Write 2 Done Sent Done false 4341 Acked Read 7 Done Sent Done false 4342 Acked Write 7 Done Sent Done false 4343 Acked Read 7 Done Sent Done false 4344 Acked Flush 0 Done Sent Done false 4345 Acked Read 7 Done Sent Done false 4346 Acked Read 10 Done Sent Done false 4347 Acked Read 9 Done Sent Done false 4348 Acked Read 8 Done Sent Done false 4349 Acked Write 4 Done Sent Done false 4350 Acked Write 2 Done Sent Done false 4351 Acked Read 1 Done Sent Done false 4352 Acked Write 3 Done Sent Done false 4353 Acked Write 3 Done Sent Done false 4354 Acked Write 4 Done Sent Done false 4355 Acked Read 3 Done Sent Done false 4356 Acked Read 1 Done Sent Done false 4357 Acked Read 10 Done Sent Done false 4358 Acked Write 7 Done Sent Done false 4359 Acked Read 6 Done Sent Done false 4360 Acked Write 7 Done Sent Done false 4361 Acked Write 2 Done Sent Done false 4362 Acked Write 9 Done Sent Done false 4363 Acked Flush 0 Done Sent Done false 4364 Acked Read 9 Done Sent Done false 4365 Acked Write 3 Done Sent Done false 4366 Acked Flush 0 Done Sent Done false 4367 Acked Write 5 Done Sent Done false 4368 Acked Write 5 Done Sent Done false 4369 Acked Write 7 Done Sent Done false 4370 Acked Write 4 Done Sent Done false 4371 Acked Write 6 Done Sent Done false 4372 Acked Flush 0 Done Sent Done false 4373 Acked Read 1 Done Sent Done false 4374 Acked Read 5 Done Sent Done false 4375 Acked Read 4 Done Sent Done false 4376 Acked Read 2 Done Sent Done false 4377 Acked Write 6 Done Sent Done false 4378 Acked Read 6 Done Sent Done false 4379 Acked Write 8 Done Sent Done false 4380 Acked Read 4 Done Sent Done false 4381 Acked Read 2 Done Sent Done false 4382 Acked Read 5 Done Sent Done false 4383 Acked Write 9 Done Sent Done false 4384 Acked Read 1 Done Sent Done false 4385 Acked Read 10 Done Sent Done false 4386 Acked Flush 0 Done Sent Done false 4387 Acked Write 9 Done Sent Done false 4388 Acked Write 4 Done Sent Done false 4389 Acked Flush 0 Done Sent Done false 4390 Acked Write 9 Done Sent Done false 4391 Acked Read 3 Done Sent Done false 4392 Acked Read 6 Done Sent Done false 4393 Acked Write 5 Done Sent Done false 4394 Acked Read 1 Done Sent Done false 4395 Acked Read 2 Done Sent Done false 4396 Acked Write 7 Done Sent Done false 4397 Acked Write 3 Done Sent Done false 4398 Acked Read 10 Done Sent Done false 4399 Acked Read 5 Done Sent Done false 4400 Acked Write 1 Done Sent Done false 4401 Acked Read 1 Done Sent Done false 4402 Acked Write 9 Done Sent Done false 4403 Acked Write 5 Done Sent Done false 4404 Acked Write 9 Done Sent Done false 4405 Acked Write 2 Done Sent Done false 4406 Acked Read 7 Done Sent Done false 4407 Acked Write 4 Done Sent Done false 4408 Acked Write 3 Done Sent Done false 4409 Acked Write 9 Done Sent Done false 4410 Acked Read 4 Done Sent Done false 4411 Acked Flush 0 Done Sent Done false 4412 Acked Read 1 Done Sent Done false 4413 Acked Write 8 Done Sent Done false 4414 Acked Read 10 Done Sent Done false 4415 Acked Write 10 Done Sent Done false 4416 Acked Write 9 Done Sent Done false 4417 Acked Read 5 Done Sent Done false 4418 Acked Read 4 Done Sent Done false 4419 Acked Read 7 Done Sent Done false 4420 Acked Read 8 Done Sent Done false 4421 Acked Flush 0 Done Sent Done false 4422 Acked Write 8 Done Sent Done false 4423 Acked Write 10 Done Sent Done false 4424 Acked Write 7 Done Sent Done false 4425 Acked Write 4 Done Sent Done false 4426 Acked Flush 0 Done Sent Done false 4427 Acked Read 7 Done Sent Done false 4428 Acked Write 2 Done Sent Done false 4429 Acked Flush 0 Done Sent Done false 4430 Acked Write 9 Done Sent Done false 4431 Acked Flush 0 Done Sent Done false 4432 Acked Read 5 Done Sent Done false 4433 Acked Read 10 Done Sent Done false 4434 Acked Read 7 Done Sent Done false 4435 Acked Write 5 Done Sent Done false 4436 Acked Write 8 Done Sent Done false 4437 Acked Read 5 Done Sent Done false 4438 Acked Write 9 Done Sent Done false 4439 Acked Write 10 Done Sent Done false 4440 Acked Flush 0 Done Sent Done false 4441 Acked Read 6 Done Sent Done false 4442 Acked Read 1 Done Sent Done false 4443 Acked Flush 0 Done Sent Done false 4444 Acked Write 8 Done Sent Done false 4445 Acked Read 6 Done Sent Done false 4446 Acked Read 3 Done Sent Done false 4447 Acked Flush 0 Done Sent Done false 4448 Acked Write 6 Done Sent Done false 4449 Acked Read 4 Done Sent Done false 4450 Acked Read 10 Done Sent Done false 4451 Acked Write 7 Done Sent Done false 4452 Acked Read 9 Done Sent Done false 4453 Acked Read 8 Done Sent Done false 4454 Acked Write 4 Done Sent Done false 4455 Acked Write 7 Done Sent Done false 4456 Acked Read 2 Done Sent Done false 4457 Acked Flush 0 Done Sent Done false 4458 Acked Read 1 Done Sent Done false 4459 Acked Write 5 Done Sent Done false 4460 Acked Write 8 Done Sent Done false 4461 Acked Read 5 Done Sent Done false 4462 Acked Write 7 Done Sent Done false 4463 Acked Flush 0 Done Sent Done false 4464 Acked Write 10 Done Sent Done false 4465 Acked Write 5 Done Sent Done false 4466 Acked Write 2 Done Sent Done false 4467 Acked Read 9 Done Sent Done false 4468 Acked Write 6 Done Sent Done false 4469 Acked Write 7 Done Sent Done false 4470 Acked Write 2 Done Sent Done false 4471 Acked Read 3 Done Sent Done false 4472 Acked Read 8 Done Sent Done false 4473 Acked Write 10 Done Sent Done false 4474 Acked Read 3 Done Sent Done false 4475 Acked Write 5 Done Sent Done false 4476 Acked Write 10 Done Sent Done false 4477 Acked Write 1 Done Sent Done false 4478 Acked Write 8 Done Sent Done false 4479 Acked Read 6 Done Sent Done false 4480 Acked Write 9 Done Sent Done false 4481 Acked Read 7 Done Sent Done false 4482 Acked Read 10 Done Sent Done false 4483 Acked Read 10 Done Sent Done false 4484 Acked Read 2 Done Sent Done false 4485 Acked Read 10 Done Sent Done false 4486 Acked Read 10 Done Sent Done false 4487 Acked Write 7 Done Sent Done false 4488 Acked Write 10 Done Sent Done false 4489 Acked Read 8 Done Sent Done false 4490 Acked Write 10 Done Sent Done false 4491 Acked Write 2 Done Sent Done false 4492 Acked Read 10 Done Sent Done false 4493 Acked Write 1 Done Sent Done false 4494 Acked Read 4 Done Sent Done false 4495 Acked Read 2 Done Sent Done false 4496 Acked Write 10 Done Sent Done false 4497 Acked Flush 0 Done Sent Done false 4498 Acked Read 1 Done Sent Done false 4499 Acked Read 9 Done Sent Done false 4500 Acked Write 5 Done Sent Done false 4501 Acked Write 1 Done Sent Done false 4502 Acked Write 7 Done Sent Done false 4503 Acked Read 6 Done Sent Done false 4504 Acked Read 4 Done Sent Done false 4505 Acked Write 4 Done Sent Done false 4506 Acked Read 5 Done Sent Done false 4507 Acked Read 7 Done Sent Done false 4508 Acked Write 4 Done Sent Done false 4509 Acked Write 2 Done Sent Done false 4510 Acked Write 8 Done Sent Done false 4511 Acked Read 9 Done Sent Done false 4512 Acked Read 9 Done Sent Done false 4513 Acked Read 2 Done Sent Done false 4514 Acked Write 9 Done Sent Done false 4515 Acked Write 2 Done Sent Done false 4516 Acked Write 6 Done Sent Done false 4517 Acked Read 9 Done Sent Done false 4518 Acked Write 2 Done Sent Done false 4519 Acked Read 1 Done Sent Done false 4520 Acked Write 2 Done Sent Done false 4521 Acked Read 2 Done Sent Done false 4522 Acked Write 4 Done Sent Done false 4523 Acked Read 3 Done Sent Done false 4524 Acked Write 6 Done Sent Done false 4525 Acked Read 6 Done Sent Done false 4526 Acked Write 5 Done Sent Done false 4527 Acked Read 2 Done Sent Done false 4528 Acked Write 10 Done Sent Done false 4529 Acked Write 2 Done Sent Done false 4530 Acked Flush 0 Done Sent Done false 4531 Acked Read 3 Done Sent Done false 4532 Acked Write 4 Done Sent Done false 4533 Acked Write 8 Done Sent Done false 4534 Acked Write 1 Done Sent Done false 4535 Acked Write 8 Done Sent Done false 4536 Acked Read 8 Done Sent Done false 4537 Acked Read 6 Done Sent Done false 4538 Acked Read 7 Done Sent Done false 4539 Acked Read 1 Done Sent Done false 4540 Acked Write 8 Done Sent Done false 4541 Acked Write 2 Done Sent Done false 4542 Acked Read 7 Done Sent Done false 4543 Acked Read 3 Done Sent Done false 4544 Acked Write 10 Done Sent Done false 4545 Acked Write 2 Done Sent Done false 4546 Acked Flush 0 Done Sent Done false 4547 Acked Write 4 Done Sent Done false 4548 Acked Write 1 Done Sent Done false 4549 Acked Read 8 Done Sent Done false 4550 Acked Read 1 Done Sent Done false 4551 Acked Flush 0 Done Sent Done false 4552 Acked Write 9 Done Sent Done false 4553 Acked Write 2 Done Sent Done false 4554 Acked Read 4 Done Sent Done false 4555 Acked Read 7 Done Sent Done false 4556 Acked Write 7 Done Sent Done false 4557 Acked Read 4 Done Sent Done false 4558 Acked Write 3 Done Sent Done false 4559 Acked Write 9 Done Sent Done false 4560 Acked Read 2 Done Sent Done false 4561 Acked Read 6 Done Sent Done false 4562 Acked Write 6 Done Sent Done false 4563 Acked Read 10 Done Sent Done false 4564 Acked Read 2 Done Sent Done false 4565 Acked Read 9 Done Sent Done false 4566 Acked Write 8 Done Sent Done false 4567 Acked Read 10 Done Sent Done false 4568 Acked Read 4 Done Sent Done false 4569 Acked Read 10 Done Sent Done false 4570 Acked Read 5 Done Sent Done false 4571 Acked Write 7 Done Sent Done false 4572 Acked Read 6 Done Sent Done false 4573 Acked Read 10 Done Sent Done false 4574 Acked Read 7 Done Sent Done false 4575 Acked Read 3 Done Sent Done false 4576 Acked Write 6 Done Sent Done false 4577 Acked Write 9 Done Sent Done false 4578 Acked Write 4 Done Sent Done false 4579 Acked Read 9 Done Sent Done false 4580 Acked Read 1 Done Sent Done false 4581 Acked Write 2 Done Sent Done false 4582 Acked Write 9 Done Sent Done false 4583 Acked Write 10 Done Sent Done false 4584 Acked Read 5 Done Sent Done false 4585 Acked Write 9 Done Sent Done false 4586 Acked Write 4 Done Sent Done false 4587 Acked Read 4 Done Sent Done false 4588 Acked Read 6 Done Sent Done false 4589 Acked Flush 0 Done Sent Done false 4590 Acked Flush 0 Done Sent Done false 4591 Acked Read 1 Done Sent Done false 4592 Acked Flush 0 Done Sent Done false 4593 Acked Write 2 Done Sent Done false 4594 Acked Read 9 Done Sent Done false 4595 Acked Write 5 Done Sent Done false 4596 Acked Read 2 Done Sent Done false 4597 Acked Read 9 Done Sent Done false 4598 Acked Read 7 Done Sent Done false 4599 Acked Write 1 Done Sent Done false 4600 Acked Flush 0 Done Sent Done false 4601 Acked Read 6 Done Sent Done false 4602 Acked Write 3 Done Sent Done false 4603 Acked Write 8 Done Sent Done false 4604 Acked Write 1 Done Sent Done false 4605 Acked Read 1 Done Sent Done false 4606 Acked Read 8 Done Sent Done false 4607 Acked Write 4 Done Sent Done false 4608 Acked Read 7 Done Sent Done false 4609 Acked Read 7 Done Sent Done false 4610 Acked Read 5 Done Sent Done false 4611 Acked Write 7 Done Sent Done false 4612 Acked Read 4 Done Sent Done false 4613 Acked Read 7 Done Sent Done false 4614 Acked Read 8 Done Sent Done false 4615 Acked Read 10 Done Sent Done false 4616 Acked Write 10 Done Sent Done false 4617 Acked Read 2 Done Sent Done false 4618 Acked Write 1 Done Sent Done false 4619 Acked Read 6 Done Sent Done false 4620 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4318 4620 Downstairs last five completed: 4318 4317 4316 4315 4314 Upstairs last five completed: 4620 4619 4618 4617 4616 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:36.365530923Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:36.365591077Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:36.365609542Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:36.365626188Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:36.365713305Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4319 Acked Write 3 Done Sent Done false 4320 Acked Read 1 Done Sent Done false 4321 Acked Flush 0 Done Sent Done false 4322 Acked Write 5 Done Sent Done false 4323 Acked Read 4 Done Sent Done false 4324 Acked Write 8 Done Sent Done false 4325 Acked Read 2 Done Sent Done false 4326 Acked Read 9 Done Sent Done false 4327 Acked Read 5 Done Sent Done false 4328 Acked Read 2 Done Sent Done false 4329 Acked Read 9 Done Sent Done false 4330 Acked Read 1 Done Sent Done false 4331 Acked Read 9 Done Sent Done false 4332 Acked Read 5 Done Sent Done false 4333 Acked Read 7 Done Sent Done false 4334 Acked Write 6 Done Sent Done false 4335 Acked Read 2 Done Sent Done false 4336 Acked Write 7 Done Sent Done false 4337 Acked Flush 0 Done Sent Done false 4338 Acked Write 10 Done Sent Done false 4339 Acked Write 10 Done Sent Done false 4340 Acked Write 2 Done Sent Done false 4341 Acked Read 7 Done Sent Done false 4342 Acked Write 7 Done Sent Done false 4343 Acked Read 7 Done Sent Done false 4344 Acked Flush 0 Done Sent Done false 4345 Acked Read 7 Done Sent Done false 4346 Acked Read 10 Done Sent Done false 4347 Acked Read 9 Done Sent Done false 4348 Acked Read 8 Done Sent Done false 4349 Acked Write 4 Done Sent Done false 4350 Acked Write 2 Done Sent Done false 4351 Acked Read 1 Done Sent Done false 4352 Acked Write 3 Done Sent Done false 4353 Acked Write 3 Done Sent Done false 4354 Acked Write 4 Done Sent Done false 4355 Acked Read 3 Done Sent Done false 4356 Acked Read 1 Done Sent Done false 4357 Acked Read 10 Done Sent Done false 4358 Acked Write 7 Done Sent Done false 4359 Acked Read 6 Done Sent Done false 4360 Acked Write 7 Done Sent Done false 4361 Acked Write 2 Done Sent Done false 4362 Acked Write 9 Done Sent Done false 4363 Acked Flush 0 Done Sent Done false 4364 Acked Read 9 Done Sent Done false 4365 Acked Write 3 Done Sent Done false 4366 Acked Flush 0 Done Sent Done false 4367 Acked Write 5 Done Sent Done false 4368 Acked Write 5 Done Sent Done false 4369 Acked Write 7 Done Sent Done false 4370 Acked Write 4 Done Sent Done false 4371 Acked Write 6 Done Sent Done false 4372 Acked Flush 0 Done Sent Done false 4373 Acked Read 1 Done Sent Done false 4374 Acked Read 5 Done Sent Done false 4375 Acked Read 4 Done Sent Done false 4376 Acked Read 2 Done Sent Done false 4377 Acked Write 6 Done Sent Done false 4378 Acked Read 6 Done Sent Done false 4379 Acked Write 8 Done Sent Done false 4380 Acked Read 4 Done Sent Done false 4381 Acked Read 2 Done Sent Done false 4382 Acked Read 5 Done Sent Done false 4383 Acked Write 9 Done Sent Done false 4384 Acked Read 1 Done Sent Done false 4385 Acked Read 10 Done Sent Done false 4386 Acked Flush 0 Done Sent Done false 4387 Acked Write 9 Done Sent Done false 4388 Acked Write 4 Done Sent Done false 4389 Acked Flush 0 Done Sent Done false 4390 Acked Write 9 Done Sent Done false 4391 Acked Read 3 Done Sent Done false 4392 Acked Read 6 Done Sent Done false 4393 Acked Write 5 Done Sent Done false 4394 Acked Read 1 Done Sent Done false 4395 Acked Read 2 Done Sent Done false 4396 Acked Write 7 Done Sent Done false 4397 Acked Write 3 Done Sent Done false 4398 Acked Read 10 Done Sent Done false 4399 Acked Read 5 Done Sent Done false 4400 Acked Write 1 Done Sent Done false 4401 Acked Read 1 Done Sent Done false 4402 Acked Write 9 Done Sent Done false 4403 Acked Write 5 Done Sent Done false 4404 Acked Write 9 Done Sent Done false 4405 Acked Write 2 Done Sent Done false 4406 Acked Read 7 Done Sent Done false 4407 Acked Write 4 Done Sent Done false 4408 Acked Write 3 Done Sent Done false 4409 Acked Write 9 Done Sent Done false 4410 Acked Read 4 Done Sent Done false 4411 Acked Flush 0 Done Sent Done false 4412 Acked Read 1 Done Sent Done false 4413 Acked Write 8 Done Sent Done false 4414 Acked Read 10 Done Sent Done false 4415 Acked Write 10 Done Sent Done false 4416 Acked Write 9 Done Sent Done false 4417 Acked Read 5 Done Sent Done false 4418 Acked Read 4 Done Sent Done false 4419 Acked Read 7 Done Sent Done false 4420 Acked Read 8 Done Sent Done false 4421 Acked Flush 0 Done Sent Done false 4422 Acked Write 8 Done Sent Done false 4423 Acked Write 10 Done Sent Done false 4424 Acked Write 7 Done Sent Done false 4425 Acked Write 4 Done Sent Done false 4426 Acked Flush 0 Done Sent Done false 4427 Acked Read 7 Done Sent Done false 4428 Acked Write 2 Done Sent Done false 4429 Acked Flush 0 Done Sent Done false 4430 Acked Write 9 Done Sent Done false 4431 Acked Flush 0 Done Sent Done false 4432 Acked Read 5 Done Sent Done false 4433 Acked Read 10 Done Sent Done false 4434 Acked Read 7 Done Sent Done false 4435 Acked Write 5 Done Sent Done false 4436 Acked Write 8 Done Sent Done false 4437 Acked Read 5 Done Sent Done false 4438 Acked Write 9 Done Sent Done false 4439 Acked Write 10 Done Sent Done false 4440 Acked Flush 0 Done Sent Done false 4441 Acked Read 6 Done Sent Done false 4442 Acked Read 1 Done Sent Done false 4443 Acked Flush 0 Done Sent Done false 4444 Acked Write 8 Done Sent Done false 4445 Acked Read 6 Done Sent Done false 4446 Acked Read 3 Done Sent Done false 4447 Acked Flush 0 Done Sent Done false 4448 Acked Write 6 Done Sent Done false 4449 Acked Read 4 Done Sent Done false 4450 Acked Read 10 Done Sent Done false 4451 Acked Write 7 Done Sent Done false 4452 Acked Read 9 Done Sent Done false 4453 Acked Read 8 Done Sent Done false 4454 Acked Write 4 Done Sent Done false 4455 Acked Write 7 Done Sent Done false 4456 Acked Read 2 Done Sent Done false 4457 Acked Flush 0 Done Sent Done false 4458 Acked Read 1 Done Sent Done false 4459 Acked Write 5 Done Sent Done false 4460 Acked Write 8 Done Sent Done false 4461 Acked Read 5 Done Sent Done false 4462 Acked Write 7 Done Sent Done false 4463 Acked Flush 0 Done Sent Done false 4464 Acked Write 10 Done Sent Done false 4465 Acked Write 5 Done Sent Done false 4466 Acked Write 2 Done Sent Done false 4467 Acked Read 9 Done Sent Done false 4468 Acked Write 6 Done Sent Done false 4469 Acked Write 7 Done Sent Done false 4470 Acked Write 2 Done Sent Done false 4471 Acked Read 3 Done Sent Done false 4472 Acked Read 8 Done Sent Done false 4473 Acked Write 10 Done Sent Done false 4474 Acked Read 3 Done Sent Done false 4475 Acked Write 5 Done Sent Done false 4476 Acked Write 10 Done Sent Done false 4477 Acked Write 1 Done Sent Done false 4478 Acked Write 8 Done Sent Done false 4479 Acked Read 6 Done Sent Done false 4480 Acked Write 9 Done Sent Done false 4481 Acked Read 7 Done Sent Done false 4482 Acked Read 10 Done Sent Done false 4483 Acked Read 10 Done Sent Done false 4484 Acked Read 2 Done Sent Done false 4485 Acked Read 10 Done Sent Done false 4486 Acked Read 10 Done Sent Done false 4487 Acked Write 7 Done Sent Done false 4488 Acked Write 10 Done Sent Done false 4489 Acked Read 8 Done Sent Done false 4490 Acked Write 10 Done Sent Done false 4491 Acked Write 2 Done Sent Done false 4492 Acked Read 10 Done Sent Done false 4493 Acked Write 1 Done Sent Done false 4494 Acked Read 4 Done Sent Done false 4495 Acked Read 2 Done Sent Done false 4496 Acked Write 10 Done Sent Done false 4497 Acked Flush 0 Done Sent Done false 4498 Acked Read 1 Done Sent Done false 4499 Acked Read 9 Done Sent Done false 4500 Acked Write 5 Done Sent Done false 4501 Acked Write 1 Done Sent Done false 4502 Acked Write 7 Done Sent Done false 4503 Acked Read 6 Done Sent Done false 4504 Acked Read 4 Done Sent Done false 4505 Acked Write 4 Done Sent Done false 4506 Acked Read 5 Done Sent Done false 4507 Acked Read 7 Done Sent Done false 4508 Acked Write 4 Done Sent Done false 4509 Acked Write 2 Done Sent Done false 4510 Acked Write 8 Done Sent Done false 4511 Acked Read 9 Done Sent Done false 4512 Acked Read 9 Done Sent Done false 4513 Acked Read 2 Done Sent Done false 4514 Acked Write 9 Done Sent Done false 4515 Acked Write 2 Done Sent Done false 4516 Acked Write 6 Done Sent Done false 4517 Acked Read 9 Done Sent Done false 4518 Acked Write 2 Done Sent Done false 4519 Acked Read 1 Done Sent Done false 4520 Acked Write 2 Done Sent Done false 4521 Acked Read 2 Done Sent Done false 4522 Acked Write 4 Done Sent Done false 4523 Acked Read 3 Done Sent Done false 4524 Acked Write 6 Done Sent Done false 4525 Acked Read 6 Done Sent Done false 4526 Acked Write 5 Done Sent Done false 4527 Acked Read 2 Done Sent Done false 4528 Acked Write 10 Done Sent Done false 4529 Acked Write 2 Done Sent Done false 4530 Acked Flush 0 Done Sent Done false 4531 Acked Read 3 Done Sent Done false 4532 Acked Write 4 Done Sent Done false 4533 Acked Write 8 Done Sent Done false 4534 Acked Write 1 Done Sent Done false 4535 Acked Write 8 Done Sent Done false 4536 Acked Read 8 Done Sent Done false 4537 Acked Read 6 Done Sent Done false 4538 Acked Read 7 Done Sent Done false 4539 Acked Read 1 Done Sent Done false 4540 Acked Write 8 Done Sent Done false 4541 Acked Write 2 Done Sent Done false 4542 Acked Read 7 Done Sent Done false 4543 Acked Read 3 Done Sent Done false 4544 Acked Write 10 Done Sent Done false 4545 Acked Write 2 Done Sent Done false 4546 Acked Flush 0 Done Sent Done false 4547 Acked Write 4 Done Sent Done false 4548 Acked Write 1 Done Sent Done false 4549 Acked Read 8 Done Sent Done false 4550 Acked Read 1 Done Sent Done false 4551 Acked Flush 0 Done Sent Done false 4552 Acked Write 9 Done Sent Done false 4553 Acked Write 2 Done Sent Done false 4554 Acked Read 4 Done Sent Done false 4555 Acked Read 7 Done Sent Done false 4556 Acked Write 7 Done Sent Done false 4557 Acked Read 4 Done Sent Done false 4558 Acked Write 3 Done Sent Done false 4559 Acked Write 9 Done Sent Done false 4560 Acked Read 2 Done Sent Done false 4561 Acked Read 6 Done Sent Done false 4562 Acked Write 6 Done Sent Done false 4563 Acked Read 10 Done Sent Done false 4564 Acked Read 2 Done Sent Done false 4565 Acked Read 9 Done Sent Done false 4566 Acked Write 8 Done Sent Done false 4567 Acked Read 10 Done Sent Done false 4568 Acked Read 4 Done Sent Done false 4569 Acked Read 10 Done Sent Done false 4570 Acked Read 5 Done Sent Done false 4571 Acked Write 7 Done Sent Done false 4572 Acked Read 6 Done Sent Done false 4573 Acked Read 10 Done Sent Done false 4574 Acked Read 7 Done Sent Done false 4575 Acked Read 3 Done Sent Done false 4576 Acked Write 6 Done Sent Done false 4577 Acked Write 9 Done Sent Done false 4578 Acked Write 4 Done Sent Done false 4579 Acked Read 9 Done Sent Done false 4580 Acked Read 1 Done Sent Done false 4581 Acked Write 2 Done Sent Done false 4582 Acked Write 9 Done Sent Done false 4583 Acked Write 10 Done Sent Done false 4584 Acked Read 5 Done Sent Done false 4585 Acked Write 9 Done Sent Done false 4586 Acked Write 4 Done Sent Done false 4587 Acked Read 4 Done Sent Done false 4588 Acked Read 6 Done Sent Done false 4589 Acked Flush 0 Done Sent Done false 4590 Acked Flush 0 Done Sent Done false 4591 Acked Read 1 Done Sent Done false 4592 Acked Flush 0 Done Sent Done false 4593 Acked Write 2 Done Sent Done false 4594 Acked Read 9 Done Sent Done false 4595 Acked Write 5 Done Sent Done false 4596 Acked Read 2 Done Sent Done false 4597 Acked Read 9 Done Sent Done false 4598 Acked Read 7 Done Sent Done false 4599 Acked Write 1 Done Sent Done false 4600 Acked Flush 0 Done Sent Done false 4601 Acked Read 6 Done Sent Done false 4602 Acked Write 3 Done Sent Done false 4603 Acked Write 8 Done Sent Done false 4604 Acked Write 1 Done Sent Done false 4605 Acked Read 1 Done Sent Done false 4606 Acked Read 8 Done Sent Done false 4607 Acked Write 4 Done Sent Done false 4608 Acked Read 7 Done Sent Done false 4609 Acked Read 7 Done Sent Done false 4610 Acked Read 5 Done Sent Done false 4611 Acked Write 7 Done Sent Done false 4612 Acked Read 4 Done Sent Done false 4613 Acked Read 7 Done Sent Done false 4614 Acked Read 8 Done Sent Done false 4615 Acked Read 10 Done Sent Done false 4616 Acked Write 10 Done Sent Done false 4617 Acked Read 2 Done Sent Done false 4618 Acked Write 1 Done Sent Done false 4619 Acked Read 6 Done Sent Done false 4620 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4318 4620 Downstairs last five completed: 4318 4317 4316 4315 4314 Upstairs last five completed: 4620 4619 4618 4617 4616 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4319 Acked Write 3 Done Sent Done false 4320 Acked Read 1 Done Sent Done false 4321 Acked Flush 0 Done Sent Done false 4322 Acked Write 5 Done Sent Done false 4323 Acked Read 4 Done Sent Done false 4324 Acked Write 8 Done Sent Done false 4325 Acked Read 2 Done Sent Done false 4326 Acked Read 9 Done Sent Done false 4327 Acked Read 5 Done Sent Done false 4328 Acked Read 2 Done Sent Done false 4329 Acked Read 9 Done Sent Done false 4330 Acked Read 1 Done Sent Done false 4331 Acked Read 9 Done Sent Done false 4332 Acked Read 5 Done Sent Done false 4333 Acked Read 7 Done Sent Done false 4334 Acked Write 6 Done Sent Done false 4335 Acked Read 2 Done Sent Done false 4336 Acked Write 7 Done Sent Done false 4337 Acked Flush 0 Done Sent Done false 4338 Acked Write 10 Done Sent Done false 4339 Acked Write 10 Done Sent Done false 4340 Acked Write 2 Done Sent Done false 4341 Acked Read 7 Done Sent Done false 4342 Acked Write 7 Done Sent Done false 4343 Acked Read 7 Done Sent Done false 4344 Acked Flush 0 Done Sent Done false 4345 Acked Read 7 Done Sent Done false 4346 Acked Read 10 Done Sent Done false 4347 Acked Read 9 Done Sent Done false 4348 Acked Read 8 Done Sent Done false 4349 Acked Write 4 Done Sent Done false 4350 Acked Write 2 Done Sent Done false 4351 Acked Read 1 Done Sent Done false 4352 Acked Write 3 Done Sent Done false 4353 Acked Write 3 Done Sent Done false 4354 Acked Write 4 Done Sent Done false 4355 Acked Read 3 Done Sent Done false 4356 Acked Read 1 Done Sent Done false 4357 Acked Read 10 Done Sent Done false 4358 Acked Write 7 Done Sent Done false 4359 Acked Read 6 Done Sent Done false 4360 Acked Write 7 Done Sent Done false 4361 Acked Write 2 Done Sent Done false 4362 Acked Write 9 Done Sent Done false 4363 Acked Flush 0 Done Sent Done false 4364 Acked Read 9 Done Sent Done false 4365 Acked Write 3 Done Sent Done false 4366 Acked Flush 0 Done Sent Done false 4367 Acked Write 5 Done Sent Done false 4368 Acked Write 5 Done Sent Done false 4369 Acked Write 7 Done Sent Done false 4370 Acked Write 4 Done Sent Done false 4371 Acked Write 6 Done Sent Done false 4372 Acked Flush 0 Done Sent Done false 4373 Acked Read 1 Done Sent Done false 4374 Acked Read 5 Done Sent Done false 4375 Acked Read 4 Done Sent Done false 4376 Acked Read 2 Done Sent Done false 4377 Acked Write 6 Done Sent Done false 4378 Acked Read 6 Done Sent Done false 4379 Acked Write 8 Done Sent Done false 4380 Acked Read 4 Done Sent Done false 4381 Acked Read 2 Done Sent Done false 4382 Acked Read 5 Done Sent Done false 4383 Acked Write 9 Done Sent Done false 4384 Acked Read 1 Done Sent Done false 4385 Acked Read 10 Done Sent Done false 4386 Acked Flush 0 Done Sent Done false 4387 Acked Write 9 Done Sent Done false 4388 Acked Write 4 Done Sent Done false 4389 Acked Flush 0 Done Sent Done false 4390 Acked Write 9 Done Sent Done false 4391 Acked Read 3 Done Sent Done false 4392 Acked Read 6 Done Sent Done false 4393 Acked Write 5 Done Sent Done false 4394 Acked Read 1 Done Sent Done false 4395 Acked Read 2 Done Sent Done false 4396 Acked Write 7 Done Sent Done false 4397 Acked Write 3 Done Sent Done false 4398 Acked Read 10 Done Sent Done false 4399 Acked Read 5 Done Sent Done false 4400 Acked Write 1 Done Sent Done false 4401 Acked Read 1 Done Sent Done false 4402 Acked Write 9 Done Sent Done false 4403 Acked Write 5 Done Sent Done false 4404 Acked Write 9 Done Sent Done false 4405 Acked Write 2 Done Sent Done false 4406 Acked Read 7 Done Sent Done false 4407 Acked Write 4 Done Sent Done false 4408 Acked Write 3 Done Sent Done false 4409 Acked Write 9 Done Sent Done false 4410 Acked Read 4 Done Sent Done false 4411 Acked Flush 0 Done Sent Done false 4412 Acked Read 1 Done Sent Done false 4413 Acked Write 8 Done Sent Done false 4414 Acked Read 10 Done Sent Done false 4415 Acked Write 10 Done Sent Done false 4416 Acked Write 9 Done Sent Done false 4417 Acked Read 5 Done Sent Done false 4418 Acked Read 4 Done Sent Done false 4419 Acked Read 7 Done Sent Done false 4420 Acked Read 8 Done Sent Done false 4421 Acked Flush 0 Done Sent Done false 4422 Acked Write 8 Done Sent Done false 4423 Acked Write 10 Done Sent Done false 4424 Acked Write 7 Done Sent Done false 4425 Acked Write 4 Done Sent Done false 4426 Acked Flush 0 Done Sent Done false 4427 Acked Read 7 Done Sent Done false 4428 Acked Write 2 Done Sent Done false 4429 Acked Flush 0 Done Sent Done false 4430 Acked Write 9 Done Sent Done false 4431 Acked Flush 0 Done Sent Done false 4432 Acked Read 5 Done Sent Done false 4433 Acked Read 10 Done Sent Done false 4434 Acked Read 7 Done Sent Done false 4435 Acked Write 5 Done Sent Done false 4436 Acked Write 8 Done Sent Done false 4437 Acked Read 5 Done Sent Done false 4438 Acked Write 9 Done Sent Done false 4439 Acked Write 10 Done Sent Done false 4440 Acked Flush 0 Done Sent Done false 4441 Acked Read 6 Done Sent Done false 4442 Acked Read 1 Done Sent Done false 4443 Acked Flush 0 Done Sent Done false 4444 Acked Write 8 Done Sent Done false 4445 Acked Read 6 Done Sent Done false 4446 Acked Read 3 Done Sent Done false 4447 Acked Flush 0 Done Sent Done false 4448 Acked Write 6 Done Sent Done false 4449 Acked Read 4 Done Sent Done false 4450 Acked Read 10 Done Sent Done false 4451 Acked Write 7 Done Sent Done false 4452 Acked Read 9 Done Sent Done false 4453 Acked Read 8 Done Sent Done false 4454 Acked Write 4 Done Sent Done false 4455 Acked Write 7 Done Sent Done false 4456 Acked Read 2 Done Sent Done false 4457 Acked Flush 0 Done Sent Done false 4458 Acked Read 1 Done Sent Done false 4459 Acked Write 5 Done Sent Done false 4460 Acked Write 8 Done Sent Done false 4461 Acked Read 5 Done Sent Done false 4462 Acked Write 7 Done Sent Done false 4463 Acked Flush 0 Done Sent Done false 4464 Acked Write 10 Done Sent Done false 4465 Acked Write 5 Done Sent Done false 4466 Acked Write 2 Done Sent Done false 4467 Acked Read 9 Done Sent Done false 4468 Acked Write 6 Done Sent Done false 4469 Acked Write 7 Done Sent Done false 4470 Acked Write 2 Done Sent Done false 4471 Acked Read 3 Done Sent Done false 4472 Acked Read 8 Done Sent Done false 4473 Acked Write 10 Done Sent Done false 4474 Acked Read 3 Done Sent Done false 4475 Acked Write 5 Done Sent Done false 4476 Acked Write 10 Done Sent Done false 4477 Acked Write 1 Done Sent Done false 4478 Acked Write 8 Done Sent Done false 4479 Acked Read 6 Done Sent Done false 4480 Acked Write 9 Done Sent Done false 4481 Acked Read 7 Done Sent Done false 4482 Acked Read 10 Done Sent Done false 4483 Acked Read 10 Done Sent Done false 4484 Acked Read 2 Done Sent Done false 4485 Acked Read 10 Done Sent Done false 4486 Acked Read 10 Done Sent Done false 4487 Acked Write 7 Done Sent Done false 4488 Acked Write 10 Done Sent Done false 4489 Acked Read 8 Done Sent Done false 4490 Acked Write 10 Done Sent Done false 4491 Acked Write 2 Done Sent Done false 4492 Acked Read 10 Done Sent Done false 4493 Acked Write 1 Done Sent Done false 4494 Acked Read 4 Done Sent Done false 4495 Acked Read 2 Done Sent Done false 4496 Acked Write 10 Done Sent Done false 4497 Acked Flush 0 Done Sent Done false 4498 Acked Read 1 Done Sent Done false 4499 Acked Read 9 Done Sent Done false 4500 Acked Write 5 Done Sent Done false 4501 Acked Write 1 Done Sent Done false 4502 Acked Write 7 Done Sent Done false 4503 Acked Read 6 Done Sent Done false 4504 Acked Read 4 Done Sent Done false 4505 Acked Write 4 Done Sent Done false 4506 Acked Read 5 Done Sent Done false 4507 Acked Read 7 Done Sent Done false 4508 Acked Write 4 Done Sent Done false 4509 Acked Write 2 Done Sent Done false 4510 Acked Write 8 Done Sent Done false 4511 Acked Read 9 Done Sent Done false 4512 Acked Read 9 Done Sent Done false 4513 Acked Read 2 Done Sent Done false 4514 Acked Write 9 Done Sent Done false 4515 Acked Write 2 Done Sent Done false 4516 Acked Write 6 Done Sent Done false 4517 Acked Read 9 Done Sent Done false 4518 Acked Write 2 Done Sent Done false 4519 Acked Read 1 Done Sent Done false 4520 Acked Write 2 Done Sent Done false 4521 Acked Read 2 Done Sent Done false 4522 Acked Write 4 Done Sent Done false 4523 Acked Read 3 Done Sent Done false 4524 Acked Write 6 Done Sent Done false 4525 Acked Read 6 Done Sent Done false 4526 Acked Write 5 Done Sent Done false 4527 Acked Read 2 Done Sent Done false 4528 Acked Write 10 Done Sent Done false 4529 Acked Write 2 Done Sent Done false 4530 Acked Flush 0 Done Sent Done false 4531 Acked Read 3 Done Sent Done false 4532 Acked Write 4 Done Sent Done false 4533 Acked Write 8 Done Sent Done false 4534 Acked Write 1 Done Sent Done false 4535 Acked Write 8 Done Sent Done false 4536 Acked Read 8 Done Sent Done false 4537 Acked Read 6 Done Sent Done false 4538 Acked Read 7 Done Sent Done false 4539 Acked Read 1 Done Sent Done false 4540 Acked Write 8 Done Sent Done false 4541 Acked Write 2 Done Sent Done false 4542 Acked Read 7 Done Sent Done false 4543 Acked Read 3 Done Sent Done false 4544 Acked Write 10 Done Sent Done false 4545 Acked Write 2 Done Sent Done false 4546 Acked Flush 0 Done Sent Done false 4547 Acked Write 4 Done Sent Done false 4548 Acked Write 1 Done Sent Done false 4549 Acked Read 8 Done Sent Done false 4550 Acked Read 1 Done Sent Done false 4551 Acked Flush 0 Done Sent Done false 4552 Acked Write 9 Done Sent Done false 4553 Acked Write 2 Done Sent Done false 4554 Acked Read 4 Done Sent Done false 4555 Acked Read 7 Done Sent Done false 4556 Acked Write 7 Done Sent Done false 4557 Acked Read 4 Done Sent Done false 4558 Acked Write 3 Done Sent Done false 4559 Acked Write 9 Done Sent Done false 4560 Acked Read 2 Done Sent Done false 4561 Acked Read 6 Done Sent Done false 4562 Acked Write 6 Done Sent Done false 4563 Acked Read 10 Done Sent Done false 4564 Acked Read 2 Done Sent Done false 4565 Acked Read 9 Done Sent Done false 4566 Acked Write 8 Done Sent Done false 4567 Acked Read 10 Done Sent Done false 4568 Acked Read 4 Done Sent Done false 4569 Acked Read 10 Done Sent Done false 4570 Acked Read 5 Done Sent Done false 4571 Acked Write 7 Done Sent Done false 4572 Acked Read 6 Done Sent Done false 4573 Acked Read 10 Done Sent Done false 4574 Acked Read 7 Done Sent Done false 4575 Acked Read 3 Done Sent Done false 4576 Acked Write 6 Done Sent Done false 4577 Acked Write 9 Done Sent Done false 4578 Acked Write 4 Done Sent Done false 4579 Acked Read 9 Done Sent Done false 4580 Acked Read 1 Done Sent Done false 4581 Acked Write 2 Done Sent Done false 4582 Acked Write 9 Done Sent Done false 4583 Acked Write 10 Done Sent Done false 4584 Acked Read 5 Done Sent Done false 4585 Acked Write 9 Done Sent Done false 4586 Acked Write 4 Done Sent Done false 4587 Acked Read 4 Done Sent Done false 4588 Acked Read 6 Done Sent Done false 4589 Acked Flush 0 Done Sent Done false 4590 Acked Flush 0 Done Sent Done false 4591 Acked Read 1 Done Sent Done false 4592 Acked Flush 0 Done Sent Done false 4593 Acked Write 2 Done Sent Done false 4594 Acked Read 9 Done Sent Done false 4595 Acked Write 5 Done Sent Done false 4596 Acked Read 2 Done Sent Done false 4597 Acked Read 9 Done Sent Done false 4598 Acked Read 7 Done Sent Done false 4599 Acked Write 1 Done Sent Done false 4600 Acked Flush 0 Done Sent Done false 4601 Acked Read 6 Done Sent Done false 4602 Acked Write 3 Done Sent Done false 4603 Acked Write 8 Done Sent Done false 4604 Acked Write 1 Done Sent Done false 4605 Acked Read 1 Done Sent Done false 4606 Acked Read 8 Done Sent Done false 4607 Acked Write 4 Done Sent Done false 4608 Acked Read 7 Done Sent Done false 4609 Acked Read 7 Done Sent Done false 4610 Acked Read 5 Done Sent Done false 4611 Acked Write 7 Done Sent Done false 4612 Acked Read 4 Done Sent Done false 4613 Acked Read 7 Done Sent Done false 4614 Acked Read 8 Done Sent Done false 4615 Acked Read 10 Done Sent Done false 4616 Acked Write 10 Done Sent Done false 4617 Acked Read 2 Done Sent Done false 4618 Acked Write 1 Done Sent Done false 4619 Acked Read 6 Done Sent Done false 4620 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4318 4620 Downstairs last five completed: 4318 4317 4316 4315 4314 Upstairs last five completed: 4620 4619 4618 4617 4616 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.366386204Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:36463)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.366462164Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:46.367462675Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368025484Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368048558Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 4318","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368063594Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 4318","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368280035Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368303729Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368318185Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 4318","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368332211Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 4318","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.368549753Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4620 4619 4618 4617 4616 Upstairs last five completed: 4620 4619 4618 4617 4616 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:44:46.605 INFO [8810] Got stop action so:false kr:false request completed, latency_us: 314, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 6b121ce3-8d36-4d4a-b566-8907b285005b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:46.605670916Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:46.60573134Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:46.605765021Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.605791983Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.605820466Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:46.605853057Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:44:46.605 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/0, method: GET, req_id: 01dc783c-f168-4cb6-acfe-10d7c694ef7d, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:44:50.607 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 89c31043-1069-4052-9315-500beb1cc77a, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 1409 len 36864 data: 4 3 3 3 3 3 3 3 3 002/300 Write block 2537 len 16384 data: 3 3 4 5 003/300 Write block 3451 len 40960 data: 2 2 2 2 3 3 3 3 4 4 004/300 Read block 2739 len 40960 005/300 Write block 4836 len 8192 data: 2 2 006/300 Read block 1923 len 12288 007/300 Read block 2074 len 32768 008/300 Read block 3456 len 32768 009/300 Write block 885 len 28672 data: 3 5 5 4 5 5 4 010/300 Read block 360 len 32768 011/300 Read block 1217 len 24576 012/300 Read block 3546 len 16384 013/300 Flush 014/300 Read block 276 len 40960 015/300 Flush 016/300 Read block 371 len 16384 017/300 Write block 782 len 8192 data: 2 3 018/300 Read block 1488 len 12288 019/300 Write block 927 len 4096 data: 2 020/300 Flush 021/300 Read block 1911 len 8192 022/300 Flush 023/300 Write block 2264 len 28672 data: 3 3 3 4 4 4 4 024/300 Flush 025/300 Read block 1263 len 4096 026/300 Read block 129 len 12288 027/300 Write block 3489 len 20480 data: 5 5 5 3 3 028/300 Read block 2248 len 16384 029/300 Write block 524 len 28672 data: 7 8 7 5 7 7 7 030/300 Write block 1000 len 20480 data: 3 3 2 4 4 031/300 Read block 4002 len 40960 032/300 Write block 2137 len 12288 data: 3 4 4 033/300 Write block 2830 len 12288 data: 3 3 3 034/300 Read block 4697 len 8192 035/300 Write block 1320 len 32768 data: 5 3 2 2 2 2 2 2 036/300 Read block 1354 len 28672 037/300 Read block 399 len 40960 038/300 Read block 3244 len 4096 039/300 Read block 1051 len 40960 040/300 Read block 4193 len 12288 041/300 Write block 4678 len 20480 data: 4 4 4 4 4 042/300 Write block 1243 len 20480 data: 4 2 2 3 4 043/300 Flush 044/300 Write block 1103 len 36864 data: 4 3 4 4 4 5 6 5 5 045/300 Flush 046/300 Read block 3828 len 20480 047/300 Read block 2350 len 16384 048/300 Write block 119 len 12288 data: 4 3 3 049/300 Write block 181 len 32768 data: 5 5 6 6 5 2 3 3 050/300 Write block 2076 len 4096 data: 3 051/300 Write block 4684 len 20480 data: 2 2 3 3 4 052/300 Read block 33 len 8192 053/300 Read block 3116 len 40960 054/300 Write block 3225 len 12288 data: 3 3 3 055/300 Write block 1188 len 24576 data: 2 2 3 3 3 3 056/300 Read block 2671 len 20480 057/300 Read block 4725 len 16384 058/300 Read block 3674 len 16384 059/300 Read block 4456 len 40960 060/300 Read block 920 len 20480 061/300 Read block 2998 len 16384 062/300 Read block 4879 len 32768 063/300 Read block 2388 len 32768 064/300 Write block 4519 len 12288 data: 3 3 3 065/300 Write block 3829 len 40960 data: 3 3 4 4 3 3 3 3 3 3 066/300 Read block 747 len 36864 067/300 Read block 3781 len 12288 068/300 Read block 2863 len 12288 069/300 Flush 070/300 Write block 2345 len 28672 data: 5 5 6 5 6 7 7 071/300 Write block 2086 len 36864 data: 2 2 3 3 3 3 4 3 3 072/300 Read block 2003 len 32768 073/300 Write block 4787 len 12288 data: 3 2 2 074/300 Read block 3600 len 20480 075/300 Write block 2010 len 36864 data: 2 2 2 2 3 3 3 3 4 076/300 Write block 316 len 16384 data: 3 4 3 2 077/300 Write block 4560 len 40960 data: 4 5 5 6 7 6 5 5 5 4 078/300 Flush 079/300 Write block 3483 len 4096 data: 3 080/300 Write block 2916 len 40960 data: 2 2 4 3 3 3 4 5 5 4 081/300 Flush 082/300 Read block 4515 len 8192 083/300 Write block 520 len 36864 data: 3 2 3 5 8 9 8 6 8 084/300 Write block 4098 len 24576 data: 5 3 3 3 2 2 085/300 Read block 971 len 8192 086/300 Write block 4195 len 40960 data: 3 4 5 5 5 6 4 4 3 4 087/300 Read block 4761 len 24576 088/300 Read block 599 len 4096 089/300 Write block 566 len 16384 data: 2 2 3 3 090/300 Read block 1537 len 20480 091/300 Read block 821 len 32768 092/300 Write block 4215 len 16384 data: 5 4 2 2 093/300 Write block 29 len 20480 data: 4 4 5 5 5 094/300 Read block 1546 len 8192 095/300 Read block 3691 len 20480 096/300 Read block 1307 len 8192 097/300 Flush 098/300 Read block 1500 len 8192 099/300 Write block 1873 len 4096 data: 4 100/300 Write block 2305 len 16384 data: 5 6 5 4 101/300 Read block 4812 len 28672 102/300 Write block 1730 len 20480 data: 2 2 2 3 3 103/300 Read block 1949 len 36864 104/300 Flush 105/300 Write block 2456 len 24576 data: 5 6 5 6 6 5 106/300 Write block 4846 len 8192 data: 5 3 107/300 Read block 1266 len 40960 108/300 Write block 4584 len 16384 data: 4 4 2 2 109/300 Write block 1976 len 28672 data: 3 3 3 2 2 3 2 110/300 Read block 3379 len 40960 111/300 Write block 68 len 32768 data: 6 6 5 4 4 3 3 3 112/300 Flush 113/300 Write block 452 len 16384 data: 2 2 3 3 114/300 Flush 115/300 Write block 4754 len 40960 data: 3 3 3 3 4 4 4 4 4 4 116/300 Write block 3614 len 12288 data: 4 5 5 117/300 Read block 1201 len 36864 118/300 Write block 186 len 36864 data: 3 4 4 3 3 4 4 3 3 119/300 Read block 3367 len 28672 120/300 Read block 4343 len 4096 121/300 Read block 4941 len 4096 122/300 Read block 4073 len 20480 123/300 Read block 3607 len 8192 124/300 Write block 1914 len 12288 data: 5 4 3 125/300 Read block 2236 len 36864 126/300 Write block 3667 len 36864 data: 4 5 5 5 5 3 4 4 4 127/300 Write block 4673 len 32768 data: 3 3 2 3 3 5 5 5 128/300 Read block 3839 len 28672 129/300 Write block 4670 len 40960 data: 4 4 3 4 4 3 4 4 6 6 130/300 Read block 2846 len 16384 131/300 Read block 763 len 12288 132/300 Write block 2940 len 20480 data: 5 5 5 4 2 133/300 Write block 1212 len 8192 data: 3 3 134/300 Read block 522 len 16384 135/300 Read block 247 len 36864 136/300 Read block 1566 len 32768 137/300 Read block 4599 len 32768 138/300 Write block 4541 len 4096 data: 2 139/300 Read block 2188 len 4096 140/300 Read block 1197 len 20480 141/300 Write block 1211 len 20480 data: 3 4 4 3 3 142/300 Read block 264 len 12288 143/300 Write block 3240 len 4096 data: 3 144/300 Write block 3460 len 16384 data: 5 3 4 3 145/300 Write block 2449 len 32768 data: 3 3 3 3 4 5 5 6 146/300 Read block 1745 len 12288 147/300 Write block 4993 len 24576 data: 4 5 4 3 3 2 148/300 Read block 178 len 32768 149/300 Read block 3349 len 8192 150/300 Flush 151/300 Read block 270 len 12288 152/300 Flush 153/300 Read block 3754 len 32768 154/300 Read block 3519 len 20480 155/300 Read block 64 len 24576 156/300 Write block 4425 len 4096 data: 6 157/300 Write block 1992 len 40960 data: 8 7 5 5 6 7 7 7 6 6 158/300 Read block 684 len 16384 159/300 Flush 160/300 Write block 3313 len 36864 data: 4 4 4 4 4 4 5 4 4 161/300 Read block 3702 len 28672 162/300 Read block 2329 len 28672 163/300 Read block 3468 len 20480 164/300 Read block 2566 len 40960 165/300 Write block 4502 len 36864 data: 3 3 3 3 3 3 3 3 3 166/300 Flush 167/300 Read block 418 len 40960 168/300 Flush 169/300 Read block 4322 len 12288 170/300 Read block 1363 len 32768 171/300 Write block 4702 len 28672 data: 4 3 3 4 3 4 5 172/300 Flush 173/300 Read block 2674 len 32768 174/300 Write block 4166 len 28672 data: 4 2 2 2 2 2 3 175/300 Read block 1807 len 36864 176/300 Write block 937 len 32768 data: 3 3 4 4 4 4 3 3 177/300 Write block 5 len 24576 data: 3 3 3 2 2 3 178/300 Read block 2398 len 24576 179/300 Read block 4973 len 20480 180/300 Read block 4315 len 32768 181/300 Flush 182/300 Write block 2520 len 12288 data: 2 2 2 183/300 Write block 3970 len 40960 data: 2 2 2 2 2 2 2 2 2 2 184/300 Write block 3459 len 36864 data: 5 6 4 5 4 4 4 4 4 185/300 Write block 2277 len 40960 data: 4 3 3 2 2 2 2 2 3 3 186/300 Read block 2384 len 16384 187/300 Read block 1399 len 12288 188/300 Read block 3426 len 20480 189/300 Read block 4042 len 20480 190/300 Write block 4563 len 40960 data: 7 8 7 6 6 6 5 4 4 3 191/300 Read block 2357 len 32768 192/300 Read block 2173 len 16384 193/300 Read block 1479 len 28672 194/300 Read block 4519 len 12288 195/300 Read block 124 len 20480 196/300 Read block 1070 len 16384 197/300 Read block 1326 len 36864 198/300 Read block 1845 len 20480 199/300 Write block 3626 len 12288 data: 2 2 2 200/300 Read block 1678 len 32768 201/300 Read block 1759 len 16384 202/300 Write block 2753 len 4096 data: 2 203/300 Write block 1104 len 20480 data: 4 5 5 5 6 204/300 Read block 2727 len 4096 205/300 Flush 206/300 Flush 207/300 Read block 4546 len 8192 208/300 Read block 3292 len 28672 209/300 Read block 3643 len 36864 210/300 Write block 442 len 12288 data: 4 4 3 211/300 Write block 2427 len 8192 data: 2 2 212/300 Read block 386 len 4096 213/300 Write block 2486 len 4096 data: 4 214/300 Flush 215/300 Write block 4207 len 16384 data: 4 4 5 5 216/300 Read block 1671 len 36864 217/300 Write block 671 len 36864 data: 3 2 3 4 4 4 4 7 6 218/300 Flush 219/300 Read block 1698 len 24576 220/300 Flush 221/300 Write block 220 len 16384 data: 5 5 3 4 222/300 Read block 3716 len 20480 223/300 Read block 4705 len 16384 224/300 Read block 4380 len 20480 225/300 Write block 1725 len 12288 data: 4 4 4 226/300 Flush 227/300 Flush 228/300 Write block 2005 len 8192 data: 6 5 229/300 Read block 4464 len 28672 230/300 Read block 4861 len 28672 231/300 Write block 4595 len 24576 data: 4 3 3 2 3 3 232/300 Read block 1168 len 12288 233/300 Read block 3398 len 8192 234/300 Read block 954 len 28672 235/300 Write block 4236 len 16384 data: 3 5 5 3 236/300 Write block 2664 len 20480 data: 2 4 3 3 2 237/300 Read block 682 len 28672 238/300 Read block 1510 len 20480 239/300 Read block 4983 len 12288 240/300 Read block 1622 len 4096 241/300 Read block 1930 len 16384 242/300 Read block 4032 len 16384 243/300 Read block 2991 len 8192 244/300 Write block 468 len 12288 data: 4 5 5 245/300 Read block 249 len 20480 246/300 Read block 880 len 24576 247/300 Write block 3300 len 28672 data: 2 3 3 3 3 3 2 248/300 Read block 247 len 24576 249/300 Read block 4533 len 8192 250/300 Write block 2526 len 20480 data: 4 5 5 5 3 251/300 Flush 252/300 Read block 4414 len 40960 253/300 Write block 2196 len 36864 data: 2 2 2 2 2 2 2 2 2 254/300 Write block 1519 len 8192 data: 3 2 255/300 Write block 963 len 20480 data: 3 3 3 3 3 256/300 Read block 3724 len 4096 257/300 Flush 258/300 Write block 4299 len 12288 data: 2 3 3 259/300 Write block 4470 len 8192 data: 4 5 260/300 Write block 2920 len 28672 data: 4 4 5 6 6 5 3 261/300 Write block 3078 len 36864 data: 3 2 2 2 3 3 3 6 6 262/300 Read block 2493 len 32768 263/300 Write block 3688 len 40960 data: 4 4 5 4 3 3 4 4 4 5 264/300 Read block 121 len 12288 265/300 Read block 3054 len 36864 266/300 Write block 605 len 32768 data: 2 2 2 2 2 2 2 2 267/300 Write block 2370 len 24576 data: 4 4 5 7 7 8 268/300 Write block 4100 len 40960 data: 4 4 3 3 2 2 2 2 2 2 269/300 Read block 2936 len 20480 270/300 Read block 1618 len 28672 271/300 Flush 272/300 Read block 3553 len 28672 273/300 Write block 2180 len 12288 data: 3 4 4 274/300 Read block 4990 len 4096 275/300 Write block 4472 len 40960 data: 6 6 6 3 4 3 4 4 4 4 276/300 Read block 861 len 20480 277/300 Read block 3670 len 40960 278/300 Read block 2915 len 36864 279/300 Write block 814 len 20480 data: 4 4 3 3 2 280/300 Write block 724 len 36864 data: 3 3 4 4 2 2 2 2 2 281/300 Write block 95 len 36864 data: 4 3 3 3 3 4 4 4 2 282/300 Flush 283/300 Write block 1037 len 28672 data: 3 4 2 4 4 3 3 284/300 Read block 2754 len 12288 285/300 Read block 2299 len 32768 286/300 Read block 1625 len 16384 287/300 Read block 3253 len 4096 288/300 Write block 2444 len 20480 data: 3 3 4 3 3 289/300 Read block 1654 len 32768 290/300 Read block 4531 len 40960 291/300 Write block 1209 len 20480 data: 3 3 4 5 5 292/300 Write block 976 len 12288 data: 3 2 2 293/300 Write block 4208 len 36864 data: 5 6 6 5 5 5 5 6 5 294/300 Read block 4508 len 24576 295/300 Write block 3537 len 20480 data: 5 5 5 5 5 296/300 Read block 2266 len 4096 297/300 Write block 3596 len 8192 data: 3 3 298/300 Read block 4394 len 32768 299/300 Read block 2980 len 24576 300/300 Read block 2768 len 16384 301/300 Write block 4904 len 28672 data: 2 2 2 2 2 2 2 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:44:50.844 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6613760b-5f70-492f-804a-a6c9f57209ef, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4621 Acked Write 9 Sent Done Done false 4622 Acked Write 4 Sent Done Done false 4623 Acked Write 10 Sent Done Done false 4624 Acked Read 10 Sent Done Done false 4625 Acked Write 2 Sent Done Done false 4626 Acked Read 3 Sent Done Done false 4627 Acked Read 8 Sent Done Done false 4628 Acked Read 8 Sent Done Done false 4629 Acked Write 7 Sent Done Done false 4630 Acked Read 8 Sent Done Done false 4631 Acked Read 6 Sent Done Done false 4632 Acked Read 4 Sent Done Done false 4633 Acked Flush 0 Sent Done Done false 4634 Acked Read 10 Sent Done Done false 4635 Acked Flush 0 Sent Done Done false 4636 Acked Read 4 Sent Done Done false 4637 Acked Write 2 Sent Done Done false 4638 Acked Read 3 Sent Done Done false 4639 Acked Write 1 Sent Done Done false 4640 Acked Flush 0 Sent Done Done false 4641 Acked Read 2 Sent Done Done false 4642 Acked Flush 0 Sent Done Done false 4643 Acked Write 7 Sent Done Done false 4644 Acked Flush 0 Sent Done Done false 4645 Acked Read 1 Sent Done Done false 4646 Acked Read 3 Sent Done Done false 4647 Acked Write 5 Sent Done Done false 4648 Acked Read 4 Sent Done Done false 4649 Acked Write 7 Sent Done Done false 4650 Acked Write 5 Sent Done Done false 4651 Acked Read 10 Sent Done Done false 4652 Acked Write 3 Sent Done Done false 4653 Acked Write 3 Sent Done Done false 4654 Acked Read 2 Sent Done Done false 4655 Acked Write 8 Sent Done Done false 4656 Acked Read 7 Sent Done Done false 4657 Acked Read 10 Sent Done Done false 4658 Acked Read 1 Sent Done Done false 4659 Acked Read 10 Sent Done Done false 4660 Acked Read 3 Sent Done Done false 4661 Acked Write 5 Sent Done Done false 4662 Acked Write 5 Sent Done Done false 4663 Acked Flush 0 Sent Done Done false 4664 Acked Write 9 Sent Done Done false 4665 Acked Flush 0 Sent Done Done false 4666 Acked Read 5 Sent Done Done false 4667 Acked Read 4 Sent Done Done false 4668 Acked Write 3 Sent Done Done false 4669 Acked Write 8 Sent Done Done false 4670 Acked Write 1 Sent Done Done false 4671 Acked Write 5 Sent Done Done false 4672 Acked Read 2 Sent Done Done false 4673 Acked Read 10 Sent Done Done false 4674 Acked Write 3 Sent Done Done false 4675 Acked Write 6 Sent Done Done false 4676 Acked Read 5 Sent Done Done false 4677 Acked Read 4 Sent Done Done false 4678 Acked Read 4 Sent Done Done false 4679 Acked Read 10 Sent Done Done false 4680 Acked Read 5 Sent Done Done false 4681 Acked Read 4 Sent Done Done false 4682 Acked Read 8 Sent Done Done false 4683 Acked Read 8 Sent Done Done false 4684 Acked Write 3 Sent Done Done false 4685 Acked Write 10 Sent Done Done false 4686 Acked Read 9 Sent Done Done false 4687 Acked Read 3 Sent Done Done false 4688 Acked Read 3 Sent Done Done false 4689 Acked Flush 0 Sent Done Done false 4690 Acked Write 7 Sent Done Done false 4691 Acked Write 9 Sent Done Done false 4692 Acked Read 8 Sent Done Done false 4693 Acked Write 3 Sent Done Done false 4694 Acked Read 5 Sent Done Done false 4695 Acked Write 9 Sent Done Done false 4696 Acked Write 4 Sent Done Done false 4697 Acked Write 10 Sent Done Done false 4698 Acked Flush 0 Sent Done Done false 4699 Acked Write 1 Sent Done Done false 4700 Acked Write 10 Sent Done Done false 4701 Acked Flush 0 Sent Done Done false 4702 Acked Read 2 Sent Done Done false 4703 Acked Write 9 Sent Done Done false 4704 Acked Write 6 Sent Done Done false 4705 Acked Read 2 Sent Done Done false 4706 Acked Write 10 Sent Done Done false 4707 Acked Read 6 Sent Done Done false 4708 Acked Read 1 Sent Done Done false 4709 Acked Write 4 Sent Done Done false 4710 Acked Read 5 Sent Done Done false 4711 Acked Read 8 Sent Done Done false 4712 Acked Write 4 Sent Done Done false 4713 Acked Write 5 Sent Done Done false 4714 Acked Read 2 Sent Done Done false 4715 Acked Read 5 Sent Done Done false 4716 Acked Read 2 Sent Done Done false 4717 Acked Flush 0 Sent Done Done false 4718 Acked Read 2 Sent Done Done false 4719 Acked Write 1 Sent Done Done false 4720 Acked Write 4 Sent Done Done false 4721 Acked Read 7 Sent Done Done false 4722 Acked Write 5 Sent Done Done false 4723 Acked Read 9 Sent Done Done false 4724 Acked Flush 0 Sent Done Done false 4725 Acked Write 6 Sent Done Done false 4726 Acked Write 2 Sent Done Done false 4727 Acked Read 10 Sent Done Done false 4728 Acked Write 4 Sent Done Done false 4729 Acked Write 7 Sent Done Done false 4730 Acked Read 10 Sent Done Done false 4731 Acked Write 8 Sent Done Done false 4732 Acked Flush 0 Sent Done Done false 4733 Acked Write 4 Sent Done Done false 4734 Acked Flush 0 Sent Done Done false 4735 Acked Write 10 Sent Done Done false 4736 Acked Write 3 Sent Done Done false 4737 Acked Read 9 Sent Done Done false 4738 Acked Write 9 Sent Done Done false 4739 Acked Read 7 Sent Done Done false 4740 Acked Read 1 Sent Done Done false 4741 Acked Read 1 Sent Done Done false 4742 Acked Read 5 Sent Done Done false 4743 Acked Read 2 Sent Done Done false 4744 Acked Write 3 Sent Done Done false 4745 Acked Read 9 Sent Done Done false 4746 Acked Write 9 Sent Done Done false 4747 Acked Write 8 Sent Done Done false 4748 Acked Read 7 Sent Done Done false 4749 Acked Write 10 Sent Done Done false 4750 Acked Read 4 Sent Done Done false 4751 Acked Read 3 Sent Done Done false 4752 Acked Write 5 Sent Done Done false 4753 Acked Write 2 Sent Done Done false 4754 Acked Read 4 Sent Done Done false 4755 Acked Read 9 Sent Done Done false 4756 Acked Read 8 Sent Done Done false 4757 Acked Read 8 Sent Done Done false 4758 Acked Write 1 Sent Done Done false 4759 Acked Read 1 Sent Done Done false 4760 Acked Read 5 Sent Done Done false 4761 Acked Write 5 Sent Done Done false 4762 Acked Read 3 Sent Done Done false 4763 Acked Write 1 Sent Done Done false 4764 Acked Write 4 Sent Done Done false 4765 Acked Write 8 Sent Done Done false 4766 Acked Read 3 Sent Done Done false 4767 Acked Write 6 Sent Done Done false 4768 Acked Read 8 Sent Done Done false 4769 Acked Read 2 Sent Done Done false 4770 Acked Flush 0 Sent Done Done false 4771 Acked Read 3 Sent Done Done false 4772 Acked Flush 0 Sent Done Done false 4773 Acked Read 8 Sent Done Done false 4774 Acked Read 5 Sent Done Done false 4775 Acked Read 6 Sent Done Done false 4776 Acked Write 1 Sent Done Done false 4777 Acked Write 10 Sent Done Done false 4778 Acked Read 4 Sent Done Done false 4779 Acked Flush 0 Sent Done Done false 4780 Acked Write 9 Sent Done Done false 4781 Acked Read 7 Sent Done Done false 4782 Acked Read 7 Sent Done Done false 4783 Acked Read 5 Sent Done Done false 4784 Acked Read 10 Sent Done Done false 4785 Acked Write 9 Sent Done Done false 4786 Acked Flush 0 Sent Done Done false 4787 Acked Read 10 Sent Done Done false 4788 Acked Flush 0 Sent Done Done false 4789 Acked Read 3 Sent Done Done false 4790 Acked Read 8 Sent Done Done false 4791 Acked Write 7 Sent Done Done false 4792 Acked Flush 0 Sent Done Done false 4793 Acked Read 8 Sent Done Done false 4794 Acked Write 7 Sent Done Done false 4795 Acked Read 9 Sent Done Done false 4796 Acked Write 8 Sent Done Done false 4797 Acked Write 6 Sent Done Done false 4798 Acked Read 6 Sent Done Done false 4799 Acked Read 5 Sent Done Done false 4800 Acked Read 8 Sent Done Done false 4801 Acked Flush 0 Sent Done Done false 4802 Acked Write 3 Sent Done Done false 4803 Acked Write 10 Sent Done Done false 4804 Acked Write 9 Sent Done Done false 4805 Acked Write 10 Sent Done Done false 4806 Acked Read 4 Sent Done Done false 4807 Acked Read 3 Sent Done Done false 4808 Acked Read 5 Sent Done Done false 4809 Acked Read 5 Sent Done Done false 4810 Acked Write 10 Sent Done Done false 4811 Acked Read 8 Sent Done Done false 4812 Acked Read 4 Sent Done Done false 4813 Acked Read 7 Sent Done Done false 4814 Acked Read 3 Sent Done Done false 4815 Acked Read 5 Sent Done Done false 4816 Acked Read 4 Sent Done Done false 4817 Acked Read 9 Sent Done Done false 4818 Acked Read 5 Sent Done Done false 4819 Acked Write 3 Sent Done Done false 4820 Acked Read 8 Sent Done Done false 4821 Acked Read 4 Sent Done Done false 4822 Acked Write 1 Sent Done Done false 4823 Acked Write 5 Sent Done Done false 4824 Acked Read 1 Sent Done Done false 4825 Acked Flush 0 Sent Done Done false 4826 Acked Flush 0 Sent Done Done false 4827 Acked Read 2 Sent Done Done false 4828 Acked Read 7 Sent Done Done false 4829 Acked Read 9 Sent Done Done false 4830 Acked Write 3 Sent Done Done false 4831 Acked Write 2 Sent Done Done false 4832 Acked Read 1 Sent Done Done false 4833 Acked Write 1 Sent Done Done false 4834 Acked Flush 0 Sent Done Done false 4835 Acked Write 4 Sent Done Done false 4836 Acked Read 9 Sent Done Done false 4837 Acked Write 9 Sent Done Done false 4838 Acked Flush 0 Sent Done Done false 4839 Acked Read 6 Sent Done Done false 4840 Acked Flush 0 Sent Done Done false 4841 Acked Write 4 Sent Done Done false 4842 Acked Read 5 Sent Done Done false 4843 Acked Read 4 Sent Done Done false 4844 Acked Read 5 Sent Done Done false 4845 Acked Write 3 Sent Done Done false 4846 Acked Flush 0 Sent Done Done false 4847 Acked Flush 0 Sent Done Done false 4848 Acked Write 2 Sent Done Done false 4849 Acked Read 7 Sent Done Done false 4850 Acked Read 7 Sent Done Done false 4851 Acked Write 6 Sent Done Done false 4852 Acked Read 3 Sent Done Done false 4853 Acked Read 2 Sent Done Done false 4854 Acked Read 7 Sent Done Done false 4855 Acked Write 4 Sent Done Done false 4856 Acked Write 5 Sent Done Done false 4857 Acked Read 7 Sent Done Done false 4858 Acked Read 5 Sent Done Done false 4859 Acked Read 3 Sent Done Done false 4860 Acked Read 1 Sent Done Done false 4861 Acked Read 4 Sent Done Done false 4862 Acked Read 4 Sent Done Done false 4863 Acked Read 2 Sent Done Done false 4864 Acked Write 3 Sent Done Done false 4865 Acked Read 5 Sent Done Done false 4866 Acked Read 6 Sent Done Done false 4867 Acked Write 7 Sent Done Done false 4868 Acked Read 6 Sent Done Done false 4869 Acked Read 2 Sent Done Done false 4870 Acked Write 5 Sent Done Done false 4871 Acked Flush 0 Sent Done Done false 4872 Acked Read 10 Sent Done Done false 4873 Acked Write 9 Sent Done Done false 4874 Acked Write 2 Sent Done Done false 4875 Acked Write 5 Sent Done Done false 4876 Acked Read 1 Sent Done Done false 4877 Acked Flush 0 Sent Done Done false 4878 Acked Write 3 Sent Done Done false 4879 Acked Write 2 Sent Done Done false 4880 Acked Write 7 Sent Done Done false 4881 Acked Write 9 Sent Done Done false 4882 Acked Read 8 Sent Done Done false 4883 Acked Write 10 Sent Done Done false 4884 Acked Read 3 Sent Done Done false 4885 Acked Read 9 Sent Done Done false 4886 Acked Write 8 Sent Done Done false 4887 Acked Write 6 Sent Done Done false 4888 Acked Write 10 Sent Done Done false 4889 Acked Read 5 Sent Done Done false 4890 Acked Read 7 Sent Done Done false 4891 Acked Flush 0 Sent Done Done false 4892 Acked Read 7 Sent Done Done false 4893 Acked Write 3 Sent Done Done false 4894 Acked Read 1 Sent Done Done false 4895 Acked Write 10 Sent Done Done false 4896 Acked Read 5 Sent Done Done false 4897 Acked Read 10 Sent Done Done false 4898 Acked Read 9 Sent Done Done false 4899 Acked Write 5 Sent Done Done false 4900 Acked Write 9 Sent Done Done false 4901 Acked Write 9 Sent Done Done false 4902 Acked Flush 0 Sent Done Done false 4903 Acked Write 7 Sent Done Done false 4904 Acked Read 3 Sent Done Done false 4905 Acked Read 8 Sent Done Done false 4906 Acked Read 4 Sent Done Done false 4907 Acked Read 1 Sent Done Done false 4908 Acked Write 5 Sent Done Done false 4909 Acked Read 8 Sent Done Done false 4910 Acked Read 10 Sent Done Done false 4911 Acked Write 5 Sent Done Done false 4912 Acked Write 3 Sent Done Done false 4913 Acked Write 9 Sent Done Done false 4914 Acked Read 6 Sent Done Done false 4915 Acked Write 5 Sent Done Done false 4916 Acked Read 1 Sent Done Done false 4917 Acked Write 2 Sent Done Done false 4918 Acked Read 8 Sent Done Done false 4919 Acked Read 6 Sent Done Done false 4920 Acked Read 4 Sent Done Done false 4921 Acked Write 7 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4902 4902 Downstairs last five completed: 4620 4619 4618 4617 4616 Upstairs last five completed: 4921 4920 4919 4918 4917 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4621 Acked Write 9 Sent Done Done false 4622 Acked Write 4 Sent Done Done false 4623 Acked Write 10 Sent Done Done false 4624 Acked Read 10 Sent Done Done false 4625 Acked Write 2 Sent Done Done false 4626 Acked Read 3 Sent Done Done false 4627 Acked Read 8 Sent Done Done false 4628 Acked Read 8 Sent Done Done false 4629 Acked Write 7 Sent Done Done false 4630 Acked Read 8 Sent Done Done false 4631 Acked Read 6 Sent Done Done false 4632 Acked Read 4 Sent Done Done false 4633 Acked Flush 0 Sent Done Done false 4634 Acked Read 10 Sent Done Done false 4635 Acked Flush 0 Sent Done Done false 4636 Acked Read 4 Sent Done Done false 4637 Acked Write 2 Sent Done Done false 4638 Acked Read 3 Sent Done Done false 4639 Acked Write 1 Sent Done Done false 4640 Acked Flush 0 Sent Done Done false 4641 Acked Read 2 Sent Done Done false 4642 Acked Flush 0 Sent Done Done false 4643 Acked Write 7 Sent Done Done false 4644 Acked Flush 0 Sent Done Done false 4645 Acked Read 1 Sent Done Done false 4646 Acked Read 3 Sent Done Done false 4647 Acked Write 5 Sent Done Done false 4648 Acked Read 4 Sent Done Done false 4649 Acked Write 7 Sent Done Done false 4650 Acked Write 5 Sent Done Done false 4651 Acked Read 10 Sent Done Done false 4652 Acked Write 3 Sent Done Done false 4653 Acked Write 3 Sent Done Done false 4654 Acked Read 2 Sent Done Done false 4655 Acked Write 8 Sent Done Done false 4656 Acked Read 7 Sent Done Done false 4657 Acked Read 10 Sent Done Done false 4658 Acked Read 1 Sent Done Done false 4659 Acked Read 10 Sent Done Done false 4660 Acked Read 3 Sent Done Done false 4661 Acked Write 5 Sent Done Done false 4662 Acked Write 5 Sent Done Done false 4663 Acked Flush 0 Sent Done Done false 4664 Acked Write 9 Sent Done Done false 4665 Acked Flush 0 Sent Done Done false 4666 Acked Read 5 Sent Done Done false 4667 Acked Read 4 Sent Done Done false 4668 Acked Write 3 Sent Done Done false 4669 Acked Write 8 Sent Done Done false 4670 Acked Write 1 Sent Done Done false 4671 Acked Write 5 Sent Done Done false 4672 Acked Read 2 Sent Done Done false 4673 Acked Read 10 Sent Done Done false 4674 Acked Write 3 Sent Done Done false 4675 Acked Write 6 Sent Done Done false 4676 Acked Read 5 Sent Done Done false 4677 Acked Read 4 Sent Done Done false 4678 Acked Read 4 Sent Done Done false 4679 Acked Read 10 Sent Done Done false 4680 Acked Read 5 Sent Done Done false 4681 Acked Read 4 Sent Done Done false 4682 Acked Read 8 Sent Done Done false 4683 Acked Read 8 Sent Done Done false 4684 Acked Write 3 Sent Done Done false 4685 Acked Write 10 Sent Done Done false 4686 Acked Read 9 Sent Done Done false 4687 Acked Read 3 Sent Done Done false 4688 Acked Read 3 Sent Done Done false 4689 Acked Flush 0 Sent Done Done false 4690 Acked Write 7 Sent Done Done false 4691 Acked Write 9 Sent Done Done false 4692 Acked Read 8 Sent Done Done false 4693 Acked Write 3 Sent Done Done false 4694 Acked Read 5 Sent Done Done false 4695 Acked Write 9 Sent Done Done false 4696 Acked Write 4 Sent Done Done false 4697 Acked Write 10 Sent Done Done false 4698 Acked Flush 0 Sent Done Done false 4699 Acked Write 1 Sent Done Done false 4700 Acked Write 10 Sent Done Done false 4701 Acked Flush 0 Sent Done Done false 4702 Acked Read 2 Sent Done Done false 4703 Acked Write 9 Sent Done Done false 4704 Acked Write 6 Sent Done Done false 4705 Acked Read 2 Sent Done Done false 4706 Acked Write 10 Sent Done Done false 4707 Acked Read 6 Sent Done Done false 4708 Acked Read 1 Sent Done Done false 4709 Acked Write 4 Sent Done Done false 4710 Acked Read 5 Sent Done Done false 4711 Acked Read 8 Sent Done Done false 4712 Acked Write 4 Sent Done Done false 4713 Acked Write 5 Sent Done Done false 4714 Acked Read 2 Sent Done Done false 4715 Acked Read 5 Sent Done Done false 4716 Acked Read 2 Sent Done Done false 4717 Acked Flush 0 Sent Done Done false 4718 Acked Read 2 Sent Done Done false 4719 Acked Write 1 Sent Done Done false 4720 Acked Write 4 Sent Done Done false 4721 Acked Read 7 Sent Done Done false 4722 Acked Write 5 Sent Done Done false 4723 Acked Read 9 Sent Done Done false 4724 Acked Flush 0 Sent Done Done false 4725 Acked Write 6 Sent Done Done false 4726 Acked Write 2 Sent Done Done false 4727 Acked Read 10 Sent Done Done false 4728 Acked Write 4 Sent Done Done false 4729 Acked Write 7 Sent Done Done false 4730 Acked Read 10 Sent Done Done false 4731 Acked Write 8 Sent Done Done false 4732 Acked Flush 0 Sent Done Done false 4733 Acked Write 4 Sent Done Done false 4734 Acked Flush 0 Sent Done Done false 4735 Acked Write 10 Sent Done Done false 4736 Acked Write 3 Sent Done Done false 4737 Acked Read 9 Sent Done Done false 4738 Acked Write 9 Sent Done Done false 4739 Acked Read 7 Sent Done Done false 4740 Acked Read 1 Sent Done Done false 4741 Acked Read 1 Sent Done Done false 4742 Acked Read 5 Sent Done Done false 4743 Acked Read 2 Sent Done Done false 4744 Acked Write 3 Sent Done Done false 4745 Acked Read 9 Sent Done Done false 4746 Acked Write 9 Sent Done Done false 4747 Acked Write 8 Sent Done Done false 4748 Acked Read 7 Sent Done Done false 4749 Acked Write 10 Sent Done Done false 4750 Acked Read 4 Sent Done Done false 4751 Acked Read 3 Sent Done Done false 4752 Acked Write 5 Sent Done Done false 4753 Acked Write 2 Sent Done Done false 4754 Acked Read 4 Sent Done Done false 4755 Acked Read 9 Sent Done Done false 4756 Acked Read 8 Sent Done Done false 4757 Acked Read 8 Sent Done Done false 4758 Acked Write 1 Sent Done Done false 4759 Acked Read 1 Sent Done Done false 4760 Acked Read 5 Sent Done Done false 4761 Acked Write 5 Sent Done Done false 4762 Acked Read 3 Sent Done Done false 4763 Acked Write 1 Sent Done Done false 4764 Acked Write 4 Sent Done Done false 4765 Acked Write 8 Sent Done Done false 4766 Acked Read 3 Sent Done Done false 4767 Acked Write 6 Sent Done Done false 4768 Acked Read 8 Sent Done Done false 4769 Acked Read 2 Sent Done Done false 4770 Acked Flush 0 Sent Done Done false 4771 Acked Read 3 Sent Done Done false 4772 Acked Flush 0 Sent Done Done false 4773 Acked Read 8 Sent Done Done false 4774 Acked Read 5 Sent Done Done false 4775 Acked Read 6 Sent Done Done false 4776 Acked Write 1 Sent Done Done false 4777 Acked Write 10 Sent Done Done false 4778 Acked Read 4 Sent Done Done false 4779 Acked Flush 0 Sent Done Done false 4780 Acked Write 9 Sent Done Done false 4781 Acked Read 7 Sent Done Done false 4782 Acked Read 7 Sent Done Done false 4783 Acked Read 5 Sent Done Done false 4784 Acked Read 10 Sent Done Done false 4785 Acked Write 9 Sent Done Done false 4786 Acked Flush 0 Sent Done Done false 4787 Acked Read 10 Sent Done Done false 4788 Acked Flush 0 Sent Done Done false 4789 Acked Read 3 Sent Done Done false 4790 Acked Read 8 Sent Done Done false 4791 Acked Write 7 Sent Done Done false 4792 Acked Flush 0 Sent Done Done false 4793 Acked Read 8 Sent Done Done false 4794 Acked Write 7 Sent Done Done false 4795 Acked Read 9 Sent Done Done false 4796 Acked Write 8 Sent Done Done false 4797 Acked Write 6 Sent Done Done false 4798 Acked Read 6 Sent Done Done false 4799 Acked Read 5 Sent Done Done false 4800 Acked Read 8 Sent Done Done false 4801 Acked Flush 0 Sent Done Done false 4802 Acked Write 3 Sent Done Done false 4803 Acked Write 10 Sent Done Done false 4804 Acked Write 9 Sent Done Done false 4805 Acked Write 10 Sent Done Done false 4806 Acked Read 4 Sent Done Done false 4807 Acked Read 3 Sent Done Done false 4808 Acked Read 5 Sent Done Done false 4809 Acked Read 5 Sent Done Done false 4810 Acked Write 10 Sent Done Done false 4811 Acked Read 8 Sent Done Done false 4812 Acked Read 4 Sent Done Done false 4813 Acked Read 7 Sent Done Done false 4814 Acked Read 3 Sent Done Done false 4815 Acked Read 5 Sent Done Done false 4816 Acked Read 4 Sent Done Done false 4817 Acked Read 9 Sent Done Done false 4818 Acked Read 5 Sent Done Done false 4819 Acked Write 3 Sent Done Done false 4820 Acked Read 8 Sent Done Done false 4821 Acked Read 4 Sent Done Done false 4822 Acked Write 1 Sent Done Done false 4823 Acked Write 5 Sent Done Done false 4824 Acked Read 1 Sent Done Done false 4825 Acked Flush 0 Sent Done Done false 4826 Acked Flush 0 Sent Done Done false 4827 Acked Read 2 Sent Done Done false 4828 Acked Read 7 Sent Done Done false 4829 Acked Read 9 Sent Done Done false 4830 Acked Write 3 Sent Done Done false 4831 Acked Write 2 Sent Done Done false 4832 Acked Read 1 Sent Done Done false 4833 Acked Write 1 Sent Done Done false 4834 Acked Flush 0 Sent Done Done false 4835 Acked Write 4 Sent Done Done false 4836 Acked Read 9 Sent Done Done false 4837 Acked Write 9 Sent Done Done false 4838 Acked Flush 0 Sent Done Done false 4839 Acked Read 6 Sent Done Done false 4840 Acked Flush 0 Sent Done Done false 4841 Acked Write 4 Sent Done Done false 4842 Acked Read 5 Sent Done Done false 4843 Acked Read 4 Sent Done Done false 4844 Acked Read 5 Sent Done Done false 4845 Acked Write 3 Sent Done Done false 4846 Acked Flush 0 Sent Done Done false 4847 Acked Flush 0 Sent Done Done false 4848 Acked Write 2 Sent Done Done false 4849 Acked Read 7 Sent Done Done false 4850 Acked Read 7 Sent Done Done false 4851 Acked Write 6 Sent Done Done false 4852 Acked Read 3 Sent Done Done false 4853 Acked Read 2 Sent Done Done false 4854 Acked Read 7 Sent Done Done false 4855 Acked Write 4 Sent Done Done false 4856 Acked Write 5 Sent Done Done false 4857 Acked Read 7 Sent Done Done false 4858 Acked Read 5 Sent Done Done false 4859 Acked Read 3 Sent Done Done false 4860 Acked Read 1 Sent Done Done false 4861 Acked Read 4 Sent Done Done false 4862 Acked Read 4 Sent Done Done false 4863 Acked Read 2 Sent Done Done false 4864 Acked Write 3 Sent Done Done false 4865 Acked Read 5 Sent Done Done false 4866 Acked Read 6 Sent Done Done false 4867 Acked Write 7 Sent Done Done false 4868 Acked Read 6 Sent Done Done false 4869 Acked Read 2 Sent Done Done false 4870 Acked Write 5 Sent Done Done false 4871 Acked Flush 0 Sent Done Done false 4872 Acked Read 10 Sent Done Done false 4873 Acked Write 9 Sent Done Done false 4874 Acked Write 2 Sent Done Done false 4875 Acked Write 5 Sent Done Done false 4876 Acked Read 1 Sent Done Done false 4877 Acked Flush 0 Sent Done Done false 4878 Acked Write 3 Sent Done Done false 4879 Acked Write 2 Sent Done Done false 4880 Acked Write 7 Sent Done Done false 4881 Acked Write 9 Sent Done Done false 4882 Acked Read 8 Sent Done Done false 4883 Acked Write 10 Sent Done Done false 4884 Acked Read 3 Sent Done Done false 4885 Acked Read 9 Sent Done Done false 4886 Acked Write 8 Sent Done Done false 4887 Acked Write 6 Sent Done Done false 4888 Acked Write 10 Sent Done Done false 4889 Acked Read 5 Sent Done Done false 4890 Acked Read 7 Sent Done Done false 4891 Acked Flush 0 Sent Done Done false 4892 Acked Read 7 Sent Done Done false 4893 Acked Write 3 Sent Done Done false 4894 Acked Read 1 Sent Done Done false 4895 Acked Write 10 Sent Done Done false 4896 Acked Read 5 Sent Done Done false 4897 Acked Read 10 Sent Done Done false 4898 Acked Read 9 Sent Done Done false 4899 Acked Write 5 Sent Done Done false 4900 Acked Write 9 Sent Done Done false 4901 Acked Write 9 Sent Done Done false 4902 Acked Flush 0 Sent Done Done false 4903 Acked Write 7 Sent Done Done false 4904 Acked Read 3 Sent Done Done false 4905 Acked Read 8 Sent Done Done false 4906 Acked Read 4 Sent Done Done false 4907 Acked Read 1 Sent Done Done false 4908 Acked Write 5 Sent Done Done false 4909 Acked Read 8 Sent Done Done false 4910 Acked Read 10 Sent Done Done false 4911 Acked Write 5 Sent Done Done false 4912 Acked Write 3 Sent Done Done false 4913 Acked Write 9 Sent Done Done false 4914 Acked Read 6 Sent Done Done false 4915 Acked Write 5 Sent Done Done false 4916 Acked Read 1 Sent Done Done false 4917 Acked Write 2 Sent Done Done false 4918 Acked Read 8 Sent Done Done false 4919 Acked Read 6 Sent Done Done false 4920 Acked Read 4 Sent Done Done false 4921 Acked Write 7 Sent Done Done false 4922 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4922 4922 Downstairs last five completed: 4620 4619 4618 4617 4616 Upstairs last five completed: 4922 4921 4920 4919 4918 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:56.60737119Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:56.607429464Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:56.607448179Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:44:56.607464515Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:44:56.607554681Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4621 Acked Write 9 Sent Done Done false 4622 Acked Write 4 Sent Done Done false 4623 Acked Write 10 Sent Done Done false 4624 Acked Read 10 Sent Done Done false 4625 Acked Write 2 Sent Done Done false 4626 Acked Read 3 Sent Done Done false 4627 Acked Read 8 Sent Done Done false 4628 Acked Read 8 Sent Done Done false 4629 Acked Write 7 Sent Done Done false 4630 Acked Read 8 Sent Done Done false 4631 Acked Read 6 Sent Done Done false 4632 Acked Read 4 Sent Done Done false 4633 Acked Flush 0 Sent Done Done false 4634 Acked Read 10 Sent Done Done false 4635 Acked Flush 0 Sent Done Done false 4636 Acked Read 4 Sent Done Done false 4637 Acked Write 2 Sent Done Done false 4638 Acked Read 3 Sent Done Done false 4639 Acked Write 1 Sent Done Done false 4640 Acked Flush 0 Sent Done Done false 4641 Acked Read 2 Sent Done Done false 4642 Acked Flush 0 Sent Done Done false 4643 Acked Write 7 Sent Done Done false 4644 Acked Flush 0 Sent Done Done false 4645 Acked Read 1 Sent Done Done false 4646 Acked Read 3 Sent Done Done false 4647 Acked Write 5 Sent Done Done false 4648 Acked Read 4 Sent Done Done false 4649 Acked Write 7 Sent Done Done false 4650 Acked Write 5 Sent Done Done false 4651 Acked Read 10 Sent Done Done false 4652 Acked Write 3 Sent Done Done false 4653 Acked Write 3 Sent Done Done false 4654 Acked Read 2 Sent Done Done false 4655 Acked Write 8 Sent Done Done false 4656 Acked Read 7 Sent Done Done false 4657 Acked Read 10 Sent Done Done false 4658 Acked Read 1 Sent Done Done false 4659 Acked Read 10 Sent Done Done false 4660 Acked Read 3 Sent Done Done false 4661 Acked Write 5 Sent Done Done false 4662 Acked Write 5 Sent Done Done false 4663 Acked Flush 0 Sent Done Done false 4664 Acked Write 9 Sent Done Done false 4665 Acked Flush 0 Sent Done Done false 4666 Acked Read 5 Sent Done Done false 4667 Acked Read 4 Sent Done Done false 4668 Acked Write 3 Sent Done Done false 4669 Acked Write 8 Sent Done Done false 4670 Acked Write 1 Sent Done Done false 4671 Acked Write 5 Sent Done Done false 4672 Acked Read 2 Sent Done Done false 4673 Acked Read 10 Sent Done Done false 4674 Acked Write 3 Sent Done Done false 4675 Acked Write 6 Sent Done Done false 4676 Acked Read 5 Sent Done Done false 4677 Acked Read 4 Sent Done Done false 4678 Acked Read 4 Sent Done Done false 4679 Acked Read 10 Sent Done Done false 4680 Acked Read 5 Sent Done Done false 4681 Acked Read 4 Sent Done Done false 4682 Acked Read 8 Sent Done Done false 4683 Acked Read 8 Sent Done Done false 4684 Acked Write 3 Sent Done Done false 4685 Acked Write 10 Sent Done Done false 4686 Acked Read 9 Sent Done Done false 4687 Acked Read 3 Sent Done Done false 4688 Acked Read 3 Sent Done Done false 4689 Acked Flush 0 Sent Done Done false 4690 Acked Write 7 Sent Done Done false 4691 Acked Write 9 Sent Done Done false 4692 Acked Read 8 Sent Done Done false 4693 Acked Write 3 Sent Done Done false 4694 Acked Read 5 Sent Done Done false 4695 Acked Write 9 Sent Done Done false 4696 Acked Write 4 Sent Done Done false 4697 Acked Write 10 Sent Done Done false 4698 Acked Flush 0 Sent Done Done false 4699 Acked Write 1 Sent Done Done false 4700 Acked Write 10 Sent Done Done false 4701 Acked Flush 0 Sent Done Done false 4702 Acked Read 2 Sent Done Done false 4703 Acked Write 9 Sent Done Done false 4704 Acked Write 6 Sent Done Done false 4705 Acked Read 2 Sent Done Done false 4706 Acked Write 10 Sent Done Done false 4707 Acked Read 6 Sent Done Done false 4708 Acked Read 1 Sent Done Done false 4709 Acked Write 4 Sent Done Done false 4710 Acked Read 5 Sent Done Done false 4711 Acked Read 8 Sent Done Done false 4712 Acked Write 4 Sent Done Done false 4713 Acked Write 5 Sent Done Done false 4714 Acked Read 2 Sent Done Done false 4715 Acked Read 5 Sent Done Done false 4716 Acked Read 2 Sent Done Done false 4717 Acked Flush 0 Sent Done Done false 4718 Acked Read 2 Sent Done Done false 4719 Acked Write 1 Sent Done Done false 4720 Acked Write 4 Sent Done Done false 4721 Acked Read 7 Sent Done Done false 4722 Acked Write 5 Sent Done Done false 4723 Acked Read 9 Sent Done Done false 4724 Acked Flush 0 Sent Done Done false 4725 Acked Write 6 Sent Done Done false 4726 Acked Write 2 Sent Done Done false 4727 Acked Read 10 Sent Done Done false 4728 Acked Write 4 Sent Done Done false 4729 Acked Write 7 Sent Done Done false 4730 Acked Read 10 Sent Done Done false 4731 Acked Write 8 Sent Done Done false 4732 Acked Flush 0 Sent Done Done false 4733 Acked Write 4 Sent Done Done false 4734 Acked Flush 0 Sent Done Done false 4735 Acked Write 10 Sent Done Done false 4736 Acked Write 3 Sent Done Done false 4737 Acked Read 9 Sent Done Done false 4738 Acked Write 9 Sent Done Done false 4739 Acked Read 7 Sent Done Done false 4740 Acked Read 1 Sent Done Done false 4741 Acked Read 1 Sent Done Done false 4742 Acked Read 5 Sent Done Done false 4743 Acked Read 2 Sent Done Done false 4744 Acked Write 3 Sent Done Done false 4745 Acked Read 9 Sent Done Done false 4746 Acked Write 9 Sent Done Done false 4747 Acked Write 8 Sent Done Done false 4748 Acked Read 7 Sent Done Done false 4749 Acked Write 10 Sent Done Done false 4750 Acked Read 4 Sent Done Done false 4751 Acked Read 3 Sent Done Done false 4752 Acked Write 5 Sent Done Done false 4753 Acked Write 2 Sent Done Done false 4754 Acked Read 4 Sent Done Done false 4755 Acked Read 9 Sent Done Done false 4756 Acked Read 8 Sent Done Done false 4757 Acked Read 8 Sent Done Done false 4758 Acked Write 1 Sent Done Done false 4759 Acked Read 1 Sent Done Done false 4760 Acked Read 5 Sent Done Done false 4761 Acked Write 5 Sent Done Done false 4762 Acked Read 3 Sent Done Done false 4763 Acked Write 1 Sent Done Done false 4764 Acked Write 4 Sent Done Done false 4765 Acked Write 8 Sent Done Done false 4766 Acked Read 3 Sent Done Done false 4767 Acked Write 6 Sent Done Done false 4768 Acked Read 8 Sent Done Done false 4769 Acked Read 2 Sent Done Done false 4770 Acked Flush 0 Sent Done Done false 4771 Acked Read 3 Sent Done Done false 4772 Acked Flush 0 Sent Done Done false 4773 Acked Read 8 Sent Done Done false 4774 Acked Read 5 Sent Done Done false 4775 Acked Read 6 Sent Done Done false 4776 Acked Write 1 Sent Done Done false 4777 Acked Write 10 Sent Done Done false 4778 Acked Read 4 Sent Done Done false 4779 Acked Flush 0 Sent Done Done false 4780 Acked Write 9 Sent Done Done false 4781 Acked Read 7 Sent Done Done false 4782 Acked Read 7 Sent Done Done false 4783 Acked Read 5 Sent Done Done false 4784 Acked Read 10 Sent Done Done false 4785 Acked Write 9 Sent Done Done false 4786 Acked Flush 0 Sent Done Done false 4787 Acked Read 10 Sent Done Done false 4788 Acked Flush 0 Sent Done Done false 4789 Acked Read 3 Sent Done Done false 4790 Acked Read 8 Sent Done Done false 4791 Acked Write 7 Sent Done Done false 4792 Acked Flush 0 Sent Done Done false 4793 Acked Read 8 Sent Done Done false 4794 Acked Write 7 Sent Done Done false 4795 Acked Read 9 Sent Done Done false 4796 Acked Write 8 Sent Done Done false 4797 Acked Write 6 Sent Done Done false 4798 Acked Read 6 Sent Done Done false 4799 Acked Read 5 Sent Done Done false 4800 Acked Read 8 Sent Done Done false 4801 Acked Flush 0 Sent Done Done false 4802 Acked Write 3 Sent Done Done false 4803 Acked Write 10 Sent Done Done false 4804 Acked Write 9 Sent Done Done false 4805 Acked Write 10 Sent Done Done false 4806 Acked Read 4 Sent Done Done false 4807 Acked Read 3 Sent Done Done false 4808 Acked Read 5 Sent Done Done false 4809 Acked Read 5 Sent Done Done false 4810 Acked Write 10 Sent Done Done false 4811 Acked Read 8 Sent Done Done false 4812 Acked Read 4 Sent Done Done false 4813 Acked Read 7 Sent Done Done false 4814 Acked Read 3 Sent Done Done false 4815 Acked Read 5 Sent Done Done false 4816 Acked Read 4 Sent Done Done false 4817 Acked Read 9 Sent Done Done false 4818 Acked Read 5 Sent Done Done false 4819 Acked Write 3 Sent Done Done false 4820 Acked Read 8 Sent Done Done false 4821 Acked Read 4 Sent Done Done false 4822 Acked Write 1 Sent Done Done false 4823 Acked Write 5 Sent Done Done false 4824 Acked Read 1 Sent Done Done false 4825 Acked Flush 0 Sent Done Done false 4826 Acked Flush 0 Sent Done Done false 4827 Acked Read 2 Sent Done Done false 4828 Acked Read 7 Sent Done Done false 4829 Acked Read 9 Sent Done Done false 4830 Acked Write 3 Sent Done Done false 4831 Acked Write 2 Sent Done Done false 4832 Acked Read 1 Sent Done Done false 4833 Acked Write 1 Sent Done Done false 4834 Acked Flush 0 Sent Done Done false 4835 Acked Write 4 Sent Done Done false 4836 Acked Read 9 Sent Done Done false 4837 Acked Write 9 Sent Done Done false 4838 Acked Flush 0 Sent Done Done false 4839 Acked Read 6 Sent Done Done false 4840 Acked Flush 0 Sent Done Done false 4841 Acked Write 4 Sent Done Done false 4842 Acked Read 5 Sent Done Done false 4843 Acked Read 4 Sent Done Done false 4844 Acked Read 5 Sent Done Done false 4845 Acked Write 3 Sent Done Done false 4846 Acked Flush 0 Sent Done Done false 4847 Acked Flush 0 Sent Done Done false 4848 Acked Write 2 Sent Done Done false 4849 Acked Read 7 Sent Done Done false 4850 Acked Read 7 Sent Done Done false 4851 Acked Write 6 Sent Done Done false 4852 Acked Read 3 Sent Done Done false 4853 Acked Read 2 Sent Done Done false 4854 Acked Read 7 Sent Done Done false 4855 Acked Write 4 Sent Done Done false 4856 Acked Write 5 Sent Done Done false 4857 Acked Read 7 Sent Done Done false 4858 Acked Read 5 Sent Done Done false 4859 Acked Read 3 Sent Done Done false 4860 Acked Read 1 Sent Done Done false 4861 Acked Read 4 Sent Done Done false 4862 Acked Read 4 Sent Done Done false 4863 Acked Read 2 Sent Done Done false 4864 Acked Write 3 Sent Done Done false 4865 Acked Read 5 Sent Done Done false 4866 Acked Read 6 Sent Done Done false 4867 Acked Write 7 Sent Done Done false 4868 Acked Read 6 Sent Done Done false 4869 Acked Read 2 Sent Done Done false 4870 Acked Write 5 Sent Done Done false 4871 Acked Flush 0 Sent Done Done false 4872 Acked Read 10 Sent Done Done false 4873 Acked Write 9 Sent Done Done false 4874 Acked Write 2 Sent Done Done false 4875 Acked Write 5 Sent Done Done false 4876 Acked Read 1 Sent Done Done false 4877 Acked Flush 0 Sent Done Done false 4878 Acked Write 3 Sent Done Done false 4879 Acked Write 2 Sent Done Done false 4880 Acked Write 7 Sent Done Done false 4881 Acked Write 9 Sent Done Done false 4882 Acked Read 8 Sent Done Done false 4883 Acked Write 10 Sent Done Done false 4884 Acked Read 3 Sent Done Done false 4885 Acked Read 9 Sent Done Done false 4886 Acked Write 8 Sent Done Done false 4887 Acked Write 6 Sent Done Done false 4888 Acked Write 10 Sent Done Done false 4889 Acked Read 5 Sent Done Done false 4890 Acked Read 7 Sent Done Done false 4891 Acked Flush 0 Sent Done Done false 4892 Acked Read 7 Sent Done Done false 4893 Acked Write 3 Sent Done Done false 4894 Acked Read 1 Sent Done Done false 4895 Acked Write 10 Sent Done Done false 4896 Acked Read 5 Sent Done Done false 4897 Acked Read 10 Sent Done Done false 4898 Acked Read 9 Sent Done Done false 4899 Acked Write 5 Sent Done Done false 4900 Acked Write 9 Sent Done Done false 4901 Acked Write 9 Sent Done Done false 4902 Acked Flush 0 Sent Done Done false 4903 Acked Write 7 Sent Done Done false 4904 Acked Read 3 Sent Done Done false 4905 Acked Read 8 Sent Done Done false 4906 Acked Read 4 Sent Done Done false 4907 Acked Read 1 Sent Done Done false 4908 Acked Write 5 Sent Done Done false 4909 Acked Read 8 Sent Done Done false 4910 Acked Read 10 Sent Done Done false 4911 Acked Write 5 Sent Done Done false 4912 Acked Write 3 Sent Done Done false 4913 Acked Write 9 Sent Done Done false 4914 Acked Read 6 Sent Done Done false 4915 Acked Write 5 Sent Done Done false 4916 Acked Read 1 Sent Done Done false 4917 Acked Write 2 Sent Done Done false 4918 Acked Read 8 Sent Done Done false 4919 Acked Read 6 Sent Done Done false 4920 Acked Read 4 Sent Done Done false 4921 Acked Write 7 Sent Done Done false 4922 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4922 4922 Downstairs last five completed: 4620 4619 4618 4617 4616 Upstairs last five completed: 4922 4921 4920 4919 4918 CLIENT: Up:0 ds:302 act:2 [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4621 Acked Write 9 Sent Done Done false 4622 Acked Write 4 Sent Done Done false 4623 Acked Write 10 Sent Done Done false 4624 Acked Read 10 Sent Done Done false 4625 Acked Write 2 Sent Done Done false 4626 Acked Read 3 Sent Done Done false 4627 Acked Read 8 Sent Done Done false 4628 Acked Read 8 Sent Done Done false 4629 Acked Write 7 Sent Done Done false 4630 Acked Read 8 Sent Done Done false 4631 Acked Read 6 Sent Done Done false 4632 Acked Read 4 Sent Done Done false 4633 Acked Flush 0 Sent Done Done false 4634 Acked Read 10 Sent Done Done false 4635 Acked Flush 0 Sent Done Done false 4636 Acked Read 4 Sent Done Done false 4637 Acked Write 2 Sent Done Done false 4638 Acked Read 3 Sent Done Done false 4639 Acked Write 1 Sent Done Done false 4640 Acked Flush 0 Sent Done Done false 4641 Acked Read 2 Sent Done Done false 4642 Acked Flush 0 Sent Done Done false 4643 Acked Write 7 Sent Done Done false 4644 Acked Flush 0 Sent Done Done false 4645 Acked Read 1 Sent Done Done false 4646 Acked Read 3 Sent Done Done false 4647 Acked Write 5 Sent Done Done false 4648 Acked Read 4 Sent Done Done false 4649 Acked Write 7 Sent Done Done false 4650 Acked Write 5 Sent Done Done false 4651 Acked Read 10 Sent Done Done false 4652 Acked Write 3 Sent Done Done false 4653 Acked Write 3 Sent Done Done false 4654 Acked Read 2 Sent Done Done false 4655 Acked Write 8 Sent Done Done false 4656 Acked Read 7 Sent Done Done false 4657 Acked Read 10 Sent Done Done false 4658 Acked Read 1 Sent Done Done false 4659 Acked Read 10 Sent Done Done false 4660 Acked Read 3 Sent Done Done false 4661 Acked Write 5 Sent Done Done false 4662 Acked Write 5 Sent Done Done false 4663 Acked Flush 0 Sent Done Done false 4664 Acked Write 9 Sent Done Done false 4665 Acked Flush 0 Sent Done Done false 4666 Acked Read 5 Sent Done Done false 4667 Acked Read 4 Sent Done Done false 4668 Acked Write 3 Sent Done Done false 4669 Acked Write 8 Sent Done Done false 4670 Acked Write 1 Sent Done Done false 4671 Acked Write 5 Sent Done Done false 4672 Acked Read 2 Sent Done Done false 4673 Acked Read 10 Sent Done Done false 4674 Acked Write 3 Sent Done Done false 4675 Acked Write 6 Sent Done Done false 4676 Acked Read 5 Sent Done Done false 4677 Acked Read 4 Sent Done Done false 4678 Acked Read 4 Sent Done Done false 4679 Acked Read 10 Sent Done Done false 4680 Acked Read 5 Sent Done Done false 4681 Acked Read 4 Sent Done Done false 4682 Acked Read 8 Sent Done Done false 4683 Acked Read 8 Sent Done Done false 4684 Acked Write 3 Sent Done Done false 4685 Acked Write 10 Sent Done Done false 4686 Acked Read 9 Sent Done Done false 4687 Acked Read 3 Sent Done Done false 4688 Acked Read 3 Sent Done Done false 4689 Acked Flush 0 Sent Done Done false 4690 Acked Write 7 Sent Done Done false 4691 Acked Write 9 Sent Done Done false 4692 Acked Read 8 Sent Done Done false 4693 Acked Write 3 Sent Done Done false 4694 Acked Read 5 Sent Done Done false 4695 Acked Write 9 Sent Done Done false 4696 Acked Write 4 Sent Done Done false 4697 Acked Write 10 Sent Done Done false 4698 Acked Flush 0 Sent Done Done false 4699 Acked Write 1 Sent Done Done false 4700 Acked Write 10 Sent Done Done false 4701 Acked Flush 0 Sent Done Done false 4702 Acked Read 2 Sent Done Done false 4703 Acked Write 9 Sent Done Done false 4704 Acked Write 6 Sent Done Done false 4705 Acked Read 2 Sent Done Done false 4706 Acked Write 10 Sent Done Done false 4707 Acked Read 6 Sent Done Done false 4708 Acked Read 1 Sent Done Done false 4709 Acked Write 4 Sent Done Done false 4710 Acked Read 5 Sent Done Done false 4711 Acked Read 8 Sent Done Done false 4712 Acked Write 4 Sent Done Done false 4713 Acked Write 5 Sent Done Done false 4714 Acked Read 2 Sent Done Done false 4715 Acked Read 5 Sent Done Done false 4716 Acked Read 2 Sent Done Done false 4717 Acked Flush 0 Sent Done Done false 4718 Acked Read 2 Sent Done Done false 4719 Acked Write 1 Sent Done Done false 4720 Acked Write 4 Sent Done Done false 4721 Acked Read 7 Sent Done Done false 4722 Acked Write 5 Sent Done Done false 4723 Acked Read 9 Sent Done Done false 4724 Acked Flush 0 Sent Done Done false 4725 Acked Write 6 Sent Done Done false 4726 Acked Write 2 Sent Done Done false 4727 Acked Read 10 Sent Done Done false 4728 Acked Write 4 Sent Done Done false 4729 Acked Write 7 Sent Done Done false 4730 Acked Read 10 Sent Done Done false 4731 Acked Write 8 Sent Done Done false 4732 Acked Flush 0 Sent Done Done false 4733 Acked Write 4 Sent Done Done false 4734 Acked Flush 0 Sent Done Done false 4735 Acked Write 10 Sent Done Done false 4736 Acked Write 3 Sent Done Done false 4737 Acked Read 9 Sent Done Done false 4738 Acked Write 9 Sent Done Done false 4739 Acked Read 7 Sent Done Done false 4740 Acked Read 1 Sent Done Done false 4741 Acked Read 1 Sent Done Done false 4742 Acked Read 5 Sent Done Done false 4743 Acked Read 2 Sent Done Done false 4744 Acked Write 3 Sent Done Done false 4745 Acked Read 9 Sent Done Done false 4746 Acked Write 9 Sent Done Done false 4747 Acked Write 8 Sent Done Done false 4748 Acked Read 7 Sent Done Done false 4749 Acked Write 10 Sent Done Done false 4750 Acked Read 4 Sent Done Done false 4751 Acked Read 3 Sent Done Done false 4752 Acked Write 5 Sent Done Done false 4753 Acked Write 2 Sent Done Done false 4754 Acked Read 4 Sent Done Done false 4755 Acked Read 9 Sent Done Done false 4756 Acked Read 8 Sent Done Done false 4757 Acked Read 8 Sent Done Done false 4758 Acked Write 1 Sent Done Done false 4759 Acked Read 1 Sent Done Done false 4760 Acked Read 5 Sent Done Done false 4761 Acked Write 5 Sent Done Done false 4762 Acked Read 3 Sent Done Done false 4763 Acked Write 1 Sent Done Done false 4764 Acked Write 4 Sent Done Done false 4765 Acked Write 8 Sent Done Done false 4766 Acked Read 3 Sent Done Done false 4767 Acked Write 6 Sent Done Done false 4768 Acked Read 8 Sent Done Done false 4769 Acked Read 2 Sent Done Done false 4770 Acked Flush 0 Sent Done Done false 4771 Acked Read 3 Sent Done Done false 4772 Acked Flush 0 Sent Done Done false 4773 Acked Read 8 Sent Done Done false 4774 Acked Read 5 Sent Done Done false 4775 Acked Read 6 Sent Done Done false 4776 Acked Write 1 Sent Done Done false 4777 Acked Write 10 Sent Done Done false 4778 Acked Read 4 Sent Done Done false 4779 Acked Flush 0 Sent Done Done false 4780 Acked Write 9 Sent Done Done false 4781 Acked Read 7 Sent Done Done false 4782 Acked Read 7 Sent Done Done false 4783 Acked Read 5 Sent Done Done false 4784 Acked Read 10 Sent Done Done false 4785 Acked Write 9 Sent Done Done false 4786 Acked Flush 0 Sent Done Done false 4787 Acked Read 10 Sent Done Done false 4788 Acked Flush 0 Sent Done Done false 4789 Acked Read 3 Sent Done Done false 4790 Acked Read 8 Sent Done Done false 4791 Acked Write 7 Sent Done Done false 4792 Acked Flush 0 Sent Done Done false 4793 Acked Read 8 Sent Done Done false 4794 Acked Write 7 Sent Done Done false 4795 Acked Read 9 Sent Done Done false 4796 Acked Write 8 Sent Done Done false 4797 Acked Write 6 Sent Done Done false 4798 Acked Read 6 Sent Done Done false 4799 Acked Read 5 Sent Done Done false 4800 Acked Read 8 Sent Done Done false 4801 Acked Flush 0 Sent Done Done false 4802 Acked Write 3 Sent Done Done false 4803 Acked Write 10 Sent Done Done false 4804 Acked Write 9 Sent Done Done false 4805 Acked Write 10 Sent Done Done false 4806 Acked Read 4 Sent Done Done false 4807 Acked Read 3 Sent Done Done false 4808 Acked Read 5 Sent Done Done false 4809 Acked Read 5 Sent Done Done false 4810 Acked Write 10 Sent Done Done false 4811 Acked Read 8 Sent Done Done false 4812 Acked Read 4 Sent Done Done false 4813 Acked Read 7 Sent Done Done false 4814 Acked Read 3 Sent Done Done false 4815 Acked Read 5 Sent Done Done false 4816 Acked Read 4 Sent Done Done false 4817 Acked Read 9 Sent Done Done false 4818 Acked Read 5 Sent Done Done false 4819 Acked Write 3 Sent Done Done false 4820 Acked Read 8 Sent Done Done false 4821 Acked Read 4 Sent Done Done false 4822 Acked Write 1 Sent Done Done false 4823 Acked Write 5 Sent Done Done false 4824 Acked Read 1 Sent Done Done false 4825 Acked Flush 0 Sent Done Done false 4826 Acked Flush 0 Sent Done Done false 4827 Acked Read 2 Sent Done Done false 4828 Acked Read 7 Sent Done Done false 4829 Acked Read 9 Sent Done Done false 4830 Acked Write 3 Sent Done Done false 4831 Acked Write 2 Sent Done Done false 4832 Acked Read 1 Sent Done Done false 4833 Acked Write 1 Sent Done Done false 4834 Acked Flush 0 Sent Done Done false 4835 Acked Write 4 Sent Done Done false 4836 Acked Read 9 Sent Done Done false 4837 Acked Write 9 Sent Done Done false 4838 Acked Flush 0 Sent Done Done false 4839 Acked Read 6 Sent Done Done false 4840 Acked Flush 0 Sent Done Done false 4841 Acked Write 4 Sent Done Done false 4842 Acked Read 5 Sent Done Done false 4843 Acked Read 4 Sent Done Done false 4844 Acked Read 5 Sent Done Done false 4845 Acked Write 3 Sent Done Done false 4846 Acked Flush 0 Sent Done Done false 4847 Acked Flush 0 Sent Done Done false 4848 Acked Write 2 Sent Done Done false 4849 Acked Read 7 Sent Done Done false 4850 Acked Read 7 Sent Done Done false 4851 Acked Write 6 Sent Done Done false 4852 Acked Read 3 Sent Done Done false 4853 Acked Read 2 Sent Done Done false 4854 Acked Read 7 Sent Done Done false 4855 Acked Write 4 Sent Done Done false 4856 Acked Write 5 Sent Done Done false 4857 Acked Read 7 Sent Done Done false 4858 Acked Read 5 Sent Done Done false 4859 Acked Read 3 Sent Done Done false 4860 Acked Read 1 Sent Done Done false 4861 Acked Read 4 Sent Done Done false 4862 Acked Read 4 Sent Done Done false 4863 Acked Read 2 Sent Done Done false 4864 Acked Write 3 Sent Done Done false 4865 Acked Read 5 Sent Done Done false 4866 Acked Read 6 Sent Done Done false 4867 Acked Write 7 Sent Done Done false 4868 Acked Read 6 Sent Done Done false 4869 Acked Read 2 Sent Done Done false 4870 Acked Write 5 Sent Done Done false 4871 Acked Flush 0 Sent Done Done false 4872 Acked Read 10 Sent Done Done false 4873 Acked Write 9 Sent Done Done false 4874 Acked Write 2 Sent Done Done false 4875 Acked Write 5 Sent Done Done false 4876 Acked Read 1 Sent Done Done false 4877 Acked Flush 0 Sent Done Done false 4878 Acked Write 3 Sent Done Done false 4879 Acked Write 2 Sent Done Done false 4880 Acked Write 7 Sent Done Done false 4881 Acked Write 9 Sent Done Done false 4882 Acked Read 8 Sent Done Done false 4883 Acked Write 10 Sent Done Done false 4884 Acked Read 3 Sent Done Done false 4885 Acked Read 9 Sent Done Done false 4886 Acked Write 8 Sent Done Done false 4887 Acked Write 6 Sent Done Done false 4888 Acked Write 10 Sent Done Done false 4889 Acked Read 5 Sent Done Done false 4890 Acked Read 7 Sent Done Done false 4891 Acked Flush 0 Sent Done Done false 4892 Acked Read 7 Sent Done Done false 4893 Acked Write 3 Sent Done Done false 4894 Acked Read 1 Sent Done Done false 4895 Acked Write 10 Sent Done Done false 4896 Acked Read 5 Sent Done Done false 4897 Acked Read 10 Sent Done Done false 4898 Acked Read 9 Sent Done Done false 4899 Acked Write 5 Sent Done Done false 4900 Acked Write 9 Sent Done Done false 4901 Acked Write 9 Sent Done Done false 4902 Acked Flush 0 Sent Done Done false 4903 Acked Write 7 Sent Done Done false 4904 Acked Read 3 Sent Done Done false 4905 Acked Read 8 Sent Done Done false 4906 Acked Read 4 Sent Done Done false 4907 Acked Read 1 Sent Done Done false 4908 Acked Write 5 Sent Done Done false 4909 Acked Read 8 Sent Done Done false 4910 Acked Read 10 Sent Done Done false 4911 Acked Write 5 Sent Done Done false 4912 Acked Write 3 Sent Done Done false 4913 Acked Write 9 Sent Done Done false 4914 Acked Read 6 Sent Done Done false 4915 Acked Write 5 Sent Done Done false 4916 Acked Read 1 Sent Done Done false 4917 Acked Write 2 Sent Done Done false 4918 Acked Read 8 Sent Done Done false 4919 Acked Read 6 Sent Done Done false 4920 Acked Read 4 Sent Done Done false 4921 Acked Write 7 Sent Done Done false 4922 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4620 4922 4922 Downstairs last five completed: 4620 4619 4618 4617 4616 Upstairs last five completed: 4922 4921 4920 4919 4918 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.608572297Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:38716)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.60871052Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:06.609657516Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610157742Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610183855Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 4620","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.6101992Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 4620","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610411943Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610436787Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610451923Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 4620","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610466179Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 4620","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.610699226Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4922 4921 4920 4919 4918 Upstairs last five completed: 4922 4921 4920 4919 4918 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:45:06.853 INFO request completed, latency_us:[8830] Got stop action so:false kr:false 354, response_code: 204, uri: /stop/cid/2, method: POST, req_id: aeb0a070-101f-4a16-a656-9bf07b1b896f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:06.854234217Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:06.854305908Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:06.854340698Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.854368151Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.854399342Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:06.854428085Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:45:06.854 INFO request completed, latency_us: 138, response_code: 200, uri: /state/cid/2, method: GET, req_id: 66d201c9-3b53-4cc2-a6ac-5cc8376f3c35, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:45:10.855 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 50613469-3467-4d36-8b0d-9ed464e939d6, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 2779 len 20480 002/300 Write block 1299 len 8192 data: 3 3 003/300 Write block 2136 len 16384 data: 3 4 5 5 004/300 Read block 1435 len 12288 005/300 Read block 3709 len 36864 006/300 Read block 3987 len 36864 007/300 Read block 3746 len 20480 008/300 Read block 3141 len 4096 009/300 Read block 2079 len 20480 010/300 Read block 889 len 4096 011/300 Read block 3239 len 24576 012/300 Read block 4280 len 20480 013/300 Read block 3384 len 24576 014/300 Read block 1976 len 12288 015/300 Read block 3475 len 8192 016/300 Read block 3420 len 8192 017/300 Read block 4353 len 12288 018/300 Read block 167 len 36864 019/300 Flush 020/300 Read block 2110 len 8192 021/300 Read block 4121 len 8192 022/300 Flush 023/300 Read block 3696 len 24576 024/300 Read block 4471 len 24576 025/300 Write block 4002 len 12288 data: 3 3 3 026/300 Read block 1083 len 24576 027/300 Flush 028/300 Read block 3128 len 24576 029/300 Read block 128 len 8192 030/300 Write block 3439 len 32768 data: 3 3 2 2 2 2 2 2 031/300 Read block 1730 len 12288 032/300 Read block 2353 len 36864 033/300 Write block 2883 len 28672 data: 2 2 3 3 3 4 4 034/300 Read block 654 len 28672 035/300 Read block 1978 len 40960 036/300 Write block 1017 len 36864 data: 4 3 3 3 3 3 3 3 2 037/300 Write block 814 len 4096 data: 5 038/300 Read block 475 len 4096 039/300 Write block 2910 len 20480 data: 2 2 2 2 2 040/300 Write block 76 len 40960 data: 3 2 2 3 3 3 3 3 2 2 041/300 Read block 3626 len 16384 042/300 Read block 3176 len 32768 043/300 Flush 044/300 Write block 1012 len 36864 data: 6 4 4 4 4 5 4 4 4 045/300 Read block 1836 len 24576 046/300 Write block 1094 len 20480 data: 5 5 4 5 4 047/300 Write block 2097 len 8192 data: 3 4 048/300 Write block 1582 len 20480 data: 2 2 2 2 3 049/300 Read block 4485 len 28672 050/300 Read block 4043 len 12288 051/300 Write block 1421 len 20480 data: 3 3 4 4 4 052/300 Write block 1375 len 24576 data: 4 4 4 4 3 3 053/300 Read block 255 len 24576 054/300 Read block 2923 len 12288 055/300 Write block 1950 len 24576 data: 4 4 4 4 4 3 056/300 Read block 1342 len 28672 057/300 Write block 1569 len 36864 data: 3 3 3 3 5 4 5 5 4 058/300 Read block 3785 len 24576 059/300 Read block 3724 len 32768 060/300 Write block 4534 len 32768 data: 2 2 2 2 2 2 2 3 061/300 Flush 062/300 Flush 063/300 Write block 2596 len 12288 data: 2 2 2 064/300 Read block 3844 len 24576 065/300 Write block 3209 len 12288 data: 6 5 5 066/300 Write block 1096 len 20480 data: 5 6 5 4 3 067/300 Read block 3089 len 36864 068/300 Read block 3548 len 40960 069/300 Write block 1709 len 40960 data: 4 3 3 2 2 4 3 3 3 3 070/300 Write block 57 len 12288 data: 2 2 2 071/300 Write block 1285 len 20480 data: 3 3 3 4 4 072/300 Write block 3386 len 32768 data: 4 4 2 3 3 4 4 4 073/300 Read block 2228 len 24576 074/300 Read block 332 len 28672 075/300 Write block 4134 len 4096 data: 5 076/300 Read block 4017 len 32768 077/300 Write block 342 len 16384 data: 6 6 4 4 078/300 Write block 1014 len 8192 data: 5 5 079/300 Write block 2319 len 12288 data: 2 3 3 080/300 Write block 3850 len 24576 data: 2 3 3 2 2 2 081/300 Read block 1748 len 40960 082/300 Read block 2618 len 8192 083/300 Write block 1633 len 4096 data: 4 084/300 Read block 2451 len 20480 085/300 Write block 2684 len 24576 data: 3 3 3 3 4 3 086/300 Flush 087/300 Read block 1483 len 40960 088/300 Write block 3520 len 8192 data: 2 2 089/300 Read block 110 len 32768 090/300 Write block 584 len 12288 data: 4 4 4 091/300 Read block 2714 len 40960 092/300 Write block 3 len 16384 data: 2 3 4 4 093/300 Flush 094/300 Read block 3931 len 4096 095/300 Write block 3369 len 4096 data: 3 096/300 Read block 1565 len 16384 097/300 Read block 1997 len 24576 098/300 Write block 4659 len 40960 data: 5 4 3 4 4 3 2 3 3 3 099/300 Read block 4323 len 12288 100/300 Write block 1572 len 36864 data: 4 6 5 6 6 5 3 3 3 101/300 Write block 4114 len 40960 data: 2 3 3 3 4 3 2 2 2 3 102/300 Write block 4797 len 20480 data: 6 7 6 6 5 103/300 Read block 2802 len 24576 104/300 Write block 2385 len 32768 data: 5 5 5 6 6 5 4 3 105/300 Read block 3479 len 12288 106/300 Read block 2416 len 4096 107/300 Read block 4324 len 8192 108/300 Flush 109/300 Read block 1928 len 40960 110/300 Write block 463 len 16384 data: 2 3 3 3 111/300 Write block 1931 len 20480 data: 4 4 4 4 3 112/300 Read block 949 len 16384 113/300 Read block 4112 len 4096 114/300 Read block 1764 len 20480 115/300 Read block 3777 len 16384 116/300 Read block 3957 len 40960 117/300 Flush 118/300 Read block 897 len 8192 119/300 Read block 258 len 36864 120/300 Write block 3340 len 4096 data: 3 121/300 Read block 2114 len 20480 122/300 Read block 2479 len 28672 123/300 Read block 3132 len 32768 124/300 Read block 4181 len 24576 125/300 Write block 2483 len 16384 data: 3 3 3 5 126/300 Write block 3223 len 12288 data: 2 2 4 127/300 Flush 128/300 Read block 2305 len 28672 129/300 Read block 493 len 28672 130/300 Write block 4911 len 24576 data: 2 2 2 2 2 2 131/300 Write block 702 len 12288 data: 2 3 3 132/300 Write block 1279 len 20480 data: 4 4 4 4 4 133/300 Read block 816 len 40960 134/300 Write block 3731 len 4096 data: 4 135/300 Read block 552 len 40960 136/300 Read block 4753 len 36864 137/300 Read block 2051 len 36864 138/300 Write block 252 len 12288 data: 3 4 4 139/300 Write block 161 len 40960 data: 2 2 3 3 3 4 4 5 5 5 140/300 Read block 2153 len 24576 141/300 Read block 301 len 12288 142/300 Write block 1448 len 24576 data: 3 4 4 4 4 4 143/300 Read block 569 len 20480 144/300 Read block 4005 len 24576 145/300 Write block 752 len 8192 data: 8 7 146/300 Read block 3361 len 4096 147/300 Read block 1358 len 12288 148/300 Read block 4932 len 20480 149/300 Read block 3769 len 12288 150/300 Read block 1997 len 32768 151/300 Read block 3125 len 24576 152/300 Read block 1669 len 12288 153/300 Read block 3233 len 16384 154/300 Read block 2203 len 20480 155/300 Write block 1996 len 12288 data: 7 8 8 156/300 Write block 4967 len 32768 data: 2 4 4 4 3 2 2 2 157/300 Read block 4517 len 8192 158/300 Read block 626 len 32768 159/300 Write block 4928 len 28672 data: 3 4 4 4 4 3 3 160/300 Read block 4044 len 32768 161/300 Write block 615 len 36864 data: 2 3 3 3 3 4 4 4 3 162/300 Read block 2592 len 20480 163/300 Write block 3257 len 20480 data: 5 3 3 2 2 164/300 Read block 2522 len 8192 165/300 Read block 4099 len 12288 166/300 Read block 2570 len 20480 167/300 Read block 1615 len 8192 168/300 Read block 1087 len 32768 169/300 Write block 196 len 12288 data: 2 2 2 170/300 Read block 2886 len 12288 171/300 Flush 172/300 Write block 2825 len 8192 data: 3 3 173/300 Write block 742 len 16384 data: 4 4 5 5 174/300 Write block 3518 len 4096 data: 2 175/300 Read block 2841 len 12288 176/300 Read block 2790 len 4096 177/300 Read block 1088 len 12288 178/300 Write block 4150 len 36864 data: 4 4 5 5 6 3 3 2 2 179/300 Read block 1023 len 36864 180/300 Read block 2321 len 16384 181/300 Read block 1825 len 20480 182/300 Read block 1705 len 8192 183/300 Write block 3212 len 4096 data: 4 184/300 Read block 4172 len 32768 185/300 Read block 2960 len 36864 186/300 Read block 3343 len 40960 187/300 Read block 1824 len 36864 188/300 Write block 3098 len 36864 data: 3 3 3 3 4 4 4 4 4 189/300 Write block 3725 len 8192 data: 4 5 190/300 Read block 4675 len 28672 191/300 Flush 192/300 Read block 3985 len 8192 193/300 Read block 2014 len 28672 194/300 Write block 2486 len 32768 data: 6 3 3 3 4 4 4 5 195/300 Flush 196/300 Write block 3518 len 16384 data: 3 2 3 3 197/300 Read block 4869 len 4096 198/300 Read block 790 len 28672 199/300 Write block 1164 len 36864 data: 3 4 4 4 4 4 5 6 5 200/300 Read block 2352 len 4096 201/300 Flush 202/300 Flush 203/300 Read block 2479 len 12288 204/300 Read block 3878 len 28672 205/300 Flush 206/300 Read block 1577 len 32768 207/300 Write block 2393 len 4096 data: 3 208/300 Read block 963 len 24576 209/300 Write block 1755 len 12288 data: 3 3 3 210/300 Write block 2200 len 40960 data: 3 3 3 3 3 2 2 2 3 3 211/300 Read block 104 len 8192 212/300 Write block 3517 len 4096 data: 3 213/300 Write block 3953 len 28672 data: 5 5 4 4 3 3 4 214/300 Write block 1203 len 12288 data: 3 3 3 215/300 Read block 3973 len 28672 216/300 Write block 2336 len 8192 data: 4 4 217/300 Read block 424 len 36864 218/300 Write block 3686 len 28672 data: 5 4 5 5 6 5 4 219/300 Read block 358 len 36864 220/300 Read block 21 len 12288 221/300 Read block 1643 len 32768 222/300 Write block 3916 len 32768 data: 2 3 3 2 2 2 2 3 223/300 Write block 401 len 28672 data: 2 2 2 2 2 3 4 224/300 Read block 321 len 8192 225/300 Read block 2707 len 8192 226/300 Flush 227/300 Write block 579 len 28672 data: 4 5 4 4 3 5 5 228/300 Read block 1023 len 4096 229/300 Write block 4485 len 4096 data: 4 230/300 Read block 3772 len 28672 231/300 Read block 1988 len 20480 232/300 Flush 233/300 Write block 823 len 36864 data: 4 6 7 8 8 8 8 7 6 234/300 Write block 1115 len 20480 data: 3 3 3 3 3 235/300 Read block 3887 len 24576 236/300 Read block 3410 len 4096 237/300 Read block 1998 len 40960 238/300 Read block 1929 len 16384 239/300 Write block 987 len 16384 data: 3 3 3 3 240/300 Read block 1661 len 36864 241/300 Read block 4836 len 28672 242/300 Read block 1484 len 12288 243/300 Write block 2124 len 36864 data: 3 3 3 2 3 3 3 3 4 244/300 Write block 4772 len 16384 data: 5 5 5 4 245/300 Read block 3606 len 8192 246/300 Read block 2053 len 20480 247/300 Read block 3061 len 12288 248/300 Read block 3794 len 8192 249/300 Read block 19 len 20480 250/300 Write block 4677 len 24576 data: 5 7 7 6 5 5 251/300 Read block 2795 len 40960 252/300 Read block 427 len 40960 253/300 Write block 4975 len 4096 data: 3 254/300 Read block 4015 len 12288 255/300 Flush 256/300 Read block 3450 len 16384 257/300 Read block 4732 len 36864 258/300 Read block 4800 len 28672 259/300 Flush 260/300 Read block 4199 len 8192 261/300 Write block 2648 len 36864 data: 4 4 4 4 4 4 4 4 3 262/300 Write block 1277 len 12288 data: 2 2 5 263/300 Flush 264/300 Write block 287 len 32768 data: 2 2 2 2 3 3 3 3 265/300 Read block 3284 len 28672 266/300 Read block 2281 len 8192 267/300 Read block 3533 len 8192 268/300 Read block 4769 len 28672 269/300 Read block 1869 len 20480 270/300 Read block 4219 len 40960 271/300 Write block 2517 len 16384 data: 2 3 3 3 272/300 Write block 4722 len 12288 data: 3 3 3 273/300 Read block 2176 len 16384 274/300 Read block 4959 len 32768 275/300 Read block 3520 len 32768 276/300 Write block 864 len 20480 data: 3 4 5 5 5 277/300 Read block 478 len 40960 278/300 Read block 3229 len 32768 279/300 Read block 3709 len 32768 280/300 Read block 2707 len 4096 281/300 Read block 2392 len 12288 282/300 Read block 2948 len 8192 283/300 Write block 1629 len 16384 data: 4 4 4 4 284/300 Read block 2586 len 36864 285/300 Write block 4013 len 40960 data: 2 2 2 3 3 3 2 2 2 3 286/300 Write block 2842 len 8192 data: 5 5 287/300 Write block 3105 len 36864 data: 5 5 5 5 5 4 4 4 2 288/300 Write block 1176 len 28672 data: 5 6 6 7 6 6 5 289/300 Flush 290/300 Read block 860 len 16384 291/300 Read block 2712 len 24576 292/300 Write block 2041 len 8192 data: 5 5 293/300 Read block 722 len 8192 294/300 Write block 1154 len 36864 data: 4 4 3 5 5 5 4 4 4 295/300 Write block 1296 len 24576 data: 5 5 4 4 4 3 296/300 Read block 2554 len 28672 297/300 Write block 4592 len 16384 data: 3 3 4 5 298/300 Read block 1165 len 8192 299/300 Read block 1362 len 32768 300/300 Write block 3055 len 20480 data: 2 2 2 2 2 301/300 Read block 4879 len 4096 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:45:11.101 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7f1cd18c-3c69-4828-8ca5-ed5ead55876a, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4923 Acked Read 5 Done Done Sent false 4924 Acked Write 2 Done Done Sent false 4925 Acked Write 4 Done Done Sent false 4926 Acked Read 3 Done Done Sent false 4927 Acked Read 9 Done Done Sent false 4928 Acked Read 9 Done Done Sent false 4929 Acked Read 5 Done Done Sent false 4930 Acked Read 1 Done Done Sent false 4931 Acked Read 5 Done Done Sent false 4932 Acked Read 1 Done Done Sent false 4933 Acked Read 6 Done Done Sent false 4934 Acked Read 5 Done Done Sent false 4935 Acked Read 6 Done Done Sent false 4936 Acked Read 3 Done Done Sent false 4937 Acked Read 2 Done Done Sent false 4938 Acked Read 2 Done Done Sent false 4939 Acked Read 3 Done Done Sent false 4940 Acked Read 9 Done Done Sent false 4941 Acked Flush 0 Done Done Sent false 4942 Acked Read 2 Done Done Sent false 4943 Acked Read 2 Done Done Sent false 4944 Acked Flush 0 Done Done Sent false 4945 Acked Read 6 Done Done Sent false 4946 Acked Read 6 Done Done Sent false 4947 Acked Write 3 Done Done Sent false 4948 Acked Read 6 Done Done Sent false 4949 Acked Flush 0 Done Done Sent false 4950 Acked Read 6 Done Done Sent false 4951 Acked Read 2 Done Done Sent false 4952 Acked Write 8 Done Done Sent false 4953 Acked Read 3 Done Done Sent false 4954 Acked Read 9 Done Done Sent false 4955 Acked Write 7 Done Done Sent false 4956 Acked Read 7 Done Done Sent false 4957 Acked Read 10 Done Done Sent false 4958 Acked Write 9 Done Done Sent false 4959 Acked Write 1 Done Done Sent false 4960 Acked Read 1 Done Done Sent false 4961 Acked Write 5 Done Done Sent false 4962 Acked Write 10 Done Done Sent false 4963 Acked Read 4 Done Done Sent false 4964 Acked Read 8 Done Done Sent false 4965 Acked Flush 0 Done Done Sent false 4966 Acked Write 9 Done Done Sent false 4967 Acked Read 6 Done Done Sent false 4968 Acked Write 5 Done Done Sent false 4969 Acked Write 2 Done Done Sent false 4970 Acked Write 5 Done Done Sent false 4971 Acked Read 7 Done Done Sent false 4972 Acked Read 3 Done Done Sent false 4973 Acked Write 5 Done Done Sent false 4974 Acked Write 6 Done Done Sent false 4975 Acked Read 6 Done Done Sent false 4976 Acked Read 3 Done Done Sent false 4977 Acked Write 6 Done Done Sent false 4978 Acked Read 7 Done Done Sent false 4979 Acked Write 9 Done Done Sent false 4980 Acked Read 6 Done Done Sent false 4981 Acked Read 8 Done Done Sent false 4982 Acked Write 8 Done Done Sent false 4983 Acked Flush 0 Done Done Sent false 4984 Acked Flush 0 Done Done Sent false 4985 Acked Write 3 Done Done Sent false 4986 Acked Read 6 Done Done Sent false 4987 Acked Write 3 Done Done Sent false 4988 Acked Write 5 Done Done Sent false 4989 Acked Read 9 Done Done Sent false 4990 Acked Read 10 Done Done Sent false 4991 Acked Write 10 Done Done Sent false 4992 Acked Write 3 Done Done Sent false 4993 Acked Write 5 Done Done Sent false 4994 Acked Write 8 Done Done Sent false 4995 Acked Read 6 Done Done Sent false 4996 Acked Read 7 Done Done Sent false 4997 Acked Write 1 Done Done Sent false 4998 Acked Read 8 Done Done Sent false 4999 Acked Write 4 Done Done Sent false 5000 Acked Write 2 Done Done Sent false 5001 Acked Write 3 Done Done Sent false 5002 Acked Write 6 Done Done Sent false 5003 Acked Read 10 Done Done Sent false 5004 Acked Read 2 Done Done Sent false 5005 Acked Write 1 Done Done Sent false 5006 Acked Read 5 Done Done Sent false 5007 Acked Write 6 Done Done Sent false 5008 Acked Flush 0 Done Done Sent false 5009 Acked Read 10 Done Done Sent false 5010 Acked Write 2 Done Done Sent false 5011 Acked Read 8 Done Done Sent false 5012 Acked Write 3 Done Done Sent false 5013 Acked Read 10 Done Done Sent false 5014 Acked Write 4 Done Done Sent false 5015 Acked Flush 0 Done Done Sent false 5016 Acked Read 1 Done Done Sent false 5017 Acked Write 1 Done Done Sent false 5018 Acked Read 4 Done Done Sent false 5019 Acked Read 6 Done Done Sent false 5020 Acked Write 10 Done Done Sent false 5021 Acked Read 3 Done Done Sent false 5022 Acked Write 9 Done Done Sent false 5023 Acked Write 10 Done Done Sent false 5024 Acked Write 5 Done Done Sent false 5025 Acked Read 6 Done Done Sent false 5026 Acked Write 8 Done Done Sent false 5027 Acked Read 3 Done Done Sent false 5028 Acked Read 1 Done Done Sent false 5029 Acked Read 2 Done Done Sent false 5030 Acked Flush 0 Done Done Sent false 5031 Acked Read 10 Done Done Sent false 5032 Acked Write 4 Done Done Sent false 5033 Acked Write 5 Done Done Sent false 5034 Acked Read 4 Done Done Sent false 5035 Acked Read 1 Done Done Sent false 5036 Acked Read 5 Done Done Sent false 5037 Acked Read 4 Done Done Sent false 5038 Acked Read 10 Done Done Sent false 5039 Acked Flush 0 Done Done Sent false 5040 Acked Read 2 Done Done Sent false 5041 Acked Read 9 Done Done Sent false 5042 Acked Write 1 Done Done Sent false 5043 Acked Read 5 Done Done Sent false 5044 Acked Read 7 Done Done Sent false 5045 Acked Read 8 Done Done Sent false 5046 Acked Read 6 Done Done Sent false 5047 Acked Write 4 Done Done Sent false 5048 Acked Write 3 Done Done Sent false 5049 Acked Flush 0 Done Done Sent false 5050 Acked Read 7 Done Done Sent false 5051 Acked Read 7 Done Done Sent false 5052 Acked Write 6 Done Done Sent false 5053 Acked Write 3 Done Done Sent false 5054 Acked Write 5 Done Done Sent false 5055 Acked Read 10 Done Done Sent false 5056 Acked Write 1 Done Done Sent false 5057 Acked Read 10 Done Done Sent false 5058 Acked Read 9 Done Done Sent false 5059 Acked Read 9 Done Done Sent false 5060 Acked Write 3 Done Done Sent false 5061 Acked Write 10 Done Done Sent false 5062 Acked Read 6 Done Done Sent false 5063 Acked Read 3 Done Done Sent false 5064 Acked Write 6 Done Done Sent false 5065 Acked Read 5 Done Done Sent false 5066 Acked Read 6 Done Done Sent false 5067 Acked Write 2 Done Done Sent false 5068 Acked Read 1 Done Done Sent false 5069 Acked Read 3 Done Done Sent false 5070 Acked Read 5 Done Done Sent false 5071 Acked Read 3 Done Done Sent false 5072 Acked Read 8 Done Done Sent false 5073 Acked Read 6 Done Done Sent false 5074 Acked Read 3 Done Done Sent false 5075 Acked Read 4 Done Done Sent false 5076 Acked Read 5 Done Done Sent false 5077 Acked Write 3 Done Done Sent false 5078 Acked Write 8 Done Done Sent false 5079 Acked Read 2 Done Done Sent false 5080 Acked Read 8 Done Done Sent false 5081 Acked Write 7 Done Done Sent false 5082 Acked Read 8 Done Done Sent false 5083 Acked Write 9 Done Done Sent false 5084 Acked Read 5 Done Done Sent false 5085 Acked Write 5 Done Done Sent false 5086 Acked Read 2 Done Done Sent false 5087 Acked Read 3 Done Done Sent false 5088 Acked Read 5 Done Done Sent false 5089 Acked Read 2 Done Done Sent false 5090 Acked Read 8 Done Done Sent false 5091 Acked Write 3 Done Done Sent false 5092 Acked Read 3 Done Done Sent false 5093 Acked Flush 0 Done Done Sent false 5094 Acked Write 2 Done Done Sent false 5095 Acked Write 4 Done Done Sent false 5096 Acked Write 1 Done Done Sent false 5097 Acked Read 3 Done Done Sent false 5098 Acked Read 1 Done Done Sent false 5099 Acked Read 3 Done Done Sent false 5100 Acked Write 9 Done Done Sent false 5101 Acked Read 9 Done Done Sent false 5102 Acked Read 4 Done Done Sent false 5103 Acked Read 5 Done Done Sent false 5104 Acked Read 2 Done Done Sent false 5105 Acked Write 1 Done Done Sent false 5106 Acked Read 8 Done Done Sent false 5107 Acked Read 9 Done Done Sent false 5108 Acked Read 10 Done Done Sent false 5109 Acked Read 9 Done Done Sent false 5110 Acked Write 9 Done Done Sent false 5111 Acked Write 2 Done Done Sent false 5112 Acked Read 7 Done Done Sent false 5113 Acked Flush 0 Done Done Sent false 5114 Acked Read 2 Done Done Sent false 5115 Acked Read 7 Done Done Sent false 5116 Acked Write 8 Done Done Sent false 5117 Acked Flush 0 Done Done Sent false 5118 Acked Write 4 Done Done Sent false 5119 Acked Read 1 Done Done Sent false 5120 Acked Read 7 Done Done Sent false 5121 Acked Write 9 Done Done Sent false 5122 Acked Read 1 Done Done Sent false 5123 Acked Flush 0 Done Done Sent false 5124 Acked Flush 0 Done Done Sent false 5125 Acked Read 3 Done Done Sent false 5126 Acked Read 7 Done Done Sent false 5127 Acked Flush 0 Done Done Sent false 5128 Acked Read 8 Done Done Sent false 5129 Acked Write 1 Done Done Sent false 5130 Acked Read 6 Done Done Sent false 5131 Acked Write 3 Done Done Sent false 5132 Acked Write 10 Done Done Sent false 5133 Acked Read 2 Done Done Sent false 5134 Acked Write 1 Done Done Sent false 5135 Acked Write 7 Done Done Sent false 5136 Acked Write 3 Done Done Sent false 5137 Acked Read 7 Done Done Sent false 5138 Acked Write 2 Done Done Sent false 5139 Acked Read 9 Done Done Sent false 5140 Acked Write 7 Done Done Sent false 5141 Acked Read 9 Done Done Sent false 5142 Acked Read 3 Done Done Sent false 5143 Acked Read 8 Done Done Sent false 5144 Acked Write 8 Done Done Sent false 5145 Acked Write 7 Done Done Sent false 5146 Acked Read 2 Done Done Sent false 5147 Acked Read 2 Done Done Sent false 5148 Acked Flush 0 Done Done Sent false 5149 Acked Write 7 Done Done Sent false 5150 Acked Read 1 Done Done Sent false 5151 Acked Write 1 Done Done Sent false 5152 Acked Read 7 Done Done Sent false 5153 Acked Read 5 Done Done Sent false 5154 Acked Flush 0 Done Done Sent false 5155 Acked Write 9 Done Done Sent false 5156 Acked Write 5 Done Done Sent false 5157 Acked Read 6 Done Done Sent false 5158 Acked Read 1 Done Done Sent false 5159 Acked Read 10 Done Done Sent false 5160 Acked Read 4 Done Done Sent false 5161 Acked Write 4 Done Done Sent false 5162 Acked Read 9 Done Done Sent false 5163 Acked Read 7 Done Done Sent false 5164 Acked Read 3 Done Done Sent false 5165 Acked Write 9 Done Done Sent false 5166 Acked Write 4 Done Done Sent false 5167 Acked Read 2 Done Done Sent false 5168 Acked Read 5 Done Done Sent false 5169 Acked Read 3 Done Done Sent false 5170 Acked Read 2 Done Done Sent false 5171 Acked Read 5 Done Done Sent false 5172 Acked Write 6 Done Done Sent false 5173 Acked Read 10 Done Done Sent false 5174 Acked Read 10 Done Done Sent false 5175 Acked Write 1 Done Done Sent false 5176 Acked Read 3 Done Done Sent false 5177 Acked Flush 0 Done Done Sent false 5178 Acked Read 4 Done Done Sent false 5179 Acked Read 9 Done Done Sent false 5180 Acked Read 7 Done Done Sent false 5181 Acked Flush 0 Done Done Sent false 5182 Acked Read 2 Done Done Sent false 5183 Acked Write 9 Done Done Sent false 5184 Acked Write 3 Done Done Sent false 5185 Acked Flush 0 Done Done Sent false 5186 Acked Write 8 Done Done Sent false 5187 Acked Read 7 Done Done Sent false 5188 Acked Read 2 Done Done Sent false 5189 Acked Read 2 Done Done Sent false 5190 Acked Read 7 Done Done Sent false 5191 Acked Read 5 Done Done Sent false 5192 Acked Read 10 Done Done Sent false 5193 Acked Write 4 Done Done Sent false 5194 Acked Write 3 Done Done Sent false 5195 Acked Read 4 Done Done Sent false 5196 Acked Read 8 Done Done Sent false 5197 Acked Read 8 Done Done Sent false 5198 Acked Write 5 Done Done Sent false 5199 Acked Read 10 Done Done Sent false 5200 Acked Read 8 Done Done Sent false 5201 Acked Read 8 Done Done Sent false 5202 Acked Read 1 Done Done Sent false 5203 Acked Read 3 Done Done Sent false 5204 Acked Read 2 Done Done Sent false 5205 Acked Write 4 Done Done Sent false 5206 Acked Read 9 Done Done Sent false 5207 Acked Write 10 Done Done Sent false 5208 Acked Write 2 Done Done Sent false 5209 Acked Write 9 Done Done Sent false 5210 Acked Write 7 Done Done Sent false 5211 Acked Flush 0 Done Done Sent false 5212 Acked Read 4 Done Done Sent false 5213 Acked Read 6 Done Done Sent false 5214 Acked Write 2 Done Done Sent false 5215 Acked Read 2 Done Done Sent false 5216 Acked Write 9 Done Done Sent false 5217 Acked Write 6 Done Done Sent false 5218 Acked Read 7 Done Done Sent false 5219 Acked Write 4 Done Done Sent false 5220 Acked Read 2 Done Done Sent false 5221 Acked Read 8 Done Done Sent false 5222 Acked Write 5 Done Done Sent false 5223 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5211 5211 4922 Downstairs last five completed: 4922 4921 4920 4919 4918 Upstairs last five completed: 5223 5222 5221 5220 5219 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 4923 Acked Read 5 Done Done Sent false 4924 Acked Write 2 Done Done Sent false 4925 Acked Write 4 Done Done Sent false 4926 Acked Read 3 Done Done Sent false 4927 Acked Read 9 Done Done Sent false 4928 Acked Read 9 Done Done Sent false 4929 Acked Read 5 Done Done Sent false 4930 Acked Read 1 Done Done Sent false 4931 Acked Read 5 Done Done Sent false 4932 Acked Read 1 Done Done Sent false 4933 Acked Read 6 Done Done Sent false 4934 Acked Read 5 Done Done Sent false 4935 Acked Read 6 Done Done Sent false 4936 Acked Read 3 Done Done Sent false 4937 Acked Read 2 Done Done Sent false 4938 Acked Read 2 Done Done Sent false 4939 Acked Read 3 Done Done Sent false 4940 Acked Read 9 Done Done Sent false 4941 Acked Flush 0 Done Done Sent false 4942 Acked Read 2 Done Done Sent false 4943 Acked Read 2 Done Done Sent false 4944 Acked Flush 0 Done Done Sent false 4945 Acked Read 6 Done Done Sent false 4946 Acked Read 6 Done Done Sent false 4947 Acked Write 3 Done Done Sent false 4948 Acked Read 6 Done Done Sent false 4949 Acked Flush 0 Done Done Sent false 4950 Acked Read 6 Done Done Sent false 4951 Acked Read 2 Done Done Sent false 4952 Acked Write 8 Done Done Sent false 4953 Acked Read 3 Done Done Sent false 4954 Acked Read 9 Done Done Sent false 4955 Acked Write 7 Done Done Sent false 4956 Acked Read 7 Done Done Sent false 4957 Acked Read 10 Done Done Sent false 4958 Acked Write 9 Done Done Sent false 4959 Acked Write 1 Done Done Sent false 4960 Acked Read 1 Done Done Sent false 4961 Acked Write 5 Done Done Sent false 4962 Acked Write 10 Done Done Sent false 4963 Acked Read 4 Done Done Sent false 4964 Acked Read 8 Done Done Sent false 4965 Acked Flush 0 Done Done Sent false 4966 Acked Write 9 Done Done Sent false 4967 Acked Read 6 Done Done Sent false 4968 Acked Write 5 Done Done Sent false 4969 Acked Write 2 Done Done Sent false 4970 Acked Write 5 Done Done Sent false 4971 Acked Read 7 Done Done Sent false 4972 Acked Read 3 Done Done Sent false 4973 Acked Write 5 Done Done Sent false 4974 Acked Write 6 Done Done Sent false 4975 Acked Read 6 Done Done Sent false 4976 Acked Read 3 Done Done Sent false 4977 Acked Write 6 Done Done Sent false 4978 Acked Read 7 Done Done Sent false 4979 Acked Write 9 Done Done Sent false 4980 Acked Read 6 Done Done Sent false 4981 Acked Read 8 Done Done Sent false 4982 Acked Write 8 Done Done Sent false 4983 Acked Flush 0 Done Done Sent false 4984 Acked Flush 0 Done Done Sent false 4985 Acked Write 3 Done Done Sent false 4986 Acked Read 6 Done Done Sent false 4987 Acked Write 3 Done Done Sent false 4988 Acked Write 5 Done Done Sent false 4989 Acked Read 9 Done Done Sent false 4990 Acked Read 10 Done Done Sent false 4991 Acked Write 10 Done Done Sent false 4992 Acked Write 3 Done Done Sent false 4993 Acked Write 5 Done Done Sent false 4994 Acked Write 8 Done Done Sent false 4995 Acked Read 6 Done Done Sent false 4996 Acked Read 7 Done Done Sent false 4997 Acked Write 1 Done Done Sent false 4998 Acked Read 8 Done Done Sent false 4999 Acked Write 4 Done Done Sent false 5000 Acked Write 2 Done Done Sent false 5001 Acked Write 3 Done Done Sent false 5002 Acked Write 6 Done Done Sent false 5003 Acked Read 10 Done Done Sent false 5004 Acked Read 2 Done Done Sent false 5005 Acked Write 1 Done Done Sent false 5006 Acked Read 5 Done Done Sent false 5007 Acked Write 6 Done Done Sent false 5008 Acked Flush 0 Done Done Sent false 5009 Acked Read 10 Done Done Sent false 5010 Acked Write 2 Done Done Sent false 5011 Acked Read 8 Done Done Sent false 5012 Acked Write 3 Done Done Sent false 5013 Acked Read 10 Done Done Sent false 5014 Acked Write 4 Done Done Sent false 5015 Acked Flush 0 Done Done Sent false 5016 Acked Read 1 Done Done Sent false 5017 Acked Write 1 Done Done Sent false 5018 Acked Read 4 Done Done Sent false 5019 Acked Read 6 Done Done Sent false 5020 Acked Write 10 Done Done Sent false 5021 Acked Read 3 Done Done Sent false 5022 Acked Write 9 Done Done Sent false 5023 Acked Write 10 Done Done Sent false 5024 Acked Write 5 Done Done Sent false 5025 Acked Read 6 Done Done Sent false 5026 Acked Write 8 Done Done Sent false 5027 Acked Read 3 Done Done Sent false 5028 Acked Read 1 Done Done Sent false 5029 Acked Read 2 Done Done Sent false 5030 Acked Flush 0 Done Done Sent false 5031 Acked Read 10 Done Done Sent false 5032 Acked Write 4 Done Done Sent false 5033 Acked Write 5 Done Done Sent false 5034 Acked Read 4 Done Done Sent false 5035 Acked Read 1 Done Done Sent false 5036 Acked Read 5 Done Done Sent false 5037 Acked Read 4 Done Done Sent false 5038 Acked Read 10 Done Done Sent false 5039 Acked Flush 0 Done Done Sent false 5040 Acked Read 2 Done Done Sent false 5041 Acked Read 9 Done Done Sent false 5042 Acked Write 1 Done Done Sent false 5043 Acked Read 5 Done Done Sent false 5044 Acked Read 7 Done Done Sent false 5045 Acked Read 8 Done Done Sent false 5046 Acked Read 6 Done Done Sent false 5047 Acked Write 4 Done Done Sent false 5048 Acked Write 3 Done Done Sent false 5049 Acked Flush 0 Done Done Sent false 5050 Acked Read 7 Done Done Sent false 5051 Acked Read 7 Done Done Sent false 5052 Acked Write 6 Done Done Sent false 5053 Acked Write 3 Done Done Sent false 5054 Acked Write 5 Done Done Sent false 5055 Acked Read 10 Done Done Sent false 5056 Acked Write 1 Done Done Sent false 5057 Acked Read 10 Done Done Sent false 5058 Acked Read 9 Done Done Sent false 5059 Acked Read 9 Done Done Sent false 5060 Acked Write 3 Done Done Sent false 5061 Acked Write 10 Done Done Sent false 5062 Acked Read 6 Done Done Sent false 5063 Acked Read 3 Done Done Sent false 5064 Acked Write 6 Done Done Sent false 5065 Acked Read 5 Done Done Sent false 5066 Acked Read 6 Done Done Sent false 5067 Acked Write 2 Done Done Sent false 5068 Acked Read 1 Done Done Sent false 5069 Acked Read 3 Done Done Sent false 5070 Acked Read 5 Done Done Sent false 5071 Acked Read 3 Done Done Sent false 5072 Acked Read 8 Done Done Sent false 5073 Acked Read 6 Done Done Sent false 5074 Acked Read 3 Done Done Sent false 5075 Acked Read 4 Done Done Sent false 5076 Acked Read 5 Done Done Sent false 5077 Acked Write 3 Done Done Sent false 5078 Acked Write 8 Done Done Sent false 5079 Acked Read 2 Done Done Sent false 5080 Acked Read 8 Done Done Sent false 5081 Acked Write 7 Done Done Sent false 5082 Acked Read 8 Done Done Sent false 5083 Acked Write 9 Done Done Sent false 5084 Acked Read 5 Done Done Sent false 5085 Acked Write 5 Done Done Sent false 5086 Acked Read 2 Done Done Sent false 5087 Acked Read 3 Done Done Sent false 5088 Acked Read 5 Done Done Sent false 5089 Acked Read 2 Done Done Sent false 5090 Acked Read 8 Done Done Sent false 5091 Acked Write 3 Done Done Sent false 5092 Acked Read 3 Done Done Sent false 5093 Acked Flush 0 Done Done Sent false 5094 Acked Write 2 Done Done Sent false 5095 Acked Write 4 Done Done Sent false 5096 Acked Write 1 Done Done Sent false 5097 Acked Read 3 Done Done Sent false 5098 Acked Read 1 Done Done Sent false 5099 Acked Read 3 Done Done Sent false 5100 Acked Write 9 Done Done Sent false 5101 Acked Read 9 Done Done Sent false 5102 Acked Read 4 Done Done Sent false 5103 Acked Read 5 Done Done Sent false 5104 Acked Read 2 Done Done Sent false 5105 Acked Write 1 Done Done Sent false 5106 Acked Read 8 Done Done Sent false 5107 Acked Read 9 Done Done Sent false 5108 Acked Read 10 Done Done Sent false 5109 Acked Read 9 Done Done Sent false 5110 Acked Write 9 Done Done Sent false 5111 Acked Write 2 Done Done Sent false 5112 Acked Read 7 Done Done Sent false 5113 Acked Flush 0 Done Done Sent false 5114 Acked Read 2 Done Done Sent false 5115 Acked Read 7 Done Done Sent false 5116 Acked Write 8 Done Done Sent false 5117 Acked Flush 0 Done Done Sent false 5118 Acked Write 4 Done Done Sent false 5119 Acked Read 1 Done Done Sent false 5120 Acked Read 7 Done Done Sent false 5121 Acked Write 9 Done Done Sent false 5122 Acked Read 1 Done Done Sent false 5123 Acked Flush 0 Done Done Sent false 5124 Acked Flush 0 Done Done Sent false 5125 Acked Read 3 Done Done Sent false 5126 Acked Read 7 Done Done Sent false 5127 Acked Flush 0 Done Done Sent false 5128 Acked Read 8 Done Done Sent false 5129 Acked Write 1 Done Done Sent false 5130 Acked Read 6 Done Done Sent false 5131 Acked Write 3 Done Done Sent false 5132 Acked Write 10 Done Done Sent false 5133 Acked Read 2 Done Done Sent false 5134 Acked Write 1 Done Done Sent false 5135 Acked Write 7 Done Done Sent false 5136 Acked Write 3 Done Done Sent false 5137 Acked Read 7 Done Done Sent false 5138 Acked Write 2 Done Done Sent false 5139 Acked Read 9 Done Done Sent false 5140 Acked Write 7 Done Done Sent false 5141 Acked Read 9 Done Done Sent false 5142 Acked Read 3 Done Done Sent false 5143 Acked Read 8 Done Done Sent false 5144 Acked Write 8 Done Done Sent false 5145 Acked Write 7 Done Done Sent false 5146 Acked Read 2 Done Done Sent false 5147 Acked Read 2 Done Done Sent false 5148 Acked Flush 0 Done Done Sent false 5149 Acked Write 7 Done Done Sent false 5150 Acked Read 1 Done Done Sent false 5151 Acked Write 1 Done Done Sent false 5152 Acked Read 7 Done Done Sent false 5153 Acked Read 5 Done Done Sent false 5154 Acked Flush 0 Done Done Sent false 5155 Acked Write 9 Done Done Sent false 5156 Acked Write 5 Done Done Sent false 5157 Acked Read 6 Done Done Sent false 5158 Acked Read 1 Done Done Sent false 5159 Acked Read 10 Done Done Sent false 5160 Acked Read 4 Done Done Sent false 5161 Acked Write 4 Done Done Sent false 5162 Acked Read 9 Done Done Sent false 5163 Acked Read 7 Done Done Sent false 5164 Acked Read 3 Done Done Sent false 5165 Acked Write 9 Done Done Sent false 5166 Acked Write 4 Done Done Sent false 5167 Acked Read 2 Done Done Sent false 5168 Acked Read 5 Done Done Sent false 5169 Acked Read 3 Done Done Sent false 5170 Acked Read 2 Done Done Sent false 5171 Acked Read 5 Done Done Sent false 5172 Acked Write 6 Done Done Sent false 5173 Acked Read 10 Done Done Sent false 5174 Acked Read 10 Done Done Sent false 5175 Acked Write 1 Done Done Sent false 5176 Acked Read 3 Done Done Sent false 5177 Acked Flush 0 Done Done Sent false 5178 Acked Read 4 Done Done Sent false 5179 Acked Read 9 Done Done Sent false 5180 Acked Read 7 Done Done Sent false 5181 Acked Flush 0 Done Done Sent false 5182 Acked Read 2 Done Done Sent false 5183 Acked Write 9 Done Done Sent false 5184 Acked Write 3 Done Done Sent false 5185 Acked Flush 0 Done Done Sent false 5186 Acked Write 8 Done Done Sent false 5187 Acked Read 7 Done Done Sent false 5188 Acked Read 2 Done Done Sent false 5189 Acked Read 2 Done Done Sent false 5190 Acked Read 7 Done Done Sent false 5191 Acked Read 5 Done Done Sent false 5192 Acked Read 10 Done Done Sent false 5193 Acked Write 4 Done Done Sent false 5194 Acked Write 3 Done Done Sent false 5195 Acked Read 4 Done Done Sent false 5196 Acked Read 8 Done Done Sent false 5197 Acked Read 8 Done Done Sent false 5198 Acked Write 5 Done Done Sent false 5199 Acked Read 10 Done Done Sent false 5200 Acked Read 8 Done Done Sent false 5201 Acked Read 8 Done Done Sent false 5202 Acked Read 1 Done Done Sent false 5203 Acked Read 3 Done Done Sent false 5204 Acked Read 2 Done Done Sent false 5205 Acked Write 4 Done Done Sent false 5206 Acked Read 9 Done Done Sent false 5207 Acked Write 10 Done Done Sent false 5208 Acked Write 2 Done Done Sent false 5209 Acked Write 9 Done Done Sent false 5210 Acked Write 7 Done Done Sent false 5211 Acked Flush 0 Done Done Sent false 5212 Acked Read 4 Done Done Sent false 5213 Acked Read 6 Done Done Sent false 5214 Acked Write 2 Done Done Sent false 5215 Acked Read 2 Done Done Sent false 5216 Acked Write 9 Done Done Sent false 5217 Acked Write 6 Done Done Sent false 5218 Acked Read 7 Done Done Sent false 5219 Acked Write 4 Done Done Sent false 5220 Acked Read 2 Done Done Sent false 5221 Acked Read 8 Done Done Sent false 5222 Acked Write 5 Done Done Sent false 5223 Acked Read 1 Done Done Sent false 5224 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5224 5224 4922 Downstairs last five completed: 4922 4921 4920 4919 4918 Upstairs last five completed: 5224 5223 5222 5221 5220 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.85508249Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:36371)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.855147802Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:16.856052199Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.85660757Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856628824Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 4922","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856642791Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 4922","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856891174Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856910369Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856924365Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 4922","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.856938881Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 4922","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:16.85716663Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5224 5223 5222 5221 5220 Upstairs last five completed: 5224 5223 5222 5221 5220 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Nov 27 01:45:19.106 INFO request completed, latency_us: 341, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d5ef7747-72ed-4e4e-8be1-76bfeefdf98f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:19.107597264Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:45:19.107 INFO request completed, latency_us: 105, response_code: 200, uri: /state/cid/2, method: GET, req_id: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:19.107665796Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 85c99c03-8a43-40d6-98cf-209401d4a864, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:19.107714193Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:19.107744754Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:19.107780205Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:19.10779989Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:45:23.109 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7d534399-60e8-4d63-a0cd-05fa64b213e1, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 1409 len 32768 data: 5 4 4 4 4 4 4 4 002/300 Write block 1686 len 20480 data: 4 4 4 4 4 003/300 Write block 2513 len 32768 data: 2 2 2 2 3 4 4 4 004/300 Flush 005/300 Write block 4646 len 32768 data: 3 3 2 2 2 2 2 2 006/300 Read block 4308 len 8192 007/300 Write block 908 len 28672 data: 3 3 2 2 2 2 2 008/300 Read block 2808 len 16384 009/300 Read block 3274 len 12288 010/300 Read block 173 len 4096 011/300 Read block 4921 len 12288 012/300 Read block 1026 len 12288 013/300 Flush 014/300 Write block 2302 len 8192 data: 3 4 015/300 Read block 537 len 32768 016/300 Write block 2747 len 4096 data: 3 017/300 Write block 3795 len 4096 data: 3 018/300 Write block 1598 len 8192 data: 4 5 019/300 Write block 2065 len 8192 data: 3 3 020/300 Write block 4296 len 16384 data: 2 2 2 3 021/300 Flush 022/300 Write block 2732 len 28672 data: 3 3 2 3 3 3 4 023/300 Read block 1814 len 36864 024/300 Read block 2570 len 40960 025/300 Write block 1543 len 8192 data: 8 6 026/300 Flush 027/300 Write block 1675 len 32768 data: 3 3 2 3 3 3 4 4 028/300 Read block 4037 len 12288 029/300 Read block 4055 len 20480 030/300 Write block 521 len 4096 data: 3 031/300 Flush 032/300 Write block 3448 len 4096 data: 3 033/300 Read block 73 len 12288 034/300 Write block 2874 len 16384 data: 2 2 2 2 035/300 Read block 296 len 36864 036/300 Write block 1415 len 40960 data: 5 5 4 3 4 4 4 4 5 5 037/300 Read block 4821 len 16384 038/300 Write block 3798 len 20480 data: 2 3 3 3 3 039/300 Read block 1775 len 12288 040/300 Write block 2692 len 36864 data: 4 4 4 5 5 4 4 4 4 041/300 Read block 628 len 16384 042/300 Read block 4306 len 36864 043/300 Read block 4895 len 8192 044/300 Write block 1700 len 20480 data: 5 4 3 3 3 045/300 Read block 2093 len 12288 046/300 Read block 2009 len 28672 047/300 Flush 048/300 Read block 4583 len 8192 049/300 Read block 2274 len 40960 050/300 Write block 3938 len 28672 data: 4 3 3 3 3 2 2 051/300 Read block 1066 len 40960 052/300 Read block 1656 len 28672 053/300 Read block 1688 len 12288 054/300 Write block 4920 len 32768 data: 2 3 3 3 3 3 3 3 055/300 Write block 4795 len 40960 data: 6 6 7 8 7 7 6 5 5 5 056/300 Read block 2045 len 40960 057/300 Write block 1036 len 28672 data: 3 4 5 3 5 5 4 058/300 Write block 3396 len 20480 data: 4 4 3 3 3 059/300 Read block 943 len 28672 060/300 Read block 971 len 20480 061/300 Read block 1097 len 40960 062/300 Write block 4124 len 40960 data: 4 4 6 6 7 7 6 4 4 4 063/300 Read block 772 len 40960 064/300 Flush 065/300 Read block 2828 len 8192 066/300 Write block 2932 len 24576 data: 2 3 3 3 4 4 067/300 Write block 149 len 36864 data: 4 3 3 2 4 4 4 4 4 068/300 Read block 1631 len 16384 069/300 Read block 2244 len 20480 070/300 Write block 2949 len 4096 data: 2 071/300 Write block 2471 len 4096 data: 6 072/300 Read block 4689 len 24576 073/300 Read block 4110 len 16384 074/300 Write block 976 len 32768 data: 4 3 3 2 2 3 3 3 075/300 Write block 898 len 24576 data: 5 5 4 4 4 5 076/300 Read block 1240 len 28672 077/300 Write block 1203 len 32768 data: 4 4 4 3 4 3 4 4 078/300 Write block 3679 len 32768 data: 5 6 6 6 6 4 5 6 079/300 Flush 080/300 Write block 551 len 16384 data: 3 3 2 2 081/300 Write block 1928 len 4096 data: 4 082/300 Write block 4389 len 12288 data: 6 5 3 083/300 Read block 3324 len 12288 084/300 Write block 2457 len 40960 data: 7 6 7 7 6 4 4 4 4 4 085/300 Write block 3803 len 12288 data: 4 5 5 086/300 Write block 2368 len 40960 data: 3 3 5 5 6 8 8 9 7 7 087/300 Write block 1507 len 16384 data: 3 4 4 3 088/300 Write block 1736 len 20480 data: 3 3 3 4 4 089/300 Read block 3994 len 12288 090/300 Read block 2744 len 16384 091/300 Write block 2146 len 8192 data: 3 3 092/300 Flush 093/300 Write block 3625 len 16384 data: 2 3 3 3 094/300 Write block 3360 len 24576 data: 5 6 6 5 4 3 095/300 Read block 4734 len 4096 096/300 Write block 777 len 32768 data: 3 3 3 3 2 3 4 3 097/300 Read block 4670 len 40960 098/300 Write block 1303 len 8192 data: 2 2 099/300 Read block 2412 len 12288 100/300 Read block 1871 len 36864 101/300 Write block 3794 len 20480 data: 3 4 3 2 3 102/300 Write block 3809 len 32768 data: 6 5 5 5 5 5 5 2 103/300 Read block 4062 len 12288 104/300 Write block 3531 len 4096 data: 3 105/300 Write block 2823 len 20480 data: 3 3 4 4 3 106/300 Read block 2201 len 12288 107/300 Flush 108/300 Flush 109/300 Write block 2187 len 36864 data: 3 3 2 2 3 3 3 3 3 110/300 Read block 2495 len 32768 111/300 Read block 2890 len 16384 112/300 Read block 2826 len 8192 113/300 Write block 1524 len 28672 data: 3 4 3 4 4 4 4 114/300 Write block 2822 len 20480 data: 3 4 4 5 5 115/300 Read block 1550 len 36864 116/300 Write block 4642 len 36864 data: 3 3 3 3 4 4 3 3 3 117/300 Read block 4144 len 16384 118/300 Read block 76 len 28672 119/300 Write block 4619 len 36864 data: 2 2 3 3 3 3 3 4 4 120/300 Flush 121/300 Write block 3603 len 16384 data: 4 4 3 2 122/300 Write block 3196 len 12288 data: 4 6 6 123/300 Write block 1673 len 12288 data: 3 3 4 124/300 Read block 2164 len 20480 125/300 Write block 2760 len 28672 data: 2 3 4 5 5 5 4 126/300 Flush 127/300 Write block 4061 len 12288 data: 2 3 4 128/300 Read block 1561 len 12288 129/300 Read block 2555 len 12288 130/300 Read block 4588 len 36864 131/300 Read block 3660 len 28672 132/300 Write block 3000 len 40960 data: 2 4 4 4 3 4 3 3 3 3 133/300 Read block 4769 len 8192 134/300 Write block 4539 len 36864 data: 3 3 4 2 2 2 2 2 2 135/300 Flush 136/300 Write block 4890 len 24576 data: 3 2 2 2 2 2 137/300 Write block 1291 len 36864 data: 3 5 5 5 5 6 6 5 5 138/300 Flush 139/300 Read block 574 len 8192 140/300 Write block 2528 len 28672 data: 6 6 4 2 2 3 3 141/300 Write block 3515 len 32768 data: 3 3 4 4 3 4 4 2 142/300 Read block 3720 len 32768 143/300 Read block 1012 len 16384 144/300 Read block 3808 len 8192 145/300 Write block 1587 len 16384 data: 4 5 4 4 146/300 Write block 2163 len 20480 data: 4 4 3 2 2 147/300 Read block 2246 len 28672 148/300 Write block 3329 len 20480 data: 5 4 4 6 5 149/300 Read block 4462 len 40960 150/300 Write block 110 len 40960 data: 5 5 5 4 4 4 5 5 4 5 151/300 Write block 3087 len 20480 data: 6 6 6 3 3 152/300 Write block 4265 len 12288 data: 4 3 3 153/300 Read block 653 len 12288 154/300 Write block 2287 len 4096 data: 3 155/300 Read block 894 len 12288 156/300 Flush 157/300 Read block 512 len 12288 158/300 Write block 1953 len 28672 data: 5 5 4 3 5 6 6 159/300 Write block 2732 len 40960 data: 4 4 3 4 4 4 5 5 6 5 160/300 Read block 1142 len 24576 161/300 Write block 2587 len 24576 data: 4 4 4 4 3 2 162/300 Read block 1350 len 28672 163/300 Write block 4840 len 32768 data: 3 4 5 4 5 5 6 4 164/300 Read block 167 len 28672 165/300 Write block 1222 len 20480 data: 7 7 6 5 5 166/300 Write block 3086 len 36864 data: 7 7 7 7 4 4 3 3 3 167/300 Read block 4643 len 4096 168/300 Flush 169/300 Read block 2959 len 40960 170/300 Write block 4130 len 16384 data: 7 5 5 5 171/300 Read block 1338 len 12288 172/300 Read block 2215 len 24576 173/300 Flush 174/300 Write block 2195 len 24576 data: 4 3 3 3 3 4 175/300 Write block 426 len 8192 data: 4 2 176/300 Write block 2902 len 40960 data: 5 4 3 3 3 3 2 2 3 3 177/300 Write block 4079 len 8192 data: 4 3 178/300 Read block 4075 len 32768 179/300 Read block 29 len 12288 180/300 Write block 4000 len 12288 data: 3 3 4 181/300 Read block 2419 len 4096 182/300 Read block 728 len 16384 183/300 Write block 3676 len 20480 data: 4 5 5 6 7 184/300 Read block 2829 len 40960 185/300 Flush 186/300 Write block 2268 len 36864 data: 5 5 5 5 3 4 4 4 4 187/300 Read block 4114 len 4096 188/300 Write block 1108 len 36864 data: 7 7 6 6 4 4 4 4 4 189/300 Write block 1343 len 24576 data: 3 3 3 3 3 2 190/300 Write block 2225 len 16384 data: 4 4 4 4 191/300 Flush 192/300 Write block 4512 len 36864 data: 4 4 4 4 4 4 4 4 4 193/300 Read block 3369 len 32768 194/300 Write block 2283 len 28672 data: 3 3 4 4 4 3 4 195/300 Write block 3328 len 16384 data: 5 6 5 5 196/300 Read block 1300 len 12288 197/300 Read block 2598 len 32768 198/300 Write block 52 len 12288 data: 3 3 3 199/300 Read block 4265 len 16384 200/300 Read block 1819 len 12288 201/300 Write block 4796 len 20480 data: 7 8 9 8 8 202/300 Read block 461 len 4096 203/300 Write block 4362 len 36864 data: 5 5 5 3 3 3 3 3 2 204/300 Write block 2385 len 36864 data: 6 6 6 7 7 6 5 4 4 205/300 Read block 2753 len 12288 206/300 Read block 3463 len 32768 207/300 Write block 1061 len 20480 data: 2 2 2 2 2 208/300 Flush 209/300 Read block 2350 len 24576 210/300 Read block 2911 len 16384 211/300 Read block 4114 len 20480 212/300 Read block 860 len 32768 213/300 Read block 108 len 40960 214/300 Write block 3472 len 24576 data: 3 3 3 2 3 3 215/300 Write block 2917 len 32768 data: 3 5 4 5 5 6 7 7 216/300 Write block 914 len 12288 data: 3 2 2 217/300 Write block 2758 len 12288 data: 2 2 3 218/300 Write block 2359 len 32768 data: 3 3 4 5 4 4 4 4 219/300 Read block 514 len 20480 220/300 Read block 4470 len 40960 221/300 Read block 1223 len 8192 222/300 Write block 4684 len 4096 data: 3 223/300 Read block 336 len 4096 224/300 Read block 3569 len 40960 225/300 Write block 4425 len 16384 data: 7 5 4 5 226/300 Write block 1680 len 28672 data: 4 5 5 5 4 4 5 227/300 Write block 3475 len 32768 data: 3 4 4 3 3 3 3 3 228/300 Read block 3058 len 40960 229/300 Write block 1850 len 36864 data: 4 4 5 4 5 4 4 4 4 230/300 Flush 231/300 Write block 2847 len 8192 data: 3 3 232/300 Read block 4825 len 20480 233/300 Read block 3998 len 4096 234/300 Write block 1122 len 20480 data: 2 3 2 2 2 235/300 Read block 2091 len 28672 236/300 Read block 3340 len 32768 237/300 Write block 1235 len 4096 data: 4 238/300 Read block 489 len 32768 239/300 Write block 1905 len 12288 data: 5 5 6 240/300 Read block 3913 len 16384 241/300 Write block 3617 len 4096 data: 4 242/300 Write block 4325 len 20480 data: 2 2 3 3 4 243/300 Write block 4287 len 28672 data: 4 4 4 4 4 4 2 244/300 Write block 4153 len 8192 data: 6 7 245/300 Write block 201 len 40960 data: 2 3 3 3 3 5 8 9 8 7 246/300 Read block 2291 len 32768 247/300 Read block 2886 len 40960 248/300 Read block 700 len 36864 249/300 Read block 4201 len 16384 250/300 Write block 47 len 36864 data: 2 2 3 3 3 4 4 4 2 251/300 Read block 3649 len 32768 252/300 Read block 2970 len 20480 253/300 Write block 2838 len 28672 data: 3 4 4 4 6 6 4 254/300 Write block 2036 len 16384 data: 3 2 3 5 255/300 Write block 3786 len 40960 data: 4 3 3 3 2 2 2 3 4 5 256/300 Flush 257/300 Write block 4763 len 4096 data: 5 258/300 Read block 4959 len 36864 259/300 Read block 2934 len 32768 260/300 Read block 2843 len 16384 261/300 Write block 829 len 24576 data: 9 8 7 6 5 5 262/300 Flush 263/300 Write block 2190 len 12288 data: 3 4 4 264/300 Read block 3322 len 40960 265/300 Read block 2523 len 32768 266/300 Write block 4626 len 24576 data: 5 5 3 3 3 2 267/300 Write block 2142 len 12288 data: 5 5 6 268/300 Read block 2550 len 4096 269/300 Read block 935 len 32768 270/300 Read block 1886 len 36864 271/300 Flush 272/300 Read block 1155 len 32768 273/300 Write block 1651 len 16384 data: 5 5 3 3 274/300 Read block 4654 len 12288 275/300 Flush 276/300 Read block 3080 len 16384 277/300 Write block 1558 len 24576 data: 5 6 5 5 4 4 278/300 Flush 279/300 Write block 4529 len 20480 data: 3 3 3 3 3 280/300 Write block 2463 len 32768 data: 5 5 5 5 5 5 5 5 281/300 Flush 282/300 Read block 3757 len 12288 283/300 Read block 56 len 28672 284/300 Write block 3584 len 28672 data: 3 3 2 2 2 3 4 285/300 Read block 2024 len 32768 286/300 Write block 2977 len 16384 data: 4 4 4 4 287/300 Write block 1277 len 4096 data: 3 288/300 Write block 2505 len 8192 data: 5 5 289/300 Read block 2953 len 8192 290/300 Read block 3172 len 32768 291/300 Read block 2006 len 24576 292/300 Read block 428 len 28672 293/300 Write block 1245 len 16384 data: 3 4 5 6 294/300 Flush 295/300 Read block 4734 len 28672 296/300 Read block 4340 len 36864 297/300 Write block 1914 len 8192 data: 6 5 298/300 Read block 2201 len 28672 299/300 Read block 3002 len 28672 300/300 Write block 1656 len 36864 data: 2 2 2 2 2 2 2 3 3 301/300 Read block 655 len 12288 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:45:23.337 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: 83434b3e-16b2-42b0-871d-f14b2f0af316, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5225 Acked Write 8 Done Done Sent false 5226 Acked Write 5 Done Done Sent false 5227 Acked Write 8 Done Done Sent false 5228 Acked Flush 0 Done Done Sent false 5229 Acked Write 8 Done Done Sent false 5230 Acked Read 2 Done Done Sent false 5231 Acked Write 7 Done Done Sent false 5232 Acked Read 4 Done Done Sent false 5233 Acked Read 3 Done Done Sent false 5234 Acked Read 1 Done Done Sent false 5235 Acked Read 3 Done Done Sent false 5236 Acked Read 3 Done Done Sent false 5237 Acked Flush 0 Done Done Sent false 5238 Acked Write 2 Done Done Sent false 5239 Acked Read 8 Done Done Sent false 5240 Acked Write 1 Done Done Sent false 5241 Acked Write 1 Done Done Sent false 5242 Acked Write 2 Done Done Sent false 5243 Acked Write 2 Done Done Sent false 5244 Acked Write 4 Done Done Sent false 5245 Acked Flush 0 Done Done Sent false 5246 Acked Write 7 Done Done Sent false 5247 Acked Read 9 Done Done Sent false 5248 Acked Read 10 Done Done Sent false 5249 Acked Write 2 Done Done Sent false 5250 Acked Flush 0 Done Done Sent false 5251 Acked Write 8 Done Done Sent false 5252 Acked Read 3 Done Done Sent false 5253 Acked Read 5 Done Done Sent false 5254 Acked Write 1 Done Done Sent false 5255 Acked Flush 0 Done Done Sent false 5256 Acked Write 1 Done Done Sent false 5257 Acked Read 3 Done Done Sent false 5258 Acked Write 4 Done Done Sent false 5259 Acked Read 9 Done Done Sent false 5260 Acked Write 10 Done Done Sent false 5261 Acked Read 4 Done Done Sent false 5262 Acked Write 5 Done Done Sent false 5263 Acked Read 3 Done Done Sent false 5264 Acked Write 9 Done Done Sent false 5265 Acked Read 4 Done Done Sent false 5266 Acked Read 9 Done Done Sent false 5267 Acked Read 2 Done Done Sent false 5268 Acked Write 5 Done Done Sent false 5269 Acked Read 3 Done Done Sent false 5270 Acked Read 7 Done Done Sent false 5271 Acked Flush 0 Done Done Sent false 5272 Acked Read 2 Done Done Sent false 5273 Acked Read 10 Done Done Sent false 5274 Acked Write 7 Done Done Sent false 5275 Acked Read 10 Done Done Sent false 5276 Acked Read 7 Done Done Sent false 5277 Acked Read 3 Done Done Sent false 5278 Acked Write 8 Done Done Sent false 5279 Acked Write 10 Done Done Sent false 5280 Acked Read 10 Done Done Sent false 5281 Acked Write 7 Done Done Sent false 5282 Acked Write 5 Done Done Sent false 5283 Acked Read 7 Done Done Sent false 5284 Acked Read 5 Done Done Sent false 5285 Acked Read 10 Done Done Sent false 5286 Acked Write 10 Done Done Sent false 5287 Acked Read 10 Done Done Sent false 5288 Acked Flush 0 Done Done Sent false 5289 Acked Read 2 Done Done Sent false 5290 Acked Write 6 Done Done Sent false 5291 Acked Write 9 Done Done Sent false 5292 Acked Read 4 Done Done Sent false 5293 Acked Read 5 Done Done Sent false 5294 Acked Write 1 Done Done Sent false 5295 Acked Write 1 Done Done Sent false 5296 Acked Read 6 Done Done Sent false 5297 Acked Read 4 Done Done Sent false 5298 Acked Write 8 Done Done Sent false 5299 Acked Write 6 Done Done Sent false 5300 Acked Read 7 Done Done Sent false 5301 Acked Write 8 Done Done Sent false 5302 Acked Write 8 Done Done Sent false 5303 Acked Flush 0 Done Done Sent false 5304 Acked Write 4 Done Done Sent false 5305 Acked Write 1 Done Done Sent false 5306 Acked Write 3 Done Done Sent false 5307 Acked Read 3 Done Done Sent false 5308 Acked Write 10 Done Done Sent false 5309 Acked Write 3 Done Done Sent false 5310 Acked Write 10 Done Done Sent false 5311 Acked Write 4 Done Done Sent false 5312 Acked Write 5 Done Done Sent false 5313 Acked Read 3 Done Done Sent false 5314 Acked Read 4 Done Done Sent false 5315 Acked Write 2 Done Done Sent false 5316 Acked Flush 0 Done Done Sent false 5317 Acked Write 4 Done Done Sent false 5318 Acked Write 6 Done Done Sent false 5319 Acked Read 1 Done Done Sent false 5320 Acked Write 8 Done Done Sent false 5321 Acked Read 10 Done Done Sent false 5322 Acked Write 2 Done Done Sent false 5323 Acked Read 3 Done Done Sent false 5324 Acked Read 9 Done Done Sent false 5325 Acked Write 5 Done Done Sent false 5326 Acked Write 8 Done Done Sent false 5327 Acked Read 3 Done Done Sent false 5328 Acked Write 1 Done Done Sent false 5329 Acked Write 5 Done Done Sent false 5330 Acked Read 3 Done Done Sent false 5331 Acked Flush 0 Done Done Sent false 5332 Acked Flush 0 Done Done Sent false 5333 Acked Write 9 Done Done Sent false 5334 Acked Read 8 Done Done Sent false 5335 Acked Read 4 Done Done Sent false 5336 Acked Read 2 Done Done Sent false 5337 Acked Write 7 Done Done Sent false 5338 Acked Write 5 Done Done Sent false 5339 Acked Read 9 Done Done Sent false 5340 Acked Write 9 Done Done Sent false 5341 Acked Read 4 Done Done Sent false 5342 Acked Read 7 Done Done Sent false 5343 Acked Write 9 Done Done Sent false 5344 Acked Flush 0 Done Done Sent false 5345 Acked Write 4 Done Done Sent false 5346 Acked Write 3 Done Done Sent false 5347 Acked Write 3 Done Done Sent false 5348 Acked Read 5 Done Done Sent false 5349 Acked Write 7 Done Done Sent false 5350 Acked Flush 0 Done Done Sent false 5351 Acked Write 3 Done Done Sent false 5352 Acked Read 3 Done Done Sent false 5353 Acked Read 3 Done Done Sent false 5354 Acked Read 9 Done Done Sent false 5355 Acked Read 7 Done Done Sent false 5356 Acked Write 10 Done Done Sent false 5357 Acked Read 2 Done Done Sent false 5358 Acked Write 9 Done Done Sent false 5359 Acked Flush 0 Done Done Sent false 5360 Acked Write 6 Done Done Sent false 5361 Acked Write 9 Done Done Sent false 5362 Acked Flush 0 Done Done Sent false 5363 Acked Read 2 Done Done Sent false 5364 Acked Write 7 Done Done Sent false 5365 Acked Write 8 Done Done Sent false 5366 Acked Read 8 Done Done Sent false 5367 Acked Read 4 Done Done Sent false 5368 Acked Read 2 Done Done Sent false 5369 Acked Write 4 Done Done Sent false 5370 Acked Write 5 Done Done Sent false 5371 Acked Read 7 Done Done Sent false 5372 Acked Write 5 Done Done Sent false 5373 Acked Read 10 Done Done Sent false 5374 Acked Write 10 Done Done Sent false 5375 Acked Write 5 Done Done Sent false 5376 Acked Write 3 Done Done Sent false 5377 Acked Read 3 Done Done Sent false 5378 Acked Write 1 Done Done Sent false 5379 Acked Read 3 Done Done Sent false 5380 Acked Flush 0 Done Done Sent false 5381 Acked Read 3 Done Done Sent false 5382 Acked Write 7 Done Done Sent false 5383 Acked Write 10 Done Done Sent false 5384 Acked Read 6 Done Done Sent false 5385 Acked Write 6 Done Done Sent false 5386 Acked Read 7 Done Done Sent false 5387 Acked Write 8 Done Done Sent false 5388 Acked Read 7 Done Done Sent false 5389 Acked Write 5 Done Done Sent false 5390 Acked Write 9 Done Done Sent false 5391 Acked Read 1 Done Done Sent false 5392 Acked Flush 0 Done Done Sent false 5393 Acked Read 10 Done Done Sent false 5394 Acked Write 4 Done Done Sent false 5395 Acked Read 3 Done Done Sent false 5396 Acked Read 6 Done Done Sent false 5397 Acked Flush 0 Done Done Sent false 5398 Acked Write 6 Done Done Sent false 5399 Acked Write 2 Done Done Sent false 5400 Acked Write 10 Done Done Sent false 5401 Acked Write 2 Done Done Sent false 5402 Acked Read 8 Done Done Sent false 5403 Acked Read 3 Done Done Sent false 5404 Acked Write 3 Done Done Sent false 5405 Acked Read 1 Done Done Sent false 5406 Acked Read 4 Done Done Sent false 5407 Acked Write 5 Done Done Sent false 5408 Acked Read 10 Done Done Sent false 5409 Acked Flush 0 Done Done Sent false 5410 Acked Write 9 Done Done Sent false 5411 Acked Read 1 Done Done Sent false 5412 Acked Write 9 Done Done Sent false 5413 Acked Write 6 Done Done Sent false 5414 Acked Write 4 Done Done Sent false 5415 Acked Flush 0 Done Done Sent false 5416 Acked Write 9 Done Done Sent false 5417 Acked Read 8 Done Done Sent false 5418 Acked Write 7 Done Done Sent false 5419 Acked Write 4 Done Done Sent false 5420 Acked Read 3 Done Done Sent false 5421 Acked Read 8 Done Done Sent false 5422 Acked Write 3 Done Done Sent false 5423 Acked Read 4 Done Done Sent false 5424 Acked Read 3 Done Done Sent false 5425 Acked Write 5 Done Done Sent false 5426 Acked Read 1 Done Done Sent false 5427 Acked Write 9 Done Done Sent false 5428 Acked Write 9 Done Done Sent false 5429 Acked Read 3 Done Done Sent false 5430 Acked Read 8 Done Done Sent false 5431 Acked Write 5 Done Done Sent false 5432 Acked Flush 0 Done Done Sent false 5433 Acked Read 6 Done Done Sent false 5434 Acked Read 4 Done Done Sent false 5435 Acked Read 5 Done Done Sent false 5436 Acked Read 8 Done Done Sent false 5437 Acked Read 10 Done Done Sent false 5438 Acked Write 6 Done Done Sent false 5439 Acked Write 8 Done Done Sent false 5440 Acked Write 3 Done Done Sent false 5441 Acked Write 3 Done Done Sent false 5442 Acked Write 8 Done Done Sent false 5443 Acked Read 5 Done Done Sent false 5444 Acked Read 10 Done Done Sent false 5445 Acked Read 2 Done Done Sent false 5446 Acked Write 1 Done Done Sent false 5447 Acked Read 1 Done Done Sent false 5448 Acked Read 10 Done Done Sent false 5449 Acked Write 4 Done Done Sent false 5450 Acked Write 7 Done Done Sent false 5451 Acked Write 8 Done Done Sent false 5452 Acked Read 10 Done Done Sent false 5453 Acked Write 9 Done Done Sent false 5454 Acked Flush 0 Done Done Sent false 5455 Acked Write 2 Done Done Sent false 5456 Acked Read 5 Done Done Sent false 5457 Acked Read 1 Done Done Sent false 5458 Acked Write 5 Done Done Sent false 5459 Acked Read 7 Done Done Sent false 5460 Acked Read 8 Done Done Sent false 5461 Acked Write 1 Done Done Sent false 5462 Acked Read 8 Done Done Sent false 5463 Acked Write 3 Done Done Sent false 5464 Acked Read 4 Done Done Sent false 5465 Acked Write 1 Done Done Sent false 5466 Acked Write 5 Done Done Sent false 5467 Acked Write 7 Done Done Sent false 5468 Acked Write 2 Done Done Sent false 5469 Acked Write 10 Done Done Sent false 5470 Acked Read 8 Done Done Sent false 5471 Acked Read 10 Done Done Sent false 5472 Acked Read 9 Done Done Sent false 5473 Acked Read 4 Done Done Sent false 5474 Acked Write 9 Done Done Sent false 5475 Acked Read 8 Done Done Sent false 5476 Acked Read 5 Done Done Sent false 5477 Acked Write 7 Done Done Sent false 5478 Acked Write 4 Done Done Sent false 5479 Acked Write 10 Done Done Sent false 5480 Acked Flush 0 Done Done Sent false 5481 Acked Write 1 Done Done Sent false 5482 Acked Read 9 Done Done Sent false 5483 Acked Read 8 Done Done Sent false 5484 Acked Read 4 Done Done Sent false 5485 Acked Write 6 Done Done Sent false 5486 Acked Flush 0 Done Done Sent false 5487 Acked Write 3 Done Done Sent false 5488 Acked Read 10 Done Done Sent false 5489 Acked Read 8 Done Done Sent false 5490 Acked Write 6 Done Done Sent false 5491 Acked Write 3 Done Done Sent false 5492 Acked Read 1 Done Done Sent false 5493 Acked Read 8 Done Done Sent false 5494 Acked Read 9 Done Done Sent false 5495 Acked Flush 0 Done Done Sent false 5496 Acked Read 8 Done Done Sent false 5497 Acked Write 4 Done Done Sent false 5498 Acked Read 3 Done Done Sent false 5499 Acked Flush 0 Done Done Sent false 5500 Acked Read 4 Done Done Sent false 5501 Acked Write 6 Done Done Sent false 5502 Acked Flush 0 Done Done Sent false 5503 Acked Write 5 Done Done Sent false 5504 Acked Write 8 Done Done Sent false 5505 Acked Flush 0 Done Done Sent false 5506 Acked Read 3 Done Done Sent false 5507 Acked Read 7 Done Done Sent false 5508 Acked Write 7 Done Done Sent false 5509 Acked Read 8 Done Done Sent false 5510 Acked Write 4 Done Done Sent false 5511 Acked Write 1 Done Done Sent false 5512 Acked Write 2 Done Done Sent false 5513 Acked Read 2 Done Done Sent false 5514 Acked Read 8 Done Done Sent false 5515 Acked Read 6 Done Done Sent false 5516 Acked Read 7 Done Done Sent false 5517 Acked Write 4 Done Done Sent false 5518 Acked Flush 0 Done Done Sent false 5519 Acked Read 7 Done Done Sent false 5520 Acked Read 9 Done Done Sent false 5521 Acked Write 2 Done Done Sent false 5522 Acked Read 7 Done Done Sent false 5523 Acked Read 7 Done Done Sent false 5524 Acked Write 9 Done Done Sent false 5525 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5518 5518 5224 Downstairs last five completed: 5224 5223 5222 5221 5220 Upstairs last five completed: 5525 5524 5523 5522 5521 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5225 Acked Write 8 Done Done Sent false 5226 Acked Write 5 Done Done Sent false 5227 Acked Write 8 Done Done Sent false 5228 Acked Flush 0 Done Done Sent false 5229 Acked Write 8 Done Done Sent false 5230 Acked Read 2 Done Done Sent false 5231 Acked Write 7 Done Done Sent false 5232 Acked Read 4 Done Done Sent false 5233 Acked Read 3 Done Done Sent false 5234 Acked Read 1 Done Done Sent false 5235 Acked Read 3 Done Done Sent false 5236 Acked Read 3 Done Done Sent false 5237 Acked Flush 0 Done Done Sent false 5238 Acked Write 2 Done Done Sent false 5239 Acked Read 8 Done Done Sent false 5240 Acked Write 1 Done Done Sent false 5241 Acked Write 1 Done Done Sent false 5242 Acked Write 2 Done Done Sent false 5243 Acked Write 2 Done Done Sent false 5244 Acked Write 4 Done Done Sent false 5245 Acked Flush 0 Done Done Sent false 5246 Acked Write 7 Done Done Sent false 5247 Acked Read 9 Done Done Sent false 5248 Acked Read 10 Done Done Sent false 5249 Acked Write 2 Done Done Sent false 5250 Acked Flush 0 Done Done Sent false 5251 Acked Write 8 Done Done Sent false 5252 Acked Read 3 Done Done Sent false 5253 Acked Read 5 Done Done Sent false 5254 Acked Write 1 Done Done Sent false 5255 Acked Flush 0 Done Done Sent false 5256 Acked Write 1 Done Done Sent false 5257 Acked Read 3 Done Done Sent false 5258 Acked Write 4 Done Done Sent false 5259 Acked Read 9 Done Done Sent false 5260 Acked Write 10 Done Done Sent false 5261 Acked Read 4 Done Done Sent false 5262 Acked Write 5 Done Done Sent false 5263 Acked Read 3 Done Done Sent false 5264 Acked Write 9 Done Done Sent false 5265 Acked Read 4 Done Done Sent false 5266 Acked Read 9 Done Done Sent false 5267 Acked Read 2 Done Done Sent false 5268 Acked Write 5 Done Done Sent false 5269 Acked Read 3 Done Done Sent false 5270 Acked Read 7 Done Done Sent false 5271 Acked Flush 0 Done Done Sent false 5272 Acked Read 2 Done Done Sent false 5273 Acked Read 10 Done Done Sent false 5274 Acked Write 7 Done Done Sent false 5275 Acked Read 10 Done Done Sent false 5276 Acked Read 7 Done Done Sent false 5277 Acked Read 3 Done Done Sent false 5278 Acked Write 8 Done Done Sent false 5279 Acked Write 10 Done Done Sent false 5280 Acked Read 10 Done Done Sent false 5281 Acked Write 7 Done Done Sent false 5282 Acked Write 5 Done Done Sent false 5283 Acked Read 7 Done Done Sent false 5284 Acked Read 5 Done Done Sent false 5285 Acked Read 10 Done Done Sent false 5286 Acked Write 10 Done Done Sent false 5287 Acked Read 10 Done Done Sent false 5288 Acked Flush 0 Done Done Sent false 5289 Acked Read 2 Done Done Sent false 5290 Acked Write 6 Done Done Sent false 5291 Acked Write 9 Done Done Sent false 5292 Acked Read 4 Done Done Sent false 5293 Acked Read 5 Done Done Sent false 5294 Acked Write 1 Done Done Sent false 5295 Acked Write 1 Done Done Sent false 5296 Acked Read 6 Done Done Sent false 5297 Acked Read 4 Done Done Sent false 5298 Acked Write 8 Done Done Sent false 5299 Acked Write 6 Done Done Sent false 5300 Acked Read 7 Done Done Sent false 5301 Acked Write 8 Done Done Sent false 5302 Acked Write 8 Done Done Sent false 5303 Acked Flush 0 Done Done Sent false 5304 Acked Write 4 Done Done Sent false 5305 Acked Write 1 Done Done Sent false 5306 Acked Write 3 Done Done Sent false 5307 Acked Read 3 Done Done Sent false 5308 Acked Write 10 Done Done Sent false 5309 Acked Write 3 Done Done Sent false 5310 Acked Write 10 Done Done Sent false 5311 Acked Write 4 Done Done Sent false 5312 Acked Write 5 Done Done Sent false 5313 Acked Read 3 Done Done Sent false 5314 Acked Read 4 Done Done Sent false 5315 Acked Write 2 Done Done Sent false 5316 Acked Flush 0 Done Done Sent false 5317 Acked Write 4 Done Done Sent false 5318 Acked Write 6 Done Done Sent false 5319 Acked Read 1 Done Done Sent false 5320 Acked Write 8 Done Done Sent false 5321 Acked Read 10 Done Done Sent false 5322 Acked Write 2 Done Done Sent false 5323 Acked Read 3 Done Done Sent false 5324 Acked Read 9 Done Done Sent false 5325 Acked Write 5 Done Done Sent false 5326 Acked Write 8 Done Done Sent false 5327 Acked Read 3 Done Done Sent false 5328 Acked Write 1 Done Done Sent false 5329 Acked Write 5 Done Done Sent false 5330 Acked Read 3 Done Done Sent false 5331 Acked Flush 0 Done Done Sent false 5332 Acked Flush 0 Done Done Sent false 5333 Acked Write 9 Done Done Sent false 5334 Acked Read 8 Done Done Sent false 5335 Acked Read 4 Done Done Sent false 5336 Acked Read 2 Done Done Sent false 5337 Acked Write 7 Done Done Sent false 5338 Acked Write 5 Done Done Sent false 5339 Acked Read 9 Done Done Sent false 5340 Acked Write 9 Done Done Sent false 5341 Acked Read 4 Done Done Sent false 5342 Acked Read 7 Done Done Sent false 5343 Acked Write 9 Done Done Sent false 5344 Acked Flush 0 Done Done Sent false 5345 Acked Write 4 Done Done Sent false 5346 Acked Write 3 Done Done Sent false 5347 Acked Write 3 Done Done Sent false 5348 Acked Read 5 Done Done Sent false 5349 Acked Write 7 Done Done Sent false 5350 Acked Flush 0 Done Done Sent false 5351 Acked Write 3 Done Done Sent false 5352 Acked Read 3 Done Done Sent false 5353 Acked Read 3 Done Done Sent false 5354 Acked Read 9 Done Done Sent false 5355 Acked Read 7 Done Done Sent false 5356 Acked Write 10 Done Done Sent false 5357 Acked Read 2 Done Done Sent false 5358 Acked Write 9 Done Done Sent false 5359 Acked Flush 0 Done Done Sent false 5360 Acked Write 6 Done Done Sent false 5361 Acked Write 9 Done Done Sent false 5362 Acked Flush 0 Done Done Sent false 5363 Acked Read 2 Done Done Sent false 5364 Acked Write 7 Done Done Sent false 5365 Acked Write 8 Done Done Sent false 5366 Acked Read 8 Done Done Sent false 5367 Acked Read 4 Done Done Sent false 5368 Acked Read 2 Done Done Sent false 5369 Acked Write 4 Done Done Sent false 5370 Acked Write 5 Done Done Sent false 5371 Acked Read 7 Done Done Sent false 5372 Acked Write 5 Done Done Sent false 5373 Acked Read 10 Done Done Sent false 5374 Acked Write 10 Done Done Sent false 5375 Acked Write 5 Done Done Sent false 5376 Acked Write 3 Done Done Sent false 5377 Acked Read 3 Done Done Sent false 5378 Acked Write 1 Done Done Sent false 5379 Acked Read 3 Done Done Sent false 5380 Acked Flush 0 Done Done Sent false 5381 Acked Read 3 Done Done Sent false 5382 Acked Write 7 Done Done Sent false 5383 Acked Write 10 Done Done Sent false 5384 Acked Read 6 Done Done Sent false 5385 Acked Write 6 Done Done Sent false 5386 Acked Read 7 Done Done Sent false 5387 Acked Write 8 Done Done Sent false 5388 Acked Read 7 Done Done Sent false 5389 Acked Write 5 Done Done Sent false 5390 Acked Write 9 Done Done Sent false 5391 Acked Read 1 Done Done Sent false 5392 Acked Flush 0 Done Done Sent false 5393 Acked Read 10 Done Done Sent false 5394 Acked Write 4 Done Done Sent false 5395 Acked Read 3 Done Done Sent false 5396 Acked Read 6 Done Done Sent false 5397 Acked Flush 0 Done Done Sent false 5398 Acked Write 6 Done Done Sent false 5399 Acked Write 2 Done Done Sent false 5400 Acked Write 10 Done Done Sent false 5401 Acked Write 2 Done Done Sent false 5402 Acked Read 8 Done Done Sent false 5403 Acked Read 3 Done Done Sent false 5404 Acked Write 3 Done Done Sent false 5405 Acked Read 1 Done Done Sent false 5406 Acked Read 4 Done Done Sent false 5407 Acked Write 5 Done Done Sent false 5408 Acked Read 10 Done Done Sent false 5409 Acked Flush 0 Done Done Sent false 5410 Acked Write 9 Done Done Sent false 5411 Acked Read 1 Done Done Sent false 5412 Acked Write 9 Done Done Sent false 5413 Acked Write 6 Done Done Sent false 5414 Acked Write 4 Done Done Sent false 5415 Acked Flush 0 Done Done Sent false 5416 Acked Write 9 Done Done Sent false 5417 Acked Read 8 Done Done Sent false 5418 Acked Write 7 Done Done Sent false 5419 Acked Write 4 Done Done Sent false 5420 Acked Read 3 Done Done Sent false 5421 Acked Read 8 Done Done Sent false 5422 Acked Write 3 Done Done Sent false 5423 Acked Read 4 Done Done Sent false 5424 Acked Read 3 Done Done Sent false 5425 Acked Write 5 Done Done Sent false 5426 Acked Read 1 Done Done Sent false 5427 Acked Write 9 Done Done Sent false 5428 Acked Write 9 Done Done Sent false 5429 Acked Read 3 Done Done Sent false 5430 Acked Read 8 Done Done Sent false 5431 Acked Write 5 Done Done Sent false 5432 Acked Flush 0 Done Done Sent false 5433 Acked Read 6 Done Done Sent false 5434 Acked Read 4 Done Done Sent false 5435 Acked Read 5 Done Done Sent false 5436 Acked Read 8 Done Done Sent false 5437 Acked Read 10 Done Done Sent false 5438 Acked Write 6 Done Done Sent false 5439 Acked Write 8 Done Done Sent false 5440 Acked Write 3 Done Done Sent false 5441 Acked Write 3 Done Done Sent false 5442 Acked Write 8 Done Done Sent false 5443 Acked Read 5 Done Done Sent false 5444 Acked Read 10 Done Done Sent false 5445 Acked Read 2 Done Done Sent false 5446 Acked Write 1 Done Done Sent false 5447 Acked Read 1 Done Done Sent false 5448 Acked Read 10 Done Done Sent false 5449 Acked Write 4 Done Done Sent false 5450 Acked Write 7 Done Done Sent false 5451 Acked Write 8 Done Done Sent false 5452 Acked Read 10 Done Done Sent false 5453 Acked Write 9 Done Done Sent false 5454 Acked Flush 0 Done Done Sent false 5455 Acked Write 2 Done Done Sent false 5456 Acked Read 5 Done Done Sent false 5457 Acked Read 1 Done Done Sent false 5458 Acked Write 5 Done Done Sent false 5459 Acked Read 7 Done Done Sent false 5460 Acked Read 8 Done Done Sent false 5461 Acked Write 1 Done Done Sent false 5462 Acked Read 8 Done Done Sent false 5463 Acked Write 3 Done Done Sent false 5464 Acked Read 4 Done Done Sent false 5465 Acked Write 1 Done Done Sent false 5466 Acked Write 5 Done Done Sent false 5467 Acked Write 7 Done Done Sent false 5468 Acked Write 2 Done Done Sent false 5469 Acked Write 10 Done Done Sent false 5470 Acked Read 8 Done Done Sent false 5471 Acked Read 10 Done Done Sent false 5472 Acked Read 9 Done Done Sent false 5473 Acked Read 4 Done Done Sent false 5474 Acked Write 9 Done Done Sent false 5475 Acked Read 8 Done Done Sent false 5476 Acked Read 5 Done Done Sent false 5477 Acked Write 7 Done Done Sent false 5478 Acked Write 4 Done Done Sent false 5479 Acked Write 10 Done Done Sent false 5480 Acked Flush 0 Done Done Sent false 5481 Acked Write 1 Done Done Sent false 5482 Acked Read 9 Done Done Sent false 5483 Acked Read 8 Done Done Sent false 5484 Acked Read 4 Done Done Sent false 5485 Acked Write 6 Done Done Sent false 5486 Acked Flush 0 Done Done Sent false 5487 Acked Write 3 Done Done Sent false 5488 Acked Read 10 Done Done Sent false 5489 Acked Read 8 Done Done Sent false 5490 Acked Write 6 Done Done Sent false 5491 Acked Write 3 Done Done Sent false 5492 Acked Read 1 Done Done Sent false 5493 Acked Read 8 Done Done Sent false 5494 Acked Read 9 Done Done Sent false 5495 Acked Flush 0 Done Done Sent false 5496 Acked Read 8 Done Done Sent false 5497 Acked Write 4 Done Done Sent false 5498 Acked Read 3 Done Done Sent false 5499 Acked Flush 0 Done Done Sent false 5500 Acked Read 4 Done Done Sent false 5501 Acked Write 6 Done Done Sent false 5502 Acked Flush 0 Done Done Sent false 5503 Acked Write 5 Done Done Sent false 5504 Acked Write 8 Done Done Sent false 5505 Acked Flush 0 Done Done Sent false 5506 Acked Read 3 Done Done Sent false 5507 Acked Read 7 Done Done Sent false 5508 Acked Write 7 Done Done Sent false 5509 Acked Read 8 Done Done Sent false 5510 Acked Write 4 Done Done Sent false 5511 Acked Write 1 Done Done Sent false 5512 Acked Write 2 Done Done Sent false 5513 Acked Read 2 Done Done Sent false 5514 Acked Read 8 Done Done Sent false 5515 Acked Read 6 Done Done Sent false 5516 Acked Read 7 Done Done Sent false 5517 Acked Write 4 Done Done Sent false 5518 Acked Flush 0 Done Done Sent false 5519 Acked Read 7 Done Done Sent false 5520 Acked Read 9 Done Done Sent false 5521 Acked Write 2 Done Done Sent false 5522 Acked Read 7 Done Done Sent false 5523 Acked Read 7 Done Done Sent false 5524 Acked Write 9 Done Done Sent false 5525 Acked Read 3 Done Done Sent false 5526 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5526 5526 5224 Downstairs last five completed: 5224 5223 5222 5221 5220 Upstairs last five completed: 5526 5525 5524 5523 5522 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.1080955Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:63622)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.10817056Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:29.10925009Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.109801462Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.109822786Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 5224","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.109837602Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 5224","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.110059353Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.110079207Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.110093854Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 5224","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.11010774Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 5224","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:29.110381856Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5526 5525 5524 5523 5522 Upstairs last five completed: 5526 5525 5524 5523 5522 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:45:31.342 INFO request completed, latency_us: 393, response_code: [8810] Got stop action so:false kr:false 204, uri: /stop/cid/0, method: POST, req_id: 9a320a5d-9aea-4ae3-866a-72912e2e4519, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:31.343436825Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:45:31.343 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: dfbb07be-9648-46b1-bb46-23ce405cdf4d, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:31.34353109Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} remote_addr: 127.0.0.1:51788, local_addr{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:31.343564221Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} : 127.0.0.1:9998 {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:31.343600041Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:31.343618236Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:31.343639251Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:45:35.345 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/0, method: GET, req_id: e22f9ff2-b1d3-4ded-8761-ad0cbc5cbfa7, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Flush 002/300 Write block 4111 len 4096 data: 3 003/300 Write block 1826 len 20480 data: 5 4 3 3 3 004/300 Flush 005/300 Read block 1484 len 16384 006/300 Read block 4722 len 4096 007/300 Read block 2919 len 28672 008/300 Write block 1094 len 4096 data: 6 009/300 Read block 4737 len 36864 010/300 Read block 501 len 40960 011/300 Read block 188 len 24576 012/300 Write block 4324 len 24576 data: 3 3 3 4 4 5 013/300 Read block 1384 len 24576 014/300 Read block 4939 len 40960 015/300 Read block 2953 len 32768 016/300 Read block 3357 len 24576 017/300 Read block 1779 len 32768 018/300 Read block 2946 len 12288 019/300 Read block 4395 len 24576 020/300 Read block 1281 len 28672 021/300 Read block 1320 len 16384 022/300 Read block 4007 len 40960 023/300 Read block 1773 len 40960 024/300 Flush 025/300 Read block 2711 len 40960 026/300 Write block 2034 len 12288 data: 3 3 4 027/300 Write block 4536 len 16384 data: 3 3 3 4 028/300 Write block 1256 len 36864 data: 4 2 3 3 3 3 3 4 4 029/300 Write block 4677 len 32768 data: 6 8 8 7 6 6 3 4 030/300 Write block 676 len 40960 data: 5 5 8 7 6 6 5 4 4 4 031/300 Read block 2172 len 24576 032/300 Read block 4315 len 24576 033/300 Write block 1910 len 4096 data: 7 034/300 Write block 4139 len 36864 data: 3 3 2 2 2 3 3 3 3 035/300 Write block 3304 len 28672 data: 4 4 3 3 3 3 2 036/300 Write block 8 len 8192 data: 3 3 037/300 Write block 4981 len 40960 data: 5 6 6 6 6 4 4 4 4 4 038/300 Read block 1476 len 12288 039/300 Write block 4941 len 12288 data: 8 8 9 040/300 Write block 4708 len 20480 data: 6 4 4 4 4 041/300 Read block 3436 len 4096 042/300 Read block 2488 len 40960 043/300 Write block 507 len 24576 data: 5 5 5 5 4 5 044/300 Write block 1823 len 24576 data: 7 4 4 6 5 4 045/300 Write block 3359 len 20480 data: 6 6 7 7 6 046/300 Read block 3903 len 36864 047/300 Write block 1229 len 40960 data: 4 4 2 3 5 5 5 3 3 3 048/300 Write block 3268 len 20480 data: 2 3 3 3 3 049/300 Flush 050/300 Read block 675 len 28672 051/300 Write block 506 len 40960 data: 4 6 6 6 6 5 6 3 3 3 052/300 Write block 2708 len 36864 data: 2 2 2 2 2 2 2 3 4 053/300 Flush 054/300 Read block 829 len 4096 055/300 Write block 3752 len 20480 data: 2 5 5 4 4 056/300 Write block 4816 len 8192 data: 2 2 057/300 Read block 1550 len 12288 058/300 Write block 955 len 28672 data: 4 3 3 3 3 3 3 059/300 Write block 3989 len 20480 data: 3 5 5 4 3 060/300 Read block 960 len 4096 061/300 Read block 3129 len 32768 062/300 Write block 2713 len 40960 data: 3 3 4 5 4 4 4 4 4 3 063/300 Write block 146 len 4096 data: 5 064/300 Read block 2689 len 32768 065/300 Read block 4287 len 16384 066/300 Read block 2804 len 16384 067/300 Read block 1461 len 24576 068/300 Read block 3330 len 20480 069/300 Write block 807 len 36864 data: 4 4 3 3 4 4 3 6 5 070/300 Read block 2120 len 28672 071/300 Read block 3768 len 8192 072/300 Read block 3101 len 28672 073/300 Read block 886 len 4096 074/300 Read block 2063 len 32768 075/300 Write block 1316 len 12288 data: 6 5 5 076/300 Write block 2056 len 8192 data: 6 6 077/300 Write block 3940 len 40960 data: 4 4 4 3 3 2 3 3 4 5 078/300 Read block 2038 len 16384 079/300 Write block 2681 len 8192 data: 2 3 080/300 Flush 081/300 Write block 3175 len 32768 data: 3 3 2 3 3 3 2 2 082/300 Read block 2014 len 4096 083/300 Write block 671 len 4096 data: 4 084/300 Read block 2649 len 36864 085/300 Read block 1102 len 32768 086/300 Write block 489 len 12288 data: 5 5 6 087/300 Read block 4245 len 20480 088/300 Read block 2778 len 4096 089/300 Read block 2188 len 20480 090/300 Flush 091/300 Read block 3126 len 24576 092/300 Read block 2482 len 36864 093/300 Read block 1381 len 8192 094/300 Read block 204 len 28672 095/300 Read block 2258 len 4096 096/300 Write block 4886 len 4096 data: 3 097/300 Read block 3737 len 32768 098/300 Write block 455 len 20480 data: 4 2 2 2 2 099/300 Write block 1108 len 32768 data: 8 8 7 7 5 5 5 5 100/300 Write block 4442 len 16384 data: 6 7 6 5 101/300 Write block 4356 len 4096 data: 8 102/300 Write block 2634 len 20480 data: 3 2 3 3 3 103/300 Write block 2 len 28672 data: 2 3 4 5 5 4 4 104/300 Write block 579 len 20480 data: 5 6 5 5 4 105/300 Write block 4595 len 8192 data: 6 4 106/300 Read block 1234 len 36864 107/300 Flush 108/300 Read block 1730 len 12288 109/300 Read block 3962 len 4096 110/300 Read block 4581 len 32768 111/300 Read block 3819 len 16384 112/300 Read block 1291 len 36864 113/300 Write block 368 len 32768 data: 4 4 5 5 5 4 4 4 114/300 Read block 4687 len 40960 115/300 Flush 116/300 Write block 1251 len 12288 data: 5 5 7 117/300 Read block 1444 len 4096 118/300 Read block 1152 len 32768 119/300 Read block 539 len 24576 120/300 Read block 1340 len 4096 121/300 Write block 4389 len 40960 data: 7 6 4 3 4 3 3 3 3 3 122/300 Read block 166 len 12288 123/300 Read block 4128 len 28672 124/300 Write block 564 len 16384 data: 2 2 3 3 125/300 Flush 126/300 Read block 709 len 36864 127/300 Write block 4438 len 32768 data: 4 5 5 6 7 8 7 6 128/300 Read block 1449 len 12288 129/300 Write block 1333 len 32768 data: 2 3 3 3 4 4 3 3 130/300 Read block 4517 len 8192 131/300 Write block 3219 len 24576 data: 3 3 3 3 3 3 132/300 Read block 366 len 32768 133/300 Write block 105 len 24576 data: 3 3 3 5 5 6 134/300 Write block 4008 len 8192 data: 3 3 135/300 Read block 1758 len 8192 136/300 Read block 287 len 16384 137/300 Read block 4189 len 28672 138/300 Read block 2765 len 28672 139/300 Write block 2621 len 20480 data: 4 4 3 3 3 140/300 Read block 3138 len 12288 141/300 Write block 4882 len 24576 data: 3 3 3 4 4 4 142/300 Write block 3849 len 8192 data: 2 3 143/300 Write block 3658 len 20480 data: 6 5 5 5 5 144/300 Flush 145/300 Read block 3085 len 20480 146/300 Flush 147/300 Write block 444 len 20480 data: 4 4 4 4 4 148/300 Write block 4404 len 32768 data: 2 2 2 2 2 2 3 3 149/300 Read block 3121 len 16384 150/300 Write block 4184 len 8192 data: 5 5 151/300 Write block 1294 len 40960 data: 6 6 7 7 6 6 5 4 2 3 152/300 Read block 4027 len 36864 153/300 Read block 2767 len 4096 154/300 Flush 155/300 Write block 3684 len 36864 data: 5 6 7 5 6 6 7 6 5 156/300 Write block 619 len 4096 data: 4 157/300 Write block 4153 len 4096 data: 7 158/300 Flush 159/300 Read block 1376 len 12288 160/300 Write block 4390 len 36864 data: 7 5 4 5 4 4 4 4 4 161/300 Read block 2263 len 36864 162/300 Read block 1974 len 36864 163/300 Read block 95 len 36864 164/300 Write block 1975 len 32768 data: 3 4 4 4 3 3 4 3 165/300 Read block 3227 len 8192 166/300 Write block 3645 len 8192 data: 3 3 167/300 Write block 1848 len 4096 data: 2 168/300 Read block 4135 len 40960 169/300 Read block 2523 len 32768 170/300 Write block 3537 len 16384 data: 6 6 6 6 171/300 Write block 1468 len 28672 data: 7 7 7 7 7 5 4 172/300 Write block 3554 len 20480 data: 6 7 7 5 5 173/300 Read block 4201 len 28672 174/300 Read block 960 len 28672 175/300 Write block 3540 len 20480 data: 7 6 5 4 4 176/300 Write block 2411 len 32768 data: 3 3 3 3 3 3 2 2 177/300 Read block 866 len 40960 178/300 Read block 1631 len 40960 179/300 Write block 4912 len 8192 data: 3 3 180/300 Read block 3353 len 28672 181/300 Write block 931 len 16384 data: 3 3 4 4 182/300 Write block 3457 len 4096 data: 4 183/300 Read block 2026 len 12288 184/300 Read block 4380 len 24576 185/300 Read block 3 len 28672 186/300 Flush 187/300 Read block 1460 len 28672 188/300 Flush 189/300 Write block 584 len 8192 data: 6 6 190/300 Read block 3182 len 8192 191/300 Read block 4775 len 4096 192/300 Read block 2403 len 32768 193/300 Read block 3651 len 24576 194/300 Flush 195/300 Write block 2079 len 32768 data: 3 2 2 2 2 2 2 3 196/300 Read block 1433 len 32768 197/300 Read block 3973 len 28672 198/300 Read block 3773 len 12288 199/300 Write block 1582 len 32768 data: 3 3 3 3 4 5 6 5 200/300 Read block 304 len 20480 201/300 Write block 4662 len 36864 data: 5 5 4 3 4 4 4 4 5 202/300 Write block 1327 len 36864 data: 3 2 2 2 3 2 3 4 4 203/300 Flush 204/300 Read block 426 len 8192 205/300 Write block 712 len 20480 data: 6 3 4 4 4 206/300 Write block 3203 len 4096 data: 3 207/300 Flush 208/300 Write block 3608 len 36864 data: 2 2 2 3 3 4 5 6 6 209/300 Read block 2732 len 4096 210/300 Read block 2096 len 4096 211/300 Read block 4607 len 8192 212/300 Write block 1402 len 24576 data: 5 6 5 5 4 4 213/300 Write block 855 len 40960 data: 4 2 3 3 3 3 3 4 4 4 214/300 Write block 3591 len 28672 data: 4 4 4 3 2 4 4 215/300 Read block 190 len 4096 216/300 Write block 309 len 36864 data: 2 2 3 3 3 4 4 4 5 217/300 Read block 3199 len 12288 218/300 Read block 2688 len 8192 219/300 Write block 2639 len 16384 data: 3 2 2 2 220/300 Read block 3318 len 32768 221/300 Read block 4386 len 12288 222/300 Read block 981 len 4096 223/300 Write block 29 len 12288 data: 5 5 6 224/300 Read block 1075 len 8192 225/300 Read block 79 len 20480 226/300 Write block 1070 len 40960 data: 2 2 2 2 3 3 3 2 3 3 227/300 Flush 228/300 Read block 3484 len 16384 229/300 Read block 3667 len 28672 230/300 Write block 3579 len 4096 data: 3 231/300 Read block 2637 len 4096 232/300 Read block 50 len 40960 233/300 Read block 3834 len 16384 234/300 Read block 110 len 36864 235/300 Write block 2391 len 4096 data: 6 236/300 Read block 2784 len 12288 237/300 Read block 447 len 28672 238/300 Read block 1963 len 16384 239/300 Write block 242 len 24576 data: 4 4 3 3 3 3 240/300 Write block 2112 len 16384 data: 5 6 7 6 241/300 Read block 1137 len 8192 242/300 Read block 3906 len 12288 243/300 Read block 3116 len 8192 244/300 Write block 2922 len 4096 data: 7 245/300 Write block 291 len 8192 data: 4 4 246/300 Read block 2252 len 24576 247/300 Flush 248/300 Flush 249/300 Read block 2490 len 32768 250/300 Write block 3591 len 36864 data: 5 5 5 4 3 5 5 3 3 251/300 Write block 1265 len 20480 data: 5 5 4 5 4 252/300 Read block 4042 len 40960 253/300 Write block 1459 len 12288 data: 4 5 5 254/300 Flush 255/300 Write block 3194 len 16384 data: 5 4 5 7 256/300 Write block 588 len 40960 data: 2 2 2 3 3 3 3 3 3 3 257/300 Flush 258/300 Flush 259/300 Read block 3283 len 28672 260/300 Write block 2436 len 20480 data: 2 4 4 4 5 261/300 Read block 2374 len 4096 262/300 Read block 1608 len 8192 263/300 Read block 1080 len 4096 264/300 Read block 2565 len 12288 265/300 Read block 3060 len 20480 266/300 Write block 3837 len 4096 data: 4 267/300 Read block 1140 len 40960 268/300 Read block 2870 len 40960 269/300 Read block 1806 len 36864 270/300 Read block 1197 len 16384 271/300 Write block 826 len 24576 data: 9 9 9 10 9 8 272/300 Flush 273/300 Read block 2148 len 12288 274/300 Read block 3172 len 28672 275/300 Read block 4903 len 36864 276/300 Write block 1742 len 12288 data: 3 3 2 277/300 Flush 278/300 Write block 0 len 20480 data: 2 2 3 4 5 279/300 Read block 2922 len 28672 280/300 Write block 4845 len 4096 data: 6 281/300 Write block 2378 len 16384 data: 7 7 7 5 282/300 Read block 3176 len 40960 283/300 Read block 2700 len 36864 284/300 Write block 726 len 32768 data: 5 5 3 3 3 3 3 2 285/300 Write block 4255 len 20480 data: 3 3 3 3 3 286/300 Read block 2618 len 36864 287/300 Read block 55 len 8192 288/300 Read block 2053 len 36864 289/300 Write block 2781 len 16384 data: 3 3 3 4 290/300 Flush 291/300 Read block 383 len 32768 292/300 Write block 4715 len 24576 data: 3 2 2 2 3 3 293/300 Read block 2874 len 32768 294/300 Write block 2917 len 20480 data: 4 6 5 6 6 295/300 Read block 3650 len 12288 296/300 Read block 2943 len 4096 297/300 Write block 3960 len 24576 data: 3 3 4 4 3 4 298/300 Read block 624 len 12288 299/300 Read block 4098 len 36864 300/300 Flush 301/300 Flush Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:45:35.577 INFO request completed, latency_us: 241, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8ee8c79c-9d08-4572-a1ed-de5b9d26be4d, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5527 Acked Flush 0 Sent Done Done false 5528 Acked Write 1 Sent Done Done false 5529 Acked Write 5 Sent Done Done false 5530 Acked Flush 0 Sent Done Done false 5531 Acked Read 4 Sent Done Done false 5532 Acked Read 1 Sent Done Done false 5533 Acked Read 7 Sent Done Done false 5534 Acked Write 1 Sent Done Done false 5535 Acked Read 9 Sent Done Done false 5536 Acked Read 10 Sent Done Done false 5537 Acked Read 6 Sent Done Done false 5538 Acked Write 6 Sent Done Done false 5539 Acked Read 6 Sent Done Done false 5540 Acked Read 10 Sent Done Done false 5541 Acked Read 8 Sent Done Done false 5542 Acked Read 6 Sent Done Done false 5543 Acked Read 8 Sent Done Done false 5544 Acked Read 3 Sent Done Done false 5545 Acked Read 6 Sent Done Done false 5546 Acked Read 7 Sent Done Done false 5547 Acked Read 4 Sent Done Done false 5548 Acked Read 10 Sent Done Done false 5549 Acked Read 10 Sent Done Done false 5550 Acked Flush 0 Sent Done Done false 5551 Acked Read 10 Sent Done Done false 5552 Acked Write 3 Sent Done Done false 5553 Acked Write 4 Sent Done Done false 5554 Acked Write 9 Sent Done Done false 5555 Acked Write 8 Sent Done Done false 5556 Acked Write 10 Sent Done Done false 5557 Acked Read 6 Sent Done Done false 5558 Acked Read 6 Sent Done Done false 5559 Acked Write 1 Sent Done Done false 5560 Acked Write 9 Sent Done Done false 5561 Acked Write 7 Sent Done Done false 5562 Acked Write 2 Sent Done Done false 5563 Acked Write 10 Sent Done Done false 5564 Acked Read 3 Sent Done Done false 5565 Acked Write 3 Sent Done Done false 5566 Acked Write 5 Sent Done Done false 5567 Acked Read 1 Sent Done Done false 5568 Acked Read 10 Sent Done Done false 5569 Acked Write 6 Sent Done Done false 5570 Acked Write 6 Sent Done Done false 5571 Acked Write 5 Sent Done Done false 5572 Acked Read 9 Sent Done Done false 5573 Acked Write 10 Sent Done Done false 5574 Acked Write 5 Sent Done Done false 5575 Acked Flush 0 Sent Done Done false 5576 Acked Read 7 Sent Done Done false 5577 Acked Write 10 Sent Done Done false 5578 Acked Write 9 Sent Done Done false 5579 Acked Flush 0 Sent Done Done false 5580 Acked Read 1 Sent Done Done false 5581 Acked Write 5 Sent Done Done false 5582 Acked Write 2 Sent Done Done false 5583 Acked Read 3 Sent Done Done false 5584 Acked Write 7 Sent Done Done false 5585 Acked Write 5 Sent Done Done false 5586 Acked Read 1 Sent Done Done false 5587 Acked Read 8 Sent Done Done false 5588 Acked Write 10 Sent Done Done false 5589 Acked Write 1 Sent Done Done false 5590 Acked Read 8 Sent Done Done false 5591 Acked Read 4 Sent Done Done false 5592 Acked Read 4 Sent Done Done false 5593 Acked Read 6 Sent Done Done false 5594 Acked Read 5 Sent Done Done false 5595 Acked Write 9 Sent Done Done false 5596 Acked Read 7 Sent Done Done false 5597 Acked Read 2 Sent Done Done false 5598 Acked Read 7 Sent Done Done false 5599 Acked Read 1 Sent Done Done false 5600 Acked Read 8 Sent Done Done false 5601 Acked Write 3 Sent Done Done false 5602 Acked Write 2 Sent Done Done false 5603 Acked Write 10 Sent Done Done false 5604 Acked Read 4 Sent Done Done false 5605 Acked Write 2 Sent Done Done false 5606 Acked Flush 0 Sent Done Done false 5607 Acked Write 8 Sent Done Done false 5608 Acked Read 1 Sent Done Done false 5609 Acked Write 1 Sent Done Done false 5610 Acked Read 9 Sent Done Done false 5611 Acked Read 8 Sent Done Done false 5612 Acked Write 3 Sent Done Done false 5613 Acked Read 5 Sent Done Done false 5614 Acked Read 1 Sent Done Done false 5615 Acked Read 5 Sent Done Done false 5616 Acked Flush 0 Sent Done Done false 5617 Acked Read 6 Sent Done Done false 5618 Acked Read 9 Sent Done Done false 5619 Acked Read 2 Sent Done Done false 5620 Acked Read 7 Sent Done Done false 5621 Acked Read 1 Sent Done Done false 5622 Acked Write 1 Sent Done Done false 5623 Acked Read 8 Sent Done Done false 5624 Acked Write 5 Sent Done Done false 5625 Acked Write 8 Sent Done Done false 5626 Acked Write 4 Sent Done Done false 5627 Acked Write 1 Sent Done Done false 5628 Acked Write 5 Sent Done Done false 5629 Acked Write 7 Sent Done Done false 5630 Acked Write 5 Sent Done Done false 5631 Acked Write 2 Sent Done Done false 5632 Acked Read 9 Sent Done Done false 5633 Acked Flush 0 Sent Done Done false 5634 Acked Read 3 Sent Done Done false 5635 Acked Read 1 Sent Done Done false 5636 Acked Read 8 Sent Done Done false 5637 Acked Read 4 Sent Done Done false 5638 Acked Read 9 Sent Done Done false 5639 Acked Write 8 Sent Done Done false 5640 Acked Read 10 Sent Done Done false 5641 Acked Flush 0 Sent Done Done false 5642 Acked Write 3 Sent Done Done false 5643 Acked Read 1 Sent Done Done false 5644 Acked Read 8 Sent Done Done false 5645 Acked Read 6 Sent Done Done false 5646 Acked Read 1 Sent Done Done false 5647 Acked Write 10 Sent Done Done false 5648 Acked Read 3 Sent Done Done false 5649 Acked Read 7 Sent Done Done false 5650 Acked Write 4 Sent Done Done false 5651 Acked Flush 0 Sent Done Done false 5652 Acked Read 9 Sent Done Done false 5653 Acked Write 8 Sent Done Done false 5654 Acked Read 3 Sent Done Done false 5655 Acked Write 8 Sent Done Done false 5656 Acked Read 2 Sent Done Done false 5657 Acked Write 6 Sent Done Done false 5658 Acked Read 8 Sent Done Done false 5659 Acked Write 6 Sent Done Done false 5660 Acked Write 2 Sent Done Done false 5661 Acked Read 2 Sent Done Done false 5662 Acked Read 4 Sent Done Done false 5663 Acked Read 7 Sent Done Done false 5664 Acked Read 7 Sent Done Done false 5665 Acked Write 5 Sent Done Done false 5666 Acked Read 3 Sent Done Done false 5667 Acked Write 6 Sent Done Done false 5668 Acked Write 2 Sent Done Done false 5669 Acked Write 5 Sent Done Done false 5670 Acked Flush 0 Sent Done Done false 5671 Acked Read 5 Sent Done Done false 5672 Acked Flush 0 Sent Done Done false 5673 Acked Write 5 Sent Done Done false 5674 Acked Write 8 Sent Done Done false 5675 Acked Read 4 Sent Done Done false 5676 Acked Write 2 Sent Done Done false 5677 Acked Write 10 Sent Done Done false 5678 Acked Read 9 Sent Done Done false 5679 Acked Read 1 Sent Done Done false 5680 Acked Flush 0 Sent Done Done false 5681 Acked Write 9 Sent Done Done false 5682 Acked Write 1 Sent Done Done false 5683 Acked Write 1 Sent Done Done false 5684 Acked Flush 0 Sent Done Done false 5685 Acked Read 3 Sent Done Done false 5686 Acked Write 9 Sent Done Done false 5687 Acked Read 9 Sent Done Done false 5688 Acked Read 9 Sent Done Done false 5689 Acked Read 9 Sent Done Done false 5690 Acked Write 8 Sent Done Done false 5691 Acked Read 2 Sent Done Done false 5692 Acked Write 2 Sent Done Done false 5693 Acked Write 1 Sent Done Done false 5694 Acked Read 10 Sent Done Done false 5695 Acked Read 8 Sent Done Done false 5696 Acked Write 4 Sent Done Done false 5697 Acked Write 7 Sent Done Done false 5698 Acked Write 5 Sent Done Done false 5699 Acked Read 7 Sent Done Done false 5700 Acked Read 7 Sent Done Done false 5701 Acked Write 5 Sent Done Done false 5702 Acked Write 8 Sent Done Done false 5703 Acked Read 10 Sent Done Done false 5704 Acked Read 10 Sent Done Done false 5705 Acked Write 2 Sent Done Done false 5706 Acked Read 7 Sent Done Done false 5707 Acked Write 4 Sent Done Done false 5708 Acked Write 1 Sent Done Done false 5709 Acked Read 3 Sent Done Done false 5710 Acked Read 6 Sent Done Done false 5711 Acked Read 7 Sent Done Done false 5712 Acked Flush 0 Sent Done Done false 5713 Acked Read 7 Sent Done Done false 5714 Acked Flush 0 Sent Done Done false 5715 Acked Write 2 Sent Done Done false 5716 Acked Read 2 Sent Done Done false 5717 Acked Read 1 Sent Done Done false 5718 Acked Read 8 Sent Done Done false 5719 Acked Read 6 Sent Done Done false 5720 Acked Flush 0 Sent Done Done false 5721 Acked Write 8 Sent Done Done false 5722 Acked Read 8 Sent Done Done false 5723 Acked Read 7 Sent Done Done false 5724 Acked Read 3 Sent Done Done false 5725 Acked Write 8 Sent Done Done false 5726 Acked Read 5 Sent Done Done false 5727 Acked Write 9 Sent Done Done false 5728 Acked Write 9 Sent Done Done false 5729 Acked Flush 0 Sent Done Done false 5730 Acked Read 2 Sent Done Done false 5731 Acked Write 5 Sent Done Done false 5732 Acked Write 1 Sent Done Done false 5733 Acked Flush 0 Sent Done Done false 5734 Acked Write 9 Sent Done Done false 5735 Acked Read 1 Sent Done Done false 5736 Acked Read 1 Sent Done Done false 5737 Acked Read 2 Sent Done Done false 5738 Acked Write 6 Sent Done Done false 5739 Acked Write 10 Sent Done Done false 5740 Acked Write 7 Sent Done Done false 5741 Acked Read 1 Sent Done Done false 5742 Acked Write 9 Sent Done Done false 5743 Acked Read 3 Sent Done Done false 5744 Acked Read 2 Sent Done Done false 5745 Acked Write 4 Sent Done Done false 5746 Acked Read 8 Sent Done Done false 5747 Acked Read 3 Sent Done Done false 5748 Acked Read 1 Sent Done Done false 5749 Acked Write 3 Sent Done Done false 5750 Acked Read 2 Sent Done Done false 5751 Acked Read 5 Sent Done Done false 5752 Acked Write 10 Sent Done Done false 5753 Acked Flush 0 Sent Done Done false 5754 Acked Read 4 Sent Done Done false 5755 Acked Read 7 Sent Done Done false 5756 Acked Write 1 Sent Done Done false 5757 Acked Read 1 Sent Done Done false 5758 Acked Read 10 Sent Done Done false 5759 Acked Read 4 Sent Done Done false 5760 Acked Read 9 Sent Done Done false 5761 Acked Write 1 Sent Done Done false 5762 Acked Read 3 Sent Done Done false 5763 Acked Read 7 Sent Done Done false 5764 Acked Read 4 Sent Done Done false 5765 Acked Write 6 Sent Done Done false 5766 Acked Write 4 Sent Done Done false 5767 Acked Read 2 Sent Done Done false 5768 Acked Read 3 Sent Done Done false 5769 Acked Read 2 Sent Done Done false 5770 Acked Write 1 Sent Done Done false 5771 Acked Write 2 Sent Done Done false 5772 Acked Read 6 Sent Done Done false 5773 Acked Flush 0 Sent Done Done false 5774 Acked Flush 0 Sent Done Done false 5775 Acked Read 8 Sent Done Done false 5776 Acked Write 9 Sent Done Done false 5777 Acked Write 5 Sent Done Done false 5778 Acked Read 10 Sent Done Done false 5779 Acked Write 3 Sent Done Done false 5780 Acked Flush 0 Sent Done Done false 5781 Acked Write 4 Sent Done Done false 5782 Acked Write 10 Sent Done Done false 5783 Acked Flush 0 Sent Done Done false 5784 Acked Flush 0 Sent Done Done false 5785 Acked Read 7 Sent Done Done false 5786 Acked Write 5 Sent Done Done false 5787 Acked Read 1 Sent Done Done false 5788 Acked Read 2 Sent Done Done false 5789 Acked Read 1 Sent Done Done false 5790 Acked Read 3 Sent Done Done false 5791 Acked Read 5 Sent Done Done false 5792 Acked Write 1 Sent Done Done false 5793 Acked Read 10 Sent Done Done false 5794 Acked Read 10 Sent Done Done false 5795 Acked Read 9 Sent Done Done false 5796 Acked Read 4 Sent Done Done false 5797 Acked Write 6 Sent Done Done false 5798 Acked Flush 0 Sent Done Done false 5799 Acked Read 3 Sent Done Done false 5800 Acked Read 7 Sent Done Done false 5801 Acked Read 9 Sent Done Done false 5802 Acked Write 3 Sent Done Done false 5803 Acked Flush 0 Sent Done Done false 5804 Acked Write 5 Sent Done Done false 5805 Acked Read 7 Sent Done Done false 5806 Acked Write 1 Sent Done Done false 5807 Acked Write 4 Sent Done Done false 5808 Acked Read 10 Sent Done Done false 5809 Acked Read 9 Sent Done Done false 5810 Acked Write 8 Sent Done Done false 5811 Acked Write 5 Sent Done Done false 5812 Acked Read 9 Sent Done Done false 5813 Acked Read 2 Sent Done Done false 5814 Acked Read 9 Sent Done Done false 5815 Acked Write 4 Sent Done Done false 5816 Acked Flush 0 Sent Done Done false 5817 Acked Read 8 Sent Done Done false 5818 Acked Write 6 Sent Done Done false 5819 Acked Read 8 Sent Done Done false 5820 Acked Write 5 Sent Done Done false 5821 Acked Read 3 Sent Done Done false 5822 Acked Read 1 Sent Done Done false 5823 Acked Write 6 Sent Done Done false 5824 Acked Read 3 Sent Done Done false 5825 Acked Read 9 Sent Done Done false 5826 Acked Flush 0 Sent Done Done false 5827 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5526 5827 5827 Downstairs last five completed: 5526 5525 5524 5523 5522 Upstairs last five completed: 5827 5826 5825 5824 5823 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5527 Acked Flush 0 Sent Done Done false 5528 Acked Write 1 Sent Done Done false 5529 Acked Write 5 Sent Done Done false 5530 Acked Flush 0 Sent Done Done false 5531 Acked Read 4 Sent Done Done false 5532 Acked Read 1 Sent Done Done false 5533 Acked Read 7 Sent Done Done false 5534 Acked Write 1 Sent Done Done false 5535 Acked Read 9 Sent Done Done false 5536 Acked Read 10 Sent Done Done false 5537 Acked Read 6 Sent Done Done false 5538 Acked Write 6 Sent Done Done false 5539 Acked Read 6 Sent Done Done false 5540 Acked Read 10 Sent Done Done false 5541 Acked Read 8 Sent Done Done false 5542 Acked Read 6 Sent Done Done false 5543 Acked Read 8 Sent Done Done false 5544 Acked Read 3 Sent Done Done false 5545 Acked Read 6 Sent Done Done false 5546 Acked Read 7 Sent Done Done false 5547 Acked Read 4 Sent Done Done false 5548 Acked Read 10 Sent Done Done false 5549 Acked Read 10 Sent Done Done false 5550 Acked Flush 0 Sent Done Done false 5551 Acked Read 10 Sent Done Done false 5552 Acked Write 3 Sent Done Done false 5553 Acked Write 4 Sent Done Done false 5554 Acked Write 9 Sent Done Done false 5555 Acked Write 8 Sent Done Done false 5556 Acked Write 10 Sent Done Done false 5557 Acked Read 6 Sent Done Done false 5558 Acked Read 6 Sent Done Done false 5559 Acked Write 1 Sent Done Done false 5560 Acked Write 9 Sent Done Done false 5561 Acked Write 7 Sent Done Done false 5562 Acked Write 2 Sent Done Done false 5563 Acked Write 10 Sent Done Done false 5564 Acked Read 3 Sent Done Done false 5565 Acked Write 3 Sent Done Done false 5566 Acked Write 5 Sent Done Done false 5567 Acked Read 1 Sent Done Done false 5568 Acked Read 10 Sent Done Done false 5569 Acked Write 6 Sent Done Done false 5570 Acked Write 6 Sent Done Done false 5571 Acked Write 5 Sent Done Done false 5572 Acked Read 9 Sent Done Done false 5573 Acked Write 10 Sent Done Done false 5574 Acked Write 5 Sent Done Done false 5575 Acked Flush 0 Sent Done Done false 5576 Acked Read 7 Sent Done Done false 5577 Acked Write 10 Sent Done Done false 5578 Acked Write 9 Sent Done Done false 5579 Acked Flush 0 Sent Done Done false 5580 Acked Read 1 Sent Done Done false 5581 Acked Write 5 Sent Done Done false 5582 Acked Write 2 Sent Done Done false 5583 Acked Read 3 Sent Done Done false 5584 Acked Write 7 Sent Done Done false 5585 Acked Write 5 Sent Done Done false 5586 Acked Read 1 Sent Done Done false 5587 Acked Read 8 Sent Done Done false 5588 Acked Write 10 Sent Done Done false 5589 Acked Write 1 Sent Done Done false 5590 Acked Read 8 Sent Done Done false 5591 Acked Read 4 Sent Done Done false 5592 Acked Read 4 Sent Done Done false 5593 Acked Read 6 Sent Done Done false 5594 Acked Read 5 Sent Done Done false 5595 Acked Write 9 Sent Done Done false 5596 Acked Read 7 Sent Done Done false 5597 Acked Read 2 Sent Done Done false 5598 Acked Read 7 Sent Done Done false 5599 Acked Read 1 Sent Done Done false 5600 Acked Read 8 Sent Done Done false 5601 Acked Write 3 Sent Done Done false 5602 Acked Write 2 Sent Done Done false 5603 Acked Write 10 Sent Done Done false 5604 Acked Read 4 Sent Done Done false 5605 Acked Write 2 Sent Done Done false 5606 Acked Flush 0 Sent Done Done false 5607 Acked Write 8 Sent Done Done false 5608 Acked Read 1 Sent Done Done false 5609 Acked Write 1 Sent Done Done false 5610 Acked Read 9 Sent Done Done false 5611 Acked Read 8 Sent Done Done false 5612 Acked Write 3 Sent Done Done false 5613 Acked Read 5 Sent Done Done false 5614 Acked Read 1 Sent Done Done false 5615 Acked Read 5 Sent Done Done false 5616 Acked Flush 0 Sent Done Done false 5617 Acked Read 6 Sent Done Done false 5618 Acked Read 9 Sent Done Done false 5619 Acked Read 2 Sent Done Done false 5620 Acked Read 7 Sent Done Done false 5621 Acked Read 1 Sent Done Done false 5622 Acked Write 1 Sent Done Done false 5623 Acked Read 8 Sent Done Done false 5624 Acked Write 5 Sent Done Done false 5625 Acked Write 8 Sent Done Done false 5626 Acked Write 4 Sent Done Done false 5627 Acked Write 1 Sent Done Done false 5628 Acked Write 5 Sent Done Done false 5629 Acked Write 7 Sent Done Done false 5630 Acked Write 5 Sent Done Done false 5631 Acked Write 2 Sent Done Done false 5632 Acked Read 9 Sent Done Done false 5633 Acked Flush 0 Sent Done Done false 5634 Acked Read 3 Sent Done Done false 5635 Acked Read 1 Sent Done Done false 5636 Acked Read 8 Sent Done Done false 5637 Acked Read 4 Sent Done Done false 5638 Acked Read 9 Sent Done Done false 5639 Acked Write 8 Sent Done Done false 5640 Acked Read 10 Sent Done Done false 5641 Acked Flush 0 Sent Done Done false 5642 Acked Write 3 Sent Done Done false 5643 Acked Read 1 Sent Done Done false 5644 Acked Read 8 Sent Done Done false 5645 Acked Read 6 Sent Done Done false 5646 Acked Read 1 Sent Done Done false 5647 Acked Write 10 Sent Done Done false 5648 Acked Read 3 Sent Done Done false 5649 Acked Read 7 Sent Done Done false 5650 Acked Write 4 Sent Done Done false 5651 Acked Flush 0 Sent Done Done false 5652 Acked Read 9 Sent Done Done false 5653 Acked Write 8 Sent Done Done false 5654 Acked Read 3 Sent Done Done false 5655 Acked Write 8 Sent Done Done false 5656 Acked Read 2 Sent Done Done false 5657 Acked Write 6 Sent Done Done false 5658 Acked Read 8 Sent Done Done false 5659 Acked Write 6 Sent Done Done false 5660 Acked Write 2 Sent Done Done false 5661 Acked Read 2 Sent Done Done false 5662 Acked Read 4 Sent Done Done false 5663 Acked Read 7 Sent Done Done false 5664 Acked Read 7 Sent Done Done false 5665 Acked Write 5 Sent Done Done false 5666 Acked Read 3 Sent Done Done false 5667 Acked Write 6 Sent Done Done false 5668 Acked Write 2 Sent Done Done false 5669 Acked Write 5 Sent Done Done false 5670 Acked Flush 0 Sent Done Done false 5671 Acked Read 5 Sent Done Done false 5672 Acked Flush 0 Sent Done Done false 5673 Acked Write 5 Sent Done Done false 5674 Acked Write 8 Sent Done Done false 5675 Acked Read 4 Sent Done Done false 5676 Acked Write 2 Sent Done Done false 5677 Acked Write 10 Sent Done Done false 5678 Acked Read 9 Sent Done Done false 5679 Acked Read 1 Sent Done Done false 5680 Acked Flush 0 Sent Done Done false 5681 Acked Write 9 Sent Done Done false 5682 Acked Write 1 Sent Done Done false 5683 Acked Write 1 Sent Done Done false 5684 Acked Flush 0 Sent Done Done false 5685 Acked Read 3 Sent Done Done false 5686 Acked Write 9 Sent Done Done false 5687 Acked Read 9 Sent Done Done false 5688 Acked Read 9 Sent Done Done false 5689 Acked Read 9 Sent Done Done false 5690 Acked Write 8 Sent Done Done false 5691 Acked Read 2 Sent Done Done false 5692 Acked Write 2 Sent Done Done false 5693 Acked Write 1 Sent Done Done false 5694 Acked Read 10 Sent Done Done false 5695 Acked Read 8 Sent Done Done false 5696 Acked Write 4 Sent Done Done false 5697 Acked Write 7 Sent Done Done false 5698 Acked Write 5 Sent Done Done false 5699 Acked Read 7 Sent Done Done false 5700 Acked Read 7 Sent Done Done false 5701 Acked Write 5 Sent Done Done false 5702 Acked Write 8 Sent Done Done false 5703 Acked Read 10 Sent Done Done false 5704 Acked Read 10 Sent Done Done false 5705 Acked Write 2 Sent Done Done false 5706 Acked Read 7 Sent Done Done false 5707 Acked Write 4 Sent Done Done false 5708 Acked Write 1 Sent Done Done false 5709 Acked Read 3 Sent Done Done false 5710 Acked Read 6 Sent Done Done false 5711 Acked Read 7 Sent Done Done false 5712 Acked Flush 0 Sent Done Done false 5713 Acked Read 7 Sent Done Done false 5714 Acked Flush 0 Sent Done Done false 5715 Acked Write 2 Sent Done Done false 5716 Acked Read 2 Sent Done Done false 5717 Acked Read 1 Sent Done Done false 5718 Acked Read 8 Sent Done Done false 5719 Acked Read 6 Sent Done Done false 5720 Acked Flush 0 Sent Done Done false 5721 Acked Write 8 Sent Done Done false 5722 Acked Read 8 Sent Done Done false 5723 Acked Read 7 Sent Done Done false 5724 Acked Read 3 Sent Done Done false 5725 Acked Write 8 Sent Done Done false 5726 Acked Read 5 Sent Done Done false 5727 Acked Write 9 Sent Done Done false 5728 Acked Write 9 Sent Done Done false 5729 Acked Flush 0 Sent Done Done false 5730 Acked Read 2 Sent Done Done false 5731 Acked Write 5 Sent Done Done false 5732 Acked Write 1 Sent Done Done false 5733 Acked Flush 0 Sent Done Done false 5734 Acked Write 9 Sent Done Done false 5735 Acked Read 1 Sent Done Done false 5736 Acked Read 1 Sent Done Done false 5737 Acked Read 2 Sent Done Done false 5738 Acked Write 6 Sent Done Done false 5739 Acked Write 10 Sent Done Done false 5740 Acked Write 7 Sent Done Done false 5741 Acked Read 1 Sent Done Done false 5742 Acked Write 9 Sent Done Done false 5743 Acked Read 3 Sent Done Done false 5744 Acked Read 2 Sent Done Done false 5745 Acked Write 4 Sent Done Done false 5746 Acked Read 8 Sent Done Done false 5747 Acked Read 3 Sent Done Done false 5748 Acked Read 1 Sent Done Done false 5749 Acked Write 3 Sent Done Done false 5750 Acked Read 2 Sent Done Done false 5751 Acked Read 5 Sent Done Done false 5752 Acked Write 10 Sent Done Done false 5753 Acked Flush 0 Sent Done Done false 5754 Acked Read 4 Sent Done Done false 5755 Acked Read 7 Sent Done Done false 5756 Acked Write 1 Sent Done Done false 5757 Acked Read 1 Sent Done Done false 5758 Acked Read 10 Sent Done Done false 5759 Acked Read 4 Sent Done Done false 5760 Acked Read 9 Sent Done Done false 5761 Acked Write 1 Sent Done Done false 5762 Acked Read 3 Sent Done Done false 5763 Acked Read 7 Sent Done Done false 5764 Acked Read 4 Sent Done Done false 5765 Acked Write 6 Sent Done Done false 5766 Acked Write 4 Sent Done Done false 5767 Acked Read 2 Sent Done Done false 5768 Acked Read 3 Sent Done Done false 5769 Acked Read 2 Sent Done Done false 5770 Acked Write 1 Sent Done Done false 5771 Acked Write 2 Sent Done Done false 5772 Acked Read 6 Sent Done Done false 5773 Acked Flush 0 Sent Done Done false 5774 Acked Flush 0 Sent Done Done false 5775 Acked Read 8 Sent Done Done false 5776 Acked Write 9 Sent Done Done false 5777 Acked Write 5 Sent Done Done false 5778 Acked Read 10 Sent Done Done false 5779 Acked Write 3 Sent Done Done false 5780 Acked Flush 0 Sent Done Done false 5781 Acked Write 4 Sent Done Done false 5782 Acked Write 10 Sent Done Done false 5783 Acked Flush 0 Sent Done Done false 5784 Acked Flush 0 Sent Done Done false 5785 Acked Read 7 Sent Done Done false 5786 Acked Write 5 Sent Done Done false 5787 Acked Read 1 Sent Done Done false 5788 Acked Read 2 Sent Done Done false 5789 Acked Read 1 Sent Done Done false 5790 Acked Read 3 Sent Done Done false 5791 Acked Read 5 Sent Done Done false 5792 Acked Write 1 Sent Done Done false 5793 Acked Read 10 Sent Done Done false 5794 Acked Read 10 Sent Done Done false 5795 Acked Read 9 Sent Done Done false 5796 Acked Read 4 Sent Done Done false 5797 Acked Write 6 Sent Done Done false 5798 Acked Flush 0 Sent Done Done false 5799 Acked Read 3 Sent Done Done false 5800 Acked Read 7 Sent Done Done false 5801 Acked Read 9 Sent Done Done false 5802 Acked Write 3 Sent Done Done false 5803 Acked Flush 0 Sent Done Done false 5804 Acked Write 5 Sent Done Done false 5805 Acked Read 7 Sent Done Done false 5806 Acked Write 1 Sent Done Done false 5807 Acked Write 4 Sent Done Done false 5808 Acked Read 10 Sent Done Done false 5809 Acked Read 9 Sent Done Done false 5810 Acked Write 8 Sent Done Done false 5811 Acked Write 5 Sent Done Done false 5812 Acked Read 9 Sent Done Done false 5813 Acked Read 2 Sent Done Done false 5814 Acked Read 9 Sent Done Done false 5815 Acked Write 4 Sent Done Done false 5816 Acked Flush 0 Sent Done Done false 5817 Acked Read 8 Sent Done Done false 5818 Acked Write 6 Sent Done Done false 5819 Acked Read 8 Sent Done Done false 5820 Acked Write 5 Sent Done Done false 5821 Acked Read 3 Sent Done Done false 5822 Acked Read 1 Sent Done Done false 5823 Acked Write 6 Sent Done Done false 5824 Acked Read 3 Sent Done Done false 5825 Acked Read 9 Sent Done Done false 5826 Acked Flush 0 Sent Done Done false 5827 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5526 5827 5827 Downstairs last five completed: 5526 5525 5524 5523 5522 Upstairs last five completed: 5827 5826 5825 5824 5823 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:41.344958067Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:41.345025358Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:41.345048952Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:41.345069527Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:41.345166411Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5527 Acked Flush 0 Sent Done Done false 5528 Acked Write 1 Sent Done Done false 5529 Acked Write 5 Sent Done Done false 5530 Acked Flush 0 Sent Done Done false 5531 Acked Read 4 Sent Done Done false 5532 Acked Read 1 Sent Done Done false 5533 Acked Read 7 Sent Done Done false 5534 Acked Write 1 Sent Done Done false 5535 Acked Read 9 Sent Done Done false 5536 Acked Read 10 Sent Done Done false 5537 Acked Read 6 Sent Done Done false 5538 Acked Write 6 Sent Done Done false 5539 Acked Read 6 Sent Done Done false 5540 Acked Read 10 Sent Done Done false 5541 Acked Read 8 Sent Done Done false 5542 Acked Read 6 Sent Done Done false 5543 Acked Read 8 Sent Done Done false 5544 Acked Read 3 Sent Done Done false 5545 Acked Read 6 Sent Done Done false 5546 Acked Read 7 Sent Done Done false 5547 Acked Read 4 Sent Done Done false 5548 Acked Read 10 Sent Done Done false 5549 Acked Read 10 Sent Done Done false 5550 Acked Flush 0 Sent Done Done false 5551 Acked Read 10 Sent Done Done false 5552 Acked Write 3 Sent Done Done false 5553 Acked Write 4 Sent Done Done false 5554 Acked Write 9 Sent Done Done false 5555 Acked Write 8 Sent Done Done false 5556 Acked Write 10 Sent Done Done false 5557 Acked Read 6 Sent Done Done false 5558 Acked Read 6 Sent Done Done false 5559 Acked Write 1 Sent Done Done false 5560 Acked Write 9 Sent Done Done false 5561 Acked Write 7 Sent Done Done false 5562 Acked Write 2 Sent Done Done false 5563 Acked Write 10 Sent Done Done false 5564 Acked Read 3 Sent Done Done false 5565 Acked Write 3 Sent Done Done false 5566 Acked Write 5 Sent Done Done false 5567 Acked Read 1 Sent Done Done false 5568 Acked Read 10 Sent Done Done false 5569 Acked Write 6 Sent Done Done false 5570 Acked Write 6 Sent Done Done false 5571 Acked Write 5 Sent Done Done false 5572 Acked Read 9 Sent Done Done false 5573 Acked Write 10 Sent Done Done false 5574 Acked Write 5 Sent Done Done false 5575 Acked Flush 0 Sent Done Done false 5576 Acked Read 7 Sent Done Done false 5577 Acked Write 10 Sent Done Done false 5578 Acked Write 9 Sent Done Done false 5579 Acked Flush 0 Sent Done Done false 5580 Acked Read 1 Sent Done Done false 5581 Acked Write 5 Sent Done Done false 5582 Acked Write 2 Sent Done Done false 5583 Acked Read 3 Sent Done Done false 5584 Acked Write 7 Sent Done Done false 5585 Acked Write 5 Sent Done Done false 5586 Acked Read 1 Sent Done Done false 5587 Acked Read 8 Sent Done Done false 5588 Acked Write 10 Sent Done Done false 5589 Acked Write 1 Sent Done Done false 5590 Acked Read 8 Sent Done Done false 5591 Acked Read 4 Sent Done Done false 5592 Acked Read 4 Sent Done Done false 5593 Acked Read 6 Sent Done Done false 5594 Acked Read 5 Sent Done Done false 5595 Acked Write 9 Sent Done Done false 5596 Acked Read 7 Sent Done Done false 5597 Acked Read 2 Sent Done Done false 5598 Acked Read 7 Sent Done Done false 5599 Acked Read 1 Sent Done Done false 5600 Acked Read 8 Sent Done Done false 5601 Acked Write 3 Sent Done Done false 5602 Acked Write 2 Sent Done Done false 5603 Acked Write 10 Sent Done Done false 5604 Acked Read 4 Sent Done Done false 5605 Acked Write 2 Sent Done Done false 5606 Acked Flush 0 Sent Done Done false 5607 Acked Write 8 Sent Done Done false 5608 Acked Read 1 Sent Done Done false 5609 Acked Write 1 Sent Done Done false 5610 Acked Read 9 Sent Done Done false 5611 Acked Read 8 Sent Done Done false 5612 Acked Write 3 Sent Done Done false 5613 Acked Read 5 Sent Done Done false 5614 Acked Read 1 Sent Done Done false 5615 Acked Read 5 Sent Done Done false 5616 Acked Flush 0 Sent Done Done false 5617 Acked Read 6 Sent Done Done false 5618 Acked Read 9 Sent Done Done false 5619 Acked Read 2 Sent Done Done false 5620 Acked Read 7 Sent Done Done false 5621 Acked Read 1 Sent Done Done false 5622 Acked Write 1 Sent Done Done false 5623 Acked Read 8 Sent Done Done false 5624 Acked Write 5 Sent Done Done false 5625 Acked Write 8 Sent Done Done false 5626 Acked Write 4 Sent Done Done false 5627 Acked Write 1 Sent Done Done false 5628 Acked Write 5 Sent Done Done false 5629 Acked Write 7 Sent Done Done false 5630 Acked Write 5 Sent Done Done false 5631 Acked Write 2 Sent Done Done false 5632 Acked Read 9 Sent Done Done false 5633 Acked Flush 0 Sent Done Done false 5634 Acked Read 3 Sent Done Done false 5635 Acked Read 1 Sent Done Done false 5636 Acked Read 8 Sent Done Done false 5637 Acked Read 4 Sent Done Done false 5638 Acked Read 9 Sent Done Done false 5639 Acked Write 8 Sent Done Done false 5640 Acked Read 10 Sent Done Done false 5641 Acked Flush 0 Sent Done Done false 5642 Acked Write 3 Sent Done Done false 5643 Acked Read 1 Sent Done Done false 5644 Acked Read 8 Sent Done Done false 5645 Acked Read 6 Sent Done Done false 5646 Acked Read 1 Sent Done Done false 5647 Acked Write 10 Sent Done Done false 5648 Acked Read 3 Sent Done Done false 5649 Acked Read 7 Sent Done Done false 5650 Acked Write 4 Sent Done Done false 5651 Acked Flush 0 Sent Done Done false 5652 Acked Read 9 Sent Done Done false 5653 Acked Write 8 Sent Done Done false 5654 Acked Read 3 Sent Done Done false 5655 Acked Write 8 Sent Done Done false 5656 Acked Read 2 Sent Done Done false 5657 Acked Write 6 Sent Done Done false 5658 Acked Read 8 Sent Done Done false 5659 Acked Write 6 Sent Done Done false 5660 Acked Write 2 Sent Done Done false 5661 Acked Read 2 Sent Done Done false 5662 Acked Read 4 Sent Done Done false 5663 Acked Read 7 Sent Done Done false 5664 Acked Read 7 Sent Done Done false 5665 Acked Write 5 Sent Done Done false 5666 Acked Read 3 Sent Done Done false 5667 Acked Write 6 Sent Done Done false 5668 Acked Write 2 Sent Done Done false 5669 Acked Write 5 Sent Done Done false 5670 Acked Flush 0 Sent Done Done false 5671 Acked Read 5 Sent Done Done false 5672 Acked Flush 0 Sent Done Done false 5673 Acked Write 5 Sent Done Done false 5674 Acked Write 8 Sent Done Done false 5675 Acked Read 4 Sent Done Done false 5676 Acked Write 2 Sent Done Done false 5677 Acked Write 10 Sent Done Done false 5678 Acked Read 9 Sent Done Done false 5679 Acked Read 1 Sent Done Done false 5680 Acked Flush 0 Sent Done Done false 5681 Acked Write 9 Sent Done Done false 5682 Acked Write 1 Sent Done Done false 5683 Acked Write 1 Sent Done Done false 5684 Acked Flush 0 Sent Done Done false 5685 Acked Read 3 Sent Done Done false 5686 Acked Write 9 Sent Done Done false 5687 Acked Read 9 Sent Done Done false 5688 Acked Read 9 Sent Done Done false 5689 Acked Read 9 Sent Done Done false 5690 Acked Write 8 Sent Done Done false 5691 Acked Read 2 Sent Done Done false 5692 Acked Write 2 Sent Done Done false 5693 Acked Write 1 Sent Done Done false 5694 Acked Read 10 Sent Done Done false 5695 Acked Read 8 Sent Done Done false 5696 Acked Write 4 Sent Done Done false 5697 Acked Write 7 Sent Done Done false 5698 Acked Write 5 Sent Done Done false 5699 Acked Read 7 Sent Done Done false 5700 Acked Read 7 Sent Done Done false 5701 Acked Write 5 Sent Done Done false 5702 Acked Write 8 Sent Done Done false 5703 Acked Read 10 Sent Done Done false 5704 Acked Read 10 Sent Done Done false 5705 Acked Write 2 Sent Done Done false 5706 Acked Read 7 Sent Done Done false 5707 Acked Write 4 Sent Done Done false 5708 Acked Write 1 Sent Done Done false 5709 Acked Read 3 Sent Done Done false 5710 Acked Read 6 Sent Done Done false 5711 Acked Read 7 Sent Done Done false 5712 Acked Flush 0 Sent Done Done false 5713 Acked Read 7 Sent Done Done false 5714 Acked Flush 0 Sent Done Done false 5715 Acked Write 2 Sent Done Done false 5716 Acked Read 2 Sent Done Done false 5717 Acked Read 1 Sent Done Done false 5718 Acked Read 8 Sent Done Done false 5719 Acked Read 6 Sent Done Done false 5720 Acked Flush 0 Sent Done Done false 5721 Acked Write 8 Sent Done Done false 5722 Acked Read 8 Sent Done Done false 5723 Acked Read 7 Sent Done Done false 5724 Acked Read 3 Sent Done Done false 5725 Acked Write 8 Sent Done Done false 5726 Acked Read 5 Sent Done Done false 5727 Acked Write 9 Sent Done Done false 5728 Acked Write 9 Sent Done Done false 5729 Acked Flush 0 Sent Done Done false 5730 Acked Read 2 Sent Done Done false 5731 Acked Write 5 Sent Done Done false 5732 Acked Write 1 Sent Done Done false 5733 Acked Flush 0 Sent Done Done false 5734 Acked Write 9 Sent Done Done false 5735 Acked Read 1 Sent Done Done false 5736 Acked Read 1 Sent Done Done false 5737 Acked Read 2 Sent Done Done false 5738 Acked Write 6 Sent Done Done false 5739 Acked Write 10 Sent Done Done false 5740 Acked Write 7 Sent Done Done false 5741 Acked Read 1 Sent Done Done false 5742 Acked Write 9 Sent Done Done false 5743 Acked Read 3 Sent Done Done false 5744 Acked Read 2 Sent Done Done false 5745 Acked Write 4 Sent Done Done false 5746 Acked Read 8 Sent Done Done false 5747 Acked Read 3 Sent Done Done false 5748 Acked Read 1 Sent Done Done false 5749 Acked Write 3 Sent Done Done false 5750 Acked Read 2 Sent Done Done false 5751 Acked Read 5 Sent Done Done false 5752 Acked Write 10 Sent Done Done false 5753 Acked Flush 0 Sent Done Done false 5754 Acked Read 4 Sent Done Done false 5755 Acked Read 7 Sent Done Done false 5756 Acked Write 1 Sent Done Done false 5757 Acked Read 1 Sent Done Done false 5758 Acked Read 10 Sent Done Done false 5759 Acked Read 4 Sent Done Done false 5760 Acked Read 9 Sent Done Done false 5761 Acked Write 1 Sent Done Done false 5762 Acked Read 3 Sent Done Done false 5763 Acked Read 7 Sent Done Done false 5764 Acked Read 4 Sent Done Done false 5765 Acked Write 6 Sent Done Done false 5766 Acked Write 4 Sent Done Done false 5767 Acked Read 2 Sent Done Done false 5768 Acked Read 3 Sent Done Done false 5769 Acked Read 2 Sent Done Done false 5770 Acked Write 1 Sent Done Done false 5771 Acked Write 2 Sent Done Done false 5772 Acked Read 6 Sent Done Done false 5773 Acked Flush 0 Sent Done Done false 5774 Acked Flush 0 Sent Done Done false 5775 Acked Read 8 Sent Done Done false 5776 Acked Write 9 Sent Done Done false 5777 Acked Write 5 Sent Done Done false 5778 Acked Read 10 Sent Done Done false 5779 Acked Write 3 Sent Done Done false 5780 Acked Flush 0 Sent Done Done false 5781 Acked Write 4 Sent Done Done false 5782 Acked Write 10 Sent Done Done false 5783 Acked Flush 0 Sent Done Done false 5784 Acked Flush 0 Sent Done Done false 5785 Acked Read 7 Sent Done Done false 5786 Acked Write 5 Sent Done Done false 5787 Acked Read 1 Sent Done Done false 5788 Acked Read 2 Sent Done Done false 5789 Acked Read 1 Sent Done Done false 5790 Acked Read 3 Sent Done Done false 5791 Acked Read 5 Sent Done Done false 5792 Acked Write 1 Sent Done Done false 5793 Acked Read 10 Sent Done Done false 5794 Acked Read 10 Sent Done Done false 5795 Acked Read 9 Sent Done Done false 5796 Acked Read 4 Sent Done Done false 5797 Acked Write 6 Sent Done Done false 5798 Acked Flush 0 Sent Done Done false 5799 Acked Read 3 Sent Done Done false 5800 Acked Read 7 Sent Done Done false 5801 Acked Read 9 Sent Done Done false 5802 Acked Write 3 Sent Done Done false 5803 Acked Flush 0 Sent Done Done false 5804 Acked Write 5 Sent Done Done false 5805 Acked Read 7 Sent Done Done false 5806 Acked Write 1 Sent Done Done false 5807 Acked Write 4 Sent Done Done false 5808 Acked Read 10 Sent Done Done false 5809 Acked Read 9 Sent Done Done false 5810 Acked Write 8 Sent Done Done false 5811 Acked Write 5 Sent Done Done false 5812 Acked Read 9 Sent Done Done false 5813 Acked Read 2 Sent Done Done false 5814 Acked Read 9 Sent Done Done false 5815 Acked Write 4 Sent Done Done false 5816 Acked Flush 0 Sent Done Done false 5817 Acked Read 8 Sent Done Done false 5818 Acked Write 6 Sent Done Done false 5819 Acked Read 8 Sent Done Done false 5820 Acked Write 5 Sent Done Done false 5821 Acked Read 3 Sent Done Done false 5822 Acked Read 1 Sent Done Done false 5823 Acked Write 6 Sent Done Done false 5824 Acked Read 3 Sent Done Done false 5825 Acked Read 9 Sent Done Done false 5826 Acked Flush 0 Sent Done Done false 5827 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5526 5827 5827 Downstairs last five completed: 5526 5525 5524 5523 5522 Upstairs last five completed: 5827 5826 5825 5824 5823 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5527 Acked Flush 0 Sent Done Done false 5528 Acked Write 1 Sent Done Done false 5529 Acked Write 5 Sent Done Done false 5530 Acked Flush 0 Sent Done Done false 5531 Acked Read 4 Sent Done Done false 5532 Acked Read 1 Sent Done Done false 5533 Acked Read 7 Sent Done Done false 5534 Acked Write 1 Sent Done Done false 5535 Acked Read 9 Sent Done Done false 5536 Acked Read 10 Sent Done Done false 5537 Acked Read 6 Sent Done Done false 5538 Acked Write 6 Sent Done Done false 5539 Acked Read 6 Sent Done Done false 5540 Acked Read 10 Sent Done Done false 5541 Acked Read 8 Sent Done Done false 5542 Acked Read 6 Sent Done Done false 5543 Acked Read 8 Sent Done Done false 5544 Acked Read 3 Sent Done Done false 5545 Acked Read 6 Sent Done Done false 5546 Acked Read 7 Sent Done Done false 5547 Acked Read 4 Sent Done Done false 5548 Acked Read 10 Sent Done Done false 5549 Acked Read 10 Sent Done Done false 5550 Acked Flush 0 Sent Done Done false 5551 Acked Read 10 Sent Done Done false 5552 Acked Write 3 Sent Done Done false 5553 Acked Write 4 Sent Done Done false 5554 Acked Write 9 Sent Done Done false 5555 Acked Write 8 Sent Done Done false 5556 Acked Write 10 Sent Done Done false 5557 Acked Read 6 Sent Done Done false 5558 Acked Read 6 Sent Done Done false 5559 Acked Write 1 Sent Done Done false 5560 Acked Write 9 Sent Done Done false 5561 Acked Write 7 Sent Done Done false 5562 Acked Write 2 Sent Done Done false 5563 Acked Write 10 Sent Done Done false 5564 Acked Read 3 Sent Done Done false 5565 Acked Write 3 Sent Done Done false 5566 Acked Write 5 Sent Done Done false 5567 Acked Read 1 Sent Done Done false 5568 Acked Read 10 Sent Done Done false 5569 Acked Write 6 Sent Done Done false 5570 Acked Write 6 Sent Done Done false 5571 Acked Write 5 Sent Done Done false 5572 Acked Read 9 Sent Done Done false 5573 Acked Write 10 Sent Done Done false 5574 Acked Write 5 Sent Done Done false 5575 Acked Flush 0 Sent Done Done false 5576 Acked Read 7 Sent Done Done false 5577 Acked Write 10 Sent Done Done false 5578 Acked Write 9 Sent Done Done false 5579 Acked Flush 0 Sent Done Done false 5580 Acked Read 1 Sent Done Done false 5581 Acked Write 5 Sent Done Done false 5582 Acked Write 2 Sent Done Done false 5583 Acked Read 3 Sent Done Done false 5584 Acked Write 7 Sent Done Done false 5585 Acked Write 5 Sent Done Done false 5586 Acked Read 1 Sent Done Done false 5587 Acked Read 8 Sent Done Done false 5588 Acked Write 10 Sent Done Done false 5589 Acked Write 1 Sent Done Done false 5590 Acked Read 8 Sent Done Done false 5591 Acked Read 4 Sent Done Done false 5592 Acked Read 4 Sent Done Done false 5593 Acked Read 6 Sent Done Done false 5594 Acked Read 5 Sent Done Done false 5595 Acked Write 9 Sent Done Done false 5596 Acked Read 7 Sent Done Done false 5597 Acked Read 2 Sent Done Done false 5598 Acked Read 7 Sent Done Done false 5599 Acked Read 1 Sent Done Done false 5600 Acked Read 8 Sent Done Done false 5601 Acked Write 3 Sent Done Done false 5602 Acked Write 2 Sent Done Done false 5603 Acked Write 10 Sent Done Done false 5604 Acked Read 4 Sent Done Done false 5605 Acked Write 2 Sent Done Done false 5606 Acked Flush 0 Sent Done Done false 5607 Acked Write 8 Sent Done Done false 5608 Acked Read 1 Sent Done Done false 5609 Acked Write 1 Sent Done Done false 5610 Acked Read 9 Sent Done Done false 5611 Acked Read 8 Sent Done Done false 5612 Acked Write 3 Sent Done Done false 5613 Acked Read 5 Sent Done Done false 5614 Acked Read 1 Sent Done Done false 5615 Acked Read 5 Sent Done Done false 5616 Acked Flush 0 Sent Done Done false 5617 Acked Read 6 Sent Done Done false 5618 Acked Read 9 Sent Done Done false 5619 Acked Read 2 Sent Done Done false 5620 Acked Read 7 Sent Done Done false 5621 Acked Read 1 Sent Done Done false 5622 Acked Write 1 Sent Done Done false 5623 Acked Read 8 Sent Done Done false 5624 Acked Write 5 Sent Done Done false 5625 Acked Write 8 Sent Done Done false 5626 Acked Write 4 Sent Done Done false 5627 Acked Write 1 Sent Done Done false 5628 Acked Write 5 Sent Done Done false 5629 Acked Write 7 Sent Done Done false 5630 Acked Write 5 Sent Done Done false 5631 Acked Write 2 Sent Done Done false 5632 Acked Read 9 Sent Done Done false 5633 Acked Flush 0 Sent Done Done false 5634 Acked Read 3 Sent Done Done false 5635 Acked Read 1 Sent Done Done false 5636 Acked Read 8 Sent Done Done false 5637 Acked Read 4 Sent Done Done false 5638 Acked Read 9 Sent Done Done false 5639 Acked Write 8 Sent Done Done false 5640 Acked Read 10 Sent Done Done false 5641 Acked Flush 0 Sent Done Done false 5642 Acked Write 3 Sent Done Done false 5643 Acked Read 1 Sent Done Done false 5644 Acked Read 8 Sent Done Done false 5645 Acked Read 6 Sent Done Done false 5646 Acked Read 1 Sent Done Done false 5647 Acked Write 10 Sent Done Done false 5648 Acked Read 3 Sent Done Done false 5649 Acked Read 7 Sent Done Done false 5650 Acked Write 4 Sent Done Done false 5651 Acked Flush 0 Sent Done Done false 5652 Acked Read 9 Sent Done Done false 5653 Acked Write 8 Sent Done Done false 5654 Acked Read 3 Sent Done Done false 5655 Acked Write 8 Sent Done Done false 5656 Acked Read 2 Sent Done Done false 5657 Acked Write 6 Sent Done Done false 5658 Acked Read 8 Sent Done Done false 5659 Acked Write 6 Sent Done Done false 5660 Acked Write 2 Sent Done Done false 5661 Acked Read 2 Sent Done Done false 5662 Acked Read 4 Sent Done Done false 5663 Acked Read 7 Sent Done Done false 5664 Acked Read 7 Sent Done Done false 5665 Acked Write 5 Sent Done Done false 5666 Acked Read 3 Sent Done Done false 5667 Acked Write 6 Sent Done Done false 5668 Acked Write 2 Sent Done Done false 5669 Acked Write 5 Sent Done Done false 5670 Acked Flush 0 Sent Done Done false 5671 Acked Read 5 Sent Done Done false 5672 Acked Flush 0 Sent Done Done false 5673 Acked Write 5 Sent Done Done false 5674 Acked Write 8 Sent Done Done false 5675 Acked Read 4 Sent Done Done false 5676 Acked Write 2 Sent Done Done false 5677 Acked Write 10 Sent Done Done false 5678 Acked Read 9 Sent Done Done false 5679 Acked Read 1 Sent Done Done false 5680 Acked Flush 0 Sent Done Done false 5681 Acked Write 9 Sent Done Done false 5682 Acked Write 1 Sent Done Done false 5683 Acked Write 1 Sent Done Done false 5684 Acked Flush 0 Sent Done Done false 5685 Acked Read 3 Sent Done Done false 5686 Acked Write 9 Sent Done Done false 5687 Acked Read 9 Sent Done Done false 5688 Acked Read 9 Sent Done Done false 5689 Acked Read 9 Sent Done Done false 5690 Acked Write 8 Sent Done Done false 5691 Acked Read 2 Sent Done Done false 5692 Acked Write 2 Sent Done Done false 5693 Acked Write 1 Sent Done Done false 5694 Acked Read 10 Sent Done Done false 5695 Acked Read 8 Sent Done Done false 5696 Acked Write 4 Sent Done Done false 5697 Acked Write 7 Sent Done Done false 5698 Acked Write 5 Sent Done Done false 5699 Acked Read 7 Sent Done Done false 5700 Acked Read 7 Sent Done Done false 5701 Acked Write 5 Sent Done Done false 5702 Acked Write 8 Sent Done Done false 5703 Acked Read 10 Sent Done Done false 5704 Acked Read 10 Sent Done Done false 5705 Acked Write 2 Sent Done Done false 5706 Acked Read 7 Sent Done Done false 5707 Acked Write 4 Sent Done Done false 5708 Acked Write 1 Sent Done Done false 5709 Acked Read 3 Sent Done Done false 5710 Acked Read 6 Sent Done Done false 5711 Acked Read 7 Sent Done Done false 5712 Acked Flush 0 Sent Done Done false 5713 Acked Read 7 Sent Done Done false 5714 Acked Flush 0 Sent Done Done false 5715 Acked Write 2 Sent Done Done false 5716 Acked Read 2 Sent Done Done false 5717 Acked Read 1 Sent Done Done false 5718 Acked Read 8 Sent Done Done false 5719 Acked Read 6 Sent Done Done false 5720 Acked Flush 0 Sent Done Done false 5721 Acked Write 8 Sent Done Done false 5722 Acked Read 8 Sent Done Done false 5723 Acked Read 7 Sent Done Done false 5724 Acked Read 3 Sent Done Done false 5725 Acked Write 8 Sent Done Done false 5726 Acked Read 5 Sent Done Done false 5727 Acked Write 9 Sent Done Done false 5728 Acked Write 9 Sent Done Done false 5729 Acked Flush 0 Sent Done Done false 5730 Acked Read 2 Sent Done Done false 5731 Acked Write 5 Sent Done Done false 5732 Acked Write 1 Sent Done Done false 5733 Acked Flush 0 Sent Done Done false 5734 Acked Write 9 Sent Done Done false 5735 Acked Read 1 Sent Done Done false 5736 Acked Read 1 Sent Done Done false 5737 Acked Read 2 Sent Done Done false 5738 Acked Write 6 Sent Done Done false 5739 Acked Write 10 Sent Done Done false 5740 Acked Write 7 Sent Done Done false 5741 Acked Read 1 Sent Done Done false 5742 Acked Write 9 Sent Done Done false 5743 Acked Read 3 Sent Done Done false 5744 Acked Read 2 Sent Done Done false 5745 Acked Write 4 Sent Done Done false 5746 Acked Read 8 Sent Done Done false 5747 Acked Read 3 Sent Done Done false 5748 Acked Read 1 Sent Done Done false 5749 Acked Write 3 Sent Done Done false 5750 Acked Read 2 Sent Done Done false 5751 Acked Read 5 Sent Done Done false 5752 Acked Write 10 Sent Done Done false 5753 Acked Flush 0 Sent Done Done false 5754 Acked Read 4 Sent Done Done false 5755 Acked Read 7 Sent Done Done false 5756 Acked Write 1 Sent Done Done false 5757 Acked Read 1 Sent Done Done false 5758 Acked Read 10 Sent Done Done false 5759 Acked Read 4 Sent Done Done false 5760 Acked Read 9 Sent Done Done false 5761 Acked Write 1 Sent Done Done false 5762 Acked Read 3 Sent Done Done false 5763 Acked Read 7 Sent Done Done false 5764 Acked Read 4 Sent Done Done false 5765 Acked Write 6 Sent Done Done false 5766 Acked Write 4 Sent Done Done false 5767 Acked Read 2 Sent Done Done false 5768 Acked Read 3 Sent Done Done false 5769 Acked Read 2 Sent Done Done false 5770 Acked Write 1 Sent Done Done false 5771 Acked Write 2 Sent Done Done false 5772 Acked Read 6 Sent Done Done false 5773 Acked Flush 0 Sent Done Done false 5774 Acked Flush 0 Sent Done Done false 5775 Acked Read 8 Sent Done Done false 5776 Acked Write 9 Sent Done Done false 5777 Acked Write 5 Sent Done Done false 5778 Acked Read 10 Sent Done Done false 5779 Acked Write 3 Sent Done Done false 5780 Acked Flush 0 Sent Done Done false 5781 Acked Write 4 Sent Done Done false 5782 Acked Write 10 Sent Done Done false 5783 Acked Flush 0 Sent Done Done false 5784 Acked Flush 0 Sent Done Done false 5785 Acked Read 7 Sent Done Done false 5786 Acked Write 5 Sent Done Done false 5787 Acked Read 1 Sent Done Done false 5788 Acked Read 2 Sent Done Done false 5789 Acked Read 1 Sent Done Done false 5790 Acked Read 3 Sent Done Done false 5791 Acked Read 5 Sent Done Done false 5792 Acked Write 1 Sent Done Done false 5793 Acked Read 10 Sent Done Done false 5794 Acked Read 10 Sent Done Done false 5795 Acked Read 9 Sent Done Done false 5796 Acked Read 4 Sent Done Done false 5797 Acked Write 6 Sent Done Done false 5798 Acked Flush 0 Sent Done Done false 5799 Acked Read 3 Sent Done Done false 5800 Acked Read 7 Sent Done Done false 5801 Acked Read 9 Sent Done Done false 5802 Acked Write 3 Sent Done Done false 5803 Acked Flush 0 Sent Done Done false 5804 Acked Write 5 Sent Done Done false 5805 Acked Read 7 Sent Done Done false 5806 Acked Write 1 Sent Done Done false 5807 Acked Write 4 Sent Done Done false 5808 Acked Read 10 Sent Done Done false 5809 Acked Read 9 Sent Done Done false 5810 Acked Write 8 Sent Done Done false 5811 Acked Write 5 Sent Done Done false 5812 Acked Read 9 Sent Done Done false 5813 Acked Read 2 Sent Done Done false 5814 Acked Read 9 Sent Done Done false 5815 Acked Write 4 Sent Done Done false 5816 Acked Flush 0 Sent Done Done false 5817 Acked Read 8 Sent Done Done false 5818 Acked Write 6 Sent Done Done false 5819 Acked Read 8 Sent Done Done false 5820 Acked Write 5 Sent Done Done false 5821 Acked Read 3 Sent Done Done false 5822 Acked Read 1 Sent Done Done false 5823 Acked Write 6 Sent Done Done false 5824 Acked Read 3 Sent Done Done false 5825 Acked Read 9 Sent Done Done false 5826 Acked Flush 0 Sent Done Done false 5827 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5526 5827 5827 Downstairs last five completed: 5526 5525 5524 5523 5522 Upstairs last five completed: 5827 5826 5825 5824 5823 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.346455105Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:58707)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.346527575Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:51.347541893Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348057484Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348083077Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 5526","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348098033Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 5526","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348308857Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348337469Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348352745Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 301 jobs since flush 5526","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348366911Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 301 jobs since flush: 5526","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.348563559Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5827 5826 5825 5824 5823 Upstairs last five completed: 5827 5826 5825 5824 5823 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 Nov 27 01:45:51.586[8810] Got stop action so:false kr:false INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: dfce71d2-4eee-4e37-bf0a-98702c0897b3, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:51.587333769Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:51.587384695Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:45:51.587417106Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.587444809Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.587472591Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:45:51.587501454Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:45:51.587 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4c3978ed-f336-423f-9569-9824d2c2b3c5, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:45:55.588 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: d211bec5-80ca-415c-84a2-74b200d71d4f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 73 len 8192 002/300 Read block 4041 len 12288 003/300 Read block 1106 len 36864 004/300 Write block 4565 len 28672 data: 8 7 7 7 6 5 5 005/300 Write block 2857 len 28672 data: 4 5 6 5 4 3 2 006/300 Flush 007/300 Write block 4716 len 32768 data: 3 3 3 4 4 3 4 4 008/300 Write block 4385 len 28672 data: 4 4 4 5 8 8 6 009/300 Read block 4819 len 20480 010/300 Write block 1624 len 36864 data: 3 3 4 4 5 5 5 5 5 011/300 Write block 773 len 20480 data: 4 4 4 4 4 012/300 Read block 1530 len 28672 013/300 Flush 014/300 Write block 370 len 4096 data: 6 015/300 Read block 3986 len 40960 016/300 Flush 017/300 Write block 3697 len 36864 data: 6 4 6 7 6 6 5 4 4 018/300 Write block 2552 len 36864 data: 5 5 4 5 6 6 6 6 7 019/300 Write block 2741 len 40960 data: 6 5 5 3 4 4 4 2 2 2 020/300 Write block 2116 len 32768 data: 5 5 5 5 5 4 3 2 021/300 Write block 4156 len 40960 data: 4 3 3 2 2 3 3 3 3 3 022/300 Read block 1994 len 12288 023/300 Read block 2445 len 20480 024/300 Write block 937 len 40960 data: 4 4 5 5 5 5 4 4 3 4 025/300 Read block 3258 len 24576 026/300 Read block 4508 len 20480 027/300 Read block 2383 len 24576 028/300 Write block 1464 len 20480 data: 4 4 3 6 8 029/300 Read block 1735 len 16384 030/300 Read block 2049 len 24576 031/300 Write block 3926 len 24576 data: 2 3 3 2 2 2 032/300 Write block 1347 len 16384 data: 4 3 2 2 033/300 Read block 2178 len 32768 034/300 Read block 4443 len 32768 035/300 Write block 4070 len 40960 data: 5 4 5 5 5 5 4 4 4 5 036/300 Write block 989 len 28672 data: 4 4 3 4 4 4 4 037/300 Write block 3599 len 16384 data: 4 4 4 4 038/300 Read block 2862 len 32768 039/300 Read block 4099 len 40960 040/300 Flush 041/300 Write block 935 len 4096 data: 3 042/300 Read block 1697 len 24576 043/300 Read block 1615 len 40960 044/300 Read block 4728 len 16384 045/300 Flush 046/300 Read block 2012 len 36864 047/300 Read block 4875 len 12288 048/300 Read block 2770 len 16384 049/300 Read block 303 len 16384 050/300 Read block 2374 len 16384 051/300 Read block 1270 len 36864 052/300 Write block 4434 len 20480 data: 6 6 5 5 5 053/300 Flush 054/300 Flush 055/300 Read block 689 len 32768 056/300 Flush 057/300 Read block 2483 len 28672 058/300 Read block 4059 len 16384 059/300 Read block 4069 len 8192 060/300 Read block 2948 len 8192 061/300 Read block 2773 len 24576 062/300 Write block 4874 len 12288 data: 2 2 3 063/300 Read block 758 len 16384 064/300 Read block 4276 len 24576 065/300 Read block 109 len 20480 066/300 Read block 2281 len 8192 067/300 Write block 4480 len 12288 data: 5 5 4 068/300 Read block 2033 len 8192 069/300 Read block 2347 len 40960 070/300 Read block 0 len 16384 071/300 Write block 2525 len 20480 data: 4 5 6 7 7 072/300 Write block 1102 len 4096 data: 4 073/300 Write block 4377 len 8192 data: 3 3 074/300 Write block 1307 len 12288 data: 2 3 3 075/300 Read block 779 len 36864 076/300 Read block 4929 len 28672 077/300 Read block 1300 len 4096 078/300 Read block 4067 len 12288 079/300 Write block 1358 len 8192 data: 3 3 080/300 Write block 226 len 32768 data: 5 5 5 4 5 5 6 5 081/300 Read block 4176 len 20480 082/300 Write block 2301 len 20480 data: 3 4 5 4 6 083/300 Write block 728 len 36864 data: 4 4 4 4 4 3 2 2 2 084/300 Write block 4191 len 24576 data: 2 2 2 3 4 5 085/300 Read block 3152 len 24576 086/300 Write block 1514 len 4096 data: 3 087/300 Write block 2078 len 32768 data: 4 4 3 3 3 3 3 3 088/300 Read block 1617 len 4096 089/300 Read block 806 len 40960 090/300 Write block 4698 len 28672 data: 3 4 4 4 5 4 4 091/300 Write block 2097 len 8192 data: 4 5 092/300 Write block 638 len 20480 data: 3 3 2 3 3 093/300 Write block 530 len 16384 data: 8 7 7 5 094/300 Flush 095/300 Write block 1688 len 40960 data: 5 5 5 2 3 3 3 4 4 4 096/300 Write block 3683 len 8192 data: 7 6 097/300 Read block 1861 len 16384 098/300 Read block 2951 len 36864 099/300 Write block 2307 len 40960 data: 6 5 5 5 5 5 4 3 2 2 100/300 Read block 1366 len 8192 101/300 Read block 1233 len 8192 102/300 Read block 3769 len 16384 103/300 Write block 1313 len 24576 data: 6 7 7 7 6 6 104/300 Write block 785 len 36864 data: 3 2 2 2 3 3 3 2 2 105/300 Read block 2068 len 24576 106/300 Read block 1192 len 40960 107/300 Write block 584 len 12288 data: 7 7 5 108/300 Read block 2786 len 4096 109/300 Write block 4785 len 4096 data: 3 110/300 Flush 111/300 Read block 4223 len 40960 112/300 Read block 4176 len 12288 113/300 Flush 114/300 Read block 4839 len 36864 115/300 Write block 3775 len 8192 data: 5 5 116/300 Read block 749 len 40960 117/300 Write block 3938 len 4096 data: 5 118/300 Read block 752 len 40960 119/300 Read block 1847 len 12288 120/300 Write block 959 len 24576 data: 4 4 4 4 4 4 121/300 Read block 2970 len 20480 122/300 Write block 4149 len 4096 data: 3 123/300 Read block 2293 len 16384 124/300 Write block 4313 len 36864 data: 3 4 4 4 5 4 4 3 3 125/300 Write block 4702 len 40960 data: 6 5 5 5 4 5 7 5 5 5 126/300 Read block 3116 len 40960 127/300 Read block 4432 len 32768 128/300 Write block 4250 len 32768 data: 2 2 2 2 2 4 4 4 129/300 Read block 2531 len 8192 130/300 Flush 131/300 Write block 4037 len 24576 data: 4 4 4 3 3 2 132/300 Read block 254 len 20480 133/300 Read block 2419 len 24576 134/300 Write block 2956 len 28672 data: 4 6 5 5 5 5 4 135/300 Flush 136/300 Flush 137/300 Flush 138/300 Read block 4339 len 36864 139/300 Write block 166 len 16384 data: 5 5 6 6 140/300 Flush 141/300 Flush 142/300 Write block 4041 len 40960 data: 4 3 2 2 3 3 3 3 4 5 143/300 Read block 1784 len 36864 144/300 Read block 1405 len 40960 145/300 Write block 1689 len 20480 data: 6 6 3 4 4 146/300 Read block 3448 len 12288 147/300 Write block 3218 len 32768 data: 3 4 4 4 4 4 4 5 148/300 Read block 3637 len 32768 149/300 Write block 2003 len 16384 data: 5 5 7 6 150/300 Read block 4198 len 16384 151/300 Write block 3947 len 12288 data: 4 5 6 152/300 Write block 3876 len 24576 data: 2 2 3 3 3 3 153/300 Write block 3316 len 24576 data: 5 5 5 6 5 5 154/300 Read block 4745 len 8192 155/300 Read block 3039 len 28672 156/300 Write block 1764 len 32768 data: 2 2 2 3 2 3 2 3 157/300 Flush 158/300 Read block 2438 len 12288 159/300 Read block 3062 len 36864 160/300 Write block 81 len 36864 data: 4 4 4 3 3 2 3 3 3 161/300 Read block 754 len 8192 162/300 Read block 4296 len 24576 163/300 Read block 1043 len 24576 164/300 Write block 1293 len 4096 data: 6 165/300 Read block 2393 len 12288 166/300 Write block 1573 len 36864 data: 7 6 7 7 6 4 4 4 3 167/300 Read block 3596 len 32768 168/300 Write block 591 len 32768 data: 4 4 4 4 4 4 4 2 169/300 Read block 2142 len 36864 170/300 Write block 2024 len 40960 data: 5 5 4 4 4 4 3 3 3 3 171/300 Read block 2552 len 28672 172/300 Read block 4527 len 24576 173/300 Write block 3327 len 36864 data: 5 6 7 6 6 7 6 4 3 174/300 Read block 1247 len 36864 175/300 Read block 1050 len 40960 176/300 Read block 3929 len 28672 177/300 Write block 1028 len 16384 data: 2 2 2 2 178/300 Read block 4691 len 32768 179/300 Flush 180/300 Read block 214 len 8192 181/300 Read block 1586 len 4096 182/300 Write block 3768 len 20480 data: 4 3 4 3 3 183/300 Read block 3123 len 40960 184/300 Read block 4528 len 40960 185/300 Read block 778 len 24576 186/300 Write block 2888 len 36864 data: 5 5 4 4 3 4 4 4 4 187/300 Read block 4850 len 28672 188/300 Read block 28 len 12288 189/300 Write block 3378 len 16384 data: 5 6 5 5 190/300 Read block 2981 len 20480 191/300 Write block 2210 len 24576 data: 3 2 2 2 2 3 192/300 Flush 193/300 Write block 862 len 32768 data: 5 5 5 5 6 6 6 5 194/300 Flush 195/300 Read block 1887 len 24576 196/300 Write block 2887 len 20480 data: 4 6 6 5 5 197/300 Write block 4817 len 4096 data: 3 198/300 Read block 815 len 4096 199/300 Write block 2443 len 4096 data: 4 200/300 Write block 99 len 28672 data: 4 5 5 5 3 3 4 201/300 Write block 3523 len 12288 data: 2 3 3 202/300 Write block 4174 len 4096 data: 3 203/300 Read block 4093 len 24576 204/300 Flush 205/300 Read block 9 len 32768 206/300 Flush 207/300 Read block 4385 len 16384 208/300 Write block 2319 len 40960 data: 3 4 4 3 3 3 3 3 3 2 209/300 Flush 210/300 Flush 211/300 Write block 847 len 4096 data: 3 212/300 Write block 1922 len 36864 data: 2 2 3 3 3 3 5 4 4 213/300 Read block 4745 len 36864 214/300 Read block 312 len 32768 215/300 Read block 3307 len 36864 216/300 Write block 3255 len 16384 data: 3 4 6 4 217/300 Read block 4373 len 28672 218/300 Read block 3020 len 24576 219/300 Read block 3586 len 32768 220/300 Write block 4828 len 12288 data: 6 5 5 221/300 Read block 3990 len 40960 222/300 Write block 2080 len 32768 data: 4 4 4 4 4 4 4 3 223/300 Write block 2163 len 40960 data: 5 5 4 3 3 3 3 2 2 2 224/300 Write block 1702 len 16384 data: 4 4 4 3 225/300 Read block 3105 len 20480 226/300 Read block 860 len 36864 227/300 Write block 475 len 40960 data: 3 4 4 4 2 2 3 3 3 4 228/300 Read block 4162 len 24576 229/300 Read block 4506 len 40960 230/300 Flush 231/300 Write block 106 len 16384 data: 4 4 6 6 232/300 Write block 2754 len 28672 data: 2 2 2 2 3 3 4 233/300 Read block 1441 len 12288 234/300 Read block 4043 len 36864 235/300 Write block 13 len 16384 data: 2 3 4 4 236/300 Flush 237/300 Read block 453 len 16384 238/300 Read block 1680 len 36864 239/300 Write block 2962 len 12288 data: 5 4 4 240/300 Read block 3178 len 28672 241/300 Flush 242/300 Write block 3213 len 12288 data: 5 4 3 243/300 Flush 244/300 Flush 245/300 Write block 2609 len 8192 data: 3 3 246/300 Write block 3648 len 36864 data: 4 4 3 2 2 4 4 5 5 247/300 Write block 4023 len 40960 data: 2 2 2 2 2 2 3 5 6 6 248/300 Read block 852 len 32768 249/300 Read block 1102 len 40960 250/300 Write block 1490 len 40960 data: 2 2 2 2 3 4 4 5 4 5 251/300 Write block 4580 len 40960 data: 3 4 4 5 5 5 3 3 2 3 252/300 Write block 1199 len 24576 data: 5 6 5 4 5 5 253/300 Write block 3102 len 24576 data: 5 5 5 6 6 6 254/300 Read block 3995 len 36864 255/300 Write block 4505 len 12288 data: 4 4 4 256/300 Flush 257/300 Write block 2300 len 36864 data: 3 4 5 6 5 7 7 7 6 258/300 Write block 876 len 16384 data: 4 4 4 4 259/300 Write block 2259 len 36864 data: 5 5 4 3 3 4 4 4 5 260/300 Write block 2637 len 12288 data: 4 4 4 261/300 Read block 4772 len 4096 262/300 Write block 3102 len 12288 data: 6 6 6 263/300 Read block 1763 len 4096 264/300 Read block 3134 len 24576 265/300 Flush 266/300 Write block 772 len 12288 data: 4 5 5 267/300 Read block 2404 len 32768 268/300 Read block 4793 len 36864 269/300 Flush 270/300 Write block 3623 len 32768 data: 4 4 3 4 4 4 2 3 271/300 Write block 3097 len 16384 data: 3 4 4 4 272/300 Write block 1686 len 20480 data: 6 5 6 7 7 273/300 Write block 4280 len 32768 data: 6 5 4 4 4 4 4 5 274/300 Write block 3609 len 16384 data: 3 3 4 4 275/300 Read block 2864 len 32768 276/300 Read block 2254 len 28672 277/300 Write block 2813 len 12288 data: 4 6 4 278/300 Flush 279/300 Read block 2247 len 4096 280/300 Read block 3970 len 4096 281/300 Write block 4530 len 12288 data: 4 4 4 282/300 Read block 2247 len 8192 283/300 Write block 1027 len 40960 data: 2 3 3 3 3 5 6 5 3 4 284/300 Read block 1622 len 4096 285/300 Flush 286/300 Write block 4825 len 20480 data: 2 3 5 7 6 287/300 Read block 1360 len 32768 288/300 Read block 582 len 40960 289/300 Read block 3955 len 24576 290/300 Write block 1793 len 12288 data: 4 4 3 291/300 Write block 4663 len 12288 data: 6 5 4 292/300 Flush 293/300 Write block 4955 len 24576 data: 4 4 3 3 4 4 294/300 Write block 2233 len 4096 data: 2 295/300 Write block 2504 len 40960 data: 5 6 6 5 4 4 3 3 2 3 296/300 Read block 3282 len 12288 297/300 Read block 1604 len 20480 298/300 Flush 299/300 Write block 1264 len 32768 data: 5 6 6 5 6 5 4 4 300/300 Read block 1850 len 32768 301/300 Read block 1057 len 32768 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:45:55.825 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/0, method: POST, req_id: a2da409d-306d-4c68-bfca-31c7fdba969e, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5828 Acked Read 2 Sent Done Done false 5829 Acked Read 3 Sent Done Done false 5830 Acked Read 9 Sent Done Done false 5831 Acked Write 7 Sent Done Done false 5832 Acked Write 7 Sent Done Done false 5833 Acked Flush 0 Sent Done Done false 5834 Acked Write 8 Sent Done Done false 5835 Acked Write 7 Sent Done Done false 5836 Acked Read 5 Sent Done Done false 5837 Acked Write 9 Sent Done Done false 5838 Acked Write 5 Sent Done Done false 5839 Acked Read 7 Sent Done Done false 5840 Acked Flush 0 Sent Done Done false 5841 Acked Write 1 Sent Done Done false 5842 Acked Read 10 Sent Done Done false 5843 Acked Flush 0 Sent Done Done false 5844 Acked Write 9 Sent Done Done false 5845 Acked Write 9 Sent Done Done false 5846 Acked Write 10 Sent Done Done false 5847 Acked Write 8 Sent Done Done false 5848 Acked Write 10 Sent Done Done false 5849 Acked Read 3 Sent Done Done false 5850 Acked Read 5 Sent Done Done false 5851 Acked Write 10 Sent Done Done false 5852 Acked Read 6 Sent Done Done false 5853 Acked Read 5 Sent Done Done false 5854 Acked Read 6 Sent Done Done false 5855 Acked Write 5 Sent Done Done false 5856 Acked Read 4 Sent Done Done false 5857 Acked Read 6 Sent Done Done false 5858 Acked Write 6 Sent Done Done false 5859 Acked Write 4 Sent Done Done false 5860 Acked Read 8 Sent Done Done false 5861 Acked Read 8 Sent Done Done false 5862 Acked Write 10 Sent Done Done false 5863 Acked Write 7 Sent Done Done false 5864 Acked Write 4 Sent Done Done false 5865 Acked Read 8 Sent Done Done false 5866 Acked Read 10 Sent Done Done false 5867 Acked Flush 0 Sent Done Done false 5868 Acked Write 1 Sent Done Done false 5869 Acked Read 6 Sent Done Done false 5870 Acked Read 10 Sent Done Done false 5871 Acked Read 4 Sent Done Done false 5872 Acked Flush 0 Sent Done Done false 5873 Acked Read 9 Sent Done Done false 5874 Acked Read 3 Sent Done Done false 5875 Acked Read 4 Sent Done Done false 5876 Acked Read 4 Sent Done Done false 5877 Acked Read 4 Sent Done Done false 5878 Acked Read 9 Sent Done Done false 5879 Acked Write 5 Sent Done Done false 5880 Acked Flush 0 Sent Done Done false 5881 Acked Flush 0 Sent Done Done false 5882 Acked Read 8 Sent Done Done false 5883 Acked Flush 0 Sent Done Done false 5884 Acked Read 7 Sent Done Done false 5885 Acked Read 4 Sent Done Done false 5886 Acked Read 2 Sent Done Done false 5887 Acked Read 2 Sent Done Done false 5888 Acked Read 6 Sent Done Done false 5889 Acked Write 3 Sent Done Done false 5890 Acked Read 4 Sent Done Done false 5891 Acked Read 6 Sent Done Done false 5892 Acked Read 5 Sent Done Done false 5893 Acked Read 2 Sent Done Done false 5894 Acked Write 3 Sent Done Done false 5895 Acked Read 2 Sent Done Done false 5896 Acked Read 10 Sent Done Done false 5897 Acked Read 4 Sent Done Done false 5898 Acked Write 5 Sent Done Done false 5899 Acked Write 1 Sent Done Done false 5900 Acked Write 2 Sent Done Done false 5901 Acked Write 3 Sent Done Done false 5902 Acked Read 9 Sent Done Done false 5903 Acked Read 7 Sent Done Done false 5904 Acked Read 1 Sent Done Done false 5905 Acked Read 3 Sent Done Done false 5906 Acked Write 2 Sent Done Done false 5907 Acked Write 8 Sent Done Done false 5908 Acked Read 5 Sent Done Done false 5909 Acked Write 5 Sent Done Done false 5910 Acked Write 9 Sent Done Done false 5911 Acked Write 6 Sent Done Done false 5912 Acked Read 6 Sent Done Done false 5913 Acked Write 1 Sent Done Done false 5914 Acked Write 8 Sent Done Done false 5915 Acked Read 1 Sent Done Done false 5916 Acked Read 10 Sent Done Done false 5917 Acked Write 7 Sent Done Done false 5918 Acked Write 2 Sent Done Done false 5919 Acked Write 5 Sent Done Done false 5920 Acked Write 4 Sent Done Done false 5921 Acked Flush 0 Sent Done Done false 5922 Acked Write 10 Sent Done Done false 5923 Acked Write 2 Sent Done Done false 5924 Acked Read 4 Sent Done Done false 5925 Acked Read 9 Sent Done Done false 5926 Acked Write 10 Sent Done Done false 5927 Acked Read 2 Sent Done Done false 5928 Acked Read 2 Sent Done Done false 5929 Acked Read 4 Sent Done Done false 5930 Acked Write 6 Sent Done Done false 5931 Acked Write 9 Sent Done Done false 5932 Acked Read 6 Sent Done Done false 5933 Acked Read 10 Sent Done Done false 5934 Acked Write 3 Sent Done Done false 5935 Acked Read 1 Sent Done Done false 5936 Acked Write 1 Sent Done Done false 5937 Acked Flush 0 Sent Done Done false 5938 Acked Read 10 Sent Done Done false 5939 Acked Read 3 Sent Done Done false 5940 Acked Flush 0 Sent Done Done false 5941 Acked Read 9 Sent Done Done false 5942 Acked Write 2 Sent Done Done false 5943 Acked Read 10 Sent Done Done false 5944 Acked Write 1 Sent Done Done false 5945 Acked Read 10 Sent Done Done false 5946 Acked Read 3 Sent Done Done false 5947 Acked Write 6 Sent Done Done false 5948 Acked Read 5 Sent Done Done false 5949 Acked Write 1 Sent Done Done false 5950 Acked Read 4 Sent Done Done false 5951 Acked Write 9 Sent Done Done false 5952 Acked Write 10 Sent Done Done false 5953 Acked Read 10 Sent Done Done false 5954 Acked Read 8 Sent Done Done false 5955 Acked Write 8 Sent Done Done false 5956 Acked Read 2 Sent Done Done false 5957 Acked Flush 0 Sent Done Done false 5958 Acked Write 6 Sent Done Done false 5959 Acked Read 5 Sent Done Done false 5960 Acked Read 6 Sent Done Done false 5961 Acked Write 7 Sent Done Done false 5962 Acked Flush 0 Sent Done Done false 5963 Acked Flush 0 Sent Done Done false 5964 Acked Flush 0 Sent Done Done false 5965 Acked Read 9 Sent Done Done false 5966 Acked Write 4 Sent Done Done false 5967 Acked Flush 0 Sent Done Done false 5968 Acked Flush 0 Sent Done Done false 5969 Acked Write 10 Sent Done Done false 5970 Acked Read 9 Sent Done Done false 5971 Acked Read 10 Sent Done Done false 5972 Acked Write 5 Sent Done Done false 5973 Acked Read 3 Sent Done Done false 5974 Acked Write 8 Sent Done Done false 5975 Acked Read 8 Sent Done Done false 5976 Acked Write 4 Sent Done Done false 5977 Acked Read 4 Sent Done Done false 5978 Acked Write 3 Sent Done Done false 5979 Acked Write 6 Sent Done Done false 5980 Acked Write 6 Sent Done Done false 5981 Acked Read 2 Sent Done Done false 5982 Acked Read 7 Sent Done Done false 5983 Acked Write 8 Sent Done Done false 5984 Acked Flush 0 Sent Done Done false 5985 Acked Read 3 Sent Done Done false 5986 Acked Read 9 Sent Done Done false 5987 Acked Write 9 Sent Done Done false 5988 Acked Read 2 Sent Done Done false 5989 Acked Read 6 Sent Done Done false 5990 Acked Read 6 Sent Done Done false 5991 Acked Write 1 Sent Done Done false 5992 Acked Read 3 Sent Done Done false 5993 Acked Write 9 Sent Done Done false 5994 Acked Read 8 Sent Done Done false 5995 Acked Write 8 Sent Done Done false 5996 Acked Read 9 Sent Done Done false 5997 Acked Write 10 Sent Done Done false 5998 Acked Read 7 Sent Done Done false 5999 Acked Read 6 Sent Done Done false 6000 Acked Write 9 Sent Done Done false 6001 Acked Read 9 Sent Done Done false 6002 Acked Read 10 Sent Done Done false 6003 Acked Read 7 Sent Done Done false 6004 Acked Write 4 Sent Done Done false 6005 Acked Read 8 Sent Done Done false 6006 Acked Flush 0 Sent Done Done false 6007 Acked Read 2 Sent Done Done false 6008 Acked Read 1 Sent Done Done false 6009 Acked Write 5 Sent Done Done false 6010 Acked Read 10 Sent Done Done false 6011 Acked Read 10 Sent Done Done false 6012 Acked Read 6 Sent Done Done false 6013 Acked Write 9 Sent Done Done false 6014 Acked Read 7 Sent Done Done false 6015 Acked Read 3 Sent Done Done false 6016 Acked Write 4 Sent Done Done false 6017 Acked Read 5 Sent Done Done false 6018 Acked Write 6 Sent Done Done false 6019 Acked Flush 0 Sent Done Done false 6020 Acked Write 8 Sent Done Done false 6021 Acked Flush 0 Sent Done Done false 6022 Acked Read 6 Sent Done Done false 6023 Acked Write 5 Sent Done Done false 6024 Acked Write 1 Sent Done Done false 6025 Acked Read 1 Sent Done Done false 6026 Acked Write 1 Sent Done Done false 6027 Acked Write 7 Sent Done Done false 6028 Acked Write 3 Sent Done Done false 6029 Acked Write 1 Sent Done Done false 6030 Acked Read 6 Sent Done Done false 6031 Acked Flush 0 Sent Done Done false 6032 Acked Read 8 Sent Done Done false 6033 Acked Flush 0 Sent Done Done false 6034 Acked Read 4 Sent Done Done false 6035 Acked Write 10 Sent Done Done false 6036 Acked Flush 0 Sent Done Done false 6037 Acked Flush 0 Sent Done Done false 6038 Acked Write 1 Sent Done Done false 6039 Acked Write 9 Sent Done Done false 6040 Acked Read 9 Sent Done Done false 6041 Acked Read 8 Sent Done Done false 6042 Acked Read 9 Sent Done Done false 6043 Acked Write 4 Sent Done Done false 6044 Acked Read 7 Sent Done Done false 6045 Acked Read 6 Sent Done Done false 6046 Acked Read 8 Sent Done Done false 6047 Acked Write 3 Sent Done Done false 6048 Acked Read 10 Sent Done Done false 6049 Acked Write 8 Sent Done Done false 6050 Acked Write 10 Sent Done Done false 6051 Acked Write 4 Sent Done Done false 6052 Acked Read 5 Sent Done Done false 6053 Acked Read 9 Sent Done Done false 6054 Acked Write 10 Sent Done Done false 6055 Acked Read 6 Sent Done Done false 6056 Acked Read 10 Sent Done Done false 6057 Acked Flush 0 Sent Done Done false 6058 Acked Write 4 Sent Done Done false 6059 Acked Write 7 Sent Done Done false 6060 Acked Read 3 Sent Done Done false 6061 Acked Read 9 Sent Done Done false 6062 Acked Write 4 Sent Done Done false 6063 Acked Flush 0 Sent Done Done false 6064 Acked Read 4 Sent Done Done false 6065 Acked Read 9 Sent Done Done false 6066 Acked Write 3 Sent Done Done false 6067 Acked Read 7 Sent Done Done false 6068 Acked Flush 0 Sent Done Done false 6069 Acked Write 3 Sent Done Done false 6070 Acked Flush 0 Sent Done Done false 6071 Acked Flush 0 Sent Done Done false 6072 Acked Write 2 Sent Done Done false 6073 Acked Write 9 Sent Done Done false 6074 Acked Write 10 Sent Done Done false 6075 Acked Read 8 Sent Done Done false 6076 Acked Read 10 Sent Done Done false 6077 Acked Write 10 Sent Done Done false 6078 Acked Write 10 Sent Done Done false 6079 Acked Write 6 Sent Done Done false 6080 Acked Write 6 Sent Done Done false 6081 Acked Read 9 Sent Done Done false 6082 Acked Write 3 Sent Done Done false 6083 Acked Flush 0 Sent Done Done false 6084 Acked Write 9 Sent Done Done false 6085 Acked Write 4 Sent Done Done false 6086 Acked Write 9 Sent Done Done false 6087 Acked Write 3 Sent Done Done false 6088 Acked Read 1 Sent Done Done false 6089 Acked Write 3 Sent Done Done false 6090 Acked Read 1 Sent Done Done false 6091 Acked Read 6 Sent Done Done false 6092 Acked Flush 0 Sent Done Done false 6093 Acked Write 3 Sent Done Done false 6094 Acked Read 8 Sent Done Done false 6095 Acked Read 9 Sent Done Done false 6096 Acked Flush 0 Sent Done Done false 6097 Acked Write 8 Sent Done Done false 6098 Acked Write 4 Sent Done Done false 6099 Acked Write 5 Sent Done Done false 6100 Acked Write 8 Sent Done Done false 6101 Acked Write 4 Sent Done Done false 6102 Acked Read 8 Sent Done Done false 6103 Acked Read 7 Sent Done Done false 6104 Acked Write 3 Sent Done Done false 6105 Acked Flush 0 Sent Done Done false 6106 Acked Read 1 Sent Done Done false 6107 Acked Read 1 Sent Done Done false 6108 Acked Write 3 Sent Done Done false 6109 Acked Read 2 Sent Done Done false 6110 Acked Write 10 Sent Done Done false 6111 Acked Read 1 Sent Done Done false 6112 Acked Flush 0 Sent Done Done false 6113 Acked Write 5 Sent Done Done false 6114 Acked Read 8 Sent Done Done false 6115 Acked Read 10 Sent Done Done false 6116 Acked Read 6 Sent Done Done false 6117 Acked Write 3 Sent Done Done false 6118 Acked Write 3 Sent Done Done false 6119 Acked Flush 0 Sent Done Done false 6120 Acked Write 6 Sent Done Done false 6121 Acked Write 1 Sent Done Done false 6122 Acked Write 10 Sent Done Done false 6123 Acked Read 3 Sent Done Done false 6124 Acked Read 5 Sent Done Done false 6125 Acked Flush 0 Sent Done Done false 6126 Acked Write 8 Sent Done Done false 6127 Acked Read 8 Sent Done Done false 6128 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5827 6125 6125 Downstairs last five completed: 5827 5826 5825 5824 5823 Upstairs last five completed: 6128 6127 6126 6125 6124 CLIENT: Up:0 ds:301 act:2 [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5828 Acked Read 2 Sent Done Done false 5829 Acked Read 3 Sent Done Done false 5830 Acked Read 9 Sent Done Done false 5831 Acked Write 7 Sent Done Done false 5832 Acked Write 7 Sent Done Done false 5833 Acked Flush 0 Sent Done Done false 5834 Acked Write 8 Sent Done Done false 5835 Acked Write 7 Sent Done Done false 5836 Acked Read 5 Sent Done Done false 5837 Acked Write 9 Sent Done Done false 5838 Acked Write 5 Sent Done Done false 5839 Acked Read 7 Sent Done Done false 5840 Acked Flush 0 Sent Done Done false 5841 Acked Write 1 Sent Done Done false 5842 Acked Read 10 Sent Done Done false 5843 Acked Flush 0 Sent Done Done false 5844 Acked Write 9 Sent Done Done false 5845 Acked Write 9 Sent Done Done false 5846 Acked Write 10 Sent Done Done false 5847 Acked Write 8 Sent Done Done false 5848 Acked Write 10 Sent Done Done false 5849 Acked Read 3 Sent Done Done false 5850 Acked Read 5 Sent Done Done false 5851 Acked Write 10 Sent Done Done false 5852 Acked Read 6 Sent Done Done false 5853 Acked Read 5 Sent Done Done false 5854 Acked Read 6 Sent Done Done false 5855 Acked Write 5 Sent Done Done false 5856 Acked Read 4 Sent Done Done false 5857 Acked Read 6 Sent Done Done false 5858 Acked Write 6 Sent Done Done false 5859 Acked Write 4 Sent Done Done false 5860 Acked Read 8 Sent Done Done false 5861 Acked Read 8 Sent Done Done false 5862 Acked Write 10 Sent Done Done false 5863 Acked Write 7 Sent Done Done false 5864 Acked Write 4 Sent Done Done false 5865 Acked Read 8 Sent Done Done false 5866 Acked Read 10 Sent Done Done false 5867 Acked Flush 0 Sent Done Done false 5868 Acked Write 1 Sent Done Done false 5869 Acked Read 6 Sent Done Done false 5870 Acked Read 10 Sent Done Done false 5871 Acked Read 4 Sent Done Done false 5872 Acked Flush 0 Sent Done Done false 5873 Acked Read 9 Sent Done Done false 5874 Acked Read 3 Sent Done Done false 5875 Acked Read 4 Sent Done Done false 5876 Acked Read 4 Sent Done Done false 5877 Acked Read 4 Sent Done Done false 5878 Acked Read 9 Sent Done Done false 5879 Acked Write 5 Sent Done Done false 5880 Acked Flush 0 Sent Done Done false 5881 Acked Flush 0 Sent Done Done false 5882 Acked Read 8 Sent Done Done false 5883 Acked Flush 0 Sent Done Done false 5884 Acked Read 7 Sent Done Done false 5885 Acked Read 4 Sent Done Done false 5886 Acked Read 2 Sent Done Done false 5887 Acked Read 2 Sent Done Done false 5888 Acked Read 6 Sent Done Done false 5889 Acked Write 3 Sent Done Done false 5890 Acked Read 4 Sent Done Done false 5891 Acked Read 6 Sent Done Done false 5892 Acked Read 5 Sent Done Done false 5893 Acked Read 2 Sent Done Done false 5894 Acked Write 3 Sent Done Done false 5895 Acked Read 2 Sent Done Done false 5896 Acked Read 10 Sent Done Done false 5897 Acked Read 4 Sent Done Done false 5898 Acked Write 5 Sent Done Done false 5899 Acked Write 1 Sent Done Done false 5900 Acked Write 2 Sent Done Done false 5901 Acked Write 3 Sent Done Done false 5902 Acked Read 9 Sent Done Done false 5903 Acked Read 7 Sent Done Done false 5904 Acked Read 1 Sent Done Done false 5905 Acked Read 3 Sent Done Done false 5906 Acked Write 2 Sent Done Done false 5907 Acked Write 8 Sent Done Done false 5908 Acked Read 5 Sent Done Done false 5909 Acked Write 5 Sent Done Done false 5910 Acked Write 9 Sent Done Done false 5911 Acked Write 6 Sent Done Done false 5912 Acked Read 6 Sent Done Done false 5913 Acked Write 1 Sent Done Done false 5914 Acked Write 8 Sent Done Done false 5915 Acked Read 1 Sent Done Done false 5916 Acked Read 10 Sent Done Done false 5917 Acked Write 7 Sent Done Done false 5918 Acked Write 2 Sent Done Done false 5919 Acked Write 5 Sent Done Done false 5920 Acked Write 4 Sent Done Done false 5921 Acked Flush 0 Sent Done Done false 5922 Acked Write 10 Sent Done Done false 5923 Acked Write 2 Sent Done Done false 5924 Acked Read 4 Sent Done Done false 5925 Acked Read 9 Sent Done Done false 5926 Acked Write 10 Sent Done Done false 5927 Acked Read 2 Sent Done Done false 5928 Acked Read 2 Sent Done Done false 5929 Acked Read 4 Sent Done Done false 5930 Acked Write 6 Sent Done Done false 5931 Acked Write 9 Sent Done Done false 5932 Acked Read 6 Sent Done Done false 5933 Acked Read 10 Sent Done Done false 5934 Acked Write 3 Sent Done Done false 5935 Acked Read 1 Sent Done Done false 5936 Acked Write 1 Sent Done Done false 5937 Acked Flush 0 Sent Done Done false 5938 Acked Read 10 Sent Done Done false 5939 Acked Read 3 Sent Done Done false 5940 Acked Flush 0 Sent Done Done false 5941 Acked Read 9 Sent Done Done false 5942 Acked Write 2 Sent Done Done false 5943 Acked Read 10 Sent Done Done false 5944 Acked Write 1 Sent Done Done false 5945 Acked Read 10 Sent Done Done false 5946 Acked Read 3 Sent Done Done false 5947 Acked Write 6 Sent Done Done false 5948 Acked Read 5 Sent Done Done false 5949 Acked Write 1 Sent Done Done false 5950 Acked Read 4 Sent Done Done false 5951 Acked Write 9 Sent Done Done false 5952 Acked Write 10 Sent Done Done false 5953 Acked Read 10 Sent Done Done false 5954 Acked Read 8 Sent Done Done false 5955 Acked Write 8 Sent Done Done false 5956 Acked Read 2 Sent Done Done false 5957 Acked Flush 0 Sent Done Done false 5958 Acked Write 6 Sent Done Done false 5959 Acked Read 5 Sent Done Done false 5960 Acked Read 6 Sent Done Done false 5961 Acked Write 7 Sent Done Done false 5962 Acked Flush 0 Sent Done Done false 5963 Acked Flush 0 Sent Done Done false 5964 Acked Flush 0 Sent Done Done false 5965 Acked Read 9 Sent Done Done false 5966 Acked Write 4 Sent Done Done false 5967 Acked Flush 0 Sent Done Done false 5968 Acked Flush 0 Sent Done Done false 5969 Acked Write 10 Sent Done Done false 5970 Acked Read 9 Sent Done Done false 5971 Acked Read 10 Sent Done Done false 5972 Acked Write 5 Sent Done Done false 5973 Acked Read 3 Sent Done Done false 5974 Acked Write 8 Sent Done Done false 5975 Acked Read 8 Sent Done Done false 5976 Acked Write 4 Sent Done Done false 5977 Acked Read 4 Sent Done Done false 5978 Acked Write 3 Sent Done Done false 5979 Acked Write 6 Sent Done Done false 5980 Acked Write 6 Sent Done Done false 5981 Acked Read 2 Sent Done Done false 5982 Acked Read 7 Sent Done Done false 5983 Acked Write 8 Sent Done Done false 5984 Acked Flush 0 Sent Done Done false 5985 Acked Read 3 Sent Done Done false 5986 Acked Read 9 Sent Done Done false 5987 Acked Write 9 Sent Done Done false 5988 Acked Read 2 Sent Done Done false 5989 Acked Read 6 Sent Done Done false 5990 Acked Read 6 Sent Done Done false 5991 Acked Write 1 Sent Done Done false 5992 Acked Read 3 Sent Done Done false 5993 Acked Write 9 Sent Done Done false 5994 Acked Read 8 Sent Done Done false 5995 Acked Write 8 Sent Done Done false 5996 Acked Read 9 Sent Done Done false 5997 Acked Write 10 Sent Done Done false 5998 Acked Read 7 Sent Done Done false 5999 Acked Read 6 Sent Done Done false 6000 Acked Write 9 Sent Done Done false 6001 Acked Read 9 Sent Done Done false 6002 Acked Read 10 Sent Done Done false 6003 Acked Read 7 Sent Done Done false 6004 Acked Write 4 Sent Done Done false 6005 Acked Read 8 Sent Done Done false 6006 Acked Flush 0 Sent Done Done false 6007 Acked Read 2 Sent Done Done false 6008 Acked Read 1 Sent Done Done false 6009 Acked Write 5 Sent Done Done false 6010 Acked Read 10 Sent Done Done false 6011 Acked Read 10 Sent Done Done false 6012 Acked Read 6 Sent Done Done false 6013 Acked Write 9 Sent Done Done false 6014 Acked Read 7 Sent Done Done false 6015 Acked Read 3 Sent Done Done false 6016 Acked Write 4 Sent Done Done false 6017 Acked Read 5 Sent Done Done false 6018 Acked Write 6 Sent Done Done false 6019 Acked Flush 0 Sent Done Done false 6020 Acked Write 8 Sent Done Done false 6021 Acked Flush 0 Sent Done Done false 6022 Acked Read 6 Sent Done Done false 6023 Acked Write 5 Sent Done Done false 6024 Acked Write 1 Sent Done Done false 6025 Acked Read 1 Sent Done Done false 6026 Acked Write 1 Sent Done Done false 6027 Acked Write 7 Sent Done Done false 6028 Acked Write 3 Sent Done Done false 6029 Acked Write 1 Sent Done Done false 6030 Acked Read 6 Sent Done Done false 6031 Acked Flush 0 Sent Done Done false 6032 Acked Read 8 Sent Done Done false 6033 Acked Flush 0 Sent Done Done false 6034 Acked Read 4 Sent Done Done false 6035 Acked Write 10 Sent Done Done false 6036 Acked Flush 0 Sent Done Done false 6037 Acked Flush 0 Sent Done Done false 6038 Acked Write 1 Sent Done Done false 6039 Acked Write 9 Sent Done Done false 6040 Acked Read 9 Sent Done Done false 6041 Acked Read 8 Sent Done Done false 6042 Acked Read 9 Sent Done Done false 6043 Acked Write 4 Sent Done Done false 6044 Acked Read 7 Sent Done Done false 6045 Acked Read 6 Sent Done Done false 6046 Acked Read 8 Sent Done Done false 6047 Acked Write 3 Sent Done Done false 6048 Acked Read 10 Sent Done Done false 6049 Acked Write 8 Sent Done Done false 6050 Acked Write 10 Sent Done Done false 6051 Acked Write 4 Sent Done Done false 6052 Acked Read 5 Sent Done Done false 6053 Acked Read 9 Sent Done Done false 6054 Acked Write 10 Sent Done Done false 6055 Acked Read 6 Sent Done Done false 6056 Acked Read 10 Sent Done Done false 6057 Acked Flush 0 Sent Done Done false 6058 Acked Write 4 Sent Done Done false 6059 Acked Write 7 Sent Done Done false 6060 Acked Read 3 Sent Done Done false 6061 Acked Read 9 Sent Done Done false 6062 Acked Write 4 Sent Done Done false 6063 Acked Flush 0 Sent Done Done false 6064 Acked Read 4 Sent Done Done false 6065 Acked Read 9 Sent Done Done false 6066 Acked Write 3 Sent Done Done false 6067 Acked Read 7 Sent Done Done false 6068 Acked Flush 0 Sent Done Done false 6069 Acked Write 3 Sent Done Done false 6070 Acked Flush 0 Sent Done Done false 6071 Acked Flush 0 Sent Done Done false 6072 Acked Write 2 Sent Done Done false 6073 Acked Write 9 Sent Done Done false 6074 Acked Write 10 Sent Done Done false 6075 Acked Read 8 Sent Done Done false 6076 Acked Read 10 Sent Done Done false 6077 Acked Write 10 Sent Done Done false 6078 Acked Write 10 Sent Done Done false 6079 Acked Write 6 Sent Done Done false 6080 Acked Write 6 Sent Done Done false 6081 Acked Read 9 Sent Done Done false 6082 Acked Write 3 Sent Done Done false 6083 Acked Flush 0 Sent Done Done false 6084 Acked Write 9 Sent Done Done false 6085 Acked Write 4 Sent Done Done false 6086 Acked Write 9 Sent Done Done false 6087 Acked Write 3 Sent Done Done false 6088 Acked Read 1 Sent Done Done false 6089 Acked Write 3 Sent Done Done false 6090 Acked Read 1 Sent Done Done false 6091 Acked Read 6 Sent Done Done false 6092 Acked Flush 0 Sent Done Done false 6093 Acked Write 3 Sent Done Done false 6094 Acked Read 8 Sent Done Done false 6095 Acked Read 9 Sent Done Done false 6096 Acked Flush 0 Sent Done Done false 6097 Acked Write 8 Sent Done Done false 6098 Acked Write 4 Sent Done Done false 6099 Acked Write 5 Sent Done Done false 6100 Acked Write 8 Sent Done Done false 6101 Acked Write 4 Sent Done Done false 6102 Acked Read 8 Sent Done Done false 6103 Acked Read 7 Sent Done Done false 6104 Acked Write 3 Sent Done Done false 6105 Acked Flush 0 Sent Done Done false 6106 Acked Read 1 Sent Done Done false 6107 Acked Read 1 Sent Done Done false 6108 Acked Write 3 Sent Done Done false 6109 Acked Read 2 Sent Done Done false 6110 Acked Write 10 Sent Done Done false 6111 Acked Read 1 Sent Done Done false 6112 Acked Flush 0 Sent Done Done false 6113 Acked Write 5 Sent Done Done false 6114 Acked Read 8 Sent Done Done false 6115 Acked Read 10 Sent Done Done false 6116 Acked Read 6 Sent Done Done false 6117 Acked Write 3 Sent Done Done false 6118 Acked Write 3 Sent Done Done false 6119 Acked Flush 0 Sent Done Done false 6120 Acked Write 6 Sent Done Done false 6121 Acked Write 1 Sent Done Done false 6122 Acked Write 10 Sent Done Done false 6123 Acked Read 3 Sent Done Done false 6124 Acked Read 5 Sent Done Done false 6125 Acked Flush 0 Sent Done Done false 6126 Acked Write 8 Sent Done Done false 6127 Acked Read 8 Sent Done Done false 6128 Acked Read 8 Sent Done Done false 6129 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5827 6129 6129 Downstairs last five completed: 5827 5826 5825 5824 5823 Upstairs last five completed: 6129 6128 6127 6126 6125 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.588102563Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:36994)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.58822528Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:01.58919175Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.589794879Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.58982631Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 5827","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.589843685Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 5827","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.590097737Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.590118382Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.590133617Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 5827","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.590147634Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 5827","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:01.590376622Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6129 6128 6127 6126 6125 Upstairs last five completed: 6129 6128 6127 6126 6125 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:46:03.830 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false , req_id: 6bba9de4-41b5-4320-a700-861f4fd9f348, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:03.831154087Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:03.831239534Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:03.831265177Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:03.83129128Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:03.831313694Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:46:03.831 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8f9b400a-45a3-42f2-ab04-19a456a5b527, remote_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:03.831350644Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:46:07.833 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: aa644fd4-edaa-472c-8038-b1baf4d79ef0, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 1963 len 36864 002/300 Write block 1979 len 24576 data: 4 4 5 4 2 2 003/300 Read block 4836 len 40960 004/300 Write block 1872 len 20480 data: 5 5 3 3 3 005/300 Write block 4685 len 24576 data: 3 4 4 5 4 4 006/300 Flush 007/300 Write block 1061 len 24576 data: 3 3 3 3 3 2 008/300 Read block 4069 len 36864 009/300 Write block 1373 len 24576 data: 5 4 5 5 5 5 010/300 Write block 890 len 36864 data: 6 5 4 4 4 3 3 3 6 011/300 Read block 500 len 4096 012/300 Read block 4145 len 8192 013/300 Read block 944 len 20480 014/300 Read block 3048 len 16384 015/300 Flush 016/300 Read block 2579 len 8192 017/300 Read block 2399 len 4096 018/300 Read block 3989 len 36864 019/300 Write block 3901 len 12288 data: 4 4 2 020/300 Read block 1972 len 28672 021/300 Read block 3458 len 4096 022/300 Read block 1965 len 4096 023/300 Flush 024/300 Read block 3591 len 20480 025/300 Write block 3907 len 8192 data: 3 3 026/300 Write block 1866 len 36864 data: 5 5 5 5 5 6 6 6 4 027/300 Read block 614 len 16384 028/300 Write block 3522 len 36864 data: 3 3 4 4 3 3 3 3 3 029/300 Flush 030/300 Read block 4597 len 12288 031/300 Write block 4299 len 12288 data: 4 4 4 032/300 Write block 4711 len 24576 data: 6 5 4 4 4 4 033/300 Write block 4888 len 20480 data: 4 5 4 3 3 034/300 Read block 4816 len 4096 035/300 Read block 1145 len 32768 036/300 Read block 1812 len 40960 037/300 Write block 1160 len 16384 data: 5 5 5 3 038/300 Write block 1750 len 36864 data: 4 5 6 5 4 4 4 4 3 039/300 Read block 4474 len 16384 040/300 Read block 3730 len 12288 041/300 Read block 419 len 24576 042/300 Write block 2672 len 8192 data: 2 2 043/300 Write block 3756 len 4096 data: 5 044/300 Write block 725 len 4096 data: 4 045/300 Read block 3527 len 40960 046/300 Read block 4445 len 24576 047/300 Flush 048/300 Read block 1929 len 16384 049/300 Read block 3209 len 16384 050/300 Read block 3899 len 40960 051/300 Write block 2568 len 40960 data: 6 6 5 5 5 3 2 2 2 3 052/300 Write block 3254 len 28672 data: 3 4 5 7 5 4 3 053/300 Write block 3025 len 32768 data: 3 3 3 3 3 5 4 5 054/300 Write block 2946 len 16384 data: 3 3 3 3 055/300 Read block 254 len 32768 056/300 Write block 4441 len 4096 data: 7 057/300 Read block 3828 len 16384 058/300 Read block 3595 len 20480 059/300 Write block 2260 len 20480 data: 6 5 4 4 5 060/300 Read block 983 len 32768 061/300 Read block 4597 len 4096 062/300 Write block 4381 len 24576 data: 2 2 3 3 5 5 063/300 Read block 2200 len 16384 064/300 Read block 490 len 40960 065/300 Write block 2034 len 36864 data: 4 4 5 3 4 6 5 6 6 066/300 Read block 1496 len 12288 067/300 Flush 068/300 Read block 1905 len 20480 069/300 Read block 4432 len 16384 070/300 Write block 2321 len 36864 data: 5 4 4 4 4 4 4 3 2 071/300 Write block 1879 len 20480 data: 3 3 3 3 5 072/300 Read block 1760 len 32768 073/300 Write block 4749 len 40960 data: 5 5 4 3 4 4 4 4 4 5 074/300 Read block 1892 len 32768 075/300 Flush 076/300 Write block 1954 len 8192 data: 6 5 077/300 Read block 176 len 28672 078/300 Read block 2218 len 28672 079/300 Write block 4588 len 8192 data: 3 4 080/300 Write block 3460 len 4096 data: 7 081/300 Write block 13 len 28672 data: 3 4 5 5 4 4 4 082/300 Read block 1742 len 8192 083/300 Write block 3698 len 8192 data: 5 7 084/300 Flush 085/300 Write block 3943 len 4096 data: 4 086/300 Read block 4929 len 24576 087/300 Read block 4073 len 28672 088/300 Write block 2616 len 12288 data: 3 3 2 089/300 Write block 671 len 36864 data: 5 3 4 5 5 6 6 9 8 090/300 Write block 1973 len 8192 data: 2 3 091/300 Read block 2492 len 8192 092/300 Read block 3697 len 4096 093/300 Read block 383 len 4096 094/300 Read block 1140 len 24576 095/300 Read block 2143 len 28672 096/300 Write block 4078 len 20480 data: 5 6 4 3 4 097/300 Read block 1995 len 8192 098/300 Read block 4198 len 20480 099/300 Write block 1215 len 12288 data: 4 2 4 100/300 Read block 3027 len 12288 101/300 Read block 4625 len 32768 102/300 Write block 1396 len 24576 data: 3 3 3 3 3 4 103/300 Flush 104/300 Flush 105/300 Write block 2840 len 4096 data: 5 106/300 Read block 2277 len 8192 107/300 Read block 1495 len 12288 108/300 Write block 3990 len 24576 data: 6 6 5 4 3 3 109/300 Write block 1704 len 28672 data: 5 4 3 3 4 5 4 110/300 Read block 4190 len 28672 111/300 Read block 4047 len 20480 112/300 Write block 1200 len 8192 data: 7 6 113/300 Write block 4640 len 40960 data: 3 3 4 4 4 4 5 5 4 4 114/300 Flush 115/300 Read block 1656 len 8192 116/300 Write block 3465 len 40960 data: 5 5 5 4 5 5 5 4 4 4 117/300 Read block 2142 len 24576 118/300 Write block 3564 len 24576 data: 3 3 2 2 2 4 119/300 Read block 1995 len 36864 120/300 Read block 1955 len 36864 121/300 Read block 225 len 8192 122/300 Write block 1149 len 12288 data: 5 5 4 123/300 Read block 2175 len 16384 124/300 Read block 4298 len 16384 125/300 Write block 1899 len 24576 data: 4 3 3 3 4 4 126/300 Write block 462 len 40960 data: 3 3 4 4 4 4 5 6 6 4 127/300 Write block 860 len 8192 data: 4 4 128/300 Read block 2072 len 4096 129/300 Read block 3787 len 40960 130/300 Read block 181 len 36864 131/300 Read block 4679 len 16384 132/300 Read block 4645 len 16384 133/300 Read block 3029 len 40960 134/300 Read block 4636 len 36864 135/300 Write block 4069 len 24576 data: 5 6 5 6 6 6 136/300 Write block 944 len 28672 data: 5 4 5 4 4 3 3 137/300 Write block 530 len 28672 data: 9 8 8 6 4 4 4 138/300 Flush 139/300 Read block 413 len 16384 140/300 Write block 2861 len 40960 data: 5 4 3 3 3 3 4 4 4 4 141/300 Read block 2691 len 8192 142/300 Write block 3228 len 28672 data: 3 3 4 5 4 4 4 143/300 Write block 4629 len 24576 data: 4 4 3 3 4 4 144/300 Flush 145/300 Read block 2170 len 16384 146/300 Read block 4461 len 16384 147/300 Write block 2641 len 8192 data: 3 3 148/300 Write block 149 len 32768 data: 5 4 4 3 5 5 5 5 149/300 Write block 4930 len 16384 data: 5 5 5 4 150/300 Write block 3363 len 24576 data: 7 5 4 3 3 3 151/300 Write block 274 len 40960 data: 3 4 4 4 3 3 3 3 2 2 152/300 Write block 3034 len 8192 data: 5 5 153/300 Flush 154/300 Read block 2023 len 16384 155/300 Read block 2738 len 16384 156/300 Flush 157/300 Read block 2508 len 20480 158/300 Read block 99 len 24576 159/300 Flush 160/300 Write block 634 len 20480 data: 4 5 3 3 4 161/300 Write block 2123 len 20480 data: 3 4 4 4 3 162/300 Read block 2984 len 36864 163/300 Read block 3442 len 20480 164/300 Flush 165/300 Read block 2235 len 20480 166/300 Read block 4775 len 28672 167/300 Read block 241 len 8192 168/300 Read block 3567 len 28672 169/300 Write block 2787 len 40960 data: 3 4 4 3 3 4 4 4 5 5 170/300 Write block 2419 len 8192 data: 2 2 171/300 Flush 172/300 Read block 2377 len 4096 173/300 Write block 3831 len 40960 data: 5 5 4 4 4 4 5 4 3 2 174/300 Read block 892 len 20480 175/300 Write block 1494 len 20480 data: 4 5 5 6 5 176/300 Read block 1775 len 20480 177/300 Read block 2468 len 36864 178/300 Write block 3348 len 40960 data: 3 3 3 3 3 3 3 4 4 5 179/300 Read block 196 len 4096 180/300 Write block 3924 len 36864 data: 3 3 3 4 4 3 3 3 2 181/300 Read block 3457 len 16384 182/300 Read block 4889 len 12288 183/300 Read block 4439 len 16384 184/300 Write block 4588 len 4096 data: 4 185/300 Read block 2960 len 4096 186/300 Read block 2311 len 32768 187/300 Read block 3461 len 4096 188/300 Flush 189/300 Write block 4408 len 32768 data: 3 3 4 4 3 5 6 5 190/300 Read block 979 len 4096 191/300 Write block 4857 len 28672 data: 4 4 5 6 5 6 4 192/300 Read block 4701 len 8192 193/300 Write block 832 len 24576 data: 7 6 6 2 3 3 194/300 Write block 4087 len 16384 data: 2 2 3 4 195/300 Flush 196/300 Write block 3758 len 8192 data: 4 5 197/300 Read block 4281 len 8192 198/300 Write block 4071 len 32768 data: 6 7 7 7 6 5 5 6 199/300 Read block 44 len 16384 200/300 Read block 2302 len 36864 201/300 Read block 3384 len 20480 202/300 Read block 2402 len 4096 203/300 Read block 4256 len 8192 204/300 Flush 205/300 Read block 2814 len 12288 206/300 Read block 1251 len 36864 207/300 Write block 4928 len 20480 data: 4 5 6 6 6 208/300 Read block 2614 len 20480 209/300 Read block 2376 len 32768 210/300 Read block 2013 len 40960 211/300 Write block 1888 len 8192 data: 6 5 212/300 Flush 213/300 Write block 3875 len 12288 data: 2 3 3 214/300 Write block 4679 len 36864 data: 9 8 7 7 4 5 4 5 5 215/300 Write block 1607 len 36864 data: 5 3 4 3 4 5 4 5 4 216/300 Flush 217/300 Write block 2847 len 40960 data: 4 4 3 4 5 5 6 6 5 5 218/300 Flush 219/300 Write block 970 len 40960 data: 2 2 2 2 2 2 5 4 4 3 220/300 Write block 668 len 8192 data: 3 3 221/300 Write block 1866 len 32768 data: 6 6 6 6 6 7 7 7 222/300 Write block 4836 len 20480 data: 3 3 3 3 4 223/300 Write block 538 len 16384 data: 4 3 4 5 224/300 Read block 2048 len 4096 225/300 Read block 4233 len 4096 226/300 Write block 1425 len 20480 data: 5 4 4 3 3 227/300 Write block 3889 len 36864 data: 4 5 5 4 4 4 4 3 4 228/300 Read block 1797 len 28672 229/300 Write block 4215 len 36864 data: 7 6 3 3 3 3 2 2 2 230/300 Write block 4846 len 12288 data: 7 5 3 231/300 Read block 2638 len 4096 232/300 Read block 3648 len 36864 233/300 Read block 2015 len 36864 234/300 Write block 2285 len 40960 data: 5 5 5 4 5 3 3 2 2 2 235/300 Write block 4694 len 28672 data: 2 2 2 3 4 5 5 236/300 Read block 4320 len 20480 237/300 Read block 1047 len 8192 238/300 Write block 4027 len 40960 data: 3 3 4 6 7 7 5 4 5 4 239/300 Read block 4848 len 40960 240/300 Read block 811 len 32768 241/300 Flush 242/300 Read block 76 len 4096 243/300 Read block 1096 len 20480 244/300 Write block 1669 len 36864 data: 3 3 2 3 4 4 5 4 3 245/300 Read block 580 len 20480 246/300 Read block 2514 len 4096 247/300 Write block 4483 len 16384 data: 4 5 5 2 248/300 Write block 2311 len 28672 data: 6 6 5 4 3 3 2 249/300 Flush 250/300 Read block 789 len 12288 251/300 Read block 4595 len 36864 252/300 Flush 253/300 Flush 254/300 Write block 3577 len 8192 data: 3 3 255/300 Read block 880 len 4096 256/300 Read block 2390 len 24576 257/300 Write block 1493 len 20480 data: 3 5 6 6 7 258/300 Read block 1959 len 12288 259/300 Write block 4172 len 36864 data: 4 3 4 3 3 2 3 4 4 260/300 Read block 1006 len 16384 261/300 Write block 2014 len 32768 data: 4 4 4 4 5 4 5 5 262/300 Write block 2309 len 8192 data: 6 6 263/300 Read block 3839 len 20480 264/300 Read block 736 len 36864 265/300 Write block 2268 len 32768 data: 6 6 6 6 4 5 5 5 266/300 Read block 4367 len 24576 267/300 Read block 528 len 36864 268/300 Read block 565 len 36864 269/300 Read block 331 len 4096 270/300 Write block 825 len 8192 data: 8 10 271/300 Read block 445 len 12288 272/300 Write block 3179 len 40960 data: 4 4 3 3 3 2 3 3 4 4 273/300 Write block 1009 len 8192 data: 4 4 274/300 Write block 3304 len 8192 data: 5 5 275/300 Write block 1229 len 24576 data: 5 5 3 4 6 6 276/300 Read block 332 len 28672 277/300 Write block 4790 len 28672 data: 3 4 4 5 6 7 8 278/300 Read block 2463 len 12288 279/300 Read block 1282 len 4096 280/300 Read block 3766 len 16384 281/300 Flush 282/300 Read block 2515 len 28672 283/300 Flush 284/300 Write block 4864 len 12288 data: 3 3 3 285/300 Write block 1988 len 8192 data: 5 6 286/300 Read block 1585 len 40960 287/300 Read block 4326 len 28672 288/300 Write block 2734 len 8192 data: 4 5 289/300 Write block 1767 len 12288 data: 4 3 4 290/300 Flush 291/300 Write block 1627 len 40960 data: 5 6 6 6 6 6 5 4 3 2 292/300 Write block 2734 len 40960 data: 5 6 5 5 6 6 7 7 6 6 293/300 Read block 797 len 8192 294/300 Read block 3043 len 28672 295/300 Write block 4464 len 20480 data: 3 3 4 4 5 296/300 Write block 1525 len 4096 data: 5 297/300 Read block 1145 len 4096 298/300 Read block 1715 len 36864 299/300 Read block 211 len 8192 300/300 Read block 2216 len 40960 301/300 Read block 2547 len 20480 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:46:08.061 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: 456f2ec3-e16a-4e83-ad00-8a86504fb6d8, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6130 Acked Read 9 Done Done Sent false 6131 Acked Write 6 Done Done Sent false 6132 Acked Read 10 Done Done Sent false 6133 Acked Write 5 Done Done Sent false 6134 Acked Write 6 Done Done Sent false 6135 Acked Flush 0 Done Done Sent false 6136 Acked Write 6 Done Done Sent false 6137 Acked Read 9 Done Done Sent false 6138 Acked Write 6 Done Done Sent false 6139 Acked Write 9 Done Done Sent false 6140 Acked Read 1 Done Done Sent false 6141 Acked Read 2 Done Done Sent false 6142 Acked Read 5 Done Done Sent false 6143 Acked Read 4 Done Done Sent false 6144 Acked Flush 0 Done Done Sent false 6145 Acked Read 2 Done Done Sent false 6146 Acked Read 1 Done Done Sent false 6147 Acked Read 9 Done Done Sent false 6148 Acked Write 3 Done Done Sent false 6149 Acked Read 7 Done Done Sent false 6150 Acked Read 1 Done Done Sent false 6151 Acked Read 1 Done Done Sent false 6152 Acked Flush 0 Done Done Sent false 6153 Acked Read 5 Done Done Sent false 6154 Acked Write 2 Done Done Sent false 6155 Acked Write 9 Done Done Sent false 6156 Acked Read 4 Done Done Sent false 6157 Acked Write 9 Done Done Sent false 6158 Acked Flush 0 Done Done Sent false 6159 Acked Read 3 Done Done Sent false 6160 Acked Write 3 Done Done Sent false 6161 Acked Write 6 Done Done Sent false 6162 Acked Write 5 Done Done Sent false 6163 Acked Read 1 Done Done Sent false 6164 Acked Read 8 Done Done Sent false 6165 Acked Read 10 Done Done Sent false 6166 Acked Write 4 Done Done Sent false 6167 Acked Write 9 Done Done Sent false 6168 Acked Read 4 Done Done Sent false 6169 Acked Read 3 Done Done Sent false 6170 Acked Read 6 Done Done Sent false 6171 Acked Write 2 Done Done Sent false 6172 Acked Write 1 Done Done Sent false 6173 Acked Write 1 Done Done Sent false 6174 Acked Read 10 Done Done Sent false 6175 Acked Read 6 Done Done Sent false 6176 Acked Flush 0 Done Done Sent false 6177 Acked Read 4 Done Done Sent false 6178 Acked Read 4 Done Done Sent false 6179 Acked Read 10 Done Done Sent false 6180 Acked Write 10 Done Done Sent false 6181 Acked Write 7 Done Done Sent false 6182 Acked Write 8 Done Done Sent false 6183 Acked Write 4 Done Done Sent false 6184 Acked Read 8 Done Done Sent false 6185 Acked Write 1 Done Done Sent false 6186 Acked Read 4 Done Done Sent false 6187 Acked Read 5 Done Done Sent false 6188 Acked Write 5 Done Done Sent false 6189 Acked Read 8 Done Done Sent false 6190 Acked Read 1 Done Done Sent false 6191 Acked Write 6 Done Done Sent false 6192 Acked Read 4 Done Done Sent false 6193 Acked Read 10 Done Done Sent false 6194 Acked Write 9 Done Done Sent false 6195 Acked Read 3 Done Done Sent false 6196 Acked Flush 0 Done Done Sent false 6197 Acked Read 5 Done Done Sent false 6198 Acked Read 4 Done Done Sent false 6199 Acked Write 9 Done Done Sent false 6200 Acked Write 5 Done Done Sent false 6201 Acked Read 8 Done Done Sent false 6202 Acked Write 10 Done Done Sent false 6203 Acked Read 8 Done Done Sent false 6204 Acked Flush 0 Done Done Sent false 6205 Acked Write 2 Done Done Sent false 6206 Acked Read 7 Done Done Sent false 6207 Acked Read 7 Done Done Sent false 6208 Acked Write 2 Done Done Sent false 6209 Acked Write 1 Done Done Sent false 6210 Acked Write 7 Done Done Sent false 6211 Acked Read 2 Done Done Sent false 6212 Acked Write 2 Done Done Sent false 6213 Acked Flush 0 Done Done Sent false 6214 Acked Write 1 Done Done Sent false 6215 Acked Read 6 Done Done Sent false 6216 Acked Read 7 Done Done Sent false 6217 Acked Write 3 Done Done Sent false 6218 Acked Write 9 Done Done Sent false 6219 Acked Write 2 Done Done Sent false 6220 Acked Read 2 Done Done Sent false 6221 Acked Read 1 Done Done Sent false 6222 Acked Read 1 Done Done Sent false 6223 Acked Read 6 Done Done Sent false 6224 Acked Read 7 Done Done Sent false 6225 Acked Write 5 Done Done Sent false 6226 Acked Read 2 Done Done Sent false 6227 Acked Read 5 Done Done Sent false 6228 Acked Write 3 Done Done Sent false 6229 Acked Read 3 Done Done Sent false 6230 Acked Read 8 Done Done Sent false 6231 Acked Write 6 Done Done Sent false 6232 Acked Flush 0 Done Done Sent false 6233 Acked Flush 0 Done Done Sent false 6234 Acked Write 1 Done Done Sent false 6235 Acked Read 2 Done Done Sent false 6236 Acked Read 3 Done Done Sent false 6237 Acked Write 6 Done Done Sent false 6238 Acked Write 7 Done Done Sent false 6239 Acked Read 7 Done Done Sent false 6240 Acked Read 5 Done Done Sent false 6241 Acked Write 2 Done Done Sent false 6242 Acked Write 10 Done Done Sent false 6243 Acked Flush 0 Done Done Sent false 6244 Acked Read 2 Done Done Sent false 6245 Acked Write 10 Done Done Sent false 6246 Acked Read 6 Done Done Sent false 6247 Acked Write 6 Done Done Sent false 6248 Acked Read 9 Done Done Sent false 6249 Acked Read 9 Done Done Sent false 6250 Acked Read 2 Done Done Sent false 6251 Acked Write 3 Done Done Sent false 6252 Acked Read 4 Done Done Sent false 6253 Acked Read 4 Done Done Sent false 6254 Acked Write 6 Done Done Sent false 6255 Acked Write 10 Done Done Sent false 6256 Acked Write 2 Done Done Sent false 6257 Acked Read 1 Done Done Sent false 6258 Acked Read 10 Done Done Sent false 6259 Acked Read 9 Done Done Sent false 6260 Acked Read 4 Done Done Sent false 6261 Acked Read 4 Done Done Sent false 6262 Acked Read 10 Done Done Sent false 6263 Acked Read 9 Done Done Sent false 6264 Acked Write 6 Done Done Sent false 6265 Acked Write 7 Done Done Sent false 6266 Acked Write 7 Done Done Sent false 6267 Acked Flush 0 Done Done Sent false 6268 Acked Read 4 Done Done Sent false 6269 Acked Write 10 Done Done Sent false 6270 Acked Read 2 Done Done Sent false 6271 Acked Write 7 Done Done Sent false 6272 Acked Write 6 Done Done Sent false 6273 Acked Flush 0 Done Done Sent false 6274 Acked Read 4 Done Done Sent false 6275 Acked Read 4 Done Done Sent false 6276 Acked Write 2 Done Done Sent false 6277 Acked Write 8 Done Done Sent false 6278 Acked Write 4 Done Done Sent false 6279 Acked Write 6 Done Done Sent false 6280 Acked Write 10 Done Done Sent false 6281 Acked Write 2 Done Done Sent false 6282 Acked Flush 0 Done Done Sent false 6283 Acked Read 4 Done Done Sent false 6284 Acked Read 4 Done Done Sent false 6285 Acked Flush 0 Done Done Sent false 6286 Acked Read 5 Done Done Sent false 6287 Acked Read 6 Done Done Sent false 6288 Acked Flush 0 Done Done Sent false 6289 Acked Write 5 Done Done Sent false 6290 Acked Write 5 Done Done Sent false 6291 Acked Read 9 Done Done Sent false 6292 Acked Read 5 Done Done Sent false 6293 Acked Flush 0 Done Done Sent false 6294 Acked Read 5 Done Done Sent false 6295 Acked Read 7 Done Done Sent false 6296 Acked Read 2 Done Done Sent false 6297 Acked Read 7 Done Done Sent false 6298 Acked Write 10 Done Done Sent false 6299 Acked Write 2 Done Done Sent false 6300 Acked Flush 0 Done Done Sent false 6301 Acked Read 1 Done Done Sent false 6302 Acked Write 10 Done Done Sent false 6303 Acked Read 5 Done Done Sent false 6304 Acked Write 5 Done Done Sent false 6305 Acked Read 5 Done Done Sent false 6306 Acked Read 9 Done Done Sent false 6307 Acked Write 10 Done Done Sent false 6308 Acked Read 1 Done Done Sent false 6309 Acked Write 9 Done Done Sent false 6310 Acked Read 4 Done Done Sent false 6311 Acked Read 3 Done Done Sent false 6312 Acked Read 4 Done Done Sent false 6313 Acked Write 1 Done Done Sent false 6314 Acked Read 1 Done Done Sent false 6315 Acked Read 8 Done Done Sent false 6316 Acked Read 1 Done Done Sent false 6317 Acked Flush 0 Done Done Sent false 6318 Acked Write 8 Done Done Sent false 6319 Acked Read 1 Done Done Sent false 6320 Acked Write 7 Done Done Sent false 6321 Acked Read 2 Done Done Sent false 6322 Acked Write 6 Done Done Sent false 6323 Acked Write 4 Done Done Sent false 6324 Acked Flush 0 Done Done Sent false 6325 Acked Write 2 Done Done Sent false 6326 Acked Read 2 Done Done Sent false 6327 Acked Write 8 Done Done Sent false 6328 Acked Read 4 Done Done Sent false 6329 Acked Read 9 Done Done Sent false 6330 Acked Read 5 Done Done Sent false 6331 Acked Read 1 Done Done Sent false 6332 Acked Read 2 Done Done Sent false 6333 Acked Flush 0 Done Done Sent false 6334 Acked Read 3 Done Done Sent false 6335 Acked Read 9 Done Done Sent false 6336 Acked Write 5 Done Done Sent false 6337 Acked Read 5 Done Done Sent false 6338 Acked Read 8 Done Done Sent false 6339 Acked Read 10 Done Done Sent false 6340 Acked Write 2 Done Done Sent false 6341 Acked Flush 0 Done Done Sent false 6342 Acked Write 3 Done Done Sent false 6343 Acked Write 9 Done Done Sent false 6344 Acked Write 9 Done Done Sent false 6345 Acked Flush 0 Done Done Sent false 6346 Acked Write 10 Done Done Sent false 6347 Acked Flush 0 Done Done Sent false 6348 Acked Write 10 Done Done Sent false 6349 Acked Write 2 Done Done Sent false 6350 Acked Write 8 Done Done Sent false 6351 Acked Write 5 Done Done Sent false 6352 Acked Write 4 Done Done Sent false 6353 Acked Read 1 Done Done Sent false 6354 Acked Read 1 Done Done Sent false 6355 Acked Write 5 Done Done Sent false 6356 Acked Write 9 Done Done Sent false 6357 Acked Read 7 Done Done Sent false 6358 Acked Write 9 Done Done Sent false 6359 Acked Write 3 Done Done Sent false 6360 Acked Read 1 Done Done Sent false 6361 Acked Read 9 Done Done Sent false 6362 Acked Read 9 Done Done Sent false 6363 Acked Write 10 Done Done Sent false 6364 Acked Write 7 Done Done Sent false 6365 Acked Read 5 Done Done Sent false 6366 Acked Read 2 Done Done Sent false 6367 Acked Write 10 Done Done Sent false 6368 Acked Read 10 Done Done Sent false 6369 Acked Read 8 Done Done Sent false 6370 Acked Flush 0 Done Done Sent false 6371 Acked Read 1 Done Done Sent false 6372 Acked Read 5 Done Done Sent false 6373 Acked Write 9 Done Done Sent false 6374 Acked Read 5 Done Done Sent false 6375 Acked Read 1 Done Done Sent false 6376 Acked Write 4 Done Done Sent false 6377 Acked Write 7 Done Done Sent false 6378 Acked Flush 0 Done Done Sent false 6379 Acked Read 3 Done Done Sent false 6380 Acked Read 9 Done Done Sent false 6381 Acked Flush 0 Done Done Sent false 6382 Acked Flush 0 Done Done Sent false 6383 Acked Write 2 Done Done Sent false 6384 Acked Read 1 Done Done Sent false 6385 Acked Read 6 Done Done Sent false 6386 Acked Write 5 Done Done Sent false 6387 Acked Read 3 Done Done Sent false 6388 Acked Write 9 Done Done Sent false 6389 Acked Read 4 Done Done Sent false 6390 Acked Write 8 Done Done Sent false 6391 Acked Write 2 Done Done Sent false 6392 Acked Read 5 Done Done Sent false 6393 Acked Read 9 Done Done Sent false 6394 Acked Write 8 Done Done Sent false 6395 Acked Read 6 Done Done Sent false 6396 Acked Read 9 Done Done Sent false 6397 Acked Read 9 Done Done Sent false 6398 Acked Read 1 Done Done Sent false 6399 Acked Write 2 Done Done Sent false 6400 Acked Read 3 Done Done Sent false 6401 Acked Write 10 Done Done Sent false 6402 Acked Write 2 Done Done Sent false 6403 Acked Write 2 Done Done Sent false 6404 Acked Write 6 Done Done Sent false 6405 Acked Read 7 Done Done Sent false 6406 Acked Write 7 Done Done Sent false 6407 Acked Read 3 Done Done Sent false 6408 Acked Read 1 Done Done Sent false 6409 Acked Read 4 Done Done Sent false 6410 Acked Flush 0 Done Done Sent false 6411 Acked Read 7 Done Done Sent false 6412 Acked Flush 0 Done Done Sent false 6413 Acked Write 3 Done Done Sent false 6414 Acked Write 2 Done Done Sent false 6415 Acked Read 10 Done Done Sent false 6416 Acked Read 7 Done Done Sent false 6417 Acked Write 2 Done Done Sent false 6418 Acked Write 3 Done Done Sent false 6419 Acked Flush 0 Done Done Sent false 6420 Acked Write 10 Done Done Sent false 6421 Acked Write 10 Done Done Sent false 6422 Acked Read 2 Done Done Sent false 6423 Acked Read 7 Done Done Sent false 6424 Acked Write 5 Done Done Sent false 6425 Acked Write 1 Done Done Sent false 6426 Acked Read 1 Done Done Sent false 6427 Acked Read 9 Done Done Sent false 6428 Acked Read 2 Done Done Sent false 6429 Acked Read 10 Done Done Sent false 6430 Acked Read 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6419 6419 6129 Downstairs last five completed: 6129 6128 6127 6126 6125 Upstairs last five completed: 6430 6429 6428 6427 6426 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6130 Acked Read 9 Done Done Sent false 6131 Acked Write 6 Done Done Sent false 6132 Acked Read 10 Done Done Sent false 6133 Acked Write 5 Done Done Sent false 6134 Acked Write 6 Done Done Sent false 6135 Acked Flush 0 Done Done Sent false 6136 Acked Write 6 Done Done Sent false 6137 Acked Read 9 Done Done Sent false 6138 Acked Write 6 Done Done Sent false 6139 Acked Write 9 Done Done Sent false 6140 Acked Read 1 Done Done Sent false 6141 Acked Read 2 Done Done Sent false 6142 Acked Read 5 Done Done Sent false 6143 Acked Read 4 Done Done Sent false 6144 Acked Flush 0 Done Done Sent false 6145 Acked Read 2 Done Done Sent false 6146 Acked Read 1 Done Done Sent false 6147 Acked Read 9 Done Done Sent false 6148 Acked Write 3 Done Done Sent false 6149 Acked Read 7 Done Done Sent false 6150 Acked Read 1 Done Done Sent false 6151 Acked Read 1 Done Done Sent false 6152 Acked Flush 0 Done Done Sent false 6153 Acked Read 5 Done Done Sent false 6154 Acked Write 2 Done Done Sent false 6155 Acked Write 9 Done Done Sent false 6156 Acked Read 4 Done Done Sent false 6157 Acked Write 9 Done Done Sent false 6158 Acked Flush 0 Done Done Sent false 6159 Acked Read 3 Done Done Sent false 6160 Acked Write 3 Done Done Sent false 6161 Acked Write 6 Done Done Sent false 6162 Acked Write 5 Done Done Sent false 6163 Acked Read 1 Done Done Sent false 6164 Acked Read 8 Done Done Sent false 6165 Acked Read 10 Done Done Sent false 6166 Acked Write 4 Done Done Sent false 6167 Acked Write 9 Done Done Sent false 6168 Acked Read 4 Done Done Sent false 6169 Acked Read 3 Done Done Sent false 6170 Acked Read 6 Done Done Sent false 6171 Acked Write 2 Done Done Sent false 6172 Acked Write 1 Done Done Sent false 6173 Acked Write 1 Done Done Sent false 6174 Acked Read 10 Done Done Sent false 6175 Acked Read 6 Done Done Sent false 6176 Acked Flush 0 Done Done Sent false 6177 Acked Read 4 Done Done Sent false 6178 Acked Read 4 Done Done Sent false 6179 Acked Read 10 Done Done Sent false 6180 Acked Write 10 Done Done Sent false 6181 Acked Write 7 Done Done Sent false 6182 Acked Write 8 Done Done Sent false 6183 Acked Write 4 Done Done Sent false 6184 Acked Read 8 Done Done Sent false 6185 Acked Write 1 Done Done Sent false 6186 Acked Read 4 Done Done Sent false 6187 Acked Read 5 Done Done Sent false 6188 Acked Write 5 Done Done Sent false 6189 Acked Read 8 Done Done Sent false 6190 Acked Read 1 Done Done Sent false 6191 Acked Write 6 Done Done Sent false 6192 Acked Read 4 Done Done Sent false 6193 Acked Read 10 Done Done Sent false 6194 Acked Write 9 Done Done Sent false 6195 Acked Read 3 Done Done Sent false 6196 Acked Flush 0 Done Done Sent false 6197 Acked Read 5 Done Done Sent false 6198 Acked Read 4 Done Done Sent false 6199 Acked Write 9 Done Done Sent false 6200 Acked Write 5 Done Done Sent false 6201 Acked Read 8 Done Done Sent false 6202 Acked Write 10 Done Done Sent false 6203 Acked Read 8 Done Done Sent false 6204 Acked Flush 0 Done Done Sent false 6205 Acked Write 2 Done Done Sent false 6206 Acked Read 7 Done Done Sent false 6207 Acked Read 7 Done Done Sent false 6208 Acked Write 2 Done Done Sent false 6209 Acked Write 1 Done Done Sent false 6210 Acked Write 7 Done Done Sent false 6211 Acked Read 2 Done Done Sent false 6212 Acked Write 2 Done Done Sent false 6213 Acked Flush 0 Done Done Sent false 6214 Acked Write 1 Done Done Sent false 6215 Acked Read 6 Done Done Sent false 6216 Acked Read 7 Done Done Sent false 6217 Acked Write 3 Done Done Sent false 6218 Acked Write 9 Done Done Sent false 6219 Acked Write 2 Done Done Sent false 6220 Acked Read 2 Done Done Sent false 6221 Acked Read 1 Done Done Sent false 6222 Acked Read 1 Done Done Sent false 6223 Acked Read 6 Done Done Sent false 6224 Acked Read 7 Done Done Sent false 6225 Acked Write 5 Done Done Sent false 6226 Acked Read 2 Done Done Sent false 6227 Acked Read 5 Done Done Sent false 6228 Acked Write 3 Done Done Sent false 6229 Acked Read 3 Done Done Sent false 6230 Acked Read 8 Done Done Sent false 6231 Acked Write 6 Done Done Sent false 6232 Acked Flush 0 Done Done Sent false 6233 Acked Flush 0 Done Done Sent false 6234 Acked Write 1 Done Done Sent false 6235 Acked Read 2 Done Done Sent false 6236 Acked Read 3 Done Done Sent false 6237 Acked Write 6 Done Done Sent false 6238 Acked Write 7 Done Done Sent false 6239 Acked Read 7 Done Done Sent false 6240 Acked Read 5 Done Done Sent false 6241 Acked Write 2 Done Done Sent false 6242 Acked Write 10 Done Done Sent false 6243 Acked Flush 0 Done Done Sent false 6244 Acked Read 2 Done Done Sent false 6245 Acked Write 10 Done Done Sent false 6246 Acked Read 6 Done Done Sent false 6247 Acked Write 6 Done Done Sent false 6248 Acked Read 9 Done Done Sent false 6249 Acked Read 9 Done Done Sent false 6250 Acked Read 2 Done Done Sent false 6251 Acked Write 3 Done Done Sent false 6252 Acked Read 4 Done Done Sent false 6253 Acked Read 4 Done Done Sent false 6254 Acked Write 6 Done Done Sent false 6255 Acked Write 10 Done Done Sent false 6256 Acked Write 2 Done Done Sent false 6257 Acked Read 1 Done Done Sent false 6258 Acked Read 10 Done Done Sent false 6259 Acked Read 9 Done Done Sent false 6260 Acked Read 4 Done Done Sent false 6261 Acked Read 4 Done Done Sent false 6262 Acked Read 10 Done Done Sent false 6263 Acked Read 9 Done Done Sent false 6264 Acked Write 6 Done Done Sent false 6265 Acked Write 7 Done Done Sent false 6266 Acked Write 7 Done Done Sent false 6267 Acked Flush 0 Done Done Sent false 6268 Acked Read 4 Done Done Sent false 6269 Acked Write 10 Done Done Sent false 6270 Acked Read 2 Done Done Sent false 6271 Acked Write 7 Done Done Sent false 6272 Acked Write 6 Done Done Sent false 6273 Acked Flush 0 Done Done Sent false 6274 Acked Read 4 Done Done Sent false 6275 Acked Read 4 Done Done Sent false 6276 Acked Write 2 Done Done Sent false 6277 Acked Write 8 Done Done Sent false 6278 Acked Write 4 Done Done Sent false 6279 Acked Write 6 Done Done Sent false 6280 Acked Write 10 Done Done Sent false 6281 Acked Write 2 Done Done Sent false 6282 Acked Flush 0 Done Done Sent false 6283 Acked Read 4 Done Done Sent false 6284 Acked Read 4 Done Done Sent false 6285 Acked Flush 0 Done Done Sent false 6286 Acked Read 5 Done Done Sent false 6287 Acked Read 6 Done Done Sent false 6288 Acked Flush 0 Done Done Sent false 6289 Acked Write 5 Done Done Sent false 6290 Acked Write 5 Done Done Sent false 6291 Acked Read 9 Done Done Sent false 6292 Acked Read 5 Done Done Sent false 6293 Acked Flush 0 Done Done Sent false 6294 Acked Read 5 Done Done Sent false 6295 Acked Read 7 Done Done Sent false 6296 Acked Read 2 Done Done Sent false 6297 Acked Read 7 Done Done Sent false 6298 Acked Write 10 Done Done Sent false 6299 Acked Write 2 Done Done Sent false 6300 Acked Flush 0 Done Done Sent false 6301 Acked Read 1 Done Done Sent false 6302 Acked Write 10 Done Done Sent false 6303 Acked Read 5 Done Done Sent false 6304 Acked Write 5 Done Done Sent false 6305 Acked Read 5 Done Done Sent false 6306 Acked Read 9 Done Done Sent false 6307 Acked Write 10 Done Done Sent false 6308 Acked Read 1 Done Done Sent false 6309 Acked Write 9 Done Done Sent false 6310 Acked Read 4 Done Done Sent false 6311 Acked Read 3 Done Done Sent false 6312 Acked Read 4 Done Done Sent false 6313 Acked Write 1 Done Done Sent false 6314 Acked Read 1 Done Done Sent false 6315 Acked Read 8 Done Done Sent false 6316 Acked Read 1 Done Done Sent false 6317 Acked Flush 0 Done Done Sent false 6318 Acked Write 8 Done Done Sent false 6319 Acked Read 1 Done Done Sent false 6320 Acked Write 7 Done Done Sent false 6321 Acked Read 2 Done Done Sent false 6322 Acked Write 6 Done Done Sent false 6323 Acked Write 4 Done Done Sent false 6324 Acked Flush 0 Done Done Sent false 6325 Acked Write 2 Done Done Sent false 6326 Acked Read 2 Done Done Sent false 6327 Acked Write 8 Done Done Sent false 6328 Acked Read 4 Done Done Sent false 6329 Acked Read 9 Done Done Sent false 6330 Acked Read 5 Done Done Sent false 6331 Acked Read 1 Done Done Sent false 6332 Acked Read 2 Done Done Sent false 6333 Acked Flush 0 Done Done Sent false 6334 Acked Read 3 Done Done Sent false 6335 Acked Read 9 Done Done Sent false 6336 Acked Write 5 Done Done Sent false 6337 Acked Read 5 Done Done Sent false 6338 Acked Read 8 Done Done Sent false 6339 Acked Read 10 Done Done Sent false 6340 Acked Write 2 Done Done Sent false 6341 Acked Flush 0 Done Done Sent false 6342 Acked Write 3 Done Done Sent false 6343 Acked Write 9 Done Done Sent false 6344 Acked Write 9 Done Done Sent false 6345 Acked Flush 0 Done Done Sent false 6346 Acked Write 10 Done Done Sent false 6347 Acked Flush 0 Done Done Sent false 6348 Acked Write 10 Done Done Sent false 6349 Acked Write 2 Done Done Sent false 6350 Acked Write 8 Done Done Sent false 6351 Acked Write 5 Done Done Sent false 6352 Acked Write 4 Done Done Sent false 6353 Acked Read 1 Done Done Sent false 6354 Acked Read 1 Done Done Sent false 6355 Acked Write 5 Done Done Sent false 6356 Acked Write 9 Done Done Sent false 6357 Acked Read 7 Done Done Sent false 6358 Acked Write 9 Done Done Sent false 6359 Acked Write 3 Done Done Sent false 6360 Acked Read 1 Done Done Sent false 6361 Acked Read 9 Done Done Sent false 6362 Acked Read 9 Done Done Sent false 6363 Acked Write 10 Done Done Sent false 6364 Acked Write 7 Done Done Sent false 6365 Acked Read 5 Done Done Sent false 6366 Acked Read 2 Done Done Sent false 6367 Acked Write 10 Done Done Sent false 6368 Acked Read 10 Done Done Sent false 6369 Acked Read 8 Done Done Sent false 6370 Acked Flush 0 Done Done Sent false 6371 Acked Read 1 Done Done Sent false 6372 Acked Read 5 Done Done Sent false 6373 Acked Write 9 Done Done Sent false 6374 Acked Read 5 Done Done Sent false 6375 Acked Read 1 Done Done Sent false 6376 Acked Write 4 Done Done Sent false 6377 Acked Write 7 Done Done Sent false 6378 Acked Flush 0 Done Done Sent false 6379 Acked Read 3 Done Done Sent false 6380 Acked Read 9 Done Done Sent false 6381 Acked Flush 0 Done Done Sent false 6382 Acked Flush 0 Done Done Sent false 6383 Acked Write 2 Done Done Sent false 6384 Acked Read 1 Done Done Sent false 6385 Acked Read 6 Done Done Sent false 6386 Acked Write 5 Done Done Sent false 6387 Acked Read 3 Done Done Sent false 6388 Acked Write 9 Done Done Sent false 6389 Acked Read 4 Done Done Sent false 6390 Acked Write 8 Done Done Sent false 6391 Acked Write 2 Done Done Sent false 6392 Acked Read 5 Done Done Sent false 6393 Acked Read 9 Done Done Sent false 6394 Acked Write 8 Done Done Sent false 6395 Acked Read 6 Done Done Sent false 6396 Acked Read 9 Done Done Sent false 6397 Acked Read 9 Done Done Sent false 6398 Acked Read 1 Done Done Sent false 6399 Acked Write 2 Done Done Sent false 6400 Acked Read 3 Done Done Sent false 6401 Acked Write 10 Done Done Sent false 6402 Acked Write 2 Done Done Sent false 6403 Acked Write 2 Done Done Sent false 6404 Acked Write 6 Done Done Sent false 6405 Acked Read 7 Done Done Sent false 6406 Acked Write 7 Done Done Sent false 6407 Acked Read 3 Done Done Sent false 6408 Acked Read 1 Done Done Sent false 6409 Acked Read 4 Done Done Sent false 6410 Acked Flush 0 Done Done Sent false 6411 Acked Read 7 Done Done Sent false 6412 Acked Flush 0 Done Done Sent false 6413 Acked Write 3 Done Done Sent false 6414 Acked Write 2 Done Done Sent false 6415 Acked Read 10 Done Done Sent false 6416 Acked Read 7 Done Done Sent false 6417 Acked Write 2 Done Done Sent false 6418 Acked Write 3 Done Done Sent false 6419 Acked Flush 0 Done Done Sent false 6420 Acked Write 10 Done Done Sent false 6421 Acked Write 10 Done Done Sent false 6422 Acked Read 2 Done Done Sent false 6423 Acked Read 7 Done Done Sent false 6424 Acked Write 5 Done Done Sent false 6425 Acked Write 1 Done Done Sent false 6426 Acked Read 1 Done Done Sent false 6427 Acked Read 9 Done Done Sent false 6428 Acked Read 2 Done Done Sent false 6429 Acked Read 10 Done Done Sent false 6430 Acked Read 5 Done Done Sent false 6431 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6431 6431 6129 Downstairs last five completed: 6129 6128 6127 6126 6125 Upstairs last five completed: 6431 6430 6429 6428 6427 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.83274185Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:57413)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.832901627Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:13.833782341Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834362345Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834385659Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 6129","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834401004Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 6129","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834610088Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834629723Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834644049Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 6129","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834657986Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 6129","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:13.834907029Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6431 6430 6429 6428 6427 Upstairs last five completed: 6431 6430 6429 6428 6427 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 [8820] Got stop action so:false kr:false Nov 27 01:46:16.065 INFO request completed, latency_us: 413, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6155072d-43a8-4354-8468-42f22cab0b60, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:46:16.066 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:16.066217406Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} , latency_us: 135, response_code: 200, uri: /state/cid/1, method: GET, req_id: 646d02ae-e412-40ee-b279-5823c9d06c25, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:16.066289066Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .0.1:51788, local_addr: 127.0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:16.066319948Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .0.1:9998 {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:16.066342492Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:16.066366306Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:16.06638605Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:46:20.067 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 25767abb-5636-460d-a4e5-448d7a9c0e15, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Flush 002/300 Read block 1970 len 20480 003/300 Read block 4780 len 32768 004/300 Read block 1222 len 20480 005/300 Read block 4911 len 16384 006/300 Write block 1935 len 20480 data: 4 3 3 4 4 007/300 Read block 216 len 24576 008/300 Flush 009/300 Write block 4663 len 16384 data: 7 6 5 5 010/300 Read block 4199 len 28672 011/300 Write block 718 len 8192 data: 5 4 012/300 Write block 2354 len 4096 data: 5 013/300 Read block 4984 len 4096 014/300 Write block 3021 len 40960 data: 5 4 4 3 4 4 4 4 4 6 015/300 Flush 016/300 Read block 1002 len 28672 017/300 Read block 386 len 16384 018/300 Flush 019/300 Read block 4266 len 28672 020/300 Read block 2832 len 12288 021/300 Write block 2794 len 12288 data: 5 6 6 022/300 Write block 25 len 40960 data: 4 4 4 3 6 6 7 6 6 5 023/300 Write block 4612 len 32768 data: 4 3 2 2 2 2 2 3 024/300 Read block 90 len 12288 025/300 Write block 2676 len 12288 data: 3 3 3 026/300 Write block 3799 len 28672 data: 4 4 4 4 5 6 6 027/300 Flush 028/300 Read block 2575 len 32768 029/300 Read block 145 len 16384 030/300 Write block 2537 len 32768 data: 4 4 5 6 8 9 8 8 031/300 Read block 2857 len 32768 032/300 Write block 1642 len 36864 data: 2 2 3 5 5 5 5 5 5 033/300 Write block 2262 len 32768 data: 5 5 6 5 5 6 7 7 034/300 Read block 1467 len 32768 035/300 Write block 4571 len 8192 data: 6 4 036/300 Read block 3419 len 28672 037/300 Read block 3700 len 40960 038/300 Read block 1583 len 8192 039/300 Write block 384 len 24576 data: 2 2 2 2 3 4 040/300 Write block 351 len 12288 data: 3 3 3 041/300 Read block 187 len 16384 042/300 Read block 3830 len 32768 043/300 Write block 1112 len 40960 data: 6 6 6 6 5 4 4 4 3 2 044/300 Flush 045/300 Read block 3996 len 36864 046/300 Read block 4885 len 24576 047/300 Write block 4961 len 32768 data: 4 5 5 5 4 2 3 5 048/300 Read block 635 len 8192 049/300 Write block 3982 len 28672 data: 2 2 2 2 3 3 3 050/300 Write block 1112 len 28672 data: 7 7 7 7 6 5 5 051/300 Read block 1079 len 28672 052/300 Write block 3210 len 32768 data: 6 6 5 6 5 4 3 3 053/300 Write block 2658 len 8192 data: 2 2 054/300 Read block 2320 len 20480 055/300 Write block 494 len 24576 data: 5 4 3 3 3 3 056/300 Flush 057/300 Flush 058/300 Flush 059/300 Read block 1732 len 12288 060/300 Write block 1453 len 12288 data: 5 4 4 061/300 Write block 2825 len 12288 data: 6 6 4 062/300 Write block 2986 len 36864 data: 3 3 3 3 4 4 4 3 3 063/300 Write block 977 len 4096 data: 5 064/300 Write block 3732 len 4096 data: 4 065/300 Write block 2288 len 8192 data: 5 6 066/300 Flush 067/300 Read block 3261 len 12288 068/300 Flush 069/300 Read block 3601 len 4096 070/300 Flush 071/300 Read block 2369 len 12288 072/300 Read block 332 len 8192 073/300 Write block 579 len 16384 data: 6 7 6 6 074/300 Flush 075/300 Read block 2820 len 16384 076/300 Read block 2929 len 12288 077/300 Read block 3913 len 12288 078/300 Flush 079/300 Read block 917 len 40960 080/300 Read block 1452 len 8192 081/300 Read block 3830 len 36864 082/300 Read block 4291 len 12288 083/300 Read block 4132 len 40960 084/300 Read block 766 len 24576 085/300 Write block 2876 len 12288 data: 3 3 3 086/300 Write block 4380 len 12288 data: 2 3 3 087/300 Write block 4988 len 8192 data: 5 5 088/300 Read block 7 len 16384 089/300 Write block 551 len 36864 data: 4 4 3 3 3 3 2 2 2 090/300 Write block 1955 len 32768 data: 6 4 6 7 7 5 4 5 091/300 Write block 4833 len 32768 data: 4 3 2 4 4 4 4 5 092/300 Write block 814 len 8192 data: 7 6 093/300 Write block 4688 len 36864 data: 6 5 5 5 3 3 3 3 3 094/300 Read block 4848 len 24576 095/300 Write block 3245 len 36864 data: 2 3 3 2 3 3 3 3 3 096/300 Flush 097/300 Read block 4134 len 20480 098/300 Read block 1877 len 24576 099/300 Write block 2394 len 16384 data: 3 4 4 4 100/300 Write block 4962 len 4096 data: 6 101/300 Read block 1683 len 4096 102/300 Flush 103/300 Write block 4640 len 20480 data: 4 4 5 5 5 104/300 Write block 1336 len 40960 data: 4 5 5 4 4 3 3 4 4 4 105/300 Write block 2835 len 36864 data: 3 3 3 4 5 6 5 7 7 106/300 Write block 892 len 16384 data: 5 5 5 4 107/300 Write block 1202 len 20480 data: 5 6 6 5 4 108/300 Write block 1245 len 16384 data: 4 5 6 7 109/300 Write block 1479 len 4096 data: 2 110/300 Read block 4892 len 24576 111/300 Flush 112/300 Read block 3844 len 24576 113/300 Read block 4655 len 12288 114/300 Write block 4798 len 40960 data: 10 9 9 7 6 6 6 5 4 3 115/300 Read block 166 len 36864 116/300 Read block 657 len 16384 117/300 Write block 2003 len 4096 data: 6 118/300 Read block 791 len 32768 119/300 Write block 4911 len 12288 data: 3 4 4 120/300 Flush 121/300 Flush 122/300 Read block 1947 len 40960 123/300 Read block 4123 len 20480 124/300 Write block 3420 len 28672 data: 2 3 4 3 3 2 2 125/300 Write block 1883 len 20480 data: 6 5 5 5 6 126/300 Read block 431 len 8192 127/300 Flush 128/300 Write block 2635 len 40960 data: 3 4 5 5 5 3 4 4 2 3 129/300 Read block 965 len 40960 130/300 Flush 131/300 Read block 2161 len 8192 132/300 Write block 2816 len 32768 data: 4 4 4 4 3 3 4 5 133/300 Read block 1772 len 8192 134/300 Read block 3557 len 36864 135/300 Read block 2003 len 16384 136/300 Read block 385 len 24576 137/300 Write block 1571 len 28672 data: 4 5 8 7 8 8 7 138/300 Read block 3867 len 40960 139/300 Write block 2630 len 4096 data: 3 140/300 Read block 4098 len 4096 141/300 Read block 2882 len 12288 142/300 Read block 1931 len 32768 143/300 Read block 3914 len 4096 144/300 Write block 4654 len 16384 data: 3 4 4 5 145/300 Write block 3922 len 4096 data: 3 146/300 Read block 1885 len 16384 147/300 Read block 788 len 4096 148/300 Write block 3779 len 32768 data: 3 3 4 4 4 4 4 5 149/300 Read block 1693 len 16384 150/300 Read block 599 len 36864 151/300 Read block 2998 len 4096 152/300 Read block 1618 len 4096 153/300 Read block 511 len 40960 154/300 Read block 449 len 28672 155/300 Read block 4944 len 32768 156/300 Write block 1716 len 28672 data: 4 4 4 3 3 4 4 157/300 Read block 4756 len 24576 158/300 Read block 3829 len 40960 159/300 Flush 160/300 Read block 2750 len 20480 161/300 Read block 1051 len 8192 162/300 Write block 353 len 28672 data: 4 4 5 2 2 2 2 163/300 Read block 786 len 36864 164/300 Read block 1411 len 28672 165/300 Write block 412 len 12288 data: 2 2 4 166/300 Write block 1979 len 28672 data: 5 5 6 5 3 3 4 167/300 Write block 1208 len 8192 data: 4 5 168/300 Flush 169/300 Write block 3514 len 20480 data: 3 4 4 5 5 170/300 Write block 831 len 8192 data: 9 8 171/300 Write block 2403 len 8192 data: 6 6 172/300 Write block 3446 len 4096 data: 3 173/300 Read block 2848 len 28672 174/300 Write block 3753 len 36864 data: 6 6 5 6 4 5 6 6 5 175/300 Flush 176/300 Flush 177/300 Write block 3541 len 20480 data: 7 6 5 5 4 178/300 Read block 4181 len 16384 179/300 Write block 197 len 32768 data: 3 3 2 2 3 4 4 4 180/300 Read block 3141 len 16384 181/300 Read block 421 len 32768 182/300 Write block 2440 len 24576 data: 6 5 5 5 4 4 183/300 Write block 4090 len 32768 data: 5 5 5 6 7 7 6 5 184/300 Write block 4818 len 20480 data: 2 2 2 2 2 185/300 Write block 465 len 32768 data: 5 5 5 6 7 7 5 4 186/300 Write block 4250 len 16384 data: 3 3 3 3 187/300 Write block 4013 len 20480 data: 3 3 3 4 4 188/300 Write block 3485 len 12288 data: 3 3 4 189/300 Read block 2992 len 36864 190/300 Read block 3858 len 32768 191/300 Read block 3825 len 4096 192/300 Write block 4188 len 28672 data: 3 3 3 3 3 3 4 193/300 Read block 2310 len 8192 194/300 Write block 3501 len 4096 data: 2 195/300 Read block 642 len 32768 196/300 Read block 3530 len 16384 197/300 Write block 1815 len 32768 data: 6 7 7 6 6 7 7 8 198/300 Read block 2097 len 28672 199/300 Read block 4477 len 4096 200/300 Flush 201/300 Write block 3850 len 20480 data: 4 4 4 3 3 202/300 Read block 1366 len 4096 203/300 Read block 3283 len 12288 204/300 Write block 1884 len 20480 data: 6 6 6 7 7 205/300 Write block 3156 len 8192 data: 4 5 206/300 Write block 4161 len 24576 data: 4 4 4 4 4 5 207/300 Write block 1346 len 4096 data: 4 208/300 Write block 2568 len 28672 data: 7 7 6 6 6 4 3 209/300 Flush 210/300 Flush 211/300 Read block 3809 len 16384 212/300 Read block 1230 len 28672 213/300 Read block 3163 len 36864 214/300 Read block 2621 len 32768 215/300 Write block 1483 len 36864 data: 4 4 3 4 4 3 3 3 3 216/300 Write block 1746 len 40960 data: 2 2 3 3 5 6 7 6 5 5 217/300 Write block 2652 len 4096 data: 5 218/300 Read block 1665 len 20480 219/300 Flush 220/300 Flush 221/300 Read block 909 len 24576 222/300 Write block 1690 len 32768 data: 8 4 5 5 4 5 5 5 223/300 Read block 3623 len 20480 224/300 Read block 4926 len 4096 225/300 Read block 2763 len 36864 226/300 Write block 2730 len 32768 data: 4 3 5 5 6 7 6 6 227/300 Write block 2604 len 8192 data: 5 5 228/300 Read block 2921 len 24576 229/300 Read block 2889 len 12288 230/300 Read block 4012 len 4096 231/300 Write block 2832 len 32768 data: 4 2 2 4 4 4 5 6 232/300 Read block 2518 len 36864 233/300 Read block 4585 len 20480 234/300 Read block 3321 len 4096 235/300 Read block 1474 len 8192 236/300 Write block 3994 len 8192 data: 4 4 237/300 Write block 1645 len 4096 data: 6 238/300 Read block 867 len 28672 239/300 Read block 2287 len 24576 240/300 Write block 3490 len 40960 data: 6 6 4 4 4 3 2 2 2 2 241/300 Read block 4779 len 16384 242/300 Write block 2593 len 32768 data: 2 2 2 3 3 3 2 3 243/300 Write block 4839 len 36864 data: 5 6 5 6 5 6 7 8 6 244/300 Write block 1367 len 20480 data: 5 6 6 6 5 245/300 Write block 1098 len 20480 data: 6 5 4 3 5 246/300 Write block 1493 len 24576 data: 4 6 7 7 8 6 247/300 Flush 248/300 Write block 3124 len 36864 data: 4 4 4 4 5 5 5 5 5 249/300 Write block 2428 len 36864 data: 3 2 2 2 2 2 2 2 3 250/300 Read block 3797 len 24576 251/300 Read block 2560 len 4096 252/300 Write block 1263 len 20480 data: 5 6 7 7 6 253/300 Read block 943 len 28672 254/300 Read block 2105 len 4096 255/300 Read block 1470 len 40960 256/300 Write block 1848 len 28672 data: 3 4 5 5 6 5 6 257/300 Write block 1445 len 4096 data: 2 258/300 Read block 4836 len 36864 259/300 Read block 4488 len 32768 260/300 Write block 2279 len 36864 data: 4 3 3 3 4 4 6 6 6 261/300 Read block 4348 len 8192 262/300 Read block 3454 len 40960 263/300 Read block 971 len 24576 264/300 Flush 265/300 Read block 4639 len 12288 266/300 Read block 2282 len 20480 267/300 Read block 2973 len 36864 268/300 Write block 3939 len 32768 data: 4 5 5 5 5 4 3 4 269/300 Read block 879 len 24576 270/300 Write block 4935 len 16384 data: 3 2 4 7 271/300 Write block 442 len 36864 data: 5 5 5 5 5 5 5 4 2 272/300 Write block 3984 len 16384 data: 3 3 4 4 273/300 Read block 3012 len 32768 274/300 Read block 1955 len 28672 275/300 Write block 257 len 28672 data: 4 3 3 3 6 6 6 276/300 Write block 4230 len 28672 data: 3 3 4 4 3 3 4 277/300 Write block 2074 len 28672 data: 4 4 4 3 5 5 5 278/300 Flush 279/300 Read block 3793 len 4096 280/300 Read block 1418 len 16384 281/300 Read block 842 len 28672 282/300 Read block 937 len 20480 283/300 Read block 1516 len 8192 284/300 Flush 285/300 Write block 4922 len 24576 data: 4 4 4 4 4 4 286/300 Write block 2936 len 36864 data: 5 5 5 5 6 6 6 5 3 287/300 Write block 2502 len 36864 data: 6 5 6 7 7 6 5 5 4 288/300 Read block 4803 len 8192 289/300 Read block 1295 len 16384 290/300 Write block 706 len 8192 data: 3 4 291/300 Read block 1779 len 16384 292/300 Read block 1215 len 4096 293/300 Read block 4533 len 24576 294/300 Read block 4593 len 8192 295/300 Read block 4994 len 20480 296/300 Flush 297/300 Flush 298/300 Flush 299/300 Flush 300/300 Read block 3439 len 4096 301/300 Write block 1284 len 8192 data: 5 4 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:46:20.287 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/1, method: POST, req_id: f68b387d-b926-4f96-b488-f699ca09be9d, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6432 Acked Flush 0 Done Sent Done false 6433 Acked Read 5 Done Sent Done false 6434 Acked Read 8 Done Sent Done false 6435 Acked Read 5 Done Sent Done false 6436 Acked Read 4 Done Sent Done false 6437 Acked Write 5 Done Sent Done false 6438 Acked Read 6 Done Sent Done false 6439 Acked Flush 0 Done Sent Done false 6440 Acked Write 4 Done Sent Done false 6441 Acked Read 7 Done Sent Done false 6442 Acked Write 2 Done Sent Done false 6443 Acked Write 1 Done Sent Done false 6444 Acked Read 1 Done Sent Done false 6445 Acked Write 10 Done Sent Done false 6446 Acked Flush 0 Done Sent Done false 6447 Acked Read 7 Done Sent Done false 6448 Acked Read 4 Done Sent Done false 6449 Acked Flush 0 Done Sent Done false 6450 Acked Read 7 Done Sent Done false 6451 Acked Read 3 Done Sent Done false 6452 Acked Write 3 Done Sent Done false 6453 Acked Write 10 Done Sent Done false 6454 Acked Write 8 Done Sent Done false 6455 Acked Read 3 Done Sent Done false 6456 Acked Write 3 Done Sent Done false 6457 Acked Write 7 Done Sent Done false 6458 Acked Flush 0 Done Sent Done false 6459 Acked Read 8 Done Sent Done false 6460 Acked Read 4 Done Sent Done false 6461 Acked Write 8 Done Sent Done false 6462 Acked Read 8 Done Sent Done false 6463 Acked Write 9 Done Sent Done false 6464 Acked Write 8 Done Sent Done false 6465 Acked Read 8 Done Sent Done false 6466 Acked Write 2 Done Sent Done false 6467 Acked Read 7 Done Sent Done false 6468 Acked Read 10 Done Sent Done false 6469 Acked Read 2 Done Sent Done false 6470 Acked Write 6 Done Sent Done false 6471 Acked Write 3 Done Sent Done false 6472 Acked Read 4 Done Sent Done false 6473 Acked Read 8 Done Sent Done false 6474 Acked Write 10 Done Sent Done false 6475 Acked Flush 0 Done Sent Done false 6476 Acked Read 9 Done Sent Done false 6477 Acked Read 6 Done Sent Done false 6478 Acked Write 8 Done Sent Done false 6479 Acked Read 2 Done Sent Done false 6480 Acked Write 7 Done Sent Done false 6481 Acked Write 7 Done Sent Done false 6482 Acked Read 7 Done Sent Done false 6483 Acked Write 8 Done Sent Done false 6484 Acked Write 2 Done Sent Done false 6485 Acked Read 5 Done Sent Done false 6486 Acked Write 6 Done Sent Done false 6487 Acked Flush 0 Done Sent Done false 6488 Acked Flush 0 Done Sent Done false 6489 Acked Flush 0 Done Sent Done false 6490 Acked Read 3 Done Sent Done false 6491 Acked Write 3 Done Sent Done false 6492 Acked Write 3 Done Sent Done false 6493 Acked Write 9 Done Sent Done false 6494 Acked Write 1 Done Sent Done false 6495 Acked Write 1 Done Sent Done false 6496 Acked Write 2 Done Sent Done false 6497 Acked Flush 0 Done Sent Done false 6498 Acked Read 3 Done Sent Done false 6499 Acked Flush 0 Done Sent Done false 6500 Acked Read 1 Done Sent Done false 6501 Acked Flush 0 Done Sent Done false 6502 Acked Read 3 Done Sent Done false 6503 Acked Read 2 Done Sent Done false 6504 Acked Write 4 Done Sent Done false 6505 Acked Flush 0 Done Sent Done false 6506 Acked Read 4 Done Sent Done false 6507 Acked Read 3 Done Sent Done false 6508 Acked Read 3 Done Sent Done false 6509 Acked Flush 0 Done Sent Done false 6510 Acked Read 10 Done Sent Done false 6511 Acked Read 2 Done Sent Done false 6512 Acked Read 9 Done Sent Done false 6513 Acked Read 3 Done Sent Done false 6514 Acked Read 10 Done Sent Done false 6515 Acked Read 6 Done Sent Done false 6516 Acked Write 3 Done Sent Done false 6517 Acked Write 3 Done Sent Done false 6518 Acked Write 2 Done Sent Done false 6519 Acked Read 4 Done Sent Done false 6520 Acked Write 9 Done Sent Done false 6521 Acked Write 8 Done Sent Done false 6522 Acked Write 8 Done Sent Done false 6523 Acked Write 2 Done Sent Done false 6524 Acked Write 9 Done Sent Done false 6525 Acked Read 6 Done Sent Done false 6526 Acked Write 9 Done Sent Done false 6527 Acked Flush 0 Done Sent Done false 6528 Acked Read 5 Done Sent Done false 6529 Acked Read 6 Done Sent Done false 6530 Acked Write 4 Done Sent Done false 6531 Acked Write 1 Done Sent Done false 6532 Acked Read 1 Done Sent Done false 6533 Acked Flush 0 Done Sent Done false 6534 Acked Write 5 Done Sent Done false 6535 Acked Write 10 Done Sent Done false 6536 Acked Write 9 Done Sent Done false 6537 Acked Write 4 Done Sent Done false 6538 Acked Write 5 Done Sent Done false 6539 Acked Write 4 Done Sent Done false 6540 Acked Write 1 Done Sent Done false 6541 Acked Read 6 Done Sent Done false 6542 Acked Flush 0 Done Sent Done false 6543 Acked Read 6 Done Sent Done false 6544 Acked Read 3 Done Sent Done false 6545 Acked Write 10 Done Sent Done false 6546 Acked Read 9 Done Sent Done false 6547 Acked Read 4 Done Sent Done false 6548 Acked Write 1 Done Sent Done false 6549 Acked Read 8 Done Sent Done false 6550 Acked Write 3 Done Sent Done false 6551 Acked Flush 0 Done Sent Done false 6552 Acked Flush 0 Done Sent Done false 6553 Acked Read 10 Done Sent Done false 6554 Acked Read 5 Done Sent Done false 6555 Acked Write 7 Done Sent Done false 6556 Acked Write 5 Done Sent Done false 6557 Acked Read 2 Done Sent Done false 6558 Acked Flush 0 Done Sent Done false 6559 Acked Write 10 Done Sent Done false 6560 Acked Read 10 Done Sent Done false 6561 Acked Flush 0 Done Sent Done false 6562 Acked Read 2 Done Sent Done false 6563 Acked Write 8 Done Sent Done false 6564 Acked Read 2 Done Sent Done false 6565 Acked Read 9 Done Sent Done false 6566 Acked Read 4 Done Sent Done false 6567 Acked Read 6 Done Sent Done false 6568 Acked Write 7 Done Sent Done false 6569 Acked Read 10 Done Sent Done false 6570 Acked Write 1 Done Sent Done false 6571 Acked Read 1 Done Sent Done false 6572 Acked Read 3 Done Sent Done false 6573 Acked Read 8 Done Sent Done false 6574 Acked Read 1 Done Sent Done false 6575 Acked Write 4 Done Sent Done false 6576 Acked Write 1 Done Sent Done false 6577 Acked Read 4 Done Sent Done false 6578 Acked Read 1 Done Sent Done false 6579 Acked Write 8 Done Sent Done false 6580 Acked Read 4 Done Sent Done false 6581 Acked Read 9 Done Sent Done false 6582 Acked Read 1 Done Sent Done false 6583 Acked Read 1 Done Sent Done false 6584 Acked Read 10 Done Sent Done false 6585 Acked Read 7 Done Sent Done false 6586 Acked Read 8 Done Sent Done false 6587 Acked Write 7 Done Sent Done false 6588 Acked Read 6 Done Sent Done false 6589 Acked Read 10 Done Sent Done false 6590 Acked Flush 0 Done Sent Done false 6591 Acked Read 5 Done Sent Done false 6592 Acked Read 2 Done Sent Done false 6593 Acked Write 7 Done Sent Done false 6594 Acked Read 9 Done Sent Done false 6595 Acked Read 7 Done Sent Done false 6596 Acked Write 3 Done Sent Done false 6597 Acked Write 7 Done Sent Done false 6598 Acked Write 2 Done Sent Done false 6599 Acked Flush 0 Done Sent Done false 6600 Acked Write 5 Done Sent Done false 6601 Acked Write 2 Done Sent Done false 6602 Acked Write 2 Done Sent Done false 6603 Acked Write 1 Done Sent Done false 6604 Acked Read 7 Done Sent Done false 6605 Acked Write 9 Done Sent Done false 6606 Acked Flush 0 Done Sent Done false 6607 Acked Flush 0 Done Sent Done false 6608 Acked Write 5 Done Sent Done false 6609 Acked Read 4 Done Sent Done false 6610 Acked Write 8 Done Sent Done false 6611 Acked Read 4 Done Sent Done false 6612 Acked Read 8 Done Sent Done false 6613 Acked Write 6 Done Sent Done false 6614 Acked Write 8 Done Sent Done false 6615 Acked Write 5 Done Sent Done false 6616 Acked Write 8 Done Sent Done false 6617 Acked Write 4 Done Sent Done false 6618 Acked Write 5 Done Sent Done false 6619 Acked Write 3 Done Sent Done false 6620 Acked Read 9 Done Sent Done false 6621 Acked Read 8 Done Sent Done false 6622 Acked Read 1 Done Sent Done false 6623 Acked Write 7 Done Sent Done false 6624 Acked Read 2 Done Sent Done false 6625 Acked Write 1 Done Sent Done false 6626 Acked Read 8 Done Sent Done false 6627 Acked Read 4 Done Sent Done false 6628 Acked Write 8 Done Sent Done false 6629 Acked Read 7 Done Sent Done false 6630 Acked Read 1 Done Sent Done false 6631 Acked Flush 0 Done Sent Done false 6632 Acked Write 5 Done Sent Done false 6633 Acked Read 1 Done Sent Done false 6634 Acked Read 3 Done Sent Done false 6635 Acked Write 5 Done Sent Done false 6636 Acked Write 2 Done Sent Done false 6637 Acked Write 6 Done Sent Done false 6638 Acked Write 1 Done Sent Done false 6639 Acked Write 7 Done Sent Done false 6640 Acked Flush 0 Done Sent Done false 6641 Acked Flush 0 Done Sent Done false 6642 Acked Read 4 Done Sent Done false 6643 Acked Read 7 Done Sent Done false 6644 Acked Read 9 Done Sent Done false 6645 Acked Read 8 Done Sent Done false 6646 Acked Write 9 Done Sent Done false 6647 Acked Write 10 Done Sent Done false 6648 Acked Write 1 Done Sent Done false 6649 Acked Read 5 Done Sent Done false 6650 Acked Flush 0 Done Sent Done false 6651 Acked Flush 0 Done Sent Done false 6652 Acked Read 6 Done Sent Done false 6653 Acked Write 8 Done Sent Done false 6654 Acked Read 5 Done Sent Done false 6655 Acked Read 1 Done Sent Done false 6656 Acked Read 9 Done Sent Done false 6657 Acked Write 8 Done Sent Done false 6658 Acked Write 2 Done Sent Done false 6659 Acked Read 6 Done Sent Done false 6660 Acked Read 3 Done Sent Done false 6661 Acked Read 1 Done Sent Done false 6662 Acked Write 8 Done Sent Done false 6663 Acked Read 9 Done Sent Done false 6664 Acked Read 5 Done Sent Done false 6665 Acked Read 1 Done Sent Done false 6666 Acked Read 2 Done Sent Done false 6667 Acked Write 2 Done Sent Done false 6668 Acked Write 1 Done Sent Done false 6669 Acked Read 7 Done Sent Done false 6670 Acked Read 6 Done Sent Done false 6671 Acked Write 10 Done Sent Done false 6672 Acked Read 4 Done Sent Done false 6673 Acked Write 8 Done Sent Done false 6674 Acked Write 9 Done Sent Done false 6675 Acked Write 5 Done Sent Done false 6676 Acked Write 5 Done Sent Done false 6677 Acked Write 6 Done Sent Done false 6678 Acked Flush 0 Done Sent Done false 6679 Acked Write 9 Done Sent Done false 6680 Acked Write 9 Done Sent Done false 6681 Acked Read 6 Done Sent Done false 6682 Acked Read 1 Done Sent Done false 6683 Acked Write 5 Done Sent Done false 6684 Acked Read 7 Done Sent Done false 6685 Acked Read 1 Done Sent Done false 6686 Acked Read 10 Done Sent Done false 6687 Acked Write 7 Done Sent Done false 6688 Acked Write 1 Done Sent Done false 6689 Acked Read 9 Done Sent Done false 6690 Acked Read 8 Done Sent Done false 6691 Acked Write 9 Done Sent Done false 6692 Acked Read 2 Done Sent Done false 6693 Acked Read 10 Done Sent Done false 6694 Acked Read 6 Done Sent Done false 6695 Acked Flush 0 Done Sent Done false 6696 Acked Read 3 Done Sent Done false 6697 Acked Read 5 Done Sent Done false 6698 Acked Read 9 Done Sent Done false 6699 Acked Write 8 Done Sent Done false 6700 Acked Read 6 Done Sent Done false 6701 Acked Write 4 Done Sent Done false 6702 Acked Write 9 Done Sent Done false 6703 Acked Write 4 Done Sent Done false 6704 Acked Read 8 Done Sent Done false 6705 Acked Read 7 Done Sent Done false 6706 Acked Write 7 Done Sent Done false 6707 Acked Write 7 Done Sent Done false 6708 Acked Write 7 Done Sent Done false 6709 Acked Flush 0 Done Sent Done false 6710 Acked Read 1 Done Sent Done false 6711 Acked Read 4 Done Sent Done false 6712 Acked Read 7 Done Sent Done false 6713 Acked Read 5 Done Sent Done false 6714 Acked Read 2 Done Sent Done false 6715 Acked Flush 0 Done Sent Done false 6716 Acked Write 6 Done Sent Done false 6717 Acked Write 9 Done Sent Done false 6718 Acked Write 9 Done Sent Done false 6719 Acked Read 2 Done Sent Done false 6720 Acked Read 4 Done Sent Done false 6721 Acked Write 2 Done Sent Done false 6722 Acked Read 4 Done Sent Done false 6723 Acked Read 1 Done Sent Done false 6724 Acked Read 6 Done Sent Done false 6725 Acked Read 2 Done Sent Done false 6726 Acked Read 5 Done Sent Done false 6727 Acked Flush 0 Done Sent Done false 6728 Acked Flush 0 Done Sent Done false 6729 Acked Flush 0 Done Sent Done false 6730 Acked Flush 0 Done Sent Done false 6731 Acked Read 1 Done Sent Done false 6732 Acked Write 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6730 6431 6730 Downstairs last five completed: 6431 6430 6429 6428 6427 Upstairs last five completed: 6732 6731 6730 6729 6728 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6432 Acked Flush 0 Done Sent Done false 6433 Acked Read 5 Done Sent Done false 6434 Acked Read 8 Done Sent Done false 6435 Acked Read 5 Done Sent Done false 6436 Acked Read 4 Done Sent Done false 6437 Acked Write 5 Done Sent Done false 6438 Acked Read 6 Done Sent Done false 6439 Acked Flush 0 Done Sent Done false 6440 Acked Write 4 Done Sent Done false 6441 Acked Read 7 Done Sent Done false 6442 Acked Write 2 Done Sent Done false 6443 Acked Write 1 Done Sent Done false 6444 Acked Read 1 Done Sent Done false 6445 Acked Write 10 Done Sent Done false 6446 Acked Flush 0 Done Sent Done false 6447 Acked Read 7 Done Sent Done false 6448 Acked Read 4 Done Sent Done false 6449 Acked Flush 0 Done Sent Done false 6450 Acked Read 7 Done Sent Done false 6451 Acked Read 3 Done Sent Done false 6452 Acked Write 3 Done Sent Done false 6453 Acked Write 10 Done Sent Done false 6454 Acked Write 8 Done Sent Done false 6455 Acked Read 3 Done Sent Done false 6456 Acked Write 3 Done Sent Done false 6457 Acked Write 7 Done Sent Done false 6458 Acked Flush 0 Done Sent Done false 6459 Acked Read 8 Done Sent Done false 6460 Acked Read 4 Done Sent Done false 6461 Acked Write 8 Done Sent Done false 6462 Acked Read 8 Done Sent Done false 6463 Acked Write 9 Done Sent Done false 6464 Acked Write 8 Done Sent Done false 6465 Acked Read 8 Done Sent Done false 6466 Acked Write 2 Done Sent Done false 6467 Acked Read 7 Done Sent Done false 6468 Acked Read 10 Done Sent Done false 6469 Acked Read 2 Done Sent Done false 6470 Acked Write 6 Done Sent Done false 6471 Acked Write 3 Done Sent Done false 6472 Acked Read 4 Done Sent Done false 6473 Acked Read 8 Done Sent Done false 6474 Acked Write 10 Done Sent Done false 6475 Acked Flush 0 Done Sent Done false 6476 Acked Read 9 Done Sent Done false 6477 Acked Read 6 Done Sent Done false 6478 Acked Write 8 Done Sent Done false 6479 Acked Read 2 Done Sent Done false 6480 Acked Write 7 Done Sent Done false 6481 Acked Write 7 Done Sent Done false 6482 Acked Read 7 Done Sent Done false 6483 Acked Write 8 Done Sent Done false 6484 Acked Write 2 Done Sent Done false 6485 Acked Read 5 Done Sent Done false 6486 Acked Write 6 Done Sent Done false 6487 Acked Flush 0 Done Sent Done false 6488 Acked Flush 0 Done Sent Done false 6489 Acked Flush 0 Done Sent Done false 6490 Acked Read 3 Done Sent Done false 6491 Acked Write 3 Done Sent Done false 6492 Acked Write 3 Done Sent Done false 6493 Acked Write 9 Done Sent Done false 6494 Acked Write 1 Done Sent Done false 6495 Acked Write 1 Done Sent Done false 6496 Acked Write 2 Done Sent Done false 6497 Acked Flush 0 Done Sent Done false 6498 Acked Read 3 Done Sent Done false 6499 Acked Flush 0 Done Sent Done false 6500 Acked Read 1 Done Sent Done false 6501 Acked Flush 0 Done Sent Done false 6502 Acked Read 3 Done Sent Done false 6503 Acked Read 2 Done Sent Done false 6504 Acked Write 4 Done Sent Done false 6505 Acked Flush 0 Done Sent Done false 6506 Acked Read 4 Done Sent Done false 6507 Acked Read 3 Done Sent Done false 6508 Acked Read 3 Done Sent Done false 6509 Acked Flush 0 Done Sent Done false 6510 Acked Read 10 Done Sent Done false 6511 Acked Read 2 Done Sent Done false 6512 Acked Read 9 Done Sent Done false 6513 Acked Read 3 Done Sent Done false 6514 Acked Read 10 Done Sent Done false 6515 Acked Read 6 Done Sent Done false 6516 Acked Write 3 Done Sent Done false 6517 Acked Write 3 Done Sent Done false 6518 Acked Write 2 Done Sent Done false 6519 Acked Read 4 Done Sent Done false 6520 Acked Write 9 Done Sent Done false 6521 Acked Write 8 Done Sent Done false 6522 Acked Write 8 Done Sent Done false 6523 Acked Write 2 Done Sent Done false 6524 Acked Write 9 Done Sent Done false 6525 Acked Read 6 Done Sent Done false 6526 Acked Write 9 Done Sent Done false 6527 Acked Flush 0 Done Sent Done false 6528 Acked Read 5 Done Sent Done false 6529 Acked Read 6 Done Sent Done false 6530 Acked Write 4 Done Sent Done false 6531 Acked Write 1 Done Sent Done false 6532 Acked Read 1 Done Sent Done false 6533 Acked Flush 0 Done Sent Done false 6534 Acked Write 5 Done Sent Done false 6535 Acked Write 10 Done Sent Done false 6536 Acked Write 9 Done Sent Done false 6537 Acked Write 4 Done Sent Done false 6538 Acked Write 5 Done Sent Done false 6539 Acked Write 4 Done Sent Done false 6540 Acked Write 1 Done Sent Done false 6541 Acked Read 6 Done Sent Done false 6542 Acked Flush 0 Done Sent Done false 6543 Acked Read 6 Done Sent Done false 6544 Acked Read 3 Done Sent Done false 6545 Acked Write 10 Done Sent Done false 6546 Acked Read 9 Done Sent Done false 6547 Acked Read 4 Done Sent Done false 6548 Acked Write 1 Done Sent Done false 6549 Acked Read 8 Done Sent Done false 6550 Acked Write 3 Done Sent Done false 6551 Acked Flush 0 Done Sent Done false 6552 Acked Flush 0 Done Sent Done false 6553 Acked Read 10 Done Sent Done false 6554 Acked Read 5 Done Sent Done false 6555 Acked Write 7 Done Sent Done false 6556 Acked Write 5 Done Sent Done false 6557 Acked Read 2 Done Sent Done false 6558 Acked Flush 0 Done Sent Done false 6559 Acked Write 10 Done Sent Done false 6560 Acked Read 10 Done Sent Done false 6561 Acked Flush 0 Done Sent Done false 6562 Acked Read 2 Done Sent Done false 6563 Acked Write 8 Done Sent Done false 6564 Acked Read 2 Done Sent Done false 6565 Acked Read 9 Done Sent Done false 6566 Acked Read 4 Done Sent Done false 6567 Acked Read 6 Done Sent Done false 6568 Acked Write 7 Done Sent Done false 6569 Acked Read 10 Done Sent Done false 6570 Acked Write 1 Done Sent Done false 6571 Acked Read 1 Done Sent Done false 6572 Acked Read 3 Done Sent Done false 6573 Acked Read 8 Done Sent Done false 6574 Acked Read 1 Done Sent Done false 6575 Acked Write 4 Done Sent Done false 6576 Acked Write 1 Done Sent Done false 6577 Acked Read 4 Done Sent Done false 6578 Acked Read 1 Done Sent Done false 6579 Acked Write 8 Done Sent Done false 6580 Acked Read 4 Done Sent Done false 6581 Acked Read 9 Done Sent Done false 6582 Acked Read 1 Done Sent Done false 6583 Acked Read 1 Done Sent Done false 6584 Acked Read 10 Done Sent Done false 6585 Acked Read 7 Done Sent Done false 6586 Acked Read 8 Done Sent Done false 6587 Acked Write 7 Done Sent Done false 6588 Acked Read 6 Done Sent Done false 6589 Acked Read 10 Done Sent Done false 6590 Acked Flush 0 Done Sent Done false 6591 Acked Read 5 Done Sent Done false 6592 Acked Read 2 Done Sent Done false 6593 Acked Write 7 Done Sent Done false 6594 Acked Read 9 Done Sent Done false 6595 Acked Read 7 Done Sent Done false 6596 Acked Write 3 Done Sent Done false 6597 Acked Write 7 Done Sent Done false 6598 Acked Write 2 Done Sent Done false 6599 Acked Flush 0 Done Sent Done false 6600 Acked Write 5 Done Sent Done false 6601 Acked Write 2 Done Sent Done false 6602 Acked Write 2 Done Sent Done false 6603 Acked Write 1 Done Sent Done false 6604 Acked Read 7 Done Sent Done false 6605 Acked Write 9 Done Sent Done false 6606 Acked Flush 0 Done Sent Done false 6607 Acked Flush 0 Done Sent Done false 6608 Acked Write 5 Done Sent Done false 6609 Acked Read 4 Done Sent Done false 6610 Acked Write 8 Done Sent Done false 6611 Acked Read 4 Done Sent Done false 6612 Acked Read 8 Done Sent Done false 6613 Acked Write 6 Done Sent Done false 6614 Acked Write 8 Done Sent Done false 6615 Acked Write 5 Done Sent Done false 6616 Acked Write 8 Done Sent Done false 6617 Acked Write 4 Done Sent Done false 6618 Acked Write 5 Done Sent Done false 6619 Acked Write 3 Done Sent Done false 6620 Acked Read 9 Done Sent Done false 6621 Acked Read 8 Done Sent Done false 6622 Acked Read 1 Done Sent Done false 6623 Acked Write 7 Done Sent Done false 6624 Acked Read 2 Done Sent Done false 6625 Acked Write 1 Done Sent Done false 6626 Acked Read 8 Done Sent Done false 6627 Acked Read 4 Done Sent Done false 6628 Acked Write 8 Done Sent Done false 6629 Acked Read 7 Done Sent Done false 6630 Acked Read 1 Done Sent Done false 6631 Acked Flush 0 Done Sent Done false 6632 Acked Write 5 Done Sent Done false 6633 Acked Read 1 Done Sent Done false 6634 Acked Read 3 Done Sent Done false 6635 Acked Write 5 Done Sent Done false 6636 Acked Write 2 Done Sent Done false 6637 Acked Write 6 Done Sent Done false 6638 Acked Write 1 Done Sent Done false 6639 Acked Write 7 Done Sent Done false 6640 Acked Flush 0 Done Sent Done false 6641 Acked Flush 0 Done Sent Done false 6642 Acked Read 4 Done Sent Done false 6643 Acked Read 7 Done Sent Done false 6644 Acked Read 9 Done Sent Done false 6645 Acked Read 8 Done Sent Done false 6646 Acked Write 9 Done Sent Done false 6647 Acked Write 10 Done Sent Done false 6648 Acked Write 1 Done Sent Done false 6649 Acked Read 5 Done Sent Done false 6650 Acked Flush 0 Done Sent Done false 6651 Acked Flush 0 Done Sent Done false 6652 Acked Read 6 Done Sent Done false 6653 Acked Write 8 Done Sent Done false 6654 Acked Read 5 Done Sent Done false 6655 Acked Read 1 Done Sent Done false 6656 Acked Read 9 Done Sent Done false 6657 Acked Write 8 Done Sent Done false 6658 Acked Write 2 Done Sent Done false 6659 Acked Read 6 Done Sent Done false 6660 Acked Read 3 Done Sent Done false 6661 Acked Read 1 Done Sent Done false 6662 Acked Write 8 Done Sent Done false 6663 Acked Read 9 Done Sent Done false 6664 Acked Read 5 Done Sent Done false 6665 Acked Read 1 Done Sent Done false 6666 Acked Read 2 Done Sent Done false 6667 Acked Write 2 Done Sent Done false 6668 Acked Write 1 Done Sent Done false 6669 Acked Read 7 Done Sent Done false 6670 Acked Read 6 Done Sent Done false 6671 Acked Write 10 Done Sent Done false 6672 Acked Read 4 Done Sent Done false 6673 Acked Write 8 Done Sent Done false 6674 Acked Write 9 Done Sent Done false 6675 Acked Write 5 Done Sent Done false 6676 Acked Write 5 Done Sent Done false 6677 Acked Write 6 Done Sent Done false 6678 Acked Flush 0 Done Sent Done false 6679 Acked Write 9 Done Sent Done false 6680 Acked Write 9 Done Sent Done false 6681 Acked Read 6 Done Sent Done false 6682 Acked Read 1 Done Sent Done false 6683 Acked Write 5 Done Sent Done false 6684 Acked Read 7 Done Sent Done false 6685 Acked Read 1 Done Sent Done false 6686 Acked Read 10 Done Sent Done false 6687 Acked Write 7 Done Sent Done false 6688 Acked Write 1 Done Sent Done false 6689 Acked Read 9 Done Sent Done false 6690 Acked Read 8 Done Sent Done false 6691 Acked Write 9 Done Sent Done false 6692 Acked Read 2 Done Sent Done false 6693 Acked Read 10 Done Sent Done false 6694 Acked Read 6 Done Sent Done false 6695 Acked Flush 0 Done Sent Done false 6696 Acked Read 3 Done Sent Done false 6697 Acked Read 5 Done Sent Done false 6698 Acked Read 9 Done Sent Done false 6699 Acked Write 8 Done Sent Done false 6700 Acked Read 6 Done Sent Done false 6701 Acked Write 4 Done Sent Done false 6702 Acked Write 9 Done Sent Done false 6703 Acked Write 4 Done Sent Done false 6704 Acked Read 8 Done Sent Done false 6705 Acked Read 7 Done Sent Done false 6706 Acked Write 7 Done Sent Done false 6707 Acked Write 7 Done Sent Done false 6708 Acked Write 7 Done Sent Done false 6709 Acked Flush 0 Done Sent Done false 6710 Acked Read 1 Done Sent Done false 6711 Acked Read 4 Done Sent Done false 6712 Acked Read 7 Done Sent Done false 6713 Acked Read 5 Done Sent Done false 6714 Acked Read 2 Done Sent Done false 6715 Acked Flush 0 Done Sent Done false 6716 Acked Write 6 Done Sent Done false 6717 Acked Write 9 Done Sent Done false 6718 Acked Write 9 Done Sent Done false 6719 Acked Read 2 Done Sent Done false 6720 Acked Read 4 Done Sent Done false 6721 Acked Write 2 Done Sent Done false 6722 Acked Read 4 Done Sent Done false 6723 Acked Read 1 Done Sent Done false 6724 Acked Read 6 Done Sent Done false 6725 Acked Read 2 Done Sent Done false 6726 Acked Read 5 Done Sent Done false 6727 Acked Flush 0 Done Sent Done false 6728 Acked Flush 0 Done Sent Done false 6729 Acked Flush 0 Done Sent Done false 6730 Acked Flush 0 Done Sent Done false 6731 Acked Read 1 Done Sent Done false 6732 Acked Write 2 Done Sent Done false 6733 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6733 6431 6733 Downstairs last five completed: 6431 6430 6429 6428 6427 Upstairs last five completed: 6733 6732 6731 6730 6729 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:26.067630425Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:26.067693358Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:26.067717902Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:26.067739046Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:26.067754932Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6432 Acked Flush 0 Done Sent Done false 6433 Acked Read 5 Done Sent Done false 6434 Acked Read 8 Done Sent Done false 6435 Acked Read 5 Done Sent Done false 6436 Acked Read 4 Done Sent Done false 6437 Acked Write 5 Done Sent Done false 6438 Acked Read 6 Done Sent Done false 6439 Acked Flush 0 Done Sent Done false 6440 Acked Write 4 Done Sent Done false 6441 Acked Read 7 Done Sent Done false 6442 Acked Write 2 Done Sent Done false 6443 Acked Write 1 Done Sent Done false 6444 Acked Read 1 Done Sent Done false 6445 Acked Write 10 Done Sent Done false 6446 Acked Flush 0 Done Sent Done false 6447 Acked Read 7 Done Sent Done false 6448 Acked Read 4 Done Sent Done false 6449 Acked Flush 0 Done Sent Done false 6450 Acked Read 7 Done Sent Done false 6451 Acked Read 3 Done Sent Done false 6452 Acked Write 3 Done Sent Done false 6453 Acked Write 10 Done Sent Done false 6454 Acked Write 8 Done Sent Done false 6455 Acked Read 3 Done Sent Done false 6456 Acked Write 3 Done Sent Done false 6457 Acked Write 7 Done Sent Done false 6458 Acked Flush 0 Done Sent Done false 6459 Acked Read 8 Done Sent Done false 6460 Acked Read 4 Done Sent Done false 6461 Acked Write 8 Done Sent Done false 6462 Acked Read 8 Done Sent Done false 6463 Acked Write 9 Done Sent Done false 6464 Acked Write 8 Done Sent Done false 6465 Acked Read 8 Done Sent Done false 6466 Acked Write 2 Done Sent Done false 6467 Acked Read 7 Done Sent Done false 6468 Acked Read 10 Done Sent Done false 6469 Acked Read 2 Done Sent Done false 6470 Acked Write 6 Done Sent Done false 6471 Acked Write 3 Done Sent Done false 6472 Acked Read 4 Done Sent Done false 6473 Acked Read 8 Done Sent Done false 6474 Acked Write 10 Done Sent Done false 6475 Acked Flush 0 Done Sent Done false 6476 Acked Read 9 Done Sent Done false 6477 Acked Read 6 Done Sent Done false 6478 Acked Write 8 Done Sent Done false 6479 Acked Read 2 Done Sent Done false 6480 Acked Write 7 Done Sent Done false 6481 Acked Write 7 Done Sent Done false 6482 Acked Read 7 Done Sent Done false 6483 Acked Write 8 Done Sent Done false 6484 Acked Write 2 Done Sent Done false 6485 Acked Read 5 Done Sent Done false 6486 Acked Write 6 Done Sent Done false 6487 Acked Flush 0 Done Sent Done false 6488 Acked Flush 0 Done Sent Done false 6489 Acked Flush 0 Done Sent Done false 6490 Acked Read 3 Done Sent Done false 6491 Acked Write 3 Done Sent Done false 6492 Acked Write 3 Done Sent Done false 6493 Acked Write 9 Done Sent Done false 6494 Acked Write 1 Done Sent Done false 6495 Acked Write 1 Done Sent Done false 6496 Acked Write 2 Done Sent Done false 6497 Acked Flush 0 Done Sent Done false 6498 Acked Read 3 Done Sent Done false 6499 Acked Flush 0 Done Sent Done false 6500 Acked Read 1 Done Sent Done false 6501 Acked Flush 0 Done Sent Done false 6502 Acked Read 3 Done Sent Done false 6503 Acked Read 2 Done Sent Done false 6504 Acked Write 4 Done Sent Done false 6505 Acked Flush 0 Done Sent Done false 6506 Acked Read 4 Done Sent Done false 6507 Acked Read 3 Done Sent Done false 6508 Acked Read 3 Done Sent Done false 6509 Acked Flush 0 Done Sent Done false 6510 Acked Read 10 Done Sent Done false 6511 Acked Read 2 Done Sent Done false 6512 Acked Read 9 Done Sent Done false 6513 Acked Read 3 Done Sent Done false 6514 Acked Read 10 Done Sent Done false 6515 Acked Read 6 Done Sent Done false 6516 Acked Write 3 Done Sent Done false 6517 Acked Write 3 Done Sent Done false 6518 Acked Write 2 Done Sent Done false 6519 Acked Read 4 Done Sent Done false 6520 Acked Write 9 Done Sent Done false 6521 Acked Write 8 Done Sent Done false 6522 Acked Write 8 Done Sent Done false 6523 Acked Write 2 Done Sent Done false 6524 Acked Write 9 Done Sent Done false 6525 Acked Read 6 Done Sent Done false 6526 Acked Write 9 Done Sent Done false 6527 Acked Flush 0 Done Sent Done false 6528 Acked Read 5 Done Sent Done false 6529 Acked Read 6 Done Sent Done false 6530 Acked Write 4 Done Sent Done false 6531 Acked Write 1 Done Sent Done false 6532 Acked Read 1 Done Sent Done false 6533 Acked Flush 0 Done Sent Done false 6534 Acked Write 5 Done Sent Done false 6535 Acked Write 10 Done Sent Done false 6536 Acked Write 9 Done Sent Done false 6537 Acked Write 4 Done Sent Done false 6538 Acked Write 5 Done Sent Done false 6539 Acked Write 4 Done Sent Done false 6540 Acked Write 1 Done Sent Done false 6541 Acked Read 6 Done Sent Done false 6542 Acked Flush 0 Done Sent Done false 6543 Acked Read 6 Done Sent Done false 6544 Acked Read 3 Done Sent Done false 6545 Acked Write 10 Done Sent Done false 6546 Acked Read 9 Done Sent Done false 6547 Acked Read 4 Done Sent Done false 6548 Acked Write 1 Done Sent Done false 6549 Acked Read 8 Done Sent Done false 6550 Acked Write 3 Done Sent Done false 6551 Acked Flush 0 Done Sent Done false 6552 Acked Flush 0 Done Sent Done false 6553 Acked Read 10 Done Sent Done false 6554 Acked Read 5 Done Sent Done false 6555 Acked Write 7 Done Sent Done false 6556 Acked Write 5 Done Sent Done false 6557 Acked Read 2 Done Sent Done false 6558 Acked Flush 0 Done Sent Done false 6559 Acked Write 10 Done Sent Done false 6560 Acked Read 10 Done Sent Done false 6561 Acked Flush 0 Done Sent Done false 6562 Acked Read 2 Done Sent Done false 6563 Acked Write 8 Done Sent Done false 6564 Acked Read 2 Done Sent Done false 6565 Acked Read 9 Done Sent Done false 6566 Acked Read 4 Done Sent Done false 6567 Acked Read 6 Done Sent Done false 6568 Acked Write 7 Done Sent Done false 6569 Acked Read 10 Done Sent Done false 6570 Acked Write 1 Done Sent Done false 6571 Acked Read 1 Done Sent Done false 6572 Acked Read 3 Done Sent Done false 6573 Acked Read 8 Done Sent Done false 6574 Acked Read 1 Done Sent Done false 6575 Acked Write 4 Done Sent Done false 6576 Acked Write 1 Done Sent Done false 6577 Acked Read 4 Done Sent Done false 6578 Acked Read 1 Done Sent Done false 6579 Acked Write 8 Done Sent Done false 6580 Acked Read 4 Done Sent Done false 6581 Acked Read 9 Done Sent Done false 6582 Acked Read 1 Done Sent Done false 6583 Acked Read 1 Done Sent Done false 6584 Acked Read 10 Done Sent Done false 6585 Acked Read 7 Done Sent Done false 6586 Acked Read 8 Done Sent Done false 6587 Acked Write 7 Done Sent Done false 6588 Acked Read 6 Done Sent Done false 6589 Acked Read 10 Done Sent Done false 6590 Acked Flush 0 Done Sent Done false 6591 Acked Read 5 Done Sent Done false 6592 Acked Read 2 Done Sent Done false 6593 Acked Write 7 Done Sent Done false 6594 Acked Read 9 Done Sent Done false 6595 Acked Read 7 Done Sent Done false 6596 Acked Write 3 Done Sent Done false 6597 Acked Write 7 Done Sent Done false 6598 Acked Write 2 Done Sent Done false 6599 Acked Flush 0 Done Sent Done false 6600 Acked Write 5 Done Sent Done false 6601 Acked Write 2 Done Sent Done false 6602 Acked Write 2 Done Sent Done false 6603 Acked Write 1 Done Sent Done false 6604 Acked Read 7 Done Sent Done false 6605 Acked Write 9 Done Sent Done false 6606 Acked Flush 0 Done Sent Done false 6607 Acked Flush 0 Done Sent Done false 6608 Acked Write 5 Done Sent Done false 6609 Acked Read 4 Done Sent Done false 6610 Acked Write 8 Done Sent Done false 6611 Acked Read 4 Done Sent Done false 6612 Acked Read 8 Done Sent Done false 6613 Acked Write 6 Done Sent Done false 6614 Acked Write 8 Done Sent Done false 6615 Acked Write 5 Done Sent Done false 6616 Acked Write 8 Done Sent Done false 6617 Acked Write 4 Done Sent Done false 6618 Acked Write 5 Done Sent Done false 6619 Acked Write 3 Done Sent Done false 6620 Acked Read 9 Done Sent Done false 6621 Acked Read 8 Done Sent Done false 6622 Acked Read 1 Done Sent Done false 6623 Acked Write 7 Done Sent Done false 6624 Acked Read 2 Done Sent Done false 6625 Acked Write 1 Done Sent Done false 6626 Acked Read 8 Done Sent Done false 6627 Acked Read 4 Done Sent Done false 6628 Acked Write 8 Done Sent Done false 6629 Acked Read 7 Done Sent Done false 6630 Acked Read 1 Done Sent Done false 6631 Acked Flush 0 Done Sent Done false 6632 Acked Write 5 Done Sent Done false 6633 Acked Read 1 Done Sent Done false 6634 Acked Read 3 Done Sent Done false 6635 Acked Write 5 Done Sent Done false 6636 Acked Write 2 Done Sent Done false 6637 Acked Write 6 Done Sent Done false 6638 Acked Write 1 Done Sent Done false 6639 Acked Write 7 Done Sent Done false 6640 Acked Flush 0 Done Sent Done false 6641 Acked Flush 0 Done Sent Done false 6642 Acked Read 4 Done Sent Done false 6643 Acked Read 7 Done Sent Done false 6644 Acked Read 9 Done Sent Done false 6645 Acked Read 8 Done Sent Done false 6646 Acked Write 9 Done Sent Done false 6647 Acked Write 10 Done Sent Done false 6648 Acked Write 1 Done Sent Done false 6649 Acked Read 5 Done Sent Done false 6650 Acked Flush 0 Done Sent Done false 6651 Acked Flush 0 Done Sent Done false 6652 Acked Read 6 Done Sent Done false 6653 Acked Write 8 Done Sent Done false 6654 Acked Read 5 Done Sent Done false 6655 Acked Read 1 Done Sent Done false 6656 Acked Read 9 Done Sent Done false 6657 Acked Write 8 Done Sent Done false 6658 Acked Write 2 Done Sent Done false 6659 Acked Read 6 Done Sent Done false 6660 Acked Read 3 Done Sent Done false 6661 Acked Read 1 Done Sent Done false 6662 Acked Write 8 Done Sent Done false 6663 Acked Read 9 Done Sent Done false 6664 Acked Read 5 Done Sent Done false 6665 Acked Read 1 Done Sent Done false 6666 Acked Read 2 Done Sent Done false 6667 Acked Write 2 Done Sent Done false 6668 Acked Write 1 Done Sent Done false 6669 Acked Read 7 Done Sent Done false 6670 Acked Read 6 Done Sent Done false 6671 Acked Write 10 Done Sent Done false 6672 Acked Read 4 Done Sent Done false 6673 Acked Write 8 Done Sent Done false 6674 Acked Write 9 Done Sent Done false 6675 Acked Write 5 Done Sent Done false 6676 Acked Write 5 Done Sent Done false 6677 Acked Write 6 Done Sent Done false 6678 Acked Flush 0 Done Sent Done false 6679 Acked Write 9 Done Sent Done false 6680 Acked Write 9 Done Sent Done false 6681 Acked Read 6 Done Sent Done false 6682 Acked Read 1 Done Sent Done false 6683 Acked Write 5 Done Sent Done false 6684 Acked Read 7 Done Sent Done false 6685 Acked Read 1 Done Sent Done false 6686 Acked Read 10 Done Sent Done false 6687 Acked Write 7 Done Sent Done false 6688 Acked Write 1 Done Sent Done false 6689 Acked Read 9 Done Sent Done false 6690 Acked Read 8 Done Sent Done false 6691 Acked Write 9 Done Sent Done false 6692 Acked Read 2 Done Sent Done false 6693 Acked Read 10 Done Sent Done false 6694 Acked Read 6 Done Sent Done false 6695 Acked Flush 0 Done Sent Done false 6696 Acked Read 3 Done Sent Done false 6697 Acked Read 5 Done Sent Done false 6698 Acked Read 9 Done Sent Done false 6699 Acked Write 8 Done Sent Done false 6700 Acked Read 6 Done Sent Done false 6701 Acked Write 4 Done Sent Done false 6702 Acked Write 9 Done Sent Done false 6703 Acked Write 4 Done Sent Done false 6704 Acked Read 8 Done Sent Done false 6705 Acked Read 7 Done Sent Done false 6706 Acked Write 7 Done Sent Done false 6707 Acked Write 7 Done Sent Done false 6708 Acked Write 7 Done Sent Done false 6709 Acked Flush 0 Done Sent Done false 6710 Acked Read 1 Done Sent Done false 6711 Acked Read 4 Done Sent Done false 6712 Acked Read 7 Done Sent Done false 6713 Acked Read 5 Done Sent Done false 6714 Acked Read 2 Done Sent Done false 6715 Acked Flush 0 Done Sent Done false 6716 Acked Write 6 Done Sent Done false 6717 Acked Write 9 Done Sent Done false 6718 Acked Write 9 Done Sent Done false 6719 Acked Read 2 Done Sent Done false 6720 Acked Read 4 Done Sent Done false 6721 Acked Write 2 Done Sent Done false 6722 Acked Read 4 Done Sent Done false 6723 Acked Read 1 Done Sent Done false 6724 Acked Read 6 Done Sent Done false 6725 Acked Read 2 Done Sent Done false 6726 Acked Read 5 Done Sent Done false 6727 Acked Flush 0 Done Sent Done false 6728 Acked Flush 0 Done Sent Done false 6729 Acked Flush 0 Done Sent Done false 6730 Acked Flush 0 Done Sent Done false 6731 Acked Read 1 Done Sent Done false 6732 Acked Write 2 Done Sent Done false 6733 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6733 6431 6733 Downstairs last five completed: 6431 6430 6429 6428 6427 Upstairs last five completed: 6733 6732 6731 6730 6729 CLIENT: Up:0 ds:302 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6432 Acked Flush 0 Done Sent Done false 6433 Acked Read 5 Done Sent Done false 6434 Acked Read 8 Done Sent Done false 6435 Acked Read 5 Done Sent Done false 6436 Acked Read 4 Done Sent Done false 6437 Acked Write 5 Done Sent Done false 6438 Acked Read 6 Done Sent Done false 6439 Acked Flush 0 Done Sent Done false 6440 Acked Write 4 Done Sent Done false 6441 Acked Read 7 Done Sent Done false 6442 Acked Write 2 Done Sent Done false 6443 Acked Write 1 Done Sent Done false 6444 Acked Read 1 Done Sent Done false 6445 Acked Write 10 Done Sent Done false 6446 Acked Flush 0 Done Sent Done false 6447 Acked Read 7 Done Sent Done false 6448 Acked Read 4 Done Sent Done false 6449 Acked Flush 0 Done Sent Done false 6450 Acked Read 7 Done Sent Done false 6451 Acked Read 3 Done Sent Done false 6452 Acked Write 3 Done Sent Done false 6453 Acked Write 10 Done Sent Done false 6454 Acked Write 8 Done Sent Done false 6455 Acked Read 3 Done Sent Done false 6456 Acked Write 3 Done Sent Done false 6457 Acked Write 7 Done Sent Done false 6458 Acked Flush 0 Done Sent Done false 6459 Acked Read 8 Done Sent Done false 6460 Acked Read 4 Done Sent Done false 6461 Acked Write 8 Done Sent Done false 6462 Acked Read 8 Done Sent Done false 6463 Acked Write 9 Done Sent Done false 6464 Acked Write 8 Done Sent Done false 6465 Acked Read 8 Done Sent Done false 6466 Acked Write 2 Done Sent Done false 6467 Acked Read 7 Done Sent Done false 6468 Acked Read 10 Done Sent Done false 6469 Acked Read 2 Done Sent Done false 6470 Acked Write 6 Done Sent Done false 6471 Acked Write 3 Done Sent Done false 6472 Acked Read 4 Done Sent Done false 6473 Acked Read 8 Done Sent Done false 6474 Acked Write 10 Done Sent Done false 6475 Acked Flush 0 Done Sent Done false 6476 Acked Read 9 Done Sent Done false 6477 Acked Read 6 Done Sent Done false 6478 Acked Write 8 Done Sent Done false 6479 Acked Read 2 Done Sent Done false 6480 Acked Write 7 Done Sent Done false 6481 Acked Write 7 Done Sent Done false 6482 Acked Read 7 Done Sent Done false 6483 Acked Write 8 Done Sent Done false 6484 Acked Write 2 Done Sent Done false 6485 Acked Read 5 Done Sent Done false 6486 Acked Write 6 Done Sent Done false 6487 Acked Flush 0 Done Sent Done false 6488 Acked Flush 0 Done Sent Done false 6489 Acked Flush 0 Done Sent Done false 6490 Acked Read 3 Done Sent Done false 6491 Acked Write 3 Done Sent Done false 6492 Acked Write 3 Done Sent Done false 6493 Acked Write 9 Done Sent Done false 6494 Acked Write 1 Done Sent Done false 6495 Acked Write 1 Done Sent Done false 6496 Acked Write 2 Done Sent Done false 6497 Acked Flush 0 Done Sent Done false 6498 Acked Read 3 Done Sent Done false 6499 Acked Flush 0 Done Sent Done false 6500 Acked Read 1 Done Sent Done false 6501 Acked Flush 0 Done Sent Done false 6502 Acked Read 3 Done Sent Done false 6503 Acked Read 2 Done Sent Done false 6504 Acked Write 4 Done Sent Done false 6505 Acked Flush 0 Done Sent Done false 6506 Acked Read 4 Done Sent Done false 6507 Acked Read 3 Done Sent Done false 6508 Acked Read 3 Done Sent Done false 6509 Acked Flush 0 Done Sent Done false 6510 Acked Read 10 Done Sent Done false 6511 Acked Read 2 Done Sent Done false 6512 Acked Read 9 Done Sent Done false 6513 Acked Read 3 Done Sent Done false 6514 Acked Read 10 Done Sent Done false 6515 Acked Read 6 Done Sent Done false 6516 Acked Write 3 Done Sent Done false 6517 Acked Write 3 Done Sent Done false 6518 Acked Write 2 Done Sent Done false 6519 Acked Read 4 Done Sent Done false 6520 Acked Write 9 Done Sent Done false 6521 Acked Write 8 Done Sent Done false 6522 Acked Write 8 Done Sent Done false 6523 Acked Write 2 Done Sent Done false 6524 Acked Write 9 Done Sent Done false 6525 Acked Read 6 Done Sent Done false 6526 Acked Write 9 Done Sent Done false 6527 Acked Flush 0 Done Sent Done false 6528 Acked Read 5 Done Sent Done false 6529 Acked Read 6 Done Sent Done false 6530 Acked Write 4 Done Sent Done false 6531 Acked Write 1 Done Sent Done false 6532 Acked Read 1 Done Sent Done false 6533 Acked Flush 0 Done Sent Done false 6534 Acked Write 5 Done Sent Done false 6535 Acked Write 10 Done Sent Done false 6536 Acked Write 9 Done Sent Done false 6537 Acked Write 4 Done Sent Done false 6538 Acked Write 5 Done Sent Done false 6539 Acked Write 4 Done Sent Done false 6540 Acked Write 1 Done Sent Done false 6541 Acked Read 6 Done Sent Done false 6542 Acked Flush 0 Done Sent Done false 6543 Acked Read 6 Done Sent Done false 6544 Acked Read 3 Done Sent Done false 6545 Acked Write 10 Done Sent Done false 6546 Acked Read 9 Done Sent Done false 6547 Acked Read 4 Done Sent Done false 6548 Acked Write 1 Done Sent Done false 6549 Acked Read 8 Done Sent Done false 6550 Acked Write 3 Done Sent Done false 6551 Acked Flush 0 Done Sent Done false 6552 Acked Flush 0 Done Sent Done false 6553 Acked Read 10 Done Sent Done false 6554 Acked Read 5 Done Sent Done false 6555 Acked Write 7 Done Sent Done false 6556 Acked Write 5 Done Sent Done false 6557 Acked Read 2 Done Sent Done false 6558 Acked Flush 0 Done Sent Done false 6559 Acked Write 10 Done Sent Done false 6560 Acked Read 10 Done Sent Done false 6561 Acked Flush 0 Done Sent Done false 6562 Acked Read 2 Done Sent Done false 6563 Acked Write 8 Done Sent Done false 6564 Acked Read 2 Done Sent Done false 6565 Acked Read 9 Done Sent Done false 6566 Acked Read 4 Done Sent Done false 6567 Acked Read 6 Done Sent Done false 6568 Acked Write 7 Done Sent Done false 6569 Acked Read 10 Done Sent Done false 6570 Acked Write 1 Done Sent Done false 6571 Acked Read 1 Done Sent Done false 6572 Acked Read 3 Done Sent Done false 6573 Acked Read 8 Done Sent Done false 6574 Acked Read 1 Done Sent Done false 6575 Acked Write 4 Done Sent Done false 6576 Acked Write 1 Done Sent Done false 6577 Acked Read 4 Done Sent Done false 6578 Acked Read 1 Done Sent Done false 6579 Acked Write 8 Done Sent Done false 6580 Acked Read 4 Done Sent Done false 6581 Acked Read 9 Done Sent Done false 6582 Acked Read 1 Done Sent Done false 6583 Acked Read 1 Done Sent Done false 6584 Acked Read 10 Done Sent Done false 6585 Acked Read 7 Done Sent Done false 6586 Acked Read 8 Done Sent Done false 6587 Acked Write 7 Done Sent Done false 6588 Acked Read 6 Done Sent Done false 6589 Acked Read 10 Done Sent Done false 6590 Acked Flush 0 Done Sent Done false 6591 Acked Read 5 Done Sent Done false 6592 Acked Read 2 Done Sent Done false 6593 Acked Write 7 Done Sent Done false 6594 Acked Read 9 Done Sent Done false 6595 Acked Read 7 Done Sent Done false 6596 Acked Write 3 Done Sent Done false 6597 Acked Write 7 Done Sent Done false 6598 Acked Write 2 Done Sent Done false 6599 Acked Flush 0 Done Sent Done false 6600 Acked Write 5 Done Sent Done false 6601 Acked Write 2 Done Sent Done false 6602 Acked Write 2 Done Sent Done false 6603 Acked Write 1 Done Sent Done false 6604 Acked Read 7 Done Sent Done false 6605 Acked Write 9 Done Sent Done false 6606 Acked Flush 0 Done Sent Done false 6607 Acked Flush 0 Done Sent Done false 6608 Acked Write 5 Done Sent Done false 6609 Acked Read 4 Done Sent Done false 6610 Acked Write 8 Done Sent Done false 6611 Acked Read 4 Done Sent Done false 6612 Acked Read 8 Done Sent Done false 6613 Acked Write 6 Done Sent Done false 6614 Acked Write 8 Done Sent Done false 6615 Acked Write 5 Done Sent Done false 6616 Acked Write 8 Done Sent Done false 6617 Acked Write 4 Done Sent Done false 6618 Acked Write 5 Done Sent Done false 6619 Acked Write 3 Done Sent Done false 6620 Acked Read 9 Done Sent Done false 6621 Acked Read 8 Done Sent Done false 6622 Acked Read 1 Done Sent Done false 6623 Acked Write 7 Done Sent Done false 6624 Acked Read 2 Done Sent Done false 6625 Acked Write 1 Done Sent Done false 6626 Acked Read 8 Done Sent Done false 6627 Acked Read 4 Done Sent Done false 6628 Acked Write 8 Done Sent Done false 6629 Acked Read 7 Done Sent Done false 6630 Acked Read 1 Done Sent Done false 6631 Acked Flush 0 Done Sent Done false 6632 Acked Write 5 Done Sent Done false 6633 Acked Read 1 Done Sent Done false 6634 Acked Read 3 Done Sent Done false 6635 Acked Write 5 Done Sent Done false 6636 Acked Write 2 Done Sent Done false 6637 Acked Write 6 Done Sent Done false 6638 Acked Write 1 Done Sent Done false 6639 Acked Write 7 Done Sent Done false 6640 Acked Flush 0 Done Sent Done false 6641 Acked Flush 0 Done Sent Done false 6642 Acked Read 4 Done Sent Done false 6643 Acked Read 7 Done Sent Done false 6644 Acked Read 9 Done Sent Done false 6645 Acked Read 8 Done Sent Done false 6646 Acked Write 9 Done Sent Done false 6647 Acked Write 10 Done Sent Done false 6648 Acked Write 1 Done Sent Done false 6649 Acked Read 5 Done Sent Done false 6650 Acked Flush 0 Done Sent Done false 6651 Acked Flush 0 Done Sent Done false 6652 Acked Read 6 Done Sent Done false 6653 Acked Write 8 Done Sent Done false 6654 Acked Read 5 Done Sent Done false 6655 Acked Read 1 Done Sent Done false 6656 Acked Read 9 Done Sent Done false 6657 Acked Write 8 Done Sent Done false 6658 Acked Write 2 Done Sent Done false 6659 Acked Read 6 Done Sent Done false 6660 Acked Read 3 Done Sent Done false 6661 Acked Read 1 Done Sent Done false 6662 Acked Write 8 Done Sent Done false 6663 Acked Read 9 Done Sent Done false 6664 Acked Read 5 Done Sent Done false 6665 Acked Read 1 Done Sent Done false 6666 Acked Read 2 Done Sent Done false 6667 Acked Write 2 Done Sent Done false 6668 Acked Write 1 Done Sent Done false 6669 Acked Read 7 Done Sent Done false 6670 Acked Read 6 Done Sent Done false 6671 Acked Write 10 Done Sent Done false 6672 Acked Read 4 Done Sent Done false 6673 Acked Write 8 Done Sent Done false 6674 Acked Write 9 Done Sent Done false 6675 Acked Write 5 Done Sent Done false 6676 Acked Write 5 Done Sent Done false 6677 Acked Write 6 Done Sent Done false 6678 Acked Flush 0 Done Sent Done false 6679 Acked Write 9 Done Sent Done false 6680 Acked Write 9 Done Sent Done false 6681 Acked Read 6 Done Sent Done false 6682 Acked Read 1 Done Sent Done false 6683 Acked Write 5 Done Sent Done false 6684 Acked Read 7 Done Sent Done false 6685 Acked Read 1 Done Sent Done false 6686 Acked Read 10 Done Sent Done false 6687 Acked Write 7 Done Sent Done false 6688 Acked Write 1 Done Sent Done false 6689 Acked Read 9 Done Sent Done false 6690 Acked Read 8 Done Sent Done false 6691 Acked Write 9 Done Sent Done false 6692 Acked Read 2 Done Sent Done false 6693 Acked Read 10 Done Sent Done false 6694 Acked Read 6 Done Sent Done false 6695 Acked Flush 0 Done Sent Done false 6696 Acked Read 3 Done Sent Done false 6697 Acked Read 5 Done Sent Done false 6698 Acked Read 9 Done Sent Done false 6699 Acked Write 8 Done Sent Done false 6700 Acked Read 6 Done Sent Done false 6701 Acked Write 4 Done Sent Done false 6702 Acked Write 9 Done Sent Done false 6703 Acked Write 4 Done Sent Done false 6704 Acked Read 8 Done Sent Done false 6705 Acked Read 7 Done Sent Done false 6706 Acked Write 7 Done Sent Done false 6707 Acked Write 7 Done Sent Done false 6708 Acked Write 7 Done Sent Done false 6709 Acked Flush 0 Done Sent Done false 6710 Acked Read 1 Done Sent Done false 6711 Acked Read 4 Done Sent Done false 6712 Acked Read 7 Done Sent Done false 6713 Acked Read 5 Done Sent Done false 6714 Acked Read 2 Done Sent Done false 6715 Acked Flush 0 Done Sent Done false 6716 Acked Write 6 Done Sent Done false 6717 Acked Write 9 Done Sent Done false 6718 Acked Write 9 Done Sent Done false 6719 Acked Read 2 Done Sent Done false 6720 Acked Read 4 Done Sent Done false 6721 Acked Write 2 Done Sent Done false 6722 Acked Read 4 Done Sent Done false 6723 Acked Read 1 Done Sent Done false 6724 Acked Read 6 Done Sent Done false 6725 Acked Read 2 Done Sent Done false 6726 Acked Read 5 Done Sent Done false 6727 Acked Flush 0 Done Sent Done false 6728 Acked Flush 0 Done Sent Done false 6729 Acked Flush 0 Done Sent Done false 6730 Acked Flush 0 Done Sent Done false 6731 Acked Read 1 Done Sent Done false 6732 Acked Write 2 Done Sent Done false 6733 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6733 6431 6733 Downstairs last five completed: 6431 6430 6429 6428 6427 Upstairs last five completed: 6733 6732 6731 6730 6729 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.068907283Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:33324)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.068980593Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:36.070006977Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070558649Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070581003Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 6431","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070595389Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 6431","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070812931Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070833215Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070847192Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 6431","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.070860838Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 6431","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.071108572Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6733 6732 6731 6730 6729 Upstairs last five completed: 6733 6732 6731 6730 6729 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Nov 27 01:46:36.295 INFO request completed, latency_us: 317, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5281da4b-6ba6-4651-9fbf-d04d085c6538, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:46:36.296 INFO accepted connection, remote_addr: 127.0.0.1:61443, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:36.296409228Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:36.296473241Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:36.296502623Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.296516899Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.296531075Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:36.296544821Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:46:36.296 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/2, method: GET, req_id: cbb9caf6-0d7d-4cd5-a1cf-217f57eb7772, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:46:40.299 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/2, method: GET, req_id: 16a3de8a-3652-4267-8b2a-98d942654c54, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 1116 len 36864 data: 7 6 6 5 4 3 3 4 3 002/300 Read block 3015 len 28672 003/300 Flush 004/300 Write block 906 len 24576 data: 4 4 4 4 3 3 005/300 Read block 749 len 28672 006/300 Write block 333 len 12288 data: 4 4 4 007/300 Read block 4671 len 40960 008/300 Write block 1310 len 8192 data: 3 4 009/300 Read block 4252 len 16384 010/300 Flush 011/300 Write block 3594 len 32768 data: 5 4 6 6 4 5 5 5 012/300 Write block 569 len 36864 data: 4 3 4 4 4 4 4 4 4 013/300 Write block 3520 len 20480 data: 5 5 4 4 5 014/300 Write block 3474 len 28672 data: 5 4 5 5 4 4 4 015/300 Flush 016/300 Read block 2930 len 32768 017/300 Read block 1637 len 32768 018/300 Write block 2767 len 20480 data: 4 5 4 4 5 019/300 Flush 020/300 Write block 3885 len 32768 data: 2 3 3 3 5 6 6 5 021/300 Read block 1459 len 24576 022/300 Read block 2754 len 24576 023/300 Read block 1913 len 28672 024/300 Read block 2300 len 12288 025/300 Read block 4244 len 36864 026/300 Read block 4527 len 8192 027/300 Flush 028/300 Flush 029/300 Read block 3758 len 40960 030/300 Flush 031/300 Write block 136 len 40960 data: 3 3 3 3 3 3 3 2 3 3 032/300 Write block 4158 len 32768 data: 4 3 3 5 5 5 5 5 033/300 Write block 556 len 40960 data: 4 3 3 3 2 2 2 2 3 3 034/300 Write block 4387 len 16384 data: 5 6 9 9 035/300 Write block 2325 len 16384 data: 5 5 5 4 036/300 Read block 48 len 4096 037/300 Write block 4498 len 28672 data: 2 2 3 3 4 4 4 038/300 Read block 583 len 36864 039/300 Flush 040/300 Write block 2955 len 28672 data: 3 5 7 6 6 6 6 041/300 Write block 3627 len 32768 data: 5 5 3 4 3 3 4 3 042/300 Read block 4180 len 32768 043/300 Read block 1730 len 8192 044/300 Read block 2309 len 28672 045/300 Read block 2687 len 16384 046/300 Read block 89 len 28672 047/300 Write block 3704 len 28672 data: 5 5 4 4 3 3 4 048/300 Read block 3764 len 40960 049/300 Write block 821 len 8192 data: 4 4 050/300 Write block 3848 len 24576 data: 2 3 5 5 5 4 051/300 Write block 3812 len 4096 data: 6 052/300 Read block 2557 len 4096 053/300 Write block 3827 len 28672 data: 3 3 4 4 6 6 5 054/300 Write block 3496 len 4096 data: 3 055/300 Read block 1801 len 20480 056/300 Write block 1889 len 16384 data: 6 6 6 6 057/300 Read block 3293 len 8192 058/300 Write block 577 len 16384 data: 5 3 7 8 059/300 Read block 263 len 36864 060/300 Read block 4207 len 24576 061/300 Write block 4294 len 24576 data: 2 2 3 3 3 5 062/300 Read block 2708 len 28672 063/300 Write block 2189 len 24576 data: 3 4 5 5 4 4 064/300 Write block 1405 len 36864 data: 6 5 5 5 6 5 5 5 5 065/300 Write block 3172 len 4096 data: 4 066/300 Read block 4113 len 4096 067/300 Write block 4385 len 36864 data: 6 6 6 7 10 10 7 5 6 068/300 Write block 2039 len 36864 data: 7 6 7 7 7 8 8 6 3 069/300 Read block 2868 len 36864 070/300 Write block 2187 len 16384 data: 4 4 4 5 071/300 Write block 4523 len 40960 data: 4 4 4 4 3 3 4 5 5 5 072/300 Read block 1154 len 4096 073/300 Read block 35 len 8192 074/300 Write block 2046 len 40960 data: 7 4 3 3 2 3 4 4 4 4 075/300 Read block 2400 len 28672 076/300 Read block 1586 len 40960 077/300 Read block 2175 len 40960 078/300 Read block 2568 len 32768 079/300 Flush 080/300 Read block 3159 len 40960 081/300 Read block 4975 len 8192 082/300 Write block 3850 len 40960 data: 6 6 6 5 4 3 2 3 5 6 083/300 Read block 2391 len 24576 084/300 Read block 4786 len 4096 085/300 Read block 2529 len 12288 086/300 Read block 4436 len 32768 087/300 Read block 3519 len 24576 088/300 Read block 323 len 20480 089/300 Write block 65 len 8192 data: 6 6 090/300 Read block 3618 len 8192 091/300 Read block 1454 len 32768 092/300 Write block 867 len 28672 data: 7 7 6 6 5 4 5 093/300 Read block 1451 len 8192 094/300 Read block 182 len 20480 095/300 Flush 096/300 Write block 3963 len 16384 data: 5 4 5 4 097/300 Read block 3620 len 16384 098/300 Read block 2826 len 40960 099/300 Read block 1716 len 4096 100/300 Read block 2585 len 28672 101/300 Read block 3502 len 28672 102/300 Read block 3016 len 8192 103/300 Flush 104/300 Write block 31 len 16384 data: 8 7 7 6 105/300 Write block 4845 len 4096 data: 8 106/300 Read block 401 len 20480 107/300 Write block 70 len 4096 data: 6 108/300 Read block 3226 len 4096 109/300 Read block 1336 len 40960 110/300 Write block 1055 len 24576 data: 2 2 2 2 2 2 111/300 Read block 3529 len 24576 112/300 Read block 2878 len 8192 113/300 Read block 4775 len 24576 114/300 Flush 115/300 Write block 2427 len 16384 data: 3 4 3 3 116/300 Write block 4921 len 16384 data: 4 5 5 5 117/300 Read block 2575 len 40960 118/300 Flush 119/300 Write block 1694 len 4096 data: 5 120/300 Read block 4286 len 8192 121/300 Flush 122/300 Flush 123/300 Write block 864 len 12288 data: 6 6 7 124/300 Read block 1190 len 4096 125/300 Read block 2390 len 12288 126/300 Write block 718 len 32768 data: 6 5 4 3 4 4 4 5 127/300 Write block 153 len 32768 data: 6 6 6 6 5 3 3 2 128/300 Read block 2794 len 24576 129/300 Write block 4754 len 8192 data: 5 5 130/300 Flush 131/300 Read block 3050 len 32768 132/300 Write block 2327 len 40960 data: 6 5 3 2 3 3 6 5 4 5 133/300 Write block 1481 len 16384 data: 3 3 5 5 134/300 Write block 3142 len 28672 data: 3 3 3 2 2 3 2 135/300 Flush 136/300 Flush 137/300 Flush 138/300 Write block 307 len 28672 data: 2 2 3 3 4 4 4 139/300 Read block 3447 len 4096 140/300 Flush 141/300 Read block 3141 len 40960 142/300 Read block 3232 len 12288 143/300 Write block 4929 len 20480 data: 6 7 7 7 5 144/300 Read block 4431 len 28672 145/300 Flush 146/300 Flush 147/300 Write block 2695 len 4096 data: 6 148/300 Read block 3822 len 12288 149/300 Write block 2731 len 16384 data: 4 6 6 7 150/300 Flush 151/300 Read block 2891 len 4096 152/300 Write block 1538 len 16384 data: 5 6 9 8 153/300 Read block 3765 len 16384 154/300 Read block 4450 len 12288 155/300 Read block 53 len 16384 156/300 Read block 3186 len 32768 157/300 Read block 2828 len 4096 158/300 Write block 2728 len 36864 data: 4 4 5 5 7 7 8 8 7 159/300 Flush 160/300 Write block 2218 len 20480 data: 3 3 4 4 5 161/300 Read block 4752 len 4096 162/300 Write block 2180 len 32768 data: 4 5 5 4 3 3 4 5 163/300 Flush 164/300 Read block 140 len 28672 165/300 Read block 4483 len 28672 166/300 Write block 187 len 36864 data: 5 5 4 4 5 5 4 4 3 167/300 Read block 1547 len 24576 168/300 Write block 4820 len 32768 data: 3 3 3 3 3 3 4 6 169/300 Write block 4587 len 32768 data: 4 5 5 3 3 4 4 5 170/300 Write block 4144 len 12288 data: 4 4 4 171/300 Flush 172/300 Read block 4253 len 20480 173/300 Read block 1665 len 24576 174/300 Write block 1845 len 12288 data: 2 2 2 175/300 Read block 4854 len 8192 176/300 Read block 4102 len 24576 177/300 Flush 178/300 Read block 3422 len 8192 179/300 Read block 3814 len 28672 180/300 Write block 1505 len 32768 data: 3 2 4 5 5 4 3 3 181/300 Read block 577 len 32768 182/300 Read block 1934 len 4096 183/300 Read block 1837 len 16384 184/300 Read block 2133 len 28672 185/300 Flush 186/300 Read block 1795 len 12288 187/300 Read block 4542 len 16384 188/300 Write block 2707 len 40960 data: 2 3 3 3 3 3 4 4 5 6 189/300 Flush 190/300 Read block 1271 len 40960 191/300 Flush 192/300 Write block 2204 len 28672 data: 4 3 3 3 4 4 4 193/300 Write block 2312 len 16384 data: 7 6 5 4 194/300 Read block 4728 len 32768 195/300 Flush 196/300 Write block 292 len 12288 data: 5 4 4 197/300 Write block 1396 len 40960 data: 4 4 4 4 4 5 6 7 6 7 198/300 Read block 1673 len 40960 199/300 Read block 1053 len 12288 200/300 Write block 2575 len 36864 data: 3 3 4 3 3 3 3 3 3 201/300 Read block 4511 len 8192 202/300 Write block 3291 len 28672 data: 5 4 4 3 2 2 2 203/300 Read block 4070 len 24576 204/300 Read block 4836 len 32768 205/300 Write block 4443 len 40960 data: 9 8 7 2 2 2 2 2 2 2 206/300 Read block 261 len 36864 207/300 Write block 3479 len 20480 data: 5 5 4 4 4 208/300 Write block 632 len 36864 data: 6 5 5 6 4 4 5 4 3 209/300 Read block 2758 len 40960 210/300 Write block 3824 len 28672 data: 2 3 3 4 4 5 5 211/300 Write block 3097 len 40960 data: 4 5 5 5 4 7 7 7 7 7 212/300 Write block 2608 len 4096 data: 4 213/300 Write block 4138 len 40960 data: 3 4 4 3 3 3 5 5 5 4 214/300 Write block 3363 len 8192 data: 8 6 215/300 Read block 1904 len 20480 216/300 Write block 4497 len 20480 data: 2 3 3 4 4 217/300 Read block 498 len 36864 218/300 Read block 3923 len 8192 219/300 Write block 1150 len 28672 data: 6 5 3 4 5 5 4 220/300 Write block 1829 len 32768 data: 4 4 4 5 4 3 3 4 221/300 Write block 3182 len 8192 data: 4 4 222/300 Write block 4653 len 32768 data: 3 4 5 5 6 6 6 5 223/300 Write block 2395 len 4096 data: 5 224/300 Read block 2249 len 28672 225/300 Write block 102 len 16384 data: 6 4 4 5 226/300 Read block 3804 len 12288 227/300 Read block 4412 len 40960 228/300 Write block 880 len 8192 data: 4 4 229/300 Write block 1181 len 20480 data: 7 6 4 3 3 230/300 Read block 1055 len 32768 231/300 Read block 559 len 40960 232/300 Read block 520 len 8192 233/300 Read block 1535 len 36864 234/300 Write block 4182 len 12288 data: 5 5 6 235/300 Write block 3640 len 12288 data: 3 3 3 236/300 Write block 1167 len 24576 data: 5 5 5 6 7 6 237/300 Write block 4559 len 36864 data: 3 5 6 6 8 9 9 8 8 238/300 Write block 4682 len 20480 data: 8 5 6 5 6 239/300 Write block 355 len 36864 data: 6 3 3 3 3 2 2 2 2 240/300 Read block 4728 len 36864 241/300 Read block 4754 len 32768 242/300 Write block 4621 len 12288 data: 4 4 4 243/300 Write block 4627 len 16384 data: 6 4 5 5 244/300 Read block 865 len 24576 245/300 Flush 246/300 Write block 2367 len 4096 data: 3 247/300 Write block 1207 len 28672 data: 5 5 6 5 5 6 6 248/300 Write block 3787 len 20480 data: 4 4 4 3 3 249/300 Write block 3211 len 16384 data: 7 6 7 6 250/300 Read block 1129 len 28672 251/300 Write block 298 len 12288 data: 4 3 2 252/300 Write block 3732 len 8192 data: 5 5 253/300 Read block 3257 len 36864 254/300 Read block 4890 len 24576 255/300 Read block 1295 len 24576 256/300 Read block 4511 len 20480 257/300 Write block 1003 len 24576 data: 5 5 4 4 4 4 258/300 Read block 4320 len 40960 259/300 Flush 260/300 Read block 4714 len 4096 261/300 Read block 3269 len 16384 262/300 Read block 483 len 4096 263/300 Write block 3446 len 24576 data: 4 3 4 3 2 3 264/300 Write block 2925 len 32768 data: 6 4 2 2 2 2 2 3 265/300 Read block 3134 len 12288 266/300 Flush 267/300 Read block 4217 len 12288 268/300 Flush 269/300 Flush 270/300 Flush 271/300 Read block 2038 len 16384 272/300 Read block 4237 len 20480 273/300 Write block 4049 len 40960 data: 5 6 4 4 4 3 3 3 3 2 274/300 Write block 3848 len 24576 data: 3 4 7 7 7 6 275/300 Read block 3901 len 28672 276/300 Write block 2987 len 32768 data: 4 4 4 5 5 5 4 4 277/300 Read block 3038 len 16384 278/300 Read block 4461 len 16384 279/300 Write block 2034 len 4096 data: 5 280/300 Read block 3381 len 32768 281/300 Flush 282/300 Write block 516 len 12288 data: 3 3 3 283/300 Flush 284/300 Flush 285/300 Read block 3830 len 20480 286/300 Read block 3818 len 24576 287/300 Read block 3416 len 12288 288/300 Write block 2661 len 16384 data: 3 2 2 3 289/300 Write block 2022 len 16384 data: 6 5 6 6 290/300 Read block 87 len 24576 291/300 Flush 292/300 Flush 293/300 Write block 2605 len 36864 data: 6 5 4 5 4 4 4 3 3 294/300 Write block 3295 len 12288 data: 3 3 3 295/300 Read block 4340 len 24576 296/300 Read block 1170 len 12288 297/300 Write block 2786 len 32768 data: 4 4 5 5 4 4 5 5 298/300 Write block 3770 len 24576 data: 5 4 4 4 5 6 299/300 Write block 83 len 28672 data: 5 4 4 3 4 4 4 300/300 Read block 4456 len 16384 301/300 Read block 2461 len 8192 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:46:40.526 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2896b13e-41fc-473a-9018-fdee7e118008, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6734 Acked Write 9 Done Done Sent false 6735 Acked Read 7 Done Done Sent false 6736 Acked Flush 0 Done Done Sent false 6737 Acked Write 6 Done Done Sent false 6738 Acked Read 7 Done Done Sent false 6739 Acked Write 3 Done Done Sent false 6740 Acked Read 10 Done Done Sent false 6741 Acked Write 2 Done Done Sent false 6742 Acked Read 4 Done Done Sent false 6743 Acked Flush 0 Done Done Sent false 6744 Acked Write 8 Done Done Sent false 6745 Acked Write 9 Done Done Sent false 6746 Acked Write 5 Done Done Sent false 6747 Acked Write 7 Done Done Sent false 6748 Acked Flush 0 Done Done Sent false 6749 Acked Read 8 Done Done Sent false 6750 Acked Read 8 Done Done Sent false 6751 Acked Write 5 Done Done Sent false 6752 Acked Flush 0 Done Done Sent false 6753 Acked Write 8 Done Done Sent false 6754 Acked Read 6 Done Done Sent false 6755 Acked Read 6 Done Done Sent false 6756 Acked Read 7 Done Done Sent false 6757 Acked Read 3 Done Done Sent false 6758 Acked Read 9 Done Done Sent false 6759 Acked Read 2 Done Done Sent false 6760 Acked Flush 0 Done Done Sent false 6761 Acked Flush 0 Done Done Sent false 6762 Acked Read 10 Done Done Sent false 6763 Acked Flush 0 Done Done Sent false 6764 Acked Write 10 Done Done Sent false 6765 Acked Write 8 Done Done Sent false 6766 Acked Write 10 Done Done Sent false 6767 Acked Write 4 Done Done Sent false 6768 Acked Write 4 Done Done Sent false 6769 Acked Read 1 Done Done Sent false 6770 Acked Write 7 Done Done Sent false 6771 Acked Read 9 Done Done Sent false 6772 Acked Flush 0 Done Done Sent false 6773 Acked Write 7 Done Done Sent false 6774 Acked Write 8 Done Done Sent false 6775 Acked Read 8 Done Done Sent false 6776 Acked Read 2 Done Done Sent false 6777 Acked Read 7 Done Done Sent false 6778 Acked Read 4 Done Done Sent false 6779 Acked Read 7 Done Done Sent false 6780 Acked Write 7 Done Done Sent false 6781 Acked Read 10 Done Done Sent false 6782 Acked Write 2 Done Done Sent false 6783 Acked Write 6 Done Done Sent false 6784 Acked Write 1 Done Done Sent false 6785 Acked Read 1 Done Done Sent false 6786 Acked Write 7 Done Done Sent false 6787 Acked Write 1 Done Done Sent false 6788 Acked Read 5 Done Done Sent false 6789 Acked Write 4 Done Done Sent false 6790 Acked Read 2 Done Done Sent false 6791 Acked Write 4 Done Done Sent false 6792 Acked Read 9 Done Done Sent false 6793 Acked Read 6 Done Done Sent false 6794 Acked Write 6 Done Done Sent false 6795 Acked Read 7 Done Done Sent false 6796 Acked Write 6 Done Done Sent false 6797 Acked Write 9 Done Done Sent false 6798 Acked Write 1 Done Done Sent false 6799 Acked Read 1 Done Done Sent false 6800 Acked Write 9 Done Done Sent false 6801 Acked Write 9 Done Done Sent false 6802 Acked Read 9 Done Done Sent false 6803 Acked Write 4 Done Done Sent false 6804 Acked Write 10 Done Done Sent false 6805 Acked Read 1 Done Done Sent false 6806 Acked Read 2 Done Done Sent false 6807 Acked Write 10 Done Done Sent false 6808 Acked Read 7 Done Done Sent false 6809 Acked Read 10 Done Done Sent false 6810 Acked Read 10 Done Done Sent false 6811 Acked Read 8 Done Done Sent false 6812 Acked Flush 0 Done Done Sent false 6813 Acked Read 10 Done Done Sent false 6814 Acked Read 2 Done Done Sent false 6815 Acked Write 10 Done Done Sent false 6816 Acked Read 6 Done Done Sent false 6817 Acked Read 1 Done Done Sent false 6818 Acked Read 3 Done Done Sent false 6819 Acked Read 8 Done Done Sent false 6820 Acked Read 6 Done Done Sent false 6821 Acked Read 5 Done Done Sent false 6822 Acked Write 2 Done Done Sent false 6823 Acked Read 2 Done Done Sent false 6824 Acked Read 8 Done Done Sent false 6825 Acked Write 7 Done Done Sent false 6826 Acked Read 2 Done Done Sent false 6827 Acked Read 5 Done Done Sent false 6828 Acked Flush 0 Done Done Sent false 6829 Acked Write 4 Done Done Sent false 6830 Acked Read 4 Done Done Sent false 6831 Acked Read 10 Done Done Sent false 6832 Acked Read 1 Done Done Sent false 6833 Acked Read 7 Done Done Sent false 6834 Acked Read 7 Done Done Sent false 6835 Acked Read 2 Done Done Sent false 6836 Acked Flush 0 Done Done Sent false 6837 Acked Write 4 Done Done Sent false 6838 Acked Write 1 Done Done Sent false 6839 Acked Read 5 Done Done Sent false 6840 Acked Write 1 Done Done Sent false 6841 Acked Read 1 Done Done Sent false 6842 Acked Read 10 Done Done Sent false 6843 Acked Write 6 Done Done Sent false 6844 Acked Read 6 Done Done Sent false 6845 Acked Read 2 Done Done Sent false 6846 Acked Read 6 Done Done Sent false 6847 Acked Flush 0 Done Done Sent false 6848 Acked Write 4 Done Done Sent false 6849 Acked Write 4 Done Done Sent false 6850 Acked Read 10 Done Done Sent false 6851 Acked Flush 0 Done Done Sent false 6852 Acked Write 1 Done Done Sent false 6853 Acked Read 2 Done Done Sent false 6854 Acked Flush 0 Done Done Sent false 6855 Acked Flush 0 Done Done Sent false 6856 Acked Write 3 Done Done Sent false 6857 Acked Read 1 Done Done Sent false 6858 Acked Read 3 Done Done Sent false 6859 Acked Write 8 Done Done Sent false 6860 Acked Write 8 Done Done Sent false 6861 Acked Read 6 Done Done Sent false 6862 Acked Write 2 Done Done Sent false 6863 Acked Flush 0 Done Done Sent false 6864 Acked Read 8 Done Done Sent false 6865 Acked Write 10 Done Done Sent false 6866 Acked Write 4 Done Done Sent false 6867 Acked Write 7 Done Done Sent false 6868 Acked Flush 0 Done Done Sent false 6869 Acked Flush 0 Done Done Sent false 6870 Acked Flush 0 Done Done Sent false 6871 Acked Write 7 Done Done Sent false 6872 Acked Read 1 Done Done Sent false 6873 Acked Flush 0 Done Done Sent false 6874 Acked Read 10 Done Done Sent false 6875 Acked Read 3 Done Done Sent false 6876 Acked Write 5 Done Done Sent false 6877 Acked Read 7 Done Done Sent false 6878 Acked Flush 0 Done Done Sent false 6879 Acked Flush 0 Done Done Sent false 6880 Acked Write 1 Done Done Sent false 6881 Acked Read 3 Done Done Sent false 6882 Acked Write 4 Done Done Sent false 6883 Acked Flush 0 Done Done Sent false 6884 Acked Read 1 Done Done Sent false 6885 Acked Write 4 Done Done Sent false 6886 Acked Read 4 Done Done Sent false 6887 Acked Read 3 Done Done Sent false 6888 Acked Read 4 Done Done Sent false 6889 Acked Read 8 Done Done Sent false 6890 Acked Read 1 Done Done Sent false 6891 Acked Write 9 Done Done Sent false 6892 Acked Flush 0 Done Done Sent false 6893 Acked Write 5 Done Done Sent false 6894 Acked Read 1 Done Done Sent false 6895 Acked Write 8 Done Done Sent false 6896 Acked Flush 0 Done Done Sent false 6897 Acked Read 7 Done Done Sent false 6898 Acked Read 7 Done Done Sent false 6899 Acked Write 9 Done Done Sent false 6900 Acked Read 6 Done Done Sent false 6901 Acked Write 8 Done Done Sent false 6902 Acked Write 8 Done Done Sent false 6903 Acked Write 3 Done Done Sent false 6904 Acked Flush 0 Done Done Sent false 6905 Acked Read 5 Done Done Sent false 6906 Acked Read 6 Done Done Sent false 6907 Acked Write 3 Done Done Sent false 6908 Acked Read 2 Done Done Sent false 6909 Acked Read 6 Done Done Sent false 6910 Acked Flush 0 Done Done Sent false 6911 Acked Read 2 Done Done Sent false 6912 Acked Read 7 Done Done Sent false 6913 Acked Write 8 Done Done Sent false 6914 Acked Read 8 Done Done Sent false 6915 Acked Read 1 Done Done Sent false 6916 Acked Read 4 Done Done Sent false 6917 Acked Read 7 Done Done Sent false 6918 Acked Flush 0 Done Done Sent false 6919 Acked Read 3 Done Done Sent false 6920 Acked Read 4 Done Done Sent false 6921 Acked Write 10 Done Done Sent false 6922 Acked Flush 0 Done Done Sent false 6923 Acked Read 10 Done Done Sent false 6924 Acked Flush 0 Done Done Sent false 6925 Acked Write 7 Done Done Sent false 6926 Acked Write 4 Done Done Sent false 6927 Acked Read 8 Done Done Sent false 6928 Acked Flush 0 Done Done Sent false 6929 Acked Write 3 Done Done Sent false 6930 Acked Write 10 Done Done Sent false 6931 Acked Read 10 Done Done Sent false 6932 Acked Read 3 Done Done Sent false 6933 Acked Write 9 Done Done Sent false 6934 Acked Read 2 Done Done Sent false 6935 Acked Write 7 Done Done Sent false 6936 Acked Read 6 Done Done Sent false 6937 Acked Read 8 Done Done Sent false 6938 Acked Write 10 Done Done Sent false 6939 Acked Read 9 Done Done Sent false 6940 Acked Write 5 Done Done Sent false 6941 Acked Write 9 Done Done Sent false 6942 Acked Read 10 Done Done Sent false 6943 Acked Write 7 Done Done Sent false 6944 Acked Write 10 Done Done Sent false 6945 Acked Write 1 Done Done Sent false 6946 Acked Write 10 Done Done Sent false 6947 Acked Write 2 Done Done Sent false 6948 Acked Read 5 Done Done Sent false 6949 Acked Write 5 Done Done Sent false 6950 Acked Read 9 Done Done Sent false 6951 Acked Read 2 Done Done Sent false 6952 Acked Write 7 Done Done Sent false 6953 Acked Write 8 Done Done Sent false 6954 Acked Write 2 Done Done Sent false 6955 Acked Write 8 Done Done Sent false 6956 Acked Write 1 Done Done Sent false 6957 Acked Read 7 Done Done Sent false 6958 Acked Write 4 Done Done Sent false 6959 Acked Read 3 Done Done Sent false 6960 Acked Read 10 Done Done Sent false 6961 Acked Write 2 Done Done Sent false 6962 Acked Write 5 Done Done Sent false 6963 Acked Read 8 Done Done Sent false 6964 Acked Read 10 Done Done Sent false 6965 Acked Read 2 Done Done Sent false 6966 Acked Read 9 Done Done Sent false 6967 Acked Write 3 Done Done Sent false 6968 Acked Write 3 Done Done Sent false 6969 Acked Write 6 Done Done Sent false 6970 Acked Write 9 Done Done Sent false 6971 Acked Write 5 Done Done Sent false 6972 Acked Write 9 Done Done Sent false 6973 Acked Read 9 Done Done Sent false 6974 Acked Read 8 Done Done Sent false 6975 Acked Write 3 Done Done Sent false 6976 Acked Write 4 Done Done Sent false 6977 Acked Read 6 Done Done Sent false 6978 Acked Flush 0 Done Done Sent false 6979 Acked Write 1 Done Done Sent false 6980 Acked Write 7 Done Done Sent false 6981 Acked Write 5 Done Done Sent false 6982 Acked Write 4 Done Done Sent false 6983 Acked Read 7 Done Done Sent false 6984 Acked Write 3 Done Done Sent false 6985 Acked Write 2 Done Done Sent false 6986 Acked Read 9 Done Done Sent false 6987 Acked Read 6 Done Done Sent false 6988 Acked Read 6 Done Done Sent false 6989 Acked Read 5 Done Done Sent false 6990 Acked Write 6 Done Done Sent false 6991 Acked Read 10 Done Done Sent false 6992 Acked Flush 0 Done Done Sent false 6993 Acked Read 1 Done Done Sent false 6994 Acked Read 4 Done Done Sent false 6995 Acked Read 1 Done Done Sent false 6996 Acked Write 6 Done Done Sent false 6997 Acked Write 8 Done Done Sent false 6998 Acked Read 3 Done Done Sent false 6999 Acked Flush 0 Done Done Sent false 7000 Acked Read 3 Done Done Sent false 7001 Acked Flush 0 Done Done Sent false 7002 Acked Flush 0 Done Done Sent false 7003 Acked Flush 0 Done Done Sent false 7004 Acked Read 4 Done Done Sent false 7005 Acked Read 5 Done Done Sent false 7006 Acked Write 10 Done Done Sent false 7007 Acked Write 6 Done Done Sent false 7008 Acked Read 7 Done Done Sent false 7009 Acked Write 8 Done Done Sent false 7010 Acked Read 4 Done Done Sent false 7011 Acked Read 4 Done Done Sent false 7012 Acked Write 1 Done Done Sent false 7013 Acked Read 8 Done Done Sent false 7014 Acked Flush 0 Done Done Sent false 7015 Acked Write 3 Done Done Sent false 7016 Acked Flush 0 Done Done Sent false 7017 Acked Flush 0 Done Done Sent false 7018 Acked Read 5 Done Done Sent false 7019 Acked Read 6 Done Done Sent false 7020 Acked Read 3 Done Done Sent false 7021 Acked Write 4 Done Done Sent false 7022 Acked Write 4 Done Done Sent false 7023 Acked Read 6 Done Done Sent false 7024 Acked Flush 0 Done Done Sent false 7025 Acked Flush 0 Done Done Sent false 7026 Acked Write 9 Done Done Sent false 7027 Acked Write 3 Done Done Sent false 7028 Acked Read 6 Done Done Sent false 7029 Acked Read 3 Done Done Sent false 7030 Acked Write 8 Done Done Sent false 7031 Acked Write 6 Done Done Sent false 7032 Acked Write 7 Done Done Sent false 7033 Acked Read 4 Done Done Sent false 7034 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7025 7025 6733 Downstairs last five completed: 6733 6732 6731 6730 6729 Upstairs last five completed: 7034 7033 7032 7031 7030 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6734 Acked Write 9 Done Done Sent false 6735 Acked Read 7 Done Done Sent false 6736 Acked Flush 0 Done Done Sent false 6737 Acked Write 6 Done Done Sent false 6738 Acked Read 7 Done Done Sent false 6739 Acked Write 3 Done Done Sent false 6740 Acked Read 10 Done Done Sent false 6741 Acked Write 2 Done Done Sent false 6742 Acked Read 4 Done Done Sent false 6743 Acked Flush 0 Done Done Sent false 6744 Acked Write 8 Done Done Sent false 6745 Acked Write 9 Done Done Sent false 6746 Acked Write 5 Done Done Sent false 6747 Acked Write 7 Done Done Sent false 6748 Acked Flush 0 Done Done Sent false 6749 Acked Read 8 Done Done Sent false 6750 Acked Read 8 Done Done Sent false 6751 Acked Write 5 Done Done Sent false 6752 Acked Flush 0 Done Done Sent false 6753 Acked Write 8 Done Done Sent false 6754 Acked Read 6 Done Done Sent false 6755 Acked Read 6 Done Done Sent false 6756 Acked Read 7 Done Done Sent false 6757 Acked Read 3 Done Done Sent false 6758 Acked Read 9 Done Done Sent false 6759 Acked Read 2 Done Done Sent false 6760 Acked Flush 0 Done Done Sent false 6761 Acked Flush 0 Done Done Sent false 6762 Acked Read 10 Done Done Sent false 6763 Acked Flush 0 Done Done Sent false 6764 Acked Write 10 Done Done Sent false 6765 Acked Write 8 Done Done Sent false 6766 Acked Write 10 Done Done Sent false 6767 Acked Write 4 Done Done Sent false 6768 Acked Write 4 Done Done Sent false 6769 Acked Read 1 Done Done Sent false 6770 Acked Write 7 Done Done Sent false 6771 Acked Read 9 Done Done Sent false 6772 Acked Flush 0 Done Done Sent false 6773 Acked Write 7 Done Done Sent false 6774 Acked Write 8 Done Done Sent false 6775 Acked Read 8 Done Done Sent false 6776 Acked Read 2 Done Done Sent false 6777 Acked Read 7 Done Done Sent false 6778 Acked Read 4 Done Done Sent false 6779 Acked Read 7 Done Done Sent false 6780 Acked Write 7 Done Done Sent false 6781 Acked Read 10 Done Done Sent false 6782 Acked Write 2 Done Done Sent false 6783 Acked Write 6 Done Done Sent false 6784 Acked Write 1 Done Done Sent false 6785 Acked Read 1 Done Done Sent false 6786 Acked Write 7 Done Done Sent false 6787 Acked Write 1 Done Done Sent false 6788 Acked Read 5 Done Done Sent false 6789 Acked Write 4 Done Done Sent false 6790 Acked Read 2 Done Done Sent false 6791 Acked Write 4 Done Done Sent false 6792 Acked Read 9 Done Done Sent false 6793 Acked Read 6 Done Done Sent false 6794 Acked Write 6 Done Done Sent false 6795 Acked Read 7 Done Done Sent false 6796 Acked Write 6 Done Done Sent false 6797 Acked Write 9 Done Done Sent false 6798 Acked Write 1 Done Done Sent false 6799 Acked Read 1 Done Done Sent false 6800 Acked Write 9 Done Done Sent false 6801 Acked Write 9 Done Done Sent false 6802 Acked Read 9 Done Done Sent false 6803 Acked Write 4 Done Done Sent false 6804 Acked Write 10 Done Done Sent false 6805 Acked Read 1 Done Done Sent false 6806 Acked Read 2 Done Done Sent false 6807 Acked Write 10 Done Done Sent false 6808 Acked Read 7 Done Done Sent false 6809 Acked Read 10 Done Done Sent false 6810 Acked Read 10 Done Done Sent false 6811 Acked Read 8 Done Done Sent false 6812 Acked Flush 0 Done Done Sent false 6813 Acked Read 10 Done Done Sent false 6814 Acked Read 2 Done Done Sent false 6815 Acked Write 10 Done Done Sent false 6816 Acked Read 6 Done Done Sent false 6817 Acked Read 1 Done Done Sent false 6818 Acked Read 3 Done Done Sent false 6819 Acked Read 8 Done Done Sent false 6820 Acked Read 6 Done Done Sent false 6821 Acked Read 5 Done Done Sent false 6822 Acked Write 2 Done Done Sent false 6823 Acked Read 2 Done Done Sent false 6824 Acked Read 8 Done Done Sent false 6825 Acked Write 7 Done Done Sent false 6826 Acked Read 2 Done Done Sent false 6827 Acked Read 5 Done Done Sent false 6828 Acked Flush 0 Done Done Sent false 6829 Acked Write 4 Done Done Sent false 6830 Acked Read 4 Done Done Sent false 6831 Acked Read 10 Done Done Sent false 6832 Acked Read 1 Done Done Sent false 6833 Acked Read 7 Done Done Sent false 6834 Acked Read 7 Done Done Sent false 6835 Acked Read 2 Done Done Sent false 6836 Acked Flush 0 Done Done Sent false 6837 Acked Write 4 Done Done Sent false 6838 Acked Write 1 Done Done Sent false 6839 Acked Read 5 Done Done Sent false 6840 Acked Write 1 Done Done Sent false 6841 Acked Read 1 Done Done Sent false 6842 Acked Read 10 Done Done Sent false 6843 Acked Write 6 Done Done Sent false 6844 Acked Read 6 Done Done Sent false 6845 Acked Read 2 Done Done Sent false 6846 Acked Read 6 Done Done Sent false 6847 Acked Flush 0 Done Done Sent false 6848 Acked Write 4 Done Done Sent false 6849 Acked Write 4 Done Done Sent false 6850 Acked Read 10 Done Done Sent false 6851 Acked Flush 0 Done Done Sent false 6852 Acked Write 1 Done Done Sent false 6853 Acked Read 2 Done Done Sent false 6854 Acked Flush 0 Done Done Sent false 6855 Acked Flush 0 Done Done Sent false 6856 Acked Write 3 Done Done Sent false 6857 Acked Read 1 Done Done Sent false 6858 Acked Read 3 Done Done Sent false 6859 Acked Write 8 Done Done Sent false 6860 Acked Write 8 Done Done Sent false 6861 Acked Read 6 Done Done Sent false 6862 Acked Write 2 Done Done Sent false 6863 Acked Flush 0 Done Done Sent false 6864 Acked Read 8 Done Done Sent false 6865 Acked Write 10 Done Done Sent false 6866 Acked Write 4 Done Done Sent false 6867 Acked Write 7 Done Done Sent false 6868 Acked Flush 0 Done Done Sent false 6869 Acked Flush 0 Done Done Sent false 6870 Acked Flush 0 Done Done Sent false 6871 Acked Write 7 Done Done Sent false 6872 Acked Read 1 Done Done Sent false 6873 Acked Flush 0 Done Done Sent false 6874 Acked Read 10 Done Done Sent false 6875 Acked Read 3 Done Done Sent false 6876 Acked Write 5 Done Done Sent false 6877 Acked Read 7 Done Done Sent false 6878 Acked Flush 0 Done Done Sent false 6879 Acked Flush 0 Done Done Sent false 6880 Acked Write 1 Done Done Sent false 6881 Acked Read 3 Done Done Sent false 6882 Acked Write 4 Done Done Sent false 6883 Acked Flush 0 Done Done Sent false 6884 Acked Read 1 Done Done Sent false 6885 Acked Write 4 Done Done Sent false 6886 Acked Read 4 Done Done Sent false 6887 Acked Read 3 Done Done Sent false 6888 Acked Read 4 Done Done Sent false 6889 Acked Read 8 Done Done Sent false 6890 Acked Read 1 Done Done Sent false 6891 Acked Write 9 Done Done Sent false 6892 Acked Flush 0 Done Done Sent false 6893 Acked Write 5 Done Done Sent false 6894 Acked Read 1 Done Done Sent false 6895 Acked Write 8 Done Done Sent false 6896 Acked Flush 0 Done Done Sent false 6897 Acked Read 7 Done Done Sent false 6898 Acked Read 7 Done Done Sent false 6899 Acked Write 9 Done Done Sent false 6900 Acked Read 6 Done Done Sent false 6901 Acked Write 8 Done Done Sent false 6902 Acked Write 8 Done Done Sent false 6903 Acked Write 3 Done Done Sent false 6904 Acked Flush 0 Done Done Sent false 6905 Acked Read 5 Done Done Sent false 6906 Acked Read 6 Done Done Sent false 6907 Acked Write 3 Done Done Sent false 6908 Acked Read 2 Done Done Sent false 6909 Acked Read 6 Done Done Sent false 6910 Acked Flush 0 Done Done Sent false 6911 Acked Read 2 Done Done Sent false 6912 Acked Read 7 Done Done Sent false 6913 Acked Write 8 Done Done Sent false 6914 Acked Read 8 Done Done Sent false 6915 Acked Read 1 Done Done Sent false 6916 Acked Read 4 Done Done Sent false 6917 Acked Read 7 Done Done Sent false 6918 Acked Flush 0 Done Done Sent false 6919 Acked Read 3 Done Done Sent false 6920 Acked Read 4 Done Done Sent false 6921 Acked Write 10 Done Done Sent false 6922 Acked Flush 0 Done Done Sent false 6923 Acked Read 10 Done Done Sent false 6924 Acked Flush 0 Done Done Sent false 6925 Acked Write 7 Done Done Sent false 6926 Acked Write 4 Done Done Sent false 6927 Acked Read 8 Done Done Sent false 6928 Acked Flush 0 Done Done Sent false 6929 Acked Write 3 Done Done Sent false 6930 Acked Write 10 Done Done Sent false 6931 Acked Read 10 Done Done Sent false 6932 Acked Read 3 Done Done Sent false 6933 Acked Write 9 Done Done Sent false 6934 Acked Read 2 Done Done Sent false 6935 Acked Write 7 Done Done Sent false 6936 Acked Read 6 Done Done Sent false 6937 Acked Read 8 Done Done Sent false 6938 Acked Write 10 Done Done Sent false 6939 Acked Read 9 Done Done Sent false 6940 Acked Write 5 Done Done Sent false 6941 Acked Write 9 Done Done Sent false 6942 Acked Read 10 Done Done Sent false 6943 Acked Write 7 Done Done Sent false 6944 Acked Write 10 Done Done Sent false 6945 Acked Write 1 Done Done Sent false 6946 Acked Write 10 Done Done Sent false 6947 Acked Write 2 Done Done Sent false 6948 Acked Read 5 Done Done Sent false 6949 Acked Write 5 Done Done Sent false 6950 Acked Read 9 Done Done Sent false 6951 Acked Read 2 Done Done Sent false 6952 Acked Write 7 Done Done Sent false 6953 Acked Write 8 Done Done Sent false 6954 Acked Write 2 Done Done Sent false 6955 Acked Write 8 Done Done Sent false 6956 Acked Write 1 Done Done Sent false 6957 Acked Read 7 Done Done Sent false 6958 Acked Write 4 Done Done Sent false 6959 Acked Read 3 Done Done Sent false 6960 Acked Read 10 Done Done Sent false 6961 Acked Write 2 Done Done Sent false 6962 Acked Write 5 Done Done Sent false 6963 Acked Read 8 Done Done Sent false 6964 Acked Read 10 Done Done Sent false 6965 Acked Read 2 Done Done Sent false 6966 Acked Read 9 Done Done Sent false 6967 Acked Write 3 Done Done Sent false 6968 Acked Write 3 Done Done Sent false 6969 Acked Write 6 Done Done Sent false 6970 Acked Write 9 Done Done Sent false 6971 Acked Write 5 Done Done Sent false 6972 Acked Write 9 Done Done Sent false 6973 Acked Read 9 Done Done Sent false 6974 Acked Read 8 Done Done Sent false 6975 Acked Write 3 Done Done Sent false 6976 Acked Write 4 Done Done Sent false 6977 Acked Read 6 Done Done Sent false 6978 Acked Flush 0 Done Done Sent false 6979 Acked Write 1 Done Done Sent false 6980 Acked Write 7 Done Done Sent false 6981 Acked Write 5 Done Done Sent false 6982 Acked Write 4 Done Done Sent false 6983 Acked Read 7 Done Done Sent false 6984 Acked Write 3 Done Done Sent false 6985 Acked Write 2 Done Done Sent false 6986 Acked Read 9 Done Done Sent false 6987 Acked Read 6 Done Done Sent false 6988 Acked Read 6 Done Done Sent false 6989 Acked Read 5 Done Done Sent false 6990 Acked Write 6 Done Done Sent false 6991 Acked Read 10 Done Done Sent false 6992 Acked Flush 0 Done Done Sent false 6993 Acked Read 1 Done Done Sent false 6994 Acked Read 4 Done Done Sent false 6995 Acked Read 1 Done Done Sent false 6996 Acked Write 6 Done Done Sent false 6997 Acked Write 8 Done Done Sent false 6998 Acked Read 3 Done Done Sent false 6999 Acked Flush 0 Done Done Sent false 7000 Acked Read 3 Done Done Sent false 7001 Acked Flush 0 Done Done Sent false 7002 Acked Flush 0 Done Done Sent false 7003 Acked Flush 0 Done Done Sent false 7004 Acked Read 4 Done Done Sent false 7005 Acked Read 5 Done Done Sent false 7006 Acked Write 10 Done Done Sent false 7007 Acked Write 6 Done Done Sent false 7008 Acked Read 7 Done Done Sent false 7009 Acked Write 8 Done Done Sent false 7010 Acked Read 4 Done Done Sent false 7011 Acked Read 4 Done Done Sent false 7012 Acked Write 1 Done Done Sent false 7013 Acked Read 8 Done Done Sent false 7014 Acked Flush 0 Done Done Sent false 7015 Acked Write 3 Done Done Sent false 7016 Acked Flush 0 Done Done Sent false 7017 Acked Flush 0 Done Done Sent false 7018 Acked Read 5 Done Done Sent false 7019 Acked Read 6 Done Done Sent false 7020 Acked Read 3 Done Done Sent false 7021 Acked Write 4 Done Done Sent false 7022 Acked Write 4 Done Done Sent false 7023 Acked Read 6 Done Done Sent false 7024 Acked Flush 0 Done Done Sent false 7025 Acked Flush 0 Done Done Sent false 7026 Acked Write 9 Done Done Sent false 7027 Acked Write 3 Done Done Sent false 7028 Acked Read 6 Done Done Sent false 7029 Acked Read 3 Done Done Sent false 7030 Acked Write 8 Done Done Sent false 7031 Acked Write 6 Done Done Sent false 7032 Acked Write 7 Done Done Sent false 7033 Acked Read 4 Done Done Sent false 7034 Acked Read 2 Done Done Sent false 7035 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7035 7035 6733 Downstairs last five completed: 6733 6732 6731 6730 6729 Upstairs last five completed: 7035 7034 7033 7032 7031 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:46.297633799Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:46.297696462Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:46.297725895Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:46.297746889Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:46.297768043Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6734 Acked Write 9 Done Done Sent false 6735 Acked Read 7 Done Done Sent false 6736 Acked Flush 0 Done Done Sent false 6737 Acked Write 6 Done Done Sent false 6738 Acked Read 7 Done Done Sent false 6739 Acked Write 3 Done Done Sent false 6740 Acked Read 10 Done Done Sent false 6741 Acked Write 2 Done Done Sent false 6742 Acked Read 4 Done Done Sent false 6743 Acked Flush 0 Done Done Sent false 6744 Acked Write 8 Done Done Sent false 6745 Acked Write 9 Done Done Sent false 6746 Acked Write 5 Done Done Sent false 6747 Acked Write 7 Done Done Sent false 6748 Acked Flush 0 Done Done Sent false 6749 Acked Read 8 Done Done Sent false 6750 Acked Read 8 Done Done Sent false 6751 Acked Write 5 Done Done Sent false 6752 Acked Flush 0 Done Done Sent false 6753 Acked Write 8 Done Done Sent false 6754 Acked Read 6 Done Done Sent false 6755 Acked Read 6 Done Done Sent false 6756 Acked Read 7 Done Done Sent false 6757 Acked Read 3 Done Done Sent false 6758 Acked Read 9 Done Done Sent false 6759 Acked Read 2 Done Done Sent false 6760 Acked Flush 0 Done Done Sent false 6761 Acked Flush 0 Done Done Sent false 6762 Acked Read 10 Done Done Sent false 6763 Acked Flush 0 Done Done Sent false 6764 Acked Write 10 Done Done Sent false 6765 Acked Write 8 Done Done Sent false 6766 Acked Write 10 Done Done Sent false 6767 Acked Write 4 Done Done Sent false 6768 Acked Write 4 Done Done Sent false 6769 Acked Read 1 Done Done Sent false 6770 Acked Write 7 Done Done Sent false 6771 Acked Read 9 Done Done Sent false 6772 Acked Flush 0 Done Done Sent false 6773 Acked Write 7 Done Done Sent false 6774 Acked Write 8 Done Done Sent false 6775 Acked Read 8 Done Done Sent false 6776 Acked Read 2 Done Done Sent false 6777 Acked Read 7 Done Done Sent false 6778 Acked Read 4 Done Done Sent false 6779 Acked Read 7 Done Done Sent false 6780 Acked Write 7 Done Done Sent false 6781 Acked Read 10 Done Done Sent false 6782 Acked Write 2 Done Done Sent false 6783 Acked Write 6 Done Done Sent false 6784 Acked Write 1 Done Done Sent false 6785 Acked Read 1 Done Done Sent false 6786 Acked Write 7 Done Done Sent false 6787 Acked Write 1 Done Done Sent false 6788 Acked Read 5 Done Done Sent false 6789 Acked Write 4 Done Done Sent false 6790 Acked Read 2 Done Done Sent false 6791 Acked Write 4 Done Done Sent false 6792 Acked Read 9 Done Done Sent false 6793 Acked Read 6 Done Done Sent false 6794 Acked Write 6 Done Done Sent false 6795 Acked Read 7 Done Done Sent false 6796 Acked Write 6 Done Done Sent false 6797 Acked Write 9 Done Done Sent false 6798 Acked Write 1 Done Done Sent false 6799 Acked Read 1 Done Done Sent false 6800 Acked Write 9 Done Done Sent false 6801 Acked Write 9 Done Done Sent false 6802 Acked Read 9 Done Done Sent false 6803 Acked Write 4 Done Done Sent false 6804 Acked Write 10 Done Done Sent false 6805 Acked Read 1 Done Done Sent false 6806 Acked Read 2 Done Done Sent false 6807 Acked Write 10 Done Done Sent false 6808 Acked Read 7 Done Done Sent false 6809 Acked Read 10 Done Done Sent false 6810 Acked Read 10 Done Done Sent false 6811 Acked Read 8 Done Done Sent false 6812 Acked Flush 0 Done Done Sent false 6813 Acked Read 10 Done Done Sent false 6814 Acked Read 2 Done Done Sent false 6815 Acked Write 10 Done Done Sent false 6816 Acked Read 6 Done Done Sent false 6817 Acked Read 1 Done Done Sent false 6818 Acked Read 3 Done Done Sent false 6819 Acked Read 8 Done Done Sent false 6820 Acked Read 6 Done Done Sent false 6821 Acked Read 5 Done Done Sent false 6822 Acked Write 2 Done Done Sent false 6823 Acked Read 2 Done Done Sent false 6824 Acked Read 8 Done Done Sent false 6825 Acked Write 7 Done Done Sent false 6826 Acked Read 2 Done Done Sent false 6827 Acked Read 5 Done Done Sent false 6828 Acked Flush 0 Done Done Sent false 6829 Acked Write 4 Done Done Sent false 6830 Acked Read 4 Done Done Sent false 6831 Acked Read 10 Done Done Sent false 6832 Acked Read 1 Done Done Sent false 6833 Acked Read 7 Done Done Sent false 6834 Acked Read 7 Done Done Sent false 6835 Acked Read 2 Done Done Sent false 6836 Acked Flush 0 Done Done Sent false 6837 Acked Write 4 Done Done Sent false 6838 Acked Write 1 Done Done Sent false 6839 Acked Read 5 Done Done Sent false 6840 Acked Write 1 Done Done Sent false 6841 Acked Read 1 Done Done Sent false 6842 Acked Read 10 Done Done Sent false 6843 Acked Write 6 Done Done Sent false 6844 Acked Read 6 Done Done Sent false 6845 Acked Read 2 Done Done Sent false 6846 Acked Read 6 Done Done Sent false 6847 Acked Flush 0 Done Done Sent false 6848 Acked Write 4 Done Done Sent false 6849 Acked Write 4 Done Done Sent false 6850 Acked Read 10 Done Done Sent false 6851 Acked Flush 0 Done Done Sent false 6852 Acked Write 1 Done Done Sent false 6853 Acked Read 2 Done Done Sent false 6854 Acked Flush 0 Done Done Sent false 6855 Acked Flush 0 Done Done Sent false 6856 Acked Write 3 Done Done Sent false 6857 Acked Read 1 Done Done Sent false 6858 Acked Read 3 Done Done Sent false 6859 Acked Write 8 Done Done Sent false 6860 Acked Write 8 Done Done Sent false 6861 Acked Read 6 Done Done Sent false 6862 Acked Write 2 Done Done Sent false 6863 Acked Flush 0 Done Done Sent false 6864 Acked Read 8 Done Done Sent false 6865 Acked Write 10 Done Done Sent false 6866 Acked Write 4 Done Done Sent false 6867 Acked Write 7 Done Done Sent false 6868 Acked Flush 0 Done Done Sent false 6869 Acked Flush 0 Done Done Sent false 6870 Acked Flush 0 Done Done Sent false 6871 Acked Write 7 Done Done Sent false 6872 Acked Read 1 Done Done Sent false 6873 Acked Flush 0 Done Done Sent false 6874 Acked Read 10 Done Done Sent false 6875 Acked Read 3 Done Done Sent false 6876 Acked Write 5 Done Done Sent false 6877 Acked Read 7 Done Done Sent false 6878 Acked Flush 0 Done Done Sent false 6879 Acked Flush 0 Done Done Sent false 6880 Acked Write 1 Done Done Sent false 6881 Acked Read 3 Done Done Sent false 6882 Acked Write 4 Done Done Sent false 6883 Acked Flush 0 Done Done Sent false 6884 Acked Read 1 Done Done Sent false 6885 Acked Write 4 Done Done Sent false 6886 Acked Read 4 Done Done Sent false 6887 Acked Read 3 Done Done Sent false 6888 Acked Read 4 Done Done Sent false 6889 Acked Read 8 Done Done Sent false 6890 Acked Read 1 Done Done Sent false 6891 Acked Write 9 Done Done Sent false 6892 Acked Flush 0 Done Done Sent false 6893 Acked Write 5 Done Done Sent false 6894 Acked Read 1 Done Done Sent false 6895 Acked Write 8 Done Done Sent false 6896 Acked Flush 0 Done Done Sent false 6897 Acked Read 7 Done Done Sent false 6898 Acked Read 7 Done Done Sent false 6899 Acked Write 9 Done Done Sent false 6900 Acked Read 6 Done Done Sent false 6901 Acked Write 8 Done Done Sent false 6902 Acked Write 8 Done Done Sent false 6903 Acked Write 3 Done Done Sent false 6904 Acked Flush 0 Done Done Sent false 6905 Acked Read 5 Done Done Sent false 6906 Acked Read 6 Done Done Sent false 6907 Acked Write 3 Done Done Sent false 6908 Acked Read 2 Done Done Sent false 6909 Acked Read 6 Done Done Sent false 6910 Acked Flush 0 Done Done Sent false 6911 Acked Read 2 Done Done Sent false 6912 Acked Read 7 Done Done Sent false 6913 Acked Write 8 Done Done Sent false 6914 Acked Read 8 Done Done Sent false 6915 Acked Read 1 Done Done Sent false 6916 Acked Read 4 Done Done Sent false 6917 Acked Read 7 Done Done Sent false 6918 Acked Flush 0 Done Done Sent false 6919 Acked Read 3 Done Done Sent false 6920 Acked Read 4 Done Done Sent false 6921 Acked Write 10 Done Done Sent false 6922 Acked Flush 0 Done Done Sent false 6923 Acked Read 10 Done Done Sent false 6924 Acked Flush 0 Done Done Sent false 6925 Acked Write 7 Done Done Sent false 6926 Acked Write 4 Done Done Sent false 6927 Acked Read 8 Done Done Sent false 6928 Acked Flush 0 Done Done Sent false 6929 Acked Write 3 Done Done Sent false 6930 Acked Write 10 Done Done Sent false 6931 Acked Read 10 Done Done Sent false 6932 Acked Read 3 Done Done Sent false 6933 Acked Write 9 Done Done Sent false 6934 Acked Read 2 Done Done Sent false 6935 Acked Write 7 Done Done Sent false 6936 Acked Read 6 Done Done Sent false 6937 Acked Read 8 Done Done Sent false 6938 Acked Write 10 Done Done Sent false 6939 Acked Read 9 Done Done Sent false 6940 Acked Write 5 Done Done Sent false 6941 Acked Write 9 Done Done Sent false 6942 Acked Read 10 Done Done Sent false 6943 Acked Write 7 Done Done Sent false 6944 Acked Write 10 Done Done Sent false 6945 Acked Write 1 Done Done Sent false 6946 Acked Write 10 Done Done Sent false 6947 Acked Write 2 Done Done Sent false 6948 Acked Read 5 Done Done Sent false 6949 Acked Write 5 Done Done Sent false 6950 Acked Read 9 Done Done Sent false 6951 Acked Read 2 Done Done Sent false 6952 Acked Write 7 Done Done Sent false 6953 Acked Write 8 Done Done Sent false 6954 Acked Write 2 Done Done Sent false 6955 Acked Write 8 Done Done Sent false 6956 Acked Write 1 Done Done Sent false 6957 Acked Read 7 Done Done Sent false 6958 Acked Write 4 Done Done Sent false 6959 Acked Read 3 Done Done Sent false 6960 Acked Read 10 Done Done Sent false 6961 Acked Write 2 Done Done Sent false 6962 Acked Write 5 Done Done Sent false 6963 Acked Read 8 Done Done Sent false 6964 Acked Read 10 Done Done Sent false 6965 Acked Read 2 Done Done Sent false 6966 Acked Read 9 Done Done Sent false 6967 Acked Write 3 Done Done Sent false 6968 Acked Write 3 Done Done Sent false 6969 Acked Write 6 Done Done Sent false 6970 Acked Write 9 Done Done Sent false 6971 Acked Write 5 Done Done Sent false 6972 Acked Write 9 Done Done Sent false 6973 Acked Read 9 Done Done Sent false 6974 Acked Read 8 Done Done Sent false 6975 Acked Write 3 Done Done Sent false 6976 Acked Write 4 Done Done Sent false 6977 Acked Read 6 Done Done Sent false 6978 Acked Flush 0 Done Done Sent false 6979 Acked Write 1 Done Done Sent false 6980 Acked Write 7 Done Done Sent false 6981 Acked Write 5 Done Done Sent false 6982 Acked Write 4 Done Done Sent false 6983 Acked Read 7 Done Done Sent false 6984 Acked Write 3 Done Done Sent false 6985 Acked Write 2 Done Done Sent false 6986 Acked Read 9 Done Done Sent false 6987 Acked Read 6 Done Done Sent false 6988 Acked Read 6 Done Done Sent false 6989 Acked Read 5 Done Done Sent false 6990 Acked Write 6 Done Done Sent false 6991 Acked Read 10 Done Done Sent false 6992 Acked Flush 0 Done Done Sent false 6993 Acked Read 1 Done Done Sent false 6994 Acked Read 4 Done Done Sent false 6995 Acked Read 1 Done Done Sent false 6996 Acked Write 6 Done Done Sent false 6997 Acked Write 8 Done Done Sent false 6998 Acked Read 3 Done Done Sent false 6999 Acked Flush 0 Done Done Sent false 7000 Acked Read 3 Done Done Sent false 7001 Acked Flush 0 Done Done Sent false 7002 Acked Flush 0 Done Done Sent false 7003 Acked Flush 0 Done Done Sent false 7004 Acked Read 4 Done Done Sent false 7005 Acked Read 5 Done Done Sent false 7006 Acked Write 10 Done Done Sent false 7007 Acked Write 6 Done Done Sent false 7008 Acked Read 7 Done Done Sent false 7009 Acked Write 8 Done Done Sent false 7010 Acked Read 4 Done Done Sent false 7011 Acked Read 4 Done Done Sent false 7012 Acked Write 1 Done Done Sent false 7013 Acked Read 8 Done Done Sent false 7014 Acked Flush 0 Done Done Sent false 7015 Acked Write 3 Done Done Sent false 7016 Acked Flush 0 Done Done Sent false 7017 Acked Flush 0 Done Done Sent false 7018 Acked Read 5 Done Done Sent false 7019 Acked Read 6 Done Done Sent false 7020 Acked Read 3 Done Done Sent false 7021 Acked Write 4 Done Done Sent false 7022 Acked Write 4 Done Done Sent false 7023 Acked Read 6 Done Done Sent false 7024 Acked Flush 0 Done Done Sent false 7025 Acked Flush 0 Done Done Sent false 7026 Acked Write 9 Done Done Sent false 7027 Acked Write 3 Done Done Sent false 7028 Acked Read 6 Done Done Sent false 7029 Acked Read 3 Done Done Sent false 7030 Acked Write 8 Done Done Sent false 7031 Acked Write 6 Done Done Sent false 7032 Acked Write 7 Done Done Sent false 7033 Acked Read 4 Done Done Sent false 7034 Acked Read 2 Done Done Sent false 7035 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7035 7035 6733 Downstairs last five completed: 6733 6732 6731 6730 6729 Upstairs last five completed: 7035 7034 7033 7032 7031 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6734 Acked Write 9 Done Done Sent false 6735 Acked Read 7 Done Done Sent false 6736 Acked Flush 0 Done Done Sent false 6737 Acked Write 6 Done Done Sent false 6738 Acked Read 7 Done Done Sent false 6739 Acked Write 3 Done Done Sent false 6740 Acked Read 10 Done Done Sent false 6741 Acked Write 2 Done Done Sent false 6742 Acked Read 4 Done Done Sent false 6743 Acked Flush 0 Done Done Sent false 6744 Acked Write 8 Done Done Sent false 6745 Acked Write 9 Done Done Sent false 6746 Acked Write 5 Done Done Sent false 6747 Acked Write 7 Done Done Sent false 6748 Acked Flush 0 Done Done Sent false 6749 Acked Read 8 Done Done Sent false 6750 Acked Read 8 Done Done Sent false 6751 Acked Write 5 Done Done Sent false 6752 Acked Flush 0 Done Done Sent false 6753 Acked Write 8 Done Done Sent false 6754 Acked Read 6 Done Done Sent false 6755 Acked Read 6 Done Done Sent false 6756 Acked Read 7 Done Done Sent false 6757 Acked Read 3 Done Done Sent false 6758 Acked Read 9 Done Done Sent false 6759 Acked Read 2 Done Done Sent false 6760 Acked Flush 0 Done Done Sent false 6761 Acked Flush 0 Done Done Sent false 6762 Acked Read 10 Done Done Sent false 6763 Acked Flush 0 Done Done Sent false 6764 Acked Write 10 Done Done Sent false 6765 Acked Write 8 Done Done Sent false 6766 Acked Write 10 Done Done Sent false 6767 Acked Write 4 Done Done Sent false 6768 Acked Write 4 Done Done Sent false 6769 Acked Read 1 Done Done Sent false 6770 Acked Write 7 Done Done Sent false 6771 Acked Read 9 Done Done Sent false 6772 Acked Flush 0 Done Done Sent false 6773 Acked Write 7 Done Done Sent false 6774 Acked Write 8 Done Done Sent false 6775 Acked Read 8 Done Done Sent false 6776 Acked Read 2 Done Done Sent false 6777 Acked Read 7 Done Done Sent false 6778 Acked Read 4 Done Done Sent false 6779 Acked Read 7 Done Done Sent false 6780 Acked Write 7 Done Done Sent false 6781 Acked Read 10 Done Done Sent false 6782 Acked Write 2 Done Done Sent false 6783 Acked Write 6 Done Done Sent false 6784 Acked Write 1 Done Done Sent false 6785 Acked Read 1 Done Done Sent false 6786 Acked Write 7 Done Done Sent false 6787 Acked Write 1 Done Done Sent false 6788 Acked Read 5 Done Done Sent false 6789 Acked Write 4 Done Done Sent false 6790 Acked Read 2 Done Done Sent false 6791 Acked Write 4 Done Done Sent false 6792 Acked Read 9 Done Done Sent false 6793 Acked Read 6 Done Done Sent false 6794 Acked Write 6 Done Done Sent false 6795 Acked Read 7 Done Done Sent false 6796 Acked Write 6 Done Done Sent false 6797 Acked Write 9 Done Done Sent false 6798 Acked Write 1 Done Done Sent false 6799 Acked Read 1 Done Done Sent false 6800 Acked Write 9 Done Done Sent false 6801 Acked Write 9 Done Done Sent false 6802 Acked Read 9 Done Done Sent false 6803 Acked Write 4 Done Done Sent false 6804 Acked Write 10 Done Done Sent false 6805 Acked Read 1 Done Done Sent false 6806 Acked Read 2 Done Done Sent false 6807 Acked Write 10 Done Done Sent false 6808 Acked Read 7 Done Done Sent false 6809 Acked Read 10 Done Done Sent false 6810 Acked Read 10 Done Done Sent false 6811 Acked Read 8 Done Done Sent false 6812 Acked Flush 0 Done Done Sent false 6813 Acked Read 10 Done Done Sent false 6814 Acked Read 2 Done Done Sent false 6815 Acked Write 10 Done Done Sent false 6816 Acked Read 6 Done Done Sent false 6817 Acked Read 1 Done Done Sent false 6818 Acked Read 3 Done Done Sent false 6819 Acked Read 8 Done Done Sent false 6820 Acked Read 6 Done Done Sent false 6821 Acked Read 5 Done Done Sent false 6822 Acked Write 2 Done Done Sent false 6823 Acked Read 2 Done Done Sent false 6824 Acked Read 8 Done Done Sent false 6825 Acked Write 7 Done Done Sent false 6826 Acked Read 2 Done Done Sent false 6827 Acked Read 5 Done Done Sent false 6828 Acked Flush 0 Done Done Sent false 6829 Acked Write 4 Done Done Sent false 6830 Acked Read 4 Done Done Sent false 6831 Acked Read 10 Done Done Sent false 6832 Acked Read 1 Done Done Sent false 6833 Acked Read 7 Done Done Sent false 6834 Acked Read 7 Done Done Sent false 6835 Acked Read 2 Done Done Sent false 6836 Acked Flush 0 Done Done Sent false 6837 Acked Write 4 Done Done Sent false 6838 Acked Write 1 Done Done Sent false 6839 Acked Read 5 Done Done Sent false 6840 Acked Write 1 Done Done Sent false 6841 Acked Read 1 Done Done Sent false 6842 Acked Read 10 Done Done Sent false 6843 Acked Write 6 Done Done Sent false 6844 Acked Read 6 Done Done Sent false 6845 Acked Read 2 Done Done Sent false 6846 Acked Read 6 Done Done Sent false 6847 Acked Flush 0 Done Done Sent false 6848 Acked Write 4 Done Done Sent false 6849 Acked Write 4 Done Done Sent false 6850 Acked Read 10 Done Done Sent false 6851 Acked Flush 0 Done Done Sent false 6852 Acked Write 1 Done Done Sent false 6853 Acked Read 2 Done Done Sent false 6854 Acked Flush 0 Done Done Sent false 6855 Acked Flush 0 Done Done Sent false 6856 Acked Write 3 Done Done Sent false 6857 Acked Read 1 Done Done Sent false 6858 Acked Read 3 Done Done Sent false 6859 Acked Write 8 Done Done Sent false 6860 Acked Write 8 Done Done Sent false 6861 Acked Read 6 Done Done Sent false 6862 Acked Write 2 Done Done Sent false 6863 Acked Flush 0 Done Done Sent false 6864 Acked Read 8 Done Done Sent false 6865 Acked Write 10 Done Done Sent false 6866 Acked Write 4 Done Done Sent false 6867 Acked Write 7 Done Done Sent false 6868 Acked Flush 0 Done Done Sent false 6869 Acked Flush 0 Done Done Sent false 6870 Acked Flush 0 Done Done Sent false 6871 Acked Write 7 Done Done Sent false 6872 Acked Read 1 Done Done Sent false 6873 Acked Flush 0 Done Done Sent false 6874 Acked Read 10 Done Done Sent false 6875 Acked Read 3 Done Done Sent false 6876 Acked Write 5 Done Done Sent false 6877 Acked Read 7 Done Done Sent false 6878 Acked Flush 0 Done Done Sent false 6879 Acked Flush 0 Done Done Sent false 6880 Acked Write 1 Done Done Sent false 6881 Acked Read 3 Done Done Sent false 6882 Acked Write 4 Done Done Sent false 6883 Acked Flush 0 Done Done Sent false 6884 Acked Read 1 Done Done Sent false 6885 Acked Write 4 Done Done Sent false 6886 Acked Read 4 Done Done Sent false 6887 Acked Read 3 Done Done Sent false 6888 Acked Read 4 Done Done Sent false 6889 Acked Read 8 Done Done Sent false 6890 Acked Read 1 Done Done Sent false 6891 Acked Write 9 Done Done Sent false 6892 Acked Flush 0 Done Done Sent false 6893 Acked Write 5 Done Done Sent false 6894 Acked Read 1 Done Done Sent false 6895 Acked Write 8 Done Done Sent false 6896 Acked Flush 0 Done Done Sent false 6897 Acked Read 7 Done Done Sent false 6898 Acked Read 7 Done Done Sent false 6899 Acked Write 9 Done Done Sent false 6900 Acked Read 6 Done Done Sent false 6901 Acked Write 8 Done Done Sent false 6902 Acked Write 8 Done Done Sent false 6903 Acked Write 3 Done Done Sent false 6904 Acked Flush 0 Done Done Sent false 6905 Acked Read 5 Done Done Sent false 6906 Acked Read 6 Done Done Sent false 6907 Acked Write 3 Done Done Sent false 6908 Acked Read 2 Done Done Sent false 6909 Acked Read 6 Done Done Sent false 6910 Acked Flush 0 Done Done Sent false 6911 Acked Read 2 Done Done Sent false 6912 Acked Read 7 Done Done Sent false 6913 Acked Write 8 Done Done Sent false 6914 Acked Read 8 Done Done Sent false 6915 Acked Read 1 Done Done Sent false 6916 Acked Read 4 Done Done Sent false 6917 Acked Read 7 Done Done Sent false 6918 Acked Flush 0 Done Done Sent false 6919 Acked Read 3 Done Done Sent false 6920 Acked Read 4 Done Done Sent false 6921 Acked Write 10 Done Done Sent false 6922 Acked Flush 0 Done Done Sent false 6923 Acked Read 10 Done Done Sent false 6924 Acked Flush 0 Done Done Sent false 6925 Acked Write 7 Done Done Sent false 6926 Acked Write 4 Done Done Sent false 6927 Acked Read 8 Done Done Sent false 6928 Acked Flush 0 Done Done Sent false 6929 Acked Write 3 Done Done Sent false 6930 Acked Write 10 Done Done Sent false 6931 Acked Read 10 Done Done Sent false 6932 Acked Read 3 Done Done Sent false 6933 Acked Write 9 Done Done Sent false 6934 Acked Read 2 Done Done Sent false 6935 Acked Write 7 Done Done Sent false 6936 Acked Read 6 Done Done Sent false 6937 Acked Read 8 Done Done Sent false 6938 Acked Write 10 Done Done Sent false 6939 Acked Read 9 Done Done Sent false 6940 Acked Write 5 Done Done Sent false 6941 Acked Write 9 Done Done Sent false 6942 Acked Read 10 Done Done Sent false 6943 Acked Write 7 Done Done Sent false 6944 Acked Write 10 Done Done Sent false 6945 Acked Write 1 Done Done Sent false 6946 Acked Write 10 Done Done Sent false 6947 Acked Write 2 Done Done Sent false 6948 Acked Read 5 Done Done Sent false 6949 Acked Write 5 Done Done Sent false 6950 Acked Read 9 Done Done Sent false 6951 Acked Read 2 Done Done Sent false 6952 Acked Write 7 Done Done Sent false 6953 Acked Write 8 Done Done Sent false 6954 Acked Write 2 Done Done Sent false 6955 Acked Write 8 Done Done Sent false 6956 Acked Write 1 Done Done Sent false 6957 Acked Read 7 Done Done Sent false 6958 Acked Write 4 Done Done Sent false 6959 Acked Read 3 Done Done Sent false 6960 Acked Read 10 Done Done Sent false 6961 Acked Write 2 Done Done Sent false 6962 Acked Write 5 Done Done Sent false 6963 Acked Read 8 Done Done Sent false 6964 Acked Read 10 Done Done Sent false 6965 Acked Read 2 Done Done Sent false 6966 Acked Read 9 Done Done Sent false 6967 Acked Write 3 Done Done Sent false 6968 Acked Write 3 Done Done Sent false 6969 Acked Write 6 Done Done Sent false 6970 Acked Write 9 Done Done Sent false 6971 Acked Write 5 Done Done Sent false 6972 Acked Write 9 Done Done Sent false 6973 Acked Read 9 Done Done Sent false 6974 Acked Read 8 Done Done Sent false 6975 Acked Write 3 Done Done Sent false 6976 Acked Write 4 Done Done Sent false 6977 Acked Read 6 Done Done Sent false 6978 Acked Flush 0 Done Done Sent false 6979 Acked Write 1 Done Done Sent false 6980 Acked Write 7 Done Done Sent false 6981 Acked Write 5 Done Done Sent false 6982 Acked Write 4 Done Done Sent false 6983 Acked Read 7 Done Done Sent false 6984 Acked Write 3 Done Done Sent false 6985 Acked Write 2 Done Done Sent false 6986 Acked Read 9 Done Done Sent false 6987 Acked Read 6 Done Done Sent false 6988 Acked Read 6 Done Done Sent false 6989 Acked Read 5 Done Done Sent false 6990 Acked Write 6 Done Done Sent false 6991 Acked Read 10 Done Done Sent false 6992 Acked Flush 0 Done Done Sent false 6993 Acked Read 1 Done Done Sent false 6994 Acked Read 4 Done Done Sent false 6995 Acked Read 1 Done Done Sent false 6996 Acked Write 6 Done Done Sent false 6997 Acked Write 8 Done Done Sent false 6998 Acked Read 3 Done Done Sent false 6999 Acked Flush 0 Done Done Sent false 7000 Acked Read 3 Done Done Sent false 7001 Acked Flush 0 Done Done Sent false 7002 Acked Flush 0 Done Done Sent false 7003 Acked Flush 0 Done Done Sent false 7004 Acked Read 4 Done Done Sent false 7005 Acked Read 5 Done Done Sent false 7006 Acked Write 10 Done Done Sent false 7007 Acked Write 6 Done Done Sent false 7008 Acked Read 7 Done Done Sent false 7009 Acked Write 8 Done Done Sent false 7010 Acked Read 4 Done Done Sent false 7011 Acked Read 4 Done Done Sent false 7012 Acked Write 1 Done Done Sent false 7013 Acked Read 8 Done Done Sent false 7014 Acked Flush 0 Done Done Sent false 7015 Acked Write 3 Done Done Sent false 7016 Acked Flush 0 Done Done Sent false 7017 Acked Flush 0 Done Done Sent false 7018 Acked Read 5 Done Done Sent false 7019 Acked Read 6 Done Done Sent false 7020 Acked Read 3 Done Done Sent false 7021 Acked Write 4 Done Done Sent false 7022 Acked Write 4 Done Done Sent false 7023 Acked Read 6 Done Done Sent false 7024 Acked Flush 0 Done Done Sent false 7025 Acked Flush 0 Done Done Sent false 7026 Acked Write 9 Done Done Sent false 7027 Acked Write 3 Done Done Sent false 7028 Acked Read 6 Done Done Sent false 7029 Acked Read 3 Done Done Sent false 7030 Acked Write 8 Done Done Sent false 7031 Acked Write 6 Done Done Sent false 7032 Acked Write 7 Done Done Sent false 7033 Acked Read 4 Done Done Sent false 7034 Acked Read 2 Done Done Sent false 7035 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7035 7035 6733 Downstairs last five completed: 6733 6732 6731 6730 6729 Upstairs last five completed: 7035 7034 7033 7032 7031 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.298314126Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:55385)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.29837635Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:56.299243387Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.29975469Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.299777054Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 6733","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.29979253Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 6733","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.300006482Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.300031216Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.300046042Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 6733","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.300060108Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 6733","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.300278499Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7035 7034 7033 7032 7031 Upstairs last five completed: 7035 7034 7033 7032 7031 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:46:56.534 INFO [8830] Got stop action so:false kr:false request completed, latency_us: 327, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2261a19c-bc24-4394-bb23-98892b2a615c, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:46:56.535 INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9977609e-ceac-4f74-8f63-91d2566903fa, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:56.535298476Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:56.535350052Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:46:56.535368778Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.535392801Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.535412706Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:46:56.535481107Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:47:00.537 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/2, method: GET, req_id: ec294cba-51ce-4029-8480-fbbfebd75d2a, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 625 len 20480 data: 3 4 4 4 4 002/300 Read block 2877 len 16384 003/300 Read block 2597 len 36864 004/300 Write block 1529 len 16384 data: 5 5 2 3 005/300 Read block 1928 len 36864 006/300 Read block 1248 len 20480 007/300 Read block 2985 len 28672 008/300 Write block 967 len 40960 data: 4 3 3 3 3 3 3 3 3 6 009/300 Read block 2740 len 36864 010/300 Read block 2884 len 24576 011/300 Read block 3049 len 32768 012/300 Read block 2673 len 32768 013/300 Write block 2830 len 16384 data: 4 4 5 3 014/300 Read block 2601 len 32768 015/300 Write block 3129 len 12288 data: 6 6 6 016/300 Write block 1486 len 16384 data: 5 5 4 4 017/300 Read block 1314 len 12288 018/300 Write block 3408 len 32768 data: 6 6 4 4 3 2 2 2 019/300 Write block 4743 len 28672 data: 6 6 6 6 6 6 6 020/300 Flush 021/300 Read block 2363 len 8192 022/300 Write block 4188 len 36864 data: 4 4 4 4 4 4 5 5 6 023/300 Read block 2855 len 4096 024/300 Read block 2523 len 8192 025/300 Flush 026/300 Read block 4138 len 4096 027/300 Read block 1447 len 32768 028/300 Write block 4447 len 32768 data: 3 3 3 3 3 3 2 2 029/300 Read block 4941 len 20480 030/300 Write block 3988 len 12288 data: 4 4 7 031/300 Read block 2201 len 40960 032/300 Read block 3919 len 32768 033/300 Read block 1188 len 36864 034/300 Write block 3382 len 40960 data: 6 7 7 6 5 5 3 4 4 5 035/300 Read block 3861 len 36864 036/300 Write block 1117 len 12288 data: 7 7 6 037/300 Read block 2152 len 16384 038/300 Write block 4493 len 20480 data: 2 2 2 2 3 039/300 Write block 2517 len 12288 data: 4 5 5 040/300 Flush 041/300 Read block 3657 len 28672 042/300 Read block 1850 len 36864 043/300 Read block 3670 len 32768 044/300 Read block 2111 len 4096 045/300 Read block 739 len 4096 046/300 Read block 3673 len 12288 047/300 Read block 2586 len 20480 048/300 Read block 3197 len 12288 049/300 Read block 288 len 8192 050/300 Read block 2473 len 4096 051/300 Flush 052/300 Flush 053/300 Read block 3008 len 4096 054/300 Flush 055/300 Read block 1815 len 20480 056/300 Read block 574 len 4096 057/300 Read block 3647 len 20480 058/300 Read block 2839 len 4096 059/300 Read block 3961 len 16384 060/300 Write block 3536 len 24576 data: 5 7 7 7 8 8 061/300 Write block 2445 len 20480 data: 5 5 4 4 4 062/300 Read block 1794 len 36864 063/300 Flush 064/300 Write block 1803 len 28672 data: 6 5 5 4 5 4 4 065/300 Read block 4712 len 32768 066/300 Read block 3309 len 40960 067/300 Write block 4362 len 40960 data: 6 6 6 4 4 4 4 4 3 3 068/300 Read block 2391 len 28672 069/300 Read block 550 len 36864 070/300 Read block 1554 len 16384 071/300 Write block 4451 len 28672 data: 4 4 3 3 2 2 2 072/300 Read block 2110 len 40960 073/300 Write block 38 len 40960 data: 3 2 2 2 2 2 2 2 2 3 074/300 Write block 4598 len 12288 data: 3 4 4 075/300 Read block 37 len 20480 076/300 Read block 1368 len 24576 077/300 Read block 2840 len 36864 078/300 Read block 2125 len 24576 079/300 Read block 3383 len 12288 080/300 Write block 3228 len 20480 data: 4 4 5 6 5 081/300 Write block 784 len 40960 data: 4 4 3 3 3 4 4 4 3 3 082/300 Write block 1662 len 8192 data: 3 4 083/300 Read block 188 len 32768 084/300 Read block 1515 len 36864 085/300 Write block 3211 len 36864 data: 8 7 8 7 5 4 4 4 5 086/300 Read block 1569 len 28672 087/300 Write block 4762 len 4096 data: 5 088/300 Write block 3318 len 28672 data: 6 7 6 6 3 3 3 089/300 Write block 4758 len 40960 data: 6 5 5 5 6 6 3 3 3 3 090/300 Read block 1300 len 4096 091/300 Read block 3737 len 12288 092/300 Flush 093/300 Read block 3009 len 40960 094/300 Read block 4085 len 24576 095/300 Flush 096/300 Read block 3750 len 20480 097/300 Read block 1996 len 16384 098/300 Write block 2479 len 12288 data: 6 6 5 099/300 Flush 100/300 Flush 101/300 Write block 2405 len 12288 data: 6 6 5 102/300 Read block 1142 len 24576 103/300 Read block 1827 len 8192 104/300 Read block 2022 len 16384 105/300 Read block 370 len 28672 106/300 Write block 928 len 20480 data: 2 2 2 4 4 107/300 Read block 1245 len 24576 108/300 Write block 1742 len 36864 data: 4 4 3 2 3 3 4 4 6 109/300 Read block 3711 len 12288 110/300 Read block 3235 len 16384 111/300 Write block 3950 len 16384 data: 5 5 5 6 112/300 Write block 3753 len 12288 data: 7 7 6 113/300 Flush 114/300 Read block 1839 len 36864 115/300 Read block 2338 len 40960 116/300 Write block 270 len 16384 data: 3 2 3 3 117/300 Write block 4264 len 16384 data: 4 5 4 4 118/300 Write block 3338 len 20480 data: 3 3 4 3 3 119/300 Write block 799 len 36864 data: 4 4 4 4 4 5 4 5 5 120/300 Read block 667 len 8192 121/300 Flush 122/300 Write block 4206 len 4096 data: 4 123/300 Read block 31 len 36864 124/300 Flush 125/300 Write block 3944 len 36864 data: 5 4 5 5 6 7 6 6 6 126/300 Read block 1752 len 24576 127/300 Write block 1414 len 40960 data: 5 6 6 5 4 5 5 5 5 6 128/300 Read block 853 len 8192 129/300 Read block 4701 len 36864 130/300 Flush 131/300 Read block 2687 len 12288 132/300 Read block 1337 len 16384 133/300 Read block 4957 len 4096 134/300 Flush 135/300 Read block 3388 len 36864 136/300 Read block 13 len 40960 137/300 Read block 3815 len 16384 138/300 Read block 324 len 24576 139/300 Write block 2395 len 12288 data: 6 5 5 140/300 Write block 2359 len 8192 data: 4 4 141/300 Read block 3001 len 24576 142/300 Flush 143/300 Write block 2490 len 32768 data: 5 5 5 6 4 4 3 3 144/300 Read block 3169 len 24576 145/300 Flush 146/300 Read block 1944 len 16384 147/300 Flush 148/300 Read block 1919 len 36864 149/300 Read block 3865 len 8192 150/300 Write block 2871 len 40960 data: 4 2 2 3 3 4 4 4 3 3 151/300 Write block 3437 len 36864 data: 3 3 4 4 3 3 3 3 3 152/300 Read block 2154 len 40960 153/300 Read block 4081 len 40960 154/300 Flush 155/300 Read block 2147 len 20480 156/300 Write block 3330 len 28672 data: 7 7 8 7 5 4 2 157/300 Read block 3988 len 36864 158/300 Write block 1052 len 40960 data: 3 3 3 3 3 3 3 3 3 4 159/300 Write block 601 len 36864 data: 3 2 2 2 3 3 3 3 3 160/300 Write block 3349 len 32768 data: 4 4 4 4 4 4 5 5 161/300 Write block 111 len 20480 data: 6 6 5 5 5 162/300 Read block 3786 len 8192 163/300 Write block 2618 len 28672 data: 3 3 4 5 5 4 4 164/300 Read block 1384 len 16384 165/300 Write block 571 len 8192 data: 5 5 166/300 Write block 975 len 4096 data: 4 167/300 Read block 4451 len 24576 168/300 Flush 169/300 Read block 2895 len 20480 170/300 Read block 675 len 24576 171/300 Read block 4315 len 28672 172/300 Read block 2272 len 40960 173/300 Read block 4507 len 24576 174/300 Write block 4388 len 4096 data: 8 175/300 Write block 3215 len 12288 data: 6 5 5 176/300 Write block 861 len 16384 data: 5 6 6 7 177/300 Write block 3443 len 16384 data: 4 4 4 5 178/300 Read block 1480 len 24576 179/300 Read block 923 len 8192 180/300 Read block 2652 len 16384 181/300 Read block 2873 len 12288 182/300 Read block 994 len 4096 183/300 Write block 1662 len 24576 data: 4 5 4 3 4 3 184/300 Read block 2384 len 32768 185/300 Read block 896 len 28672 186/300 Write block 4788 len 36864 data: 3 3 4 5 5 6 7 8 9 187/300 Write block 824 len 40960 data: 7 9 11 10 10 11 10 10 9 7 188/300 Write block 1427 len 36864 data: 5 4 4 4 3 3 3 4 4 189/300 Read block 3071 len 28672 190/300 Read block 604 len 20480 191/300 Read block 3203 len 8192 192/300 Read block 3885 len 24576 193/300 Read block 2268 len 8192 194/300 Read block 3871 len 12288 195/300 Flush 196/300 Write block 3754 len 36864 data: 8 7 7 5 6 7 7 6 5 197/300 Read block 3916 len 36864 198/300 Write block 4420 len 16384 data: 6 6 7 6 199/300 Write block 935 len 20480 data: 4 3 5 5 6 200/300 Write block 2500 len 40960 data: 4 5 7 6 7 8 8 7 6 6 201/300 Write block 4621 len 12288 data: 5 5 5 202/300 Write block 1687 len 20480 data: 6 7 8 9 5 203/300 Read block 3664 len 8192 204/300 Write block 4963 len 28672 data: 6 6 5 3 4 6 5 205/300 Read block 1550 len 28672 206/300 Write block 3136 len 20480 data: 6 5 5 4 4 207/300 Read block 2424 len 40960 208/300 Read block 3012 len 16384 209/300 Write block 3913 len 28672 data: 2 2 2 3 4 4 3 210/300 Flush 211/300 Read block 2835 len 16384 212/300 Read block 3979 len 8192 213/300 Read block 1163 len 36864 214/300 Read block 581 len 24576 215/300 Write block 430 len 12288 data: 3 2 3 216/300 Write block 1608 len 20480 data: 4 5 4 5 6 217/300 Write block 94 len 20480 data: 3 5 4 4 4 218/300 Write block 4064 len 32768 data: 4 4 5 6 5 6 7 7 219/300 Read block 2394 len 16384 220/300 Read block 622 len 32768 221/300 Write block 4709 len 32768 data: 6 6 7 6 5 5 5 5 222/300 Read block 2895 len 20480 223/300 Read block 1890 len 40960 224/300 Write block 3103 len 36864 data: 8 8 8 8 7 6 6 5 5 225/300 Read block 531 len 12288 226/300 Write block 4527 len 4096 data: 4 227/300 Read block 4712 len 40960 228/300 Write block 307 len 20480 data: 3 3 4 4 5 229/300 Read block 3846 len 4096 230/300 Write block 3215 len 16384 data: 7 6 6 5 231/300 Read block 1626 len 12288 232/300 Flush 233/300 Write block 785 len 8192 data: 5 4 234/300 Write block 2874 len 24576 data: 4 4 5 5 5 4 235/300 Read block 626 len 20480 236/300 Write block 4374 len 4096 data: 3 237/300 Write block 332 len 20480 data: 2 5 5 5 4 238/300 Write block 4946 len 12288 data: 4 4 3 239/300 Read block 3111 len 24576 240/300 Read block 1430 len 20480 241/300 Flush 242/300 Flush 243/300 Read block 3098 len 32768 244/300 Write block 4082 len 36864 data: 5 4 4 4 3 3 3 4 6 245/300 Read block 4920 len 4096 246/300 Read block 777 len 40960 247/300 Read block 1230 len 16384 248/300 Read block 1874 len 32768 249/300 Read block 41 len 16384 250/300 Write block 3664 len 16384 data: 3 3 2 5 251/300 Write block 4902 len 32768 data: 4 4 3 3 3 3 3 3 252/300 Read block 1074 len 36864 253/300 Write block 1225 len 36864 data: 6 6 4 4 6 6 4 5 7 254/300 Write block 4080 len 36864 data: 5 4 6 5 5 5 4 4 4 255/300 Read block 2138 len 36864 256/300 Write block 4793 len 12288 data: 7 8 9 257/300 Read block 2124 len 16384 258/300 Flush 259/300 Read block 2278 len 28672 260/300 Read block 3354 len 8192 261/300 Read block 0 len 12288 262/300 Write block 4984 len 20480 data: 7 7 5 5 6 263/300 Write block 1535 len 28672 data: 3 4 5 6 7 10 9 264/300 Flush 265/300 Write block 1816 len 12288 data: 8 8 7 266/300 Write block 1513 len 40960 data: 3 4 3 2 3 3 4 3 2 3 267/300 Write block 1041 len 40960 data: 6 5 4 4 5 5 4 4 4 2 268/300 Read block 336 len 40960 269/300 Read block 3780 len 40960 270/300 Flush 271/300 Write block 2050 len 28672 data: 3 4 5 5 5 5 7 272/300 Flush 273/300 Flush 274/300 Write block 3501 len 16384 data: 3 2 2 2 275/300 Read block 1267 len 4096 276/300 Write block 4655 len 32768 data: 6 6 7 7 7 6 4 6 277/300 Flush 278/300 Write block 2027 len 16384 data: 5 5 5 4 279/300 Read block 2416 len 4096 280/300 Read block 2072 len 20480 281/300 Write block 2441 len 24576 data: 6 6 6 5 6 6 282/300 Flush 283/300 Write block 4357 len 4096 data: 7 284/300 Write block 3979 len 32768 data: 3 2 2 3 3 4 4 5 285/300 Write block 3993 len 36864 data: 5 5 5 3 3 3 3 4 4 286/300 Read block 1489 len 20480 287/300 Read block 975 len 32768 288/300 Write block 2511 len 40960 data: 4 3 4 3 3 3 5 6 6 5 289/300 Flush 290/300 Read block 4779 len 12288 291/300 Read block 1580 len 20480 292/300 Write block 961 len 4096 data: 5 293/300 Read block 2016 len 40960 294/300 Flush 295/300 Flush 296/300 Write block 651 len 24576 data: 4 4 4 4 5 4 297/300 Write block 3856 len 4096 data: 3 298/300 Read block 2798 len 16384 299/300 Read block 1056 len 8192 300/300 Read block 4467 len 40960 301/300 Read block 87 len 36864 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:47:00.775 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/2, method: POST, req_id: 81195cda-8b6f-4421-80e2-214fb9a5ad9f, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7036 Acked Write 5 Done Done Sent false 7037 Acked Read 4 Done Done Sent false 7038 Acked Read 9 Done Done Sent false 7039 Acked Write 4 Done Done Sent false 7040 Acked Read 9 Done Done Sent false 7041 Acked Read 5 Done Done Sent false 7042 Acked Read 7 Done Done Sent false 7043 Acked Write 10 Done Done Sent false 7044 Acked Read 9 Done Done Sent false 7045 Acked Read 6 Done Done Sent false 7046 Acked Read 8 Done Done Sent false 7047 Acked Read 8 Done Done Sent false 7048 Acked Write 4 Done Done Sent false 7049 Acked Read 8 Done Done Sent false 7050 Acked Write 3 Done Done Sent false 7051 Acked Write 4 Done Done Sent false 7052 Acked Read 3 Done Done Sent false 7053 Acked Write 8 Done Done Sent false 7054 Acked Write 7 Done Done Sent false 7055 Acked Flush 0 Done Done Sent false 7056 Acked Read 2 Done Done Sent false 7057 Acked Write 9 Done Done Sent false 7058 Acked Read 1 Done Done Sent false 7059 Acked Read 2 Done Done Sent false 7060 Acked Flush 0 Done Done Sent false 7061 Acked Read 1 Done Done Sent false 7062 Acked Read 8 Done Done Sent false 7063 Acked Write 8 Done Done Sent false 7064 Acked Read 5 Done Done Sent false 7065 Acked Write 3 Done Done Sent false 7066 Acked Read 10 Done Done Sent false 7067 Acked Read 8 Done Done Sent false 7068 Acked Read 9 Done Done Sent false 7069 Acked Write 10 Done Done Sent false 7070 Acked Read 9 Done Done Sent false 7071 Acked Write 3 Done Done Sent false 7072 Acked Read 4 Done Done Sent false 7073 Acked Write 5 Done Done Sent false 7074 Acked Write 3 Done Done Sent false 7075 Acked Flush 0 Done Done Sent false 7076 Acked Read 7 Done Done Sent false 7077 Acked Read 9 Done Done Sent false 7078 Acked Read 8 Done Done Sent false 7079 Acked Read 1 Done Done Sent false 7080 Acked Read 1 Done Done Sent false 7081 Acked Read 3 Done Done Sent false 7082 Acked Read 5 Done Done Sent false 7083 Acked Read 3 Done Done Sent false 7084 Acked Read 2 Done Done Sent false 7085 Acked Read 1 Done Done Sent false 7086 Acked Flush 0 Done Done Sent false 7087 Acked Flush 0 Done Done Sent false 7088 Acked Read 1 Done Done Sent false 7089 Acked Flush 0 Done Done Sent false 7090 Acked Read 5 Done Done Sent false 7091 Acked Read 1 Done Done Sent false 7092 Acked Read 5 Done Done Sent false 7093 Acked Read 1 Done Done Sent false 7094 Acked Read 4 Done Done Sent false 7095 Acked Write 6 Done Done Sent false 7096 Acked Write 5 Done Done Sent false 7097 Acked Read 9 Done Done Sent false 7098 Acked Flush 0 Done Done Sent false 7099 Acked Write 7 Done Done Sent false 7100 Acked Read 8 Done Done Sent false 7101 Acked Read 10 Done Done Sent false 7102 Acked Write 10 Done Done Sent false 7103 Acked Read 7 Done Done Sent false 7104 Acked Read 9 Done Done Sent false 7105 Acked Read 4 Done Done Sent false 7106 Acked Write 7 Done Done Sent false 7107 Acked Read 10 Done Done Sent false 7108 Acked Write 10 Done Done Sent false 7109 Acked Write 3 Done Done Sent false 7110 Acked Read 5 Done Done Sent false 7111 Acked Read 6 Done Done Sent false 7112 Acked Read 9 Done Done Sent false 7113 Acked Read 6 Done Done Sent false 7114 Acked Read 3 Done Done Sent false 7115 Acked Write 5 Done Done Sent false 7116 Acked Write 10 Done Done Sent false 7117 Acked Write 2 Done Done Sent false 7118 Acked Read 8 Done Done Sent false 7119 Acked Read 9 Done Done Sent false 7120 Acked Write 9 Done Done Sent false 7121 Acked Read 7 Done Done Sent false 7122 Acked Write 1 Done Done Sent false 7123 Acked Write 7 Done Done Sent false 7124 Acked Write 10 Done Done Sent false 7125 Acked Read 1 Done Done Sent false 7126 Acked Read 3 Done Done Sent false 7127 Acked Flush 0 Done Done Sent false 7128 Acked Read 10 Done Done Sent false 7129 Acked Read 6 Done Done Sent false 7130 Acked Flush 0 Done Done Sent false 7131 Acked Read 5 Done Done Sent false 7132 Acked Read 4 Done Done Sent false 7133 Acked Write 3 Done Done Sent false 7134 Acked Flush 0 Done Done Sent false 7135 Acked Flush 0 Done Done Sent false 7136 Acked Write 3 Done Done Sent false 7137 Acked Read 6 Done Done Sent false 7138 Acked Read 2 Done Done Sent false 7139 Acked Read 4 Done Done Sent false 7140 Acked Read 7 Done Done Sent false 7141 Acked Write 5 Done Done Sent false 7142 Acked Read 6 Done Done Sent false 7143 Acked Write 9 Done Done Sent false 7144 Acked Read 3 Done Done Sent false 7145 Acked Read 4 Done Done Sent false 7146 Acked Write 4 Done Done Sent false 7147 Acked Write 3 Done Done Sent false 7148 Acked Flush 0 Done Done Sent false 7149 Acked Read 9 Done Done Sent false 7150 Acked Read 10 Done Done Sent false 7151 Acked Write 4 Done Done Sent false 7152 Acked Write 4 Done Done Sent false 7153 Acked Write 5 Done Done Sent false 7154 Acked Write 9 Done Done Sent false 7155 Acked Read 2 Done Done Sent false 7156 Acked Flush 0 Done Done Sent false 7157 Acked Write 1 Done Done Sent false 7158 Acked Read 9 Done Done Sent false 7159 Acked Flush 0 Done Done Sent false 7160 Acked Write 9 Done Done Sent false 7161 Acked Read 6 Done Done Sent false 7162 Acked Write 10 Done Done Sent false 7163 Acked Read 2 Done Done Sent false 7164 Acked Read 9 Done Done Sent false 7165 Acked Flush 0 Done Done Sent false 7166 Acked Read 3 Done Done Sent false 7167 Acked Read 4 Done Done Sent false 7168 Acked Read 1 Done Done Sent false 7169 Acked Flush 0 Done Done Sent false 7170 Acked Read 9 Done Done Sent false 7171 Acked Read 10 Done Done Sent false 7172 Acked Read 4 Done Done Sent false 7173 Acked Read 6 Done Done Sent false 7174 Acked Write 3 Done Done Sent false 7175 Acked Write 2 Done Done Sent false 7176 Acked Read 6 Done Done Sent false 7177 Acked Flush 0 Done Done Sent false 7178 Acked Write 8 Done Done Sent false 7179 Acked Read 6 Done Done Sent false 7180 Acked Flush 0 Done Done Sent false 7181 Acked Read 4 Done Done Sent false 7182 Acked Flush 0 Done Done Sent false 7183 Acked Read 9 Done Done Sent false 7184 Acked Read 2 Done Done Sent false 7185 Acked Write 10 Done Done Sent false 7186 Acked Write 9 Done Done Sent false 7187 Acked Read 10 Done Done Sent false 7188 Acked Read 10 Done Done Sent false 7189 Acked Flush 0 Done Done Sent false 7190 Acked Read 5 Done Done Sent false 7191 Acked Write 7 Done Done Sent false 7192 Acked Read 9 Done Done Sent false 7193 Acked Write 10 Done Done Sent false 7194 Acked Write 9 Done Done Sent false 7195 Acked Write 8 Done Done Sent false 7196 Acked Write 5 Done Done Sent false 7197 Acked Read 2 Done Done Sent false 7198 Acked Write 7 Done Done Sent false 7199 Acked Read 4 Done Done Sent false 7200 Acked Write 2 Done Done Sent false 7201 Acked Write 1 Done Done Sent false 7202 Acked Read 6 Done Done Sent false 7203 Acked Flush 0 Done Done Sent false 7204 Acked Read 5 Done Done Sent false 7205 Acked Read 6 Done Done Sent false 7206 Acked Read 7 Done Done Sent false 7207 Acked Read 10 Done Done Sent false 7208 Acked Read 6 Done Done Sent false 7209 Acked Write 1 Done Done Sent false 7210 Acked Write 3 Done Done Sent false 7211 Acked Write 4 Done Done Sent false 7212 Acked Write 4 Done Done Sent false 7213 Acked Read 6 Done Done Sent false 7214 Acked Read 2 Done Done Sent false 7215 Acked Read 4 Done Done Sent false 7216 Acked Read 3 Done Done Sent false 7217 Acked Read 1 Done Done Sent false 7218 Acked Write 6 Done Done Sent false 7219 Acked Read 8 Done Done Sent false 7220 Acked Read 7 Done Done Sent false 7221 Acked Write 9 Done Done Sent false 7222 Acked Write 10 Done Done Sent false 7223 Acked Write 9 Done Done Sent false 7224 Acked Read 7 Done Done Sent false 7225 Acked Read 5 Done Done Sent false 7226 Acked Read 2 Done Done Sent false 7227 Acked Read 6 Done Done Sent false 7228 Acked Read 2 Done Done Sent false 7229 Acked Read 3 Done Done Sent false 7230 Acked Flush 0 Done Done Sent false 7231 Acked Write 9 Done Done Sent false 7232 Acked Read 9 Done Done Sent false 7233 Acked Write 4 Done Done Sent false 7234 Acked Write 5 Done Done Sent false 7235 Acked Write 10 Done Done Sent false 7236 Acked Write 3 Done Done Sent false 7237 Acked Write 5 Done Done Sent false 7238 Acked Read 2 Done Done Sent false 7239 Acked Write 7 Done Done Sent false 7240 Acked Read 7 Done Done Sent false 7241 Acked Write 5 Done Done Sent false 7242 Acked Read 10 Done Done Sent false 7243 Acked Read 4 Done Done Sent false 7244 Acked Write 7 Done Done Sent false 7245 Acked Flush 0 Done Done Sent false 7246 Acked Read 4 Done Done Sent false 7247 Acked Read 2 Done Done Sent false 7248 Acked Read 9 Done Done Sent false 7249 Acked Read 6 Done Done Sent false 7250 Acked Write 3 Done Done Sent false 7251 Acked Write 5 Done Done Sent false 7252 Acked Write 5 Done Done Sent false 7253 Acked Write 8 Done Done Sent false 7254 Acked Read 4 Done Done Sent false 7255 Acked Read 8 Done Done Sent false 7256 Acked Write 8 Done Done Sent false 7257 Acked Read 5 Done Done Sent false 7258 Acked Read 10 Done Done Sent false 7259 Acked Write 9 Done Done Sent false 7260 Acked Read 3 Done Done Sent false 7261 Acked Write 1 Done Done Sent false 7262 Acked Read 10 Done Done Sent false 7263 Acked Write 5 Done Done Sent false 7264 Acked Read 1 Done Done Sent false 7265 Acked Write 4 Done Done Sent false 7266 Acked Read 3 Done Done Sent false 7267 Acked Flush 0 Done Done Sent false 7268 Acked Write 2 Done Done Sent false 7269 Acked Write 6 Done Done Sent false 7270 Acked Read 5 Done Done Sent false 7271 Acked Write 1 Done Done Sent false 7272 Acked Write 5 Done Done Sent false 7273 Acked Write 3 Done Done Sent false 7274 Acked Read 6 Done Done Sent false 7275 Acked Read 5 Done Done Sent false 7276 Acked Flush 0 Done Done Sent false 7277 Acked Flush 0 Done Done Sent false 7278 Acked Read 8 Done Done Sent false 7279 Acked Write 9 Done Done Sent false 7280 Acked Read 1 Done Done Sent false 7281 Acked Read 10 Done Done Sent false 7282 Acked Read 4 Done Done Sent false 7283 Acked Read 8 Done Done Sent false 7284 Acked Read 4 Done Done Sent false 7285 Acked Write 4 Done Done Sent false 7286 Acked Write 8 Done Done Sent false 7287 Acked Read 9 Done Done Sent false 7288 Acked Write 9 Done Done Sent false 7289 Acked Write 9 Done Done Sent false 7290 Acked Read 9 Done Done Sent false 7291 Acked Write 3 Done Done Sent false 7292 Acked Read 4 Done Done Sent false 7293 Acked Flush 0 Done Done Sent false 7294 Acked Read 7 Done Done Sent false 7295 Acked Read 2 Done Done Sent false 7296 Acked Read 3 Done Done Sent false 7297 Acked Write 5 Done Done Sent false 7298 Acked Write 7 Done Done Sent false 7299 Acked Flush 0 Done Done Sent false 7300 Acked Write 3 Done Done Sent false 7301 Acked Write 10 Done Done Sent false 7302 Acked Write 10 Done Done Sent false 7303 Acked Read 10 Done Done Sent false 7304 Acked Read 10 Done Done Sent false 7305 Acked Flush 0 Done Done Sent false 7306 Acked Write 7 Done Done Sent false 7307 Acked Flush 0 Done Done Sent false 7308 Acked Flush 0 Done Done Sent false 7309 Acked Write 4 Done Done Sent false 7310 Acked Read 1 Done Done Sent false 7311 Acked Write 8 Done Done Sent false 7312 Acked Flush 0 Done Done Sent false 7313 Acked Write 4 Done Done Sent false 7314 Acked Read 1 Done Done Sent false 7315 Acked Read 5 Done Done Sent false 7316 Acked Write 6 Done Done Sent false 7317 Acked Flush 0 Done Done Sent false 7318 Acked Write 1 Done Done Sent false 7319 Acked Write 8 Done Done Sent false 7320 Acked Write 9 Done Done Sent false 7321 Acked Read 5 Done Done Sent false 7322 Acked Read 8 Done Done Sent false 7323 Acked Write 10 Done Done Sent false 7324 Acked Flush 0 Done Done Sent false 7325 Acked Read 3 Done Done Sent false 7326 Acked Read 5 Done Done Sent false 7327 Acked Write 1 Done Done Sent false 7328 Acked Read 10 Done Done Sent false 7329 Acked Flush 0 Done Done Sent false 7330 Acked Flush 0 Done Done Sent false 7331 Acked Write 6 Done Done Sent false 7332 Acked Write 1 Done Done Sent false 7333 Acked Read 4 Done Done Sent false 7334 Acked Read 2 Done Done Sent false 7335 Acked Read 10 Done Done Sent false 7336 Acked Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7330 7330 7035 Downstairs last five completed: 7035 7034 7033 7032 7031 Upstairs last five completed: 7336 7335 7334 7333 7332 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7036 Acked Write 5 Done Done Sent false 7037 Acked Read 4 Done Done Sent false 7038 Acked Read 9 Done Done Sent false 7039 Acked Write 4 Done Done Sent false 7040 Acked Read 9 Done Done Sent false 7041 Acked Read 5 Done Done Sent false 7042 Acked Read 7 Done Done Sent false 7043 Acked Write 10 Done Done Sent false 7044 Acked Read 9 Done Done Sent false 7045 Acked Read 6 Done Done Sent false 7046 Acked Read 8 Done Done Sent false 7047 Acked Read 8 Done Done Sent false 7048 Acked Write 4 Done Done Sent false 7049 Acked Read 8 Done Done Sent false 7050 Acked Write 3 Done Done Sent false 7051 Acked Write 4 Done Done Sent false 7052 Acked Read 3 Done Done Sent false 7053 Acked Write 8 Done Done Sent false 7054 Acked Write 7 Done Done Sent false 7055 Acked Flush 0 Done Done Sent false 7056 Acked Read 2 Done Done Sent false 7057 Acked Write 9 Done Done Sent false 7058 Acked Read 1 Done Done Sent false 7059 Acked Read 2 Done Done Sent false 7060 Acked Flush 0 Done Done Sent false 7061 Acked Read 1 Done Done Sent false 7062 Acked Read 8 Done Done Sent false 7063 Acked Write 8 Done Done Sent false 7064 Acked Read 5 Done Done Sent false 7065 Acked Write 3 Done Done Sent false 7066 Acked Read 10 Done Done Sent false 7067 Acked Read 8 Done Done Sent false 7068 Acked Read 9 Done Done Sent false 7069 Acked Write 10 Done Done Sent false 7070 Acked Read 9 Done Done Sent false 7071 Acked Write 3 Done Done Sent false 7072 Acked Read 4 Done Done Sent false 7073 Acked Write 5 Done Done Sent false 7074 Acked Write 3 Done Done Sent false 7075 Acked Flush 0 Done Done Sent false 7076 Acked Read 7 Done Done Sent false 7077 Acked Read 9 Done Done Sent false 7078 Acked Read 8 Done Done Sent false 7079 Acked Read 1 Done Done Sent false 7080 Acked Read 1 Done Done Sent false 7081 Acked Read 3 Done Done Sent false 7082 Acked Read 5 Done Done Sent false 7083 Acked Read 3 Done Done Sent false 7084 Acked Read 2 Done Done Sent false 7085 Acked Read 1 Done Done Sent false 7086 Acked Flush 0 Done Done Sent false 7087 Acked Flush 0 Done Done Sent false 7088 Acked Read 1 Done Done Sent false 7089 Acked Flush 0 Done Done Sent false 7090 Acked Read 5 Done Done Sent false 7091 Acked Read 1 Done Done Sent false 7092 Acked Read 5 Done Done Sent false 7093 Acked Read 1 Done Done Sent false 7094 Acked Read 4 Done Done Sent false 7095 Acked Write 6 Done Done Sent false 7096 Acked Write 5 Done Done Sent false 7097 Acked Read 9 Done Done Sent false 7098 Acked Flush 0 Done Done Sent false 7099 Acked Write 7 Done Done Sent false 7100 Acked Read 8 Done Done Sent false 7101 Acked Read 10 Done Done Sent false 7102 Acked Write 10 Done Done Sent false 7103 Acked Read 7 Done Done Sent false 7104 Acked Read 9 Done Done Sent false 7105 Acked Read 4 Done Done Sent false 7106 Acked Write 7 Done Done Sent false 7107 Acked Read 10 Done Done Sent false 7108 Acked Write 10 Done Done Sent false 7109 Acked Write 3 Done Done Sent false 7110 Acked Read 5 Done Done Sent false 7111 Acked Read 6 Done Done Sent false 7112 Acked Read 9 Done Done Sent false 7113 Acked Read 6 Done Done Sent false 7114 Acked Read 3 Done Done Sent false 7115 Acked Write 5 Done Done Sent false 7116 Acked Write 10 Done Done Sent false 7117 Acked Write 2 Done Done Sent false 7118 Acked Read 8 Done Done Sent false 7119 Acked Read 9 Done Done Sent false 7120 Acked Write 9 Done Done Sent false 7121 Acked Read 7 Done Done Sent false 7122 Acked Write 1 Done Done Sent false 7123 Acked Write 7 Done Done Sent false 7124 Acked Write 10 Done Done Sent false 7125 Acked Read 1 Done Done Sent false 7126 Acked Read 3 Done Done Sent false 7127 Acked Flush 0 Done Done Sent false 7128 Acked Read 10 Done Done Sent false 7129 Acked Read 6 Done Done Sent false 7130 Acked Flush 0 Done Done Sent false 7131 Acked Read 5 Done Done Sent false 7132 Acked Read 4 Done Done Sent false 7133 Acked Write 3 Done Done Sent false 7134 Acked Flush 0 Done Done Sent false 7135 Acked Flush 0 Done Done Sent false 7136 Acked Write 3 Done Done Sent false 7137 Acked Read 6 Done Done Sent false 7138 Acked Read 2 Done Done Sent false 7139 Acked Read 4 Done Done Sent false 7140 Acked Read 7 Done Done Sent false 7141 Acked Write 5 Done Done Sent false 7142 Acked Read 6 Done Done Sent false 7143 Acked Write 9 Done Done Sent false 7144 Acked Read 3 Done Done Sent false 7145 Acked Read 4 Done Done Sent false 7146 Acked Write 4 Done Done Sent false 7147 Acked Write 3 Done Done Sent false 7148 Acked Flush 0 Done Done Sent false 7149 Acked Read 9 Done Done Sent false 7150 Acked Read 10 Done Done Sent false 7151 Acked Write 4 Done Done Sent false 7152 Acked Write 4 Done Done Sent false 7153 Acked Write 5 Done Done Sent false 7154 Acked Write 9 Done Done Sent false 7155 Acked Read 2 Done Done Sent false 7156 Acked Flush 0 Done Done Sent false 7157 Acked Write 1 Done Done Sent false 7158 Acked Read 9 Done Done Sent false 7159 Acked Flush 0 Done Done Sent false 7160 Acked Write 9 Done Done Sent false 7161 Acked Read 6 Done Done Sent false 7162 Acked Write 10 Done Done Sent false 7163 Acked Read 2 Done Done Sent false 7164 Acked Read 9 Done Done Sent false 7165 Acked Flush 0 Done Done Sent false 7166 Acked Read 3 Done Done Sent false 7167 Acked Read 4 Done Done Sent false 7168 Acked Read 1 Done Done Sent false 7169 Acked Flush 0 Done Done Sent false 7170 Acked Read 9 Done Done Sent false 7171 Acked Read 10 Done Done Sent false 7172 Acked Read 4 Done Done Sent false 7173 Acked Read 6 Done Done Sent false 7174 Acked Write 3 Done Done Sent false 7175 Acked Write 2 Done Done Sent false 7176 Acked Read 6 Done Done Sent false 7177 Acked Flush 0 Done Done Sent false 7178 Acked Write 8 Done Done Sent false 7179 Acked Read 6 Done Done Sent false 7180 Acked Flush 0 Done Done Sent false 7181 Acked Read 4 Done Done Sent false 7182 Acked Flush 0 Done Done Sent false 7183 Acked Read 9 Done Done Sent false 7184 Acked Read 2 Done Done Sent false 7185 Acked Write 10 Done Done Sent false 7186 Acked Write 9 Done Done Sent false 7187 Acked Read 10 Done Done Sent false 7188 Acked Read 10 Done Done Sent false 7189 Acked Flush 0 Done Done Sent false 7190 Acked Read 5 Done Done Sent false 7191 Acked Write 7 Done Done Sent false 7192 Acked Read 9 Done Done Sent false 7193 Acked Write 10 Done Done Sent false 7194 Acked Write 9 Done Done Sent false 7195 Acked Write 8 Done Done Sent false 7196 Acked Write 5 Done Done Sent false 7197 Acked Read 2 Done Done Sent false 7198 Acked Write 7 Done Done Sent false 7199 Acked Read 4 Done Done Sent false 7200 Acked Write 2 Done Done Sent false 7201 Acked Write 1 Done Done Sent false 7202 Acked Read 6 Done Done Sent false 7203 Acked Flush 0 Done Done Sent false 7204 Acked Read 5 Done Done Sent false 7205 Acked Read 6 Done Done Sent false 7206 Acked Read 7 Done Done Sent false 7207 Acked Read 10 Done Done Sent false 7208 Acked Read 6 Done Done Sent false 7209 Acked Write 1 Done Done Sent false 7210 Acked Write 3 Done Done Sent false 7211 Acked Write 4 Done Done Sent false 7212 Acked Write 4 Done Done Sent false 7213 Acked Read 6 Done Done Sent false 7214 Acked Read 2 Done Done Sent false 7215 Acked Read 4 Done Done Sent false 7216 Acked Read 3 Done Done Sent false 7217 Acked Read 1 Done Done Sent false 7218 Acked Write 6 Done Done Sent false 7219 Acked Read 8 Done Done Sent false 7220 Acked Read 7 Done Done Sent false 7221 Acked Write 9 Done Done Sent false 7222 Acked Write 10 Done Done Sent false 7223 Acked Write 9 Done Done Sent false 7224 Acked Read 7 Done Done Sent false 7225 Acked Read 5 Done Done Sent false 7226 Acked Read 2 Done Done Sent false 7227 Acked Read 6 Done Done Sent false 7228 Acked Read 2 Done Done Sent false 7229 Acked Read 3 Done Done Sent false 7230 Acked Flush 0 Done Done Sent false 7231 Acked Write 9 Done Done Sent false 7232 Acked Read 9 Done Done Sent false 7233 Acked Write 4 Done Done Sent false 7234 Acked Write 5 Done Done Sent false 7235 Acked Write 10 Done Done Sent false 7236 Acked Write 3 Done Done Sent false 7237 Acked Write 5 Done Done Sent false 7238 Acked Read 2 Done Done Sent false 7239 Acked Write 7 Done Done Sent false 7240 Acked Read 7 Done Done Sent false 7241 Acked Write 5 Done Done Sent false 7242 Acked Read 10 Done Done Sent false 7243 Acked Read 4 Done Done Sent false 7244 Acked Write 7 Done Done Sent false 7245 Acked Flush 0 Done Done Sent false 7246 Acked Read 4 Done Done Sent false 7247 Acked Read 2 Done Done Sent false 7248 Acked Read 9 Done Done Sent false 7249 Acked Read 6 Done Done Sent false 7250 Acked Write 3 Done Done Sent false 7251 Acked Write 5 Done Done Sent false 7252 Acked Write 5 Done Done Sent false 7253 Acked Write 8 Done Done Sent false 7254 Acked Read 4 Done Done Sent false 7255 Acked Read 8 Done Done Sent false 7256 Acked Write 8 Done Done Sent false 7257 Acked Read 5 Done Done Sent false 7258 Acked Read 10 Done Done Sent false 7259 Acked Write 9 Done Done Sent false 7260 Acked Read 3 Done Done Sent false 7261 Acked Write 1 Done Done Sent false 7262 Acked Read 10 Done Done Sent false 7263 Acked Write 5 Done Done Sent false 7264 Acked Read 1 Done Done Sent false 7265 Acked Write 4 Done Done Sent false 7266 Acked Read 3 Done Done Sent false 7267 Acked Flush 0 Done Done Sent false 7268 Acked Write 2 Done Done Sent false 7269 Acked Write 6 Done Done Sent false 7270 Acked Read 5 Done Done Sent false 7271 Acked Write 1 Done Done Sent false 7272 Acked Write 5 Done Done Sent false 7273 Acked Write 3 Done Done Sent false 7274 Acked Read 6 Done Done Sent false 7275 Acked Read 5 Done Done Sent false 7276 Acked Flush 0 Done Done Sent false 7277 Acked Flush 0 Done Done Sent false 7278 Acked Read 8 Done Done Sent false 7279 Acked Write 9 Done Done Sent false 7280 Acked Read 1 Done Done Sent false 7281 Acked Read 10 Done Done Sent false 7282 Acked Read 4 Done Done Sent false 7283 Acked Read 8 Done Done Sent false 7284 Acked Read 4 Done Done Sent false 7285 Acked Write 4 Done Done Sent false 7286 Acked Write 8 Done Done Sent false 7287 Acked Read 9 Done Done Sent false 7288 Acked Write 9 Done Done Sent false 7289 Acked Write 9 Done Done Sent false 7290 Acked Read 9 Done Done Sent false 7291 Acked Write 3 Done Done Sent false 7292 Acked Read 4 Done Done Sent false 7293 Acked Flush 0 Done Done Sent false 7294 Acked Read 7 Done Done Sent false 7295 Acked Read 2 Done Done Sent false 7296 Acked Read 3 Done Done Sent false 7297 Acked Write 5 Done Done Sent false 7298 Acked Write 7 Done Done Sent false 7299 Acked Flush 0 Done Done Sent false 7300 Acked Write 3 Done Done Sent false 7301 Acked Write 10 Done Done Sent false 7302 Acked Write 10 Done Done Sent false 7303 Acked Read 10 Done Done Sent false 7304 Acked Read 10 Done Done Sent false 7305 Acked Flush 0 Done Done Sent false 7306 Acked Write 7 Done Done Sent false 7307 Acked Flush 0 Done Done Sent false 7308 Acked Flush 0 Done Done Sent false 7309 Acked Write 4 Done Done Sent false 7310 Acked Read 1 Done Done Sent false 7311 Acked Write 8 Done Done Sent false 7312 Acked Flush 0 Done Done Sent false 7313 Acked Write 4 Done Done Sent false 7314 Acked Read 1 Done Done Sent false 7315 Acked Read 5 Done Done Sent false 7316 Acked Write 6 Done Done Sent false 7317 Acked Flush 0 Done Done Sent false 7318 Acked Write 1 Done Done Sent false 7319 Acked Write 8 Done Done Sent false 7320 Acked Write 9 Done Done Sent false 7321 Acked Read 5 Done Done Sent false 7322 Acked Read 8 Done Done Sent false 7323 Acked Write 10 Done Done Sent false 7324 Acked Flush 0 Done Done Sent false 7325 Acked Read 3 Done Done Sent false 7326 Acked Read 5 Done Done Sent false 7327 Acked Write 1 Done Done Sent false 7328 Acked Read 10 Done Done Sent false 7329 Acked Flush 0 Done Done Sent false 7330 Acked Flush 0 Done Done Sent false 7331 Acked Write 6 Done Done Sent false 7332 Acked Write 1 Done Done Sent false 7333 Acked Read 4 Done Done Sent false 7334 Acked Read 2 Done Done Sent false 7335 Acked Read 10 Done Done Sent false 7336 Acked Read 9 Done Done Sent false 7337 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7337 7337 7035 Downstairs last five completed: 7035 7034 7033 7032 7031 Upstairs last five completed: 7337 7336 7335 7334 7333 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.536594308Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:64460)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.536678746Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:06.537539775Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538120409Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538140573Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 7035","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.53815449Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 7035","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538409001Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538427906Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538441852Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 7035","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538455539Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 7035","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:06.538683967Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7337 7336 7335 7334 7333 Upstairs last five completed: 7337 7336 7335 7334 7333 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:47:08.780[8830] Got stop action so:false kr:false INFO request completed, latency_us: 335, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 313cdab7-a7e0-45f2-a833-3f73935ab317, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:08.780841732Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:08.780913103Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:47:08.780 INFO request completed, latency_us: 146, response_code: 200, uri: /state/cid/2, method: GET, req_id: 83044fca-b37e-42e1-8212-8387e36d16bb, remote_addr: 127.0.0.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:08.780984264Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 1:51788, local_addr: 127.0.0.1:9998 {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:08.781068201Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:08.78110983Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:08.781142651Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:47:12.783 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1cab2b25-a6a9-4be7-bfd5-418f14223cb6, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 3743 len 40960 002/300 Read block 920 len 4096 003/300 Read block 1316 len 12288 004/300 Write block 1466 len 8192 data: 4 7 005/300 Write block 4193 len 4096 data: 5 006/300 Flush 007/300 Write block 1583 len 4096 data: 4 008/300 Read block 3516 len 36864 009/300 Write block 1476 len 20480 data: 4 4 2 3 2 010/300 Read block 4627 len 16384 011/300 Read block 3121 len 32768 012/300 Read block 2731 len 24576 013/300 Read block 1229 len 12288 014/300 Read block 3927 len 36864 015/300 Read block 584 len 16384 016/300 Read block 1507 len 16384 017/300 Read block 2342 len 8192 018/300 Write block 476 len 4096 data: 5 019/300 Read block 3916 len 4096 020/300 Read block 2823 len 28672 021/300 Write block 820 len 4096 data: 3 022/300 Read block 4693 len 4096 023/300 Write block 4026 len 4096 data: 3 024/300 Write block 4664 len 40960 data: 7 6 6 5 5 5 6 5 4 5 025/300 Read block 4118 len 24576 026/300 Write block 2198 len 4096 data: 4 027/300 Read block 3195 len 16384 028/300 Write block 3529 len 20480 data: 4 4 4 3 2 029/300 Write block 882 len 20480 data: 2 3 3 4 6 030/300 Write block 1242 len 4096 data: 3 031/300 Read block 2481 len 24576 032/300 Write block 2030 len 24576 data: 5 4 4 4 6 5 033/300 Write block 2211 len 36864 data: 3 3 3 3 4 3 4 4 4 034/300 Read block 4827 len 40960 035/300 Write block 3568 len 4096 data: 3 036/300 Read block 206 len 36864 037/300 Read block 2388 len 40960 038/300 Read block 3283 len 4096 039/300 Write block 454 len 12288 data: 4 5 3 040/300 Write block 1179 len 28672 data: 8 7 8 7 5 4 4 041/300 Write block 4779 len 20480 data: 3 4 3 3 3 042/300 Write block 2306 len 24576 data: 8 8 7 7 7 7 043/300 Write block 813 len 28672 data: 4 8 7 4 4 3 2 044/300 Write block 751 len 28672 data: 7 9 8 6 6 6 5 045/300 Write block 2991 len 20480 data: 6 6 5 5 2 046/300 Read block 4518 len 28672 047/300 Read block 27 len 40960 048/300 Write block 98 len 20480 data: 5 5 6 6 7 049/300 Write block 368 len 20480 data: 5 5 7 6 6 050/300 Read block 3449 len 36864 051/300 Write block 4865 len 8192 data: 4 4 052/300 Read block 4595 len 32768 053/300 Read block 2683 len 4096 054/300 Write block 660 len 16384 data: 4 4 4 4 055/300 Read block 1427 len 36864 056/300 Write block 2280 len 24576 data: 4 4 4 5 5 7 057/300 Read block 1385 len 40960 058/300 Write block 1744 len 28672 data: 4 3 4 4 5 5 7 059/300 Read block 1074 len 20480 060/300 Write block 1914 len 24576 data: 7 6 4 3 2 2 061/300 Write block 1303 len 16384 data: 4 3 2 2 062/300 Read block 784 len 36864 063/300 Flush 064/300 Read block 4237 len 32768 065/300 Write block 1540 len 28672 data: 11 10 8 9 7 6 3 066/300 Write block 3265 len 36864 data: 2 2 2 3 4 4 4 4 3 067/300 Read block 1518 len 28672 068/300 Read block 4955 len 20480 069/300 Flush 070/300 Read block 732 len 20480 071/300 Write block 887 len 28672 data: 6 5 6 7 6 6 6 072/300 Write block 4684 len 12288 data: 7 6 7 073/300 Write block 4887 len 32768 data: 5 5 6 5 4 4 3 3 074/300 Read block 3066 len 32768 075/300 Read block 3700 len 36864 076/300 Read block 1577 len 24576 077/300 Read block 6 len 8192 078/300 Write block 3702 len 20480 data: 7 6 6 6 5 079/300 Read block 3879 len 4096 080/300 Write block 2254 len 4096 data: 2 081/300 Read block 1879 len 40960 082/300 Read block 1387 len 16384 083/300 Flush 084/300 Write block 1808 len 36864 data: 5 5 4 4 4 4 4 7 9 085/300 Read block 714 len 32768 086/300 Write block 3052 len 16384 data: 3 3 3 3 087/300 Write block 980 len 20480 data: 3 4 4 4 3 088/300 Write block 4931 len 24576 data: 8 8 6 4 4 3 089/300 Read block 2596 len 20480 090/300 Write block 2037 len 4096 data: 4 091/300 Write block 4619 len 28672 data: 4 3 6 6 6 4 4 092/300 Read block 4792 len 32768 093/300 Write block 2658 len 4096 data: 3 094/300 Read block 2079 len 4096 095/300 Read block 3084 len 16384 096/300 Read block 1659 len 8192 097/300 Flush 098/300 Write block 1277 len 36864 data: 4 3 6 5 5 5 5 6 5 099/300 Read block 2079 len 28672 100/300 Write block 1973 len 16384 data: 3 4 4 5 101/300 Read block 637 len 24576 102/300 Read block 3444 len 32768 103/300 Write block 2847 len 12288 data: 5 5 4 104/300 Read block 312 len 40960 105/300 Read block 559 len 28672 106/300 Read block 3126 len 28672 107/300 Flush 108/300 Write block 2930 len 4096 data: 3 109/300 Read block 741 len 12288 110/300 Write block 2504 len 32768 data: 8 9 9 8 7 7 5 5 111/300 Flush 112/300 Write block 121 len 40960 data: 4 3 3 4 4 4 4 5 5 4 113/300 Read block 1220 len 28672 114/300 Write block 2874 len 16384 data: 5 5 6 6 115/300 Read block 2505 len 40960 116/300 Write block 3270 len 4096 data: 5 117/300 Write block 3001 len 16384 data: 5 5 5 4 118/300 Write block 4213 len 16384 data: 6 6 8 7 119/300 Flush 120/300 Write block 570 len 40960 data: 4 6 6 5 5 5 5 6 4 8 121/300 Read block 3690 len 32768 122/300 Flush 123/300 Read block 1657 len 40960 124/300 Read block 1834 len 20480 125/300 Write block 1685 len 12288 data: 5 7 7 126/300 Write block 2688 len 12288 data: 5 4 3 127/300 Write block 518 len 36864 data: 4 3 4 4 4 6 9 10 9 128/300 Read block 2411 len 40960 129/300 Write block 2342 len 20480 data: 3 3 2 6 6 130/300 Write block 3576 len 36864 data: 4 4 4 4 4 4 3 3 4 131/300 Read block 2076 len 4096 132/300 Read block 4592 len 32768 133/300 Read block 4035 len 28672 134/300 Write block 2751 len 32768 data: 2 2 3 3 3 3 3 4 135/300 Write block 2969 len 16384 data: 5 6 6 6 136/300 Read block 1804 len 16384 137/300 Write block 3700 len 8192 data: 8 7 138/300 Write block 3396 len 36864 data: 5 5 4 4 4 3 4 4 4 139/300 Read block 3319 len 32768 140/300 Write block 4898 len 12288 data: 4 5 4 141/300 Read block 3434 len 40960 142/300 Read block 4121 len 16384 143/300 Flush 144/300 Write block 908 len 12288 data: 5 5 4 145/300 Write block 4721 len 24576 data: 4 5 5 4 4 4 146/300 Read block 419 len 8192 147/300 Write block 3850 len 8192 data: 8 8 148/300 Write block 2734 len 12288 data: 9 9 8 149/300 Read block 2643 len 28672 150/300 Read block 4905 len 36864 151/300 Read block 4440 len 40960 152/300 Write block 3219 len 24576 data: 6 5 5 5 5 5 153/300 Write block 4389 len 32768 data: 11 11 8 6 7 5 5 5 154/300 Write block 3008 len 20480 data: 4 4 3 3 4 155/300 Write block 3298 len 24576 data: 2 2 3 4 4 4 156/300 Write block 4499 len 20480 data: 4 5 5 5 5 157/300 Write block 4744 len 24576 data: 7 7 7 7 7 7 158/300 Read block 771 len 20480 159/300 Read block 1849 len 20480 160/300 Read block 1961 len 12288 161/300 Read block 4902 len 12288 162/300 Write block 280 len 32768 data: 4 4 3 3 2 2 2 3 163/300 Write block 3300 len 4096 data: 4 164/300 Write block 4829 len 36864 data: 7 6 4 4 5 4 3 5 5 165/300 Write block 1998 len 8192 data: 9 8 166/300 Read block 4914 len 20480 167/300 Read block 1287 len 24576 168/300 Read block 3022 len 32768 169/300 Write block 4097 len 12288 data: 6 6 4 170/300 Flush 171/300 Read block 3721 len 4096 172/300 Write block 3573 len 28672 data: 3 4 4 5 5 5 5 173/300 Read block 2856 len 4096 174/300 Write block 3723 len 4096 data: 4 175/300 Read block 3184 len 32768 176/300 Read block 667 len 4096 177/300 Read block 2556 len 8192 178/300 Flush 179/300 Read block 2584 len 8192 180/300 Write block 1445 len 8192 data: 3 2 181/300 Read block 2965 len 16384 182/300 Read block 4309 len 32768 183/300 Read block 3974 len 4096 184/300 Write block 874 len 36864 data: 4 3 5 5 5 5 5 5 3 185/300 Read block 3154 len 16384 186/300 Write block 4778 len 16384 data: 3 4 5 4 187/300 Read block 247 len 12288 188/300 Write block 469 len 20480 data: 8 8 6 5 4 189/300 Write block 3654 len 24576 data: 5 6 6 5 7 6 190/300 Flush 191/300 Write block 3872 len 12288 data: 2 3 2 192/300 Read block 1351 len 40960 193/300 Write block 3096 len 28672 data: 3 5 6 6 6 5 8 194/300 Write block 3043 len 8192 data: 3 3 195/300 Flush 196/300 Read block 265 len 12288 197/300 Write block 2258 len 24576 data: 5 6 7 6 6 6 198/300 Write block 3807 len 12288 data: 6 6 7 199/300 Read block 4316 len 4096 200/300 Write block 822 len 40960 data: 5 5 8 10 12 11 11 12 11 11 201/300 Write block 2914 len 24576 data: 3 2 3 5 7 6 202/300 Write block 64 len 16384 data: 5 7 7 5 203/300 Read block 761 len 8192 204/300 Read block 1766 len 32768 205/300 Write block 714 len 32768 data: 5 5 5 4 7 6 5 4 206/300 Read block 989 len 40960 207/300 Write block 1775 len 8192 data: 4 4 208/300 Read block 3506 len 40960 209/300 Write block 1389 len 12288 data: 6 5 5 210/300 Read block 3373 len 20480 211/300 Read block 1000 len 20480 212/300 Write block 1439 len 4096 data: 5 213/300 Read block 1718 len 16384 214/300 Write block 2821 len 40960 data: 4 5 6 5 7 7 5 3 3 5 215/300 Write block 447 len 16384 data: 6 6 5 3 216/300 Read block 2556 len 8192 217/300 Read block 3798 len 28672 218/300 Read block 4309 len 32768 219/300 Write block 2219 len 16384 data: 5 5 5 6 220/300 Read block 3935 len 32768 221/300 Read block 1956 len 12288 222/300 Write block 2051 len 24576 data: 5 6 6 6 6 8 223/300 Read block 2392 len 32768 224/300 Write block 1299 len 12288 data: 7 6 5 225/300 Read block 2406 len 24576 226/300 Flush 227/300 Read block 4212 len 28672 228/300 Flush 229/300 Read block 832 len 12288 230/300 Read block 3247 len 32768 231/300 Write block 243 len 36864 data: 5 4 4 4 4 2 2 2 3 232/300 Read block 2783 len 40960 233/300 Read block 845 len 4096 234/300 Read block 4 len 8192 235/300 Write block 4150 len 12288 data: 5 5 6 236/300 Write block 372 len 8192 data: 7 5 237/300 Write block 1816 len 20480 data: 10 9 8 7 8 238/300 Read block 859 len 40960 239/300 Read block 4171 len 28672 240/300 Write block 2859 len 20480 data: 7 6 6 5 4 241/300 Read block 4310 len 24576 242/300 Write block 2731 len 16384 data: 6 8 8 10 243/300 Read block 3488 len 28672 244/300 Read block 3133 len 20480 245/300 Read block 586 len 4096 246/300 Write block 693 len 12288 data: 3 4 5 247/300 Read block 2996 len 12288 248/300 Read block 628 len 4096 249/300 Read block 2348 len 16384 250/300 Write block 1389 len 36864 data: 7 6 6 5 4 2 2 5 5 251/300 Read block 4529 len 12288 252/300 Read block 3766 len 8192 253/300 Read block 848 len 28672 254/300 Flush 255/300 Write block 3592 len 36864 data: 6 6 6 5 7 7 5 6 6 256/300 Flush 257/300 Read block 4952 len 20480 258/300 Read block 2748 len 28672 259/300 Write block 1246 len 40960 data: 6 7 8 6 6 6 6 8 7 4 260/300 Write block 4826 len 8192 data: 5 7 261/300 Read block 4251 len 24576 262/300 Write block 2717 len 40960 data: 5 5 5 5 5 4 4 3 3 4 263/300 Read block 3985 len 32768 264/300 Write block 2409 len 4096 data: 3 265/300 Read block 2653 len 36864 266/300 Read block 3120 len 32768 267/300 Write block 2246 len 16384 data: 8 9 9 7 268/300 Read block 3913 len 8192 269/300 Read block 855 len 36864 270/300 Read block 25 len 20480 271/300 Write block 1302 len 36864 data: 3 5 4 3 3 3 4 4 4 272/300 Read block 803 len 12288 273/300 Read block 77 len 28672 274/300 Read block 2667 len 32768 275/300 Write block 2571 len 36864 data: 7 7 5 4 4 4 5 4 4 276/300 Read block 4893 len 4096 277/300 Read block 3134 len 28672 278/300 Flush 279/300 Flush 280/300 Read block 2115 len 32768 281/300 Write block 1074 len 12288 data: 4 4 4 282/300 Read block 3336 len 20480 283/300 Write block 1177 len 20480 data: 7 7 9 8 9 284/300 Read block 594 len 8192 285/300 Read block 755 len 20480 286/300 Write block 184 len 40960 data: 7 6 4 6 6 5 5 6 6 5 287/300 Write block 29 len 16384 data: 7 7 9 8 288/300 Write block 2245 len 4096 data: 7 289/300 Read block 783 len 24576 290/300 Read block 1445 len 36864 291/300 Read block 2063 len 36864 292/300 Flush 293/300 Read block 2866 len 40960 294/300 Read block 3103 len 8192 295/300 Write block 3682 len 20480 data: 7 8 7 7 8 296/300 Write block 4167 len 8192 data: 3 3 297/300 Write block 463 len 12288 data: 4 5 6 298/300 Read block 2582 len 40960 299/300 Read block 3962 len 40960 300/300 Read block 1417 len 16384 301/300 Read block 4519 len 20480 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:47:13.015 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0a36f009-7805-4a19-9f23-c5047d0f3213, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7338 Acked Read 10 Done Done Sent false 7339 Acked Read 1 Done Done Sent false 7340 Acked Read 3 Done Done Sent false 7341 Acked Write 2 Done Done Sent false 7342 Acked Write 1 Done Done Sent false 7343 Acked Flush 0 Done Done Sent false 7344 Acked Write 1 Done Done Sent false 7345 Acked Read 9 Done Done Sent false 7346 Acked Write 5 Done Done Sent false 7347 Acked Read 4 Done Done Sent false 7348 Acked Read 8 Done Done Sent false 7349 Acked Read 6 Done Done Sent false 7350 Acked Read 3 Done Done Sent false 7351 Acked Read 9 Done Done Sent false 7352 Acked Read 4 Done Done Sent false 7353 Acked Read 4 Done Done Sent false 7354 Acked Read 2 Done Done Sent false 7355 Acked Write 1 Done Done Sent false 7356 Acked Read 1 Done Done Sent false 7357 Acked Read 7 Done Done Sent false 7358 Acked Write 1 Done Done Sent false 7359 Acked Read 1 Done Done Sent false 7360 Acked Write 1 Done Done Sent false 7361 Acked Write 10 Done Done Sent false 7362 Acked Read 6 Done Done Sent false 7363 Acked Write 1 Done Done Sent false 7364 Acked Read 4 Done Done Sent false 7365 Acked Write 5 Done Done Sent false 7366 Acked Write 5 Done Done Sent false 7367 Acked Write 1 Done Done Sent false 7368 Acked Read 6 Done Done Sent false 7369 Acked Write 6 Done Done Sent false 7370 Acked Write 9 Done Done Sent false 7371 Acked Read 10 Done Done Sent false 7372 Acked Write 1 Done Done Sent false 7373 Acked Read 9 Done Done Sent false 7374 Acked Read 10 Done Done Sent false 7375 Acked Read 1 Done Done Sent false 7376 Acked Write 3 Done Done Sent false 7377 Acked Write 7 Done Done Sent false 7378 Acked Write 5 Done Done Sent false 7379 Acked Write 6 Done Done Sent false 7380 Acked Write 7 Done Done Sent false 7381 Acked Write 7 Done Done Sent false 7382 Acked Write 5 Done Done Sent false 7383 Acked Read 7 Done Done Sent false 7384 Acked Read 10 Done Done Sent false 7385 Acked Write 5 Done Done Sent false 7386 Acked Write 5 Done Done Sent false 7387 Acked Read 9 Done Done Sent false 7388 Acked Write 2 Done Done Sent false 7389 Acked Read 8 Done Done Sent false 7390 Acked Read 1 Done Done Sent false 7391 Acked Write 4 Done Done Sent false 7392 Acked Read 9 Done Done Sent false 7393 Acked Write 6 Done Done Sent false 7394 Acked Read 10 Done Done Sent false 7395 Acked Write 7 Done Done Sent false 7396 Acked Read 5 Done Done Sent false 7397 Acked Write 6 Done Done Sent false 7398 Acked Write 4 Done Done Sent false 7399 Acked Read 9 Done Done Sent false 7400 Acked Flush 0 Done Done Sent false 7401 Acked Read 8 Done Done Sent false 7402 Acked Write 7 Done Done Sent false 7403 Acked Write 9 Done Done Sent false 7404 Acked Read 7 Done Done Sent false 7405 Acked Read 5 Done Done Sent false 7406 Acked Flush 0 Done Done Sent false 7407 Acked Read 5 Done Done Sent false 7408 Acked Write 7 Done Done Sent false 7409 Acked Write 3 Done Done Sent false 7410 Acked Write 8 Done Done Sent false 7411 Acked Read 8 Done Done Sent false 7412 Acked Read 9 Done Done Sent false 7413 Acked Read 6 Done Done Sent false 7414 Acked Read 2 Done Done Sent false 7415 Acked Write 5 Done Done Sent false 7416 Acked Read 1 Done Done Sent false 7417 Acked Write 1 Done Done Sent false 7418 Acked Read 10 Done Done Sent false 7419 Acked Read 4 Done Done Sent false 7420 Acked Flush 0 Done Done Sent false 7421 Acked Write 9 Done Done Sent false 7422 Acked Read 8 Done Done Sent false 7423 Acked Write 4 Done Done Sent false 7424 Acked Write 5 Done Done Sent false 7425 Acked Write 6 Done Done Sent false 7426 Acked Read 5 Done Done Sent false 7427 Acked Write 1 Done Done Sent false 7428 Acked Write 7 Done Done Sent false 7429 Acked Read 8 Done Done Sent false 7430 Acked Write 1 Done Done Sent false 7431 Acked Read 1 Done Done Sent false 7432 Acked Read 4 Done Done Sent false 7433 Acked Read 2 Done Done Sent false 7434 Acked Flush 0 Done Done Sent false 7435 Acked Write 9 Done Done Sent false 7436 Acked Read 7 Done Done Sent false 7437 Acked Write 4 Done Done Sent false 7438 Acked Read 6 Done Done Sent false 7439 Acked Read 8 Done Done Sent false 7440 Acked Write 3 Done Done Sent false 7441 Acked Read 10 Done Done Sent false 7442 Acked Read 7 Done Done Sent false 7443 Acked Read 7 Done Done Sent false 7444 Acked Flush 0 Done Done Sent false 7445 Acked Write 1 Done Done Sent false 7446 Acked Read 3 Done Done Sent false 7447 Acked Write 8 Done Done Sent false 7448 Acked Flush 0 Done Done Sent false 7449 Acked Write 10 Done Done Sent false 7450 Acked Read 7 Done Done Sent false 7451 Acked Write 4 Done Done Sent false 7452 Acked Read 10 Done Done Sent false 7453 Acked Write 1 Done Done Sent false 7454 Acked Write 4 Done Done Sent false 7455 Acked Write 4 Done Done Sent false 7456 Acked Flush 0 Done Done Sent false 7457 Acked Write 10 Done Done Sent false 7458 Acked Read 8 Done Done Sent false 7459 Acked Flush 0 Done Done Sent false 7460 Acked Read 10 Done Done Sent false 7461 Acked Read 5 Done Done Sent false 7462 Acked Write 3 Done Done Sent false 7463 Acked Write 3 Done Done Sent false 7464 Acked Write 9 Done Done Sent false 7465 Acked Read 10 Done Done Sent false 7466 Acked Write 5 Done Done Sent false 7467 Acked Write 9 Done Done Sent false 7468 Acked Read 1 Done Done Sent false 7469 Acked Read 8 Done Done Sent false 7470 Acked Read 7 Done Done Sent false 7471 Acked Write 8 Done Done Sent false 7472 Acked Write 4 Done Done Sent false 7473 Acked Read 4 Done Done Sent false 7474 Acked Write 2 Done Done Sent false 7475 Acked Write 9 Done Done Sent false 7476 Acked Read 8 Done Done Sent false 7477 Acked Write 3 Done Done Sent false 7478 Acked Read 10 Done Done Sent false 7479 Acked Read 4 Done Done Sent false 7480 Acked Flush 0 Done Done Sent false 7481 Acked Write 3 Done Done Sent false 7482 Acked Write 6 Done Done Sent false 7483 Acked Read 2 Done Done Sent false 7484 Acked Write 2 Done Done Sent false 7485 Acked Write 3 Done Done Sent false 7486 Acked Read 7 Done Done Sent false 7487 Acked Read 9 Done Done Sent false 7488 Acked Read 10 Done Done Sent false 7489 Acked Write 6 Done Done Sent false 7490 Acked Write 8 Done Done Sent false 7491 Acked Write 5 Done Done Sent false 7492 Acked Write 6 Done Done Sent false 7493 Acked Write 5 Done Done Sent false 7494 Acked Write 6 Done Done Sent false 7495 Acked Read 5 Done Done Sent false 7496 Acked Read 5 Done Done Sent false 7497 Acked Read 3 Done Done Sent false 7498 Acked Read 3 Done Done Sent false 7499 Acked Write 8 Done Done Sent false 7500 Acked Write 1 Done Done Sent false 7501 Acked Write 9 Done Done Sent false 7502 Acked Write 2 Done Done Sent false 7503 Acked Read 5 Done Done Sent false 7504 Acked Read 6 Done Done Sent false 7505 Acked Read 8 Done Done Sent false 7506 Acked Write 3 Done Done Sent false 7507 Acked Flush 0 Done Done Sent false 7508 Acked Read 1 Done Done Sent false 7509 Acked Write 7 Done Done Sent false 7510 Acked Read 1 Done Done Sent false 7511 Acked Write 1 Done Done Sent false 7512 Acked Read 8 Done Done Sent false 7513 Acked Read 1 Done Done Sent false 7514 Acked Read 2 Done Done Sent false 7515 Acked Flush 0 Done Done Sent false 7516 Acked Read 2 Done Done Sent false 7517 Acked Write 2 Done Done Sent false 7518 Acked Read 4 Done Done Sent false 7519 Acked Read 8 Done Done Sent false 7520 Acked Read 1 Done Done Sent false 7521 Acked Write 9 Done Done Sent false 7522 Acked Read 4 Done Done Sent false 7523 Acked Write 4 Done Done Sent false 7524 Acked Read 3 Done Done Sent false 7525 Acked Write 5 Done Done Sent false 7526 Acked Write 6 Done Done Sent false 7527 Acked Flush 0 Done Done Sent false 7528 Acked Write 3 Done Done Sent false 7529 Acked Read 10 Done Done Sent false 7530 Acked Write 7 Done Done Sent false 7531 Acked Write 2 Done Done Sent false 7532 Acked Flush 0 Done Done Sent false 7533 Acked Read 3 Done Done Sent false 7534 Acked Write 6 Done Done Sent false 7535 Acked Write 3 Done Done Sent false 7536 Acked Read 1 Done Done Sent false 7537 Acked Write 10 Done Done Sent false 7538 Acked Write 6 Done Done Sent false 7539 Acked Write 4 Done Done Sent false 7540 Acked Read 2 Done Done Sent false 7541 Acked Read 8 Done Done Sent false 7542 Acked Write 8 Done Done Sent false 7543 Acked Read 10 Done Done Sent false 7544 Acked Write 2 Done Done Sent false 7545 Acked Read 10 Done Done Sent false 7546 Acked Write 3 Done Done Sent false 7547 Acked Read 5 Done Done Sent false 7548 Acked Read 5 Done Done Sent false 7549 Acked Write 1 Done Done Sent false 7550 Acked Read 4 Done Done Sent false 7551 Acked Write 10 Done Done Sent false 7552 Acked Write 4 Done Done Sent false 7553 Acked Read 2 Done Done Sent false 7554 Acked Read 7 Done Done Sent false 7555 Acked Read 8 Done Done Sent false 7556 Acked Write 4 Done Done Sent false 7557 Acked Read 8 Done Done Sent false 7558 Acked Read 3 Done Done Sent false 7559 Acked Write 6 Done Done Sent false 7560 Acked Read 8 Done Done Sent false 7561 Acked Write 3 Done Done Sent false 7562 Acked Read 6 Done Done Sent false 7563 Acked Flush 0 Done Done Sent false 7564 Acked Read 7 Done Done Sent false 7565 Acked Flush 0 Done Done Sent false 7566 Acked Read 3 Done Done Sent false 7567 Acked Read 8 Done Done Sent false 7568 Acked Write 9 Done Done Sent false 7569 Acked Read 10 Done Done Sent false 7570 Acked Read 1 Done Done Sent false 7571 Acked Read 2 Done Done Sent false 7572 Acked Write 3 Done Done Sent false 7573 Acked Write 2 Done Done Sent false 7574 Acked Write 5 Done Done Sent false 7575 Acked Read 10 Done Done Sent false 7576 Acked Read 7 Done Done Sent false 7577 Acked Write 5 Done Done Sent false 7578 Acked Read 6 Done Done Sent false 7579 Acked Write 4 Done Done Sent false 7580 Acked Read 7 Done Done Sent false 7581 Acked Read 5 Done Done Sent false 7582 Acked Read 1 Done Done Sent false 7583 Acked Write 3 Done Done Sent false 7584 Acked Read 3 Done Done Sent false 7585 Acked Read 1 Done Done Sent false 7586 Acked Read 4 Done Done Sent false 7587 Acked Write 9 Done Done Sent false 7588 Acked Read 3 Done Done Sent false 7589 Acked Read 2 Done Done Sent false 7590 Acked Read 7 Done Done Sent false 7591 Acked Flush 0 Done Done Sent false 7592 Acked Write 9 Done Done Sent false 7593 Acked Flush 0 Done Done Sent false 7594 Acked Read 5 Done Done Sent false 7595 Acked Read 7 Done Done Sent false 7596 Acked Write 10 Done Done Sent false 7597 Acked Write 2 Done Done Sent false 7598 Acked Read 6 Done Done Sent false 7599 Acked Write 10 Done Done Sent false 7600 Acked Read 8 Done Done Sent false 7601 Acked Write 1 Done Done Sent false 7602 Acked Read 9 Done Done Sent false 7603 Acked Read 8 Done Done Sent false 7604 Acked Write 4 Done Done Sent false 7605 Acked Read 2 Done Done Sent false 7606 Acked Read 9 Done Done Sent false 7607 Acked Read 5 Done Done Sent false 7608 Acked Write 9 Done Done Sent false 7609 Acked Read 3 Done Done Sent false 7610 Acked Read 7 Done Done Sent false 7611 Acked Read 8 Done Done Sent false 7612 Acked Write 9 Done Done Sent false 7613 Acked Read 1 Done Done Sent false 7614 Acked Read 7 Done Done Sent false 7615 Acked Flush 0 Done Done Sent false 7616 Acked Flush 0 Done Done Sent false 7617 Acked Read 8 Done Done Sent false 7618 Acked Write 3 Done Done Sent false 7619 Acked Read 5 Done Done Sent false 7620 Acked Write 5 Done Done Sent false 7621 Acked Read 2 Done Done Sent false 7622 Acked Read 5 Done Done Sent false 7623 Acked Write 10 Done Done Sent false 7624 Acked Write 4 Done Done Sent false 7625 Acked Write 1 Done Done Sent false 7626 Acked Read 6 Done Done Sent false 7627 Acked Read 9 Done Done Sent false 7628 Acked Read 9 Done Done Sent false 7629 Acked Flush 0 Done Done Sent false 7630 Acked Read 10 Done Done Sent false 7631 Acked Read 2 Done Done Sent false 7632 Acked Write 5 Done Done Sent false 7633 Acked Write 2 Done Done Sent false 7634 Acked Write 3 Done Done Sent false 7635 Acked Read 10 Done Done Sent false 7636 Acked Read 10 Done Done Sent false 7637 Acked Read 4 Done Done Sent false 7638 Acked Read 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7629 7629 7337 Downstairs last five completed: 7337 7336 7335 7334 7333 Upstairs last five completed: 7638 7637 7636 7635 7634 CLIENT: Up:0 ds:301 act:2 [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7338 Acked Read 10 Done Done Sent false 7339 Acked Read 1 Done Done Sent false 7340 Acked Read 3 Done Done Sent false 7341 Acked Write 2 Done Done Sent false 7342 Acked Write 1 Done Done Sent false 7343 Acked Flush 0 Done Done Sent false 7344 Acked Write 1 Done Done Sent false 7345 Acked Read 9 Done Done Sent false 7346 Acked Write 5 Done Done Sent false 7347 Acked Read 4 Done Done Sent false 7348 Acked Read 8 Done Done Sent false 7349 Acked Read 6 Done Done Sent false 7350 Acked Read 3 Done Done Sent false 7351 Acked Read 9 Done Done Sent false 7352 Acked Read 4 Done Done Sent false 7353 Acked Read 4 Done Done Sent false 7354 Acked Read 2 Done Done Sent false 7355 Acked Write 1 Done Done Sent false 7356 Acked Read 1 Done Done Sent false 7357 Acked Read 7 Done Done Sent false 7358 Acked Write 1 Done Done Sent false 7359 Acked Read 1 Done Done Sent false 7360 Acked Write 1 Done Done Sent false 7361 Acked Write 10 Done Done Sent false 7362 Acked Read 6 Done Done Sent false 7363 Acked Write 1 Done Done Sent false 7364 Acked Read 4 Done Done Sent false 7365 Acked Write 5 Done Done Sent false 7366 Acked Write 5 Done Done Sent false 7367 Acked Write 1 Done Done Sent false 7368 Acked Read 6 Done Done Sent false 7369 Acked Write 6 Done Done Sent false 7370 Acked Write 9 Done Done Sent false 7371 Acked Read 10 Done Done Sent false 7372 Acked Write 1 Done Done Sent false 7373 Acked Read 9 Done Done Sent false 7374 Acked Read 10 Done Done Sent false 7375 Acked Read 1 Done Done Sent false 7376 Acked Write 3 Done Done Sent false 7377 Acked Write 7 Done Done Sent false 7378 Acked Write 5 Done Done Sent false 7379 Acked Write 6 Done Done Sent false 7380 Acked Write 7 Done Done Sent false 7381 Acked Write 7 Done Done Sent false 7382 Acked Write 5 Done Done Sent false 7383 Acked Read 7 Done Done Sent false 7384 Acked Read 10 Done Done Sent false 7385 Acked Write 5 Done Done Sent false 7386 Acked Write 5 Done Done Sent false 7387 Acked Read 9 Done Done Sent false 7388 Acked Write 2 Done Done Sent false 7389 Acked Read 8 Done Done Sent false 7390 Acked Read 1 Done Done Sent false 7391 Acked Write 4 Done Done Sent false 7392 Acked Read 9 Done Done Sent false 7393 Acked Write 6 Done Done Sent false 7394 Acked Read 10 Done Done Sent false 7395 Acked Write 7 Done Done Sent false 7396 Acked Read 5 Done Done Sent false 7397 Acked Write 6 Done Done Sent false 7398 Acked Write 4 Done Done Sent false 7399 Acked Read 9 Done Done Sent false 7400 Acked Flush 0 Done Done Sent false 7401 Acked Read 8 Done Done Sent false 7402 Acked Write 7 Done Done Sent false 7403 Acked Write 9 Done Done Sent false 7404 Acked Read 7 Done Done Sent false 7405 Acked Read 5 Done Done Sent false 7406 Acked Flush 0 Done Done Sent false 7407 Acked Read 5 Done Done Sent false 7408 Acked Write 7 Done Done Sent false 7409 Acked Write 3 Done Done Sent false 7410 Acked Write 8 Done Done Sent false 7411 Acked Read 8 Done Done Sent false 7412 Acked Read 9 Done Done Sent false 7413 Acked Read 6 Done Done Sent false 7414 Acked Read 2 Done Done Sent false 7415 Acked Write 5 Done Done Sent false 7416 Acked Read 1 Done Done Sent false 7417 Acked Write 1 Done Done Sent false 7418 Acked Read 10 Done Done Sent false 7419 Acked Read 4 Done Done Sent false 7420 Acked Flush 0 Done Done Sent false 7421 Acked Write 9 Done Done Sent false 7422 Acked Read 8 Done Done Sent false 7423 Acked Write 4 Done Done Sent false 7424 Acked Write 5 Done Done Sent false 7425 Acked Write 6 Done Done Sent false 7426 Acked Read 5 Done Done Sent false 7427 Acked Write 1 Done Done Sent false 7428 Acked Write 7 Done Done Sent false 7429 Acked Read 8 Done Done Sent false 7430 Acked Write 1 Done Done Sent false 7431 Acked Read 1 Done Done Sent false 7432 Acked Read 4 Done Done Sent false 7433 Acked Read 2 Done Done Sent false 7434 Acked Flush 0 Done Done Sent false 7435 Acked Write 9 Done Done Sent false 7436 Acked Read 7 Done Done Sent false 7437 Acked Write 4 Done Done Sent false 7438 Acked Read 6 Done Done Sent false 7439 Acked Read 8 Done Done Sent false 7440 Acked Write 3 Done Done Sent false 7441 Acked Read 10 Done Done Sent false 7442 Acked Read 7 Done Done Sent false 7443 Acked Read 7 Done Done Sent false 7444 Acked Flush 0 Done Done Sent false 7445 Acked Write 1 Done Done Sent false 7446 Acked Read 3 Done Done Sent false 7447 Acked Write 8 Done Done Sent false 7448 Acked Flush 0 Done Done Sent false 7449 Acked Write 10 Done Done Sent false 7450 Acked Read 7 Done Done Sent false 7451 Acked Write 4 Done Done Sent false 7452 Acked Read 10 Done Done Sent false 7453 Acked Write 1 Done Done Sent false 7454 Acked Write 4 Done Done Sent false 7455 Acked Write 4 Done Done Sent false 7456 Acked Flush 0 Done Done Sent false 7457 Acked Write 10 Done Done Sent false 7458 Acked Read 8 Done Done Sent false 7459 Acked Flush 0 Done Done Sent false 7460 Acked Read 10 Done Done Sent false 7461 Acked Read 5 Done Done Sent false 7462 Acked Write 3 Done Done Sent false 7463 Acked Write 3 Done Done Sent false 7464 Acked Write 9 Done Done Sent false 7465 Acked Read 10 Done Done Sent false 7466 Acked Write 5 Done Done Sent false 7467 Acked Write 9 Done Done Sent false 7468 Acked Read 1 Done Done Sent false 7469 Acked Read 8 Done Done Sent false 7470 Acked Read 7 Done Done Sent false 7471 Acked Write 8 Done Done Sent false 7472 Acked Write 4 Done Done Sent false 7473 Acked Read 4 Done Done Sent false 7474 Acked Write 2 Done Done Sent false 7475 Acked Write 9 Done Done Sent false 7476 Acked Read 8 Done Done Sent false 7477 Acked Write 3 Done Done Sent false 7478 Acked Read 10 Done Done Sent false 7479 Acked Read 4 Done Done Sent false 7480 Acked Flush 0 Done Done Sent false 7481 Acked Write 3 Done Done Sent false 7482 Acked Write 6 Done Done Sent false 7483 Acked Read 2 Done Done Sent false 7484 Acked Write 2 Done Done Sent false 7485 Acked Write 3 Done Done Sent false 7486 Acked Read 7 Done Done Sent false 7487 Acked Read 9 Done Done Sent false 7488 Acked Read 10 Done Done Sent false 7489 Acked Write 6 Done Done Sent false 7490 Acked Write 8 Done Done Sent false 7491 Acked Write 5 Done Done Sent false 7492 Acked Write 6 Done Done Sent false 7493 Acked Write 5 Done Done Sent false 7494 Acked Write 6 Done Done Sent false 7495 Acked Read 5 Done Done Sent false 7496 Acked Read 5 Done Done Sent false 7497 Acked Read 3 Done Done Sent false 7498 Acked Read 3 Done Done Sent false 7499 Acked Write 8 Done Done Sent false 7500 Acked Write 1 Done Done Sent false 7501 Acked Write 9 Done Done Sent false 7502 Acked Write 2 Done Done Sent false 7503 Acked Read 5 Done Done Sent false 7504 Acked Read 6 Done Done Sent false 7505 Acked Read 8 Done Done Sent false 7506 Acked Write 3 Done Done Sent false 7507 Acked Flush 0 Done Done Sent false 7508 Acked Read 1 Done Done Sent false 7509 Acked Write 7 Done Done Sent false 7510 Acked Read 1 Done Done Sent false 7511 Acked Write 1 Done Done Sent false 7512 Acked Read 8 Done Done Sent false 7513 Acked Read 1 Done Done Sent false 7514 Acked Read 2 Done Done Sent false 7515 Acked Flush 0 Done Done Sent false 7516 Acked Read 2 Done Done Sent false 7517 Acked Write 2 Done Done Sent false 7518 Acked Read 4 Done Done Sent false 7519 Acked Read 8 Done Done Sent false 7520 Acked Read 1 Done Done Sent false 7521 Acked Write 9 Done Done Sent false 7522 Acked Read 4 Done Done Sent false 7523 Acked Write 4 Done Done Sent false 7524 Acked Read 3 Done Done Sent false 7525 Acked Write 5 Done Done Sent false 7526 Acked Write 6 Done Done Sent false 7527 Acked Flush 0 Done Done Sent false 7528 Acked Write 3 Done Done Sent false 7529 Acked Read 10 Done Done Sent false 7530 Acked Write 7 Done Done Sent false 7531 Acked Write 2 Done Done Sent false 7532 Acked Flush 0 Done Done Sent false 7533 Acked Read 3 Done Done Sent false 7534 Acked Write 6 Done Done Sent false 7535 Acked Write 3 Done Done Sent false 7536 Acked Read 1 Done Done Sent false 7537 Acked Write 10 Done Done Sent false 7538 Acked Write 6 Done Done Sent false 7539 Acked Write 4 Done Done Sent false 7540 Acked Read 2 Done Done Sent false 7541 Acked Read 8 Done Done Sent false 7542 Acked Write 8 Done Done Sent false 7543 Acked Read 10 Done Done Sent false 7544 Acked Write 2 Done Done Sent false 7545 Acked Read 10 Done Done Sent false 7546 Acked Write 3 Done Done Sent false 7547 Acked Read 5 Done Done Sent false 7548 Acked Read 5 Done Done Sent false 7549 Acked Write 1 Done Done Sent false 7550 Acked Read 4 Done Done Sent false 7551 Acked Write 10 Done Done Sent false 7552 Acked Write 4 Done Done Sent false 7553 Acked Read 2 Done Done Sent false 7554 Acked Read 7 Done Done Sent false 7555 Acked Read 8 Done Done Sent false 7556 Acked Write 4 Done Done Sent false 7557 Acked Read 8 Done Done Sent false 7558 Acked Read 3 Done Done Sent false 7559 Acked Write 6 Done Done Sent false 7560 Acked Read 8 Done Done Sent false 7561 Acked Write 3 Done Done Sent false 7562 Acked Read 6 Done Done Sent false 7563 Acked Flush 0 Done Done Sent false 7564 Acked Read 7 Done Done Sent false 7565 Acked Flush 0 Done Done Sent false 7566 Acked Read 3 Done Done Sent false 7567 Acked Read 8 Done Done Sent false 7568 Acked Write 9 Done Done Sent false 7569 Acked Read 10 Done Done Sent false 7570 Acked Read 1 Done Done Sent false 7571 Acked Read 2 Done Done Sent false 7572 Acked Write 3 Done Done Sent false 7573 Acked Write 2 Done Done Sent false 7574 Acked Write 5 Done Done Sent false 7575 Acked Read 10 Done Done Sent false 7576 Acked Read 7 Done Done Sent false 7577 Acked Write 5 Done Done Sent false 7578 Acked Read 6 Done Done Sent false 7579 Acked Write 4 Done Done Sent false 7580 Acked Read 7 Done Done Sent false 7581 Acked Read 5 Done Done Sent false 7582 Acked Read 1 Done Done Sent false 7583 Acked Write 3 Done Done Sent false 7584 Acked Read 3 Done Done Sent false 7585 Acked Read 1 Done Done Sent false 7586 Acked Read 4 Done Done Sent false 7587 Acked Write 9 Done Done Sent false 7588 Acked Read 3 Done Done Sent false 7589 Acked Read 2 Done Done Sent false 7590 Acked Read 7 Done Done Sent false 7591 Acked Flush 0 Done Done Sent false 7592 Acked Write 9 Done Done Sent false 7593 Acked Flush 0 Done Done Sent false 7594 Acked Read 5 Done Done Sent false 7595 Acked Read 7 Done Done Sent false 7596 Acked Write 10 Done Done Sent false 7597 Acked Write 2 Done Done Sent false 7598 Acked Read 6 Done Done Sent false 7599 Acked Write 10 Done Done Sent false 7600 Acked Read 8 Done Done Sent false 7601 Acked Write 1 Done Done Sent false 7602 Acked Read 9 Done Done Sent false 7603 Acked Read 8 Done Done Sent false 7604 Acked Write 4 Done Done Sent false 7605 Acked Read 2 Done Done Sent false 7606 Acked Read 9 Done Done Sent false 7607 Acked Read 5 Done Done Sent false 7608 Acked Write 9 Done Done Sent false 7609 Acked Read 3 Done Done Sent false 7610 Acked Read 7 Done Done Sent false 7611 Acked Read 8 Done Done Sent false 7612 Acked Write 9 Done Done Sent false 7613 Acked Read 1 Done Done Sent false 7614 Acked Read 7 Done Done Sent false 7615 Acked Flush 0 Done Done Sent false 7616 Acked Flush 0 Done Done Sent false 7617 Acked Read 8 Done Done Sent false 7618 Acked Write 3 Done Done Sent false 7619 Acked Read 5 Done Done Sent false 7620 Acked Write 5 Done Done Sent false 7621 Acked Read 2 Done Done Sent false 7622 Acked Read 5 Done Done Sent false 7623 Acked Write 10 Done Done Sent false 7624 Acked Write 4 Done Done Sent false 7625 Acked Write 1 Done Done Sent false 7626 Acked Read 6 Done Done Sent false 7627 Acked Read 9 Done Done Sent false 7628 Acked Read 9 Done Done Sent false 7629 Acked Flush 0 Done Done Sent false 7630 Acked Read 10 Done Done Sent false 7631 Acked Read 2 Done Done Sent false 7632 Acked Write 5 Done Done Sent false 7633 Acked Write 2 Done Done Sent false 7634 Acked Write 3 Done Done Sent false 7635 Acked Read 10 Done Done Sent false 7636 Acked Read 10 Done Done Sent false 7637 Acked Read 4 Done Done Sent false 7638 Acked Read 5 Done Done Sent false 7639 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7639 7639 7337 Downstairs last five completed: 7337 7336 7335 7334 7333 Upstairs last five completed: 7639 7638 7637 7636 7635 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.782205805Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:47690)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.782273777Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:18.783322585Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.783837167Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.783867529Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 7337","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.783883975Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 7337","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.78408967Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.784113793Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.784128119Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 7337","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.784141985Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 7337","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:18.784353439Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7639 7638 7637 7636 7635 Upstairs last five completed: 7639 7638 7637 7636 7635 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Nov 27 01:47:21.020 INFO request completed, latency_us: 313, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b2cf6a0f-21aa-4953-9c88-10b9c6ea91e7, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:47:21.021 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: ee58ef2a-f965-4857-88d8-bca1b4377516, remote_addr: 127.0.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:21.02149829Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:21.021598853Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:21.021624576Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:21.0216468Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:21.021662556Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:21.021679891Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:47:25.023 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/0, method: GET, req_id: b5766eb1-4e37-417c-815a-e1ff28713656, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 537 len 12288 002/300 Read block 3225 len 20480 003/300 Write block 1963 len 36864 data: 5 4 3 3 3 3 2 2 3 004/300 Write block 2610 len 36864 data: 5 5 4 4 3 4 4 4 4 005/300 Write block 2007 len 40960 data: 5 3 3 3 3 3 3 5 5 5 006/300 Write block 3456 len 8192 data: 4 5 007/300 Write block 2458 len 4096 data: 7 008/300 Read block 2904 len 28672 009/300 Flush 010/300 Flush 011/300 Read block 1498 len 16384 012/300 Read block 1780 len 40960 013/300 Read block 1278 len 12288 014/300 Write block 660 len 32768 data: 5 5 5 5 4 3 3 3 015/300 Write block 691 len 4096 data: 3 016/300 Read block 4045 len 16384 017/300 Write block 1439 len 40960 data: 6 4 4 4 3 3 4 3 3 4 018/300 Write block 2927 len 4096 data: 3 019/300 Read block 1903 len 24576 020/300 Write block 96 len 28672 data: 5 5 6 6 7 7 8 021/300 Flush 022/300 Read block 3976 len 32768 023/300 Read block 3506 len 8192 024/300 Read block 1988 len 36864 025/300 Read block 4642 len 20480 026/300 Read block 4938 len 32768 027/300 Write block 586 len 32768 data: 6 4 3 3 3 5 5 5 028/300 Write block 3657 len 12288 data: 6 8 7 029/300 Write block 3177 len 24576 data: 3 4 5 5 4 5 030/300 Read block 1053 len 28672 031/300 Write block 1105 len 16384 data: 6 6 6 9 032/300 Write block 2671 len 20480 data: 2 3 3 2 3 033/300 Write block 153 len 24576 data: 7 7 7 7 6 4 034/300 Flush 035/300 Read block 2466 len 20480 036/300 Write block 4425 len 12288 data: 8 6 5 037/300 Read block 3996 len 4096 038/300 Write block 1340 len 28672 data: 5 4 4 5 5 5 5 039/300 Read block 47 len 20480 040/300 Write block 2784 len 8192 data: 5 4 041/300 Read block 87 len 32768 042/300 Write block 4178 len 8192 data: 4 5 043/300 Write block 588 len 4096 data: 4 044/300 Write block 2593 len 28672 data: 3 3 3 4 4 4 3 045/300 Write block 2483 len 4096 data: 4 046/300 Flush 047/300 Write block 4840 len 24576 data: 7 6 7 6 7 9 048/300 Read block 3049 len 8192 049/300 Read block 1546 len 12288 050/300 Read block 886 len 16384 051/300 Read block 3658 len 40960 052/300 Read block 3333 len 8192 053/300 Write block 3790 len 20480 data: 4 4 3 4 5 054/300 Read block 1604 len 8192 055/300 Read block 290 len 32768 056/300 Write block 533 len 40960 data: 7 5 5 5 4 5 4 5 6 4 057/300 Read block 3230 len 32768 058/300 Read block 4075 len 4096 059/300 Write block 1858 len 12288 data: 5 4 4 060/300 Read block 2199 len 24576 061/300 Read block 1196 len 16384 062/300 Write block 2297 len 4096 data: 3 063/300 Read block 3167 len 36864 064/300 Read block 1092 len 40960 065/300 Write block 1943 len 12288 data: 2 2 3 066/300 Read block 3454 len 28672 067/300 Read block 2906 len 36864 068/300 Read block 2597 len 28672 069/300 Read block 4170 len 32768 070/300 Write block 3898 len 28672 data: 4 4 4 5 5 3 2 071/300 Read block 1061 len 20480 072/300 Read block 307 len 24576 073/300 Write block 3825 len 20480 data: 4 4 5 5 6 074/300 Flush 075/300 Read block 1384 len 4096 076/300 Read block 887 len 24576 077/300 Write block 1200 len 32768 data: 8 7 6 7 7 6 5 6 078/300 Read block 3809 len 12288 079/300 Read block 3539 len 24576 080/300 Flush 081/300 Read block 2041 len 16384 082/300 Write block 3652 len 24576 data: 3 5 6 7 7 7 083/300 Read block 2606 len 24576 084/300 Write block 3747 len 4096 data: 3 085/300 Write block 704 len 28672 data: 4 4 4 5 4 4 4 086/300 Read block 1236 len 36864 087/300 Write block 1792 len 12288 data: 4 5 5 088/300 Write block 2385 len 40960 data: 7 7 7 8 8 7 7 5 5 4 089/300 Write block 388 len 20480 data: 4 5 4 5 4 090/300 Read block 2217 len 4096 091/300 Read block 714 len 12288 092/300 Read block 4457 len 20480 093/300 Read block 1746 len 4096 094/300 Flush 095/300 Read block 4454 len 28672 096/300 Write block 3633 len 24576 data: 5 4 4 3 3 3 097/300 Write block 3558 len 16384 data: 6 4 3 2 098/300 Flush 099/300 Read block 2621 len 36864 100/300 Read block 1046 len 40960 101/300 Flush 102/300 Read block 4647 len 24576 103/300 Read block 287 len 28672 104/300 Write block 3190 len 24576 data: 3 2 4 5 6 5 105/300 Read block 3992 len 4096 106/300 Read block 11 len 32768 107/300 Flush 108/300 Read block 3352 len 4096 109/300 Read block 2358 len 20480 110/300 Write block 4577 len 12288 data: 4 4 4 111/300 Write block 2849 len 40960 data: 5 5 6 6 7 7 6 6 5 6 112/300 Read block 3640 len 24576 113/300 Read block 4160 len 4096 114/300 Read block 184 len 8192 115/300 Flush 116/300 Write block 2305 len 24576 data: 8 9 9 8 8 8 117/300 Read block 2797 len 32768 118/300 Write block 1882 len 28672 data: 4 7 7 7 7 8 8 119/300 Read block 1220 len 28672 120/300 Read block 2654 len 40960 121/300 Read block 2374 len 20480 122/300 Read block 1820 len 16384 123/300 Read block 4690 len 8192 124/300 Flush 125/300 Write block 3641 len 8192 data: 4 4 126/300 Read block 3884 len 28672 127/300 Write block 4663 len 32768 data: 8 8 7 7 6 6 6 7 128/300 Write block 889 len 20480 data: 7 8 7 7 7 129/300 Write block 2244 len 12288 data: 6 8 9 130/300 Write block 108 len 32768 data: 7 7 7 7 7 6 6 6 131/300 Read block 3530 len 24576 132/300 Read block 3845 len 4096 133/300 Read block 2454 len 20480 134/300 Write block 3749 len 40960 data: 3 3 2 3 8 9 8 8 6 7 135/300 Read block 4217 len 24576 136/300 Write block 3097 len 20480 data: 6 7 7 7 6 137/300 Read block 540 len 12288 138/300 Write block 1346 len 20480 data: 6 5 4 3 3 139/300 Write block 3400 len 32768 data: 5 4 5 5 5 5 6 6 140/300 Read block 4957 len 12288 141/300 Write block 2750 len 24576 data: 3 3 3 4 4 4 142/300 Write block 4730 len 28672 data: 4 4 4 4 3 3 3 143/300 Write block 1578 len 40960 data: 5 5 5 4 4 5 4 4 5 6 144/300 Write block 1648 len 32768 data: 6 6 6 6 6 4 4 2 145/300 Write block 4956 len 40960 data: 5 4 4 5 5 5 7 7 7 6 146/300 Read block 688 len 8192 147/300 Flush 148/300 Write block 4249 len 28672 data: 2 4 4 4 4 3 5 149/300 Write block 4172 len 40960 data: 5 4 5 4 4 3 5 6 5 4 150/300 Write block 239 len 28672 data: 4 5 5 5 6 5 5 151/300 Read block 4798 len 40960 152/300 Write block 4466 len 8192 data: 5 5 153/300 Write block 2037 len 12288 data: 5 5 8 154/300 Write block 3254 len 40960 data: 4 5 6 8 6 5 4 3 2 2 155/300 Flush 156/300 Read block 3248 len 24576 157/300 Read block 2448 len 28672 158/300 Read block 3307 len 4096 159/300 Write block 3653 len 28672 data: 6 7 8 8 8 9 8 160/300 Flush 161/300 Write block 151 len 12288 data: 5 4 8 162/300 Write block 2300 len 40960 data: 4 5 6 7 6 9 10 10 9 9 163/300 Write block 2196 len 20480 data: 4 4 5 4 5 164/300 Read block 2693 len 32768 165/300 Write block 1194 len 20480 data: 3 4 5 5 5 166/300 Read block 2623 len 40960 167/300 Write block 2930 len 20480 data: 4 3 4 4 4 168/300 Write block 1906 len 40960 data: 6 7 4 5 8 6 6 6 8 7 169/300 Write block 153 len 20480 data: 9 8 8 8 7 170/300 Write block 3703 len 4096 data: 7 171/300 Read block 1543 len 24576 172/300 Write block 771 len 12288 data: 4 5 6 173/300 Read block 711 len 8192 174/300 Write block 2912 len 12288 data: 3 3 4 175/300 Flush 176/300 Read block 4215 len 24576 177/300 Flush 178/300 Flush 179/300 Read block 3357 len 24576 180/300 Write block 973 len 20480 data: 4 4 5 7 6 181/300 Read block 1436 len 12288 182/300 Write block 510 len 28672 data: 7 6 7 4 4 4 4 183/300 Read block 1030 len 20480 184/300 Write block 3515 len 40960 data: 5 5 6 6 4 6 6 5 5 6 185/300 Read block 3674 len 40960 186/300 Read block 909 len 12288 187/300 Read block 1034 len 24576 188/300 Write block 1667 len 4096 data: 4 189/300 Read block 2902 len 16384 190/300 Write block 3195 len 12288 data: 6 6 8 191/300 Read block 1849 len 12288 192/300 Write block 1793 len 24576 data: 6 6 4 3 3 4 193/300 Read block 4573 len 32768 194/300 Read block 3721 len 36864 195/300 Read block 1925 len 28672 196/300 Write block 1584 len 40960 data: 5 5 6 7 7 6 5 3 3 2 197/300 Write block 344 len 36864 data: 5 5 5 3 3 3 3 4 4 198/300 Write block 2735 len 24576 data: 10 9 7 7 7 8 199/300 Read block 2011 len 12288 200/300 Flush 201/300 Write block 1950 len 24576 data: 5 5 5 6 7 7 202/300 Read block 4388 len 32768 203/300 Flush 204/300 Write block 3920 len 12288 data: 3 3 4 205/300 Read block 3950 len 32768 206/300 Read block 3677 len 20480 207/300 Write block 1939 len 20480 data: 5 4 3 2 3 208/300 Read block 1566 len 16384 209/300 Write block 3487 len 28672 data: 5 4 6 7 7 5 5 210/300 Read block 218 len 20480 211/300 Read block 1843 len 28672 212/300 Flush 213/300 Read block 2977 len 4096 214/300 Write block 3545 len 40960 data: 5 3 4 4 6 5 6 7 6 7 215/300 Write block 4505 len 12288 data: 5 5 5 216/300 Read block 1611 len 36864 217/300 Write block 1814 len 20480 data: 5 8 11 10 9 218/300 Write block 1606 len 40960 data: 4 6 5 6 5 6 7 5 6 5 219/300 Read block 3651 len 4096 220/300 Write block 3978 len 32768 data: 3 4 3 3 4 4 5 5 221/300 Flush 222/300 Write block 4543 len 28672 data: 3 3 3 3 3 3 5 223/300 Flush 224/300 Write block 4250 len 28672 data: 5 5 5 5 4 6 5 225/300 Flush 226/300 Write block 493 len 4096 data: 6 227/300 Read block 1609 len 4096 228/300 Read block 1260 len 12288 229/300 Read block 1508 len 4096 230/300 Flush 231/300 Flush 232/300 Read block 4473 len 8192 233/300 Write block 138 len 12288 data: 4 4 4 234/300 Read block 4982 len 20480 235/300 Write block 1144 len 8192 data: 3 3 236/300 Write block 3092 len 20480 data: 4 4 4 4 4 237/300 Flush 238/300 Read block 2102 len 40960 239/300 Read block 2177 len 20480 240/300 Read block 3546 len 36864 241/300 Read block 1560 len 12288 242/300 Write block 4897 len 8192 data: 3 5 243/300 Read block 1873 len 16384 244/300 Read block 1885 len 32768 245/300 Write block 2643 len 12288 data: 3 4 3 246/300 Read block 3883 len 28672 247/300 Read block 3318 len 12288 248/300 Write block 859 len 20480 data: 4 5 6 7 7 249/300 Flush 250/300 Read block 4224 len 36864 251/300 Read block 2648 len 32768 252/300 Read block 319 len 8192 253/300 Write block 3341 len 36864 data: 4 4 3 3 3 3 4 4 5 254/300 Read block 75 len 12288 255/300 Read block 3029 len 4096 256/300 Flush 257/300 Flush 258/300 Write block 4554 len 32768 data: 4 4 2 3 3 4 6 7 259/300 Read block 2870 len 20480 260/300 Write block 3201 len 40960 data: 4 4 4 4 4 4 4 5 7 7 261/300 Flush 262/300 Write block 2000 len 32768 data: 7 7 5 7 6 8 7 6 263/300 Write block 2364 len 40960 data: 5 5 5 4 4 4 6 6 7 9 264/300 Read block 3329 len 36864 265/300 Write block 4525 len 4096 data: 5 266/300 Read block 2288 len 8192 267/300 Write block 1157 len 12288 data: 6 6 6 268/300 Flush 269/300 Read block 3611 len 28672 270/300 Flush 271/300 Write block 762 len 20480 data: 3 3 3 2 2 272/300 Write block 2086 len 20480 data: 5 4 4 4 4 273/300 Write block 1970 len 24576 data: 3 4 2 4 5 5 274/300 Write block 2236 len 8192 data: 2 3 275/300 Flush 276/300 Read block 3059 len 40960 277/300 Flush 278/300 Write block 3107 len 40960 data: 8 7 7 6 6 5 3 2 2 2 279/300 Read block 3898 len 4096 280/300 Read block 132 len 8192 281/300 Read block 4111 len 4096 282/300 Write block 4704 len 24576 data: 6 6 5 6 8 7 283/300 Read block 1252 len 16384 284/300 Read block 4816 len 40960 285/300 Read block 1543 len 16384 286/300 Write block 2631 len 4096 data: 3 287/300 Read block 834 len 16384 288/300 Write block 608 len 36864 data: 4 4 3 3 3 2 2 3 4 289/300 Write block 1501 len 4096 data: 5 290/300 Write block 4530 len 32768 data: 6 6 6 4 3 3 4 4 291/300 Read block 4758 len 36864 292/300 Read block 2377 len 8192 293/300 Read block 3840 len 8192 294/300 Read block 1570 len 24576 295/300 Read block 469 len 8192 296/300 Write block 101 len 32768 data: 8 9 5 5 6 5 5 8 297/300 Write block 3625 len 20480 data: 4 5 6 6 4 298/300 Read block 2241 len 20480 299/300 Read block 1100 len 36864 300/300 Write block 4933 len 24576 data: 7 5 5 4 5 8 301/300 Read block 2272 len 16384 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:47:25.250 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4dd6594a-0af0-46ba-97a1-363c94df95bb, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7640 Acked Read 3 Sent Done Done false 7641 Acked Read 5 Sent Done Done false 7642 Acked Write 9 Sent Done Done false 7643 Acked Write 9 Sent Done Done false 7644 Acked Write 10 Sent Done Done false 7645 Acked Write 2 Sent Done Done false 7646 Acked Write 1 Sent Done Done false 7647 Acked Read 7 Sent Done Done false 7648 Acked Flush 0 Sent Done Done false 7649 Acked Flush 0 Sent Done Done false 7650 Acked Read 4 Sent Done Done false 7651 Acked Read 10 Sent Done Done false 7652 Acked Read 3 Sent Done Done false 7653 Acked Write 8 Sent Done Done false 7654 Acked Write 1 Sent Done Done false 7655 Acked Read 4 Sent Done Done false 7656 Acked Write 10 Sent Done Done false 7657 Acked Write 1 Sent Done Done false 7658 Acked Read 6 Sent Done Done false 7659 Acked Write 7 Sent Done Done false 7660 Acked Flush 0 Sent Done Done false 7661 Acked Read 8 Sent Done Done false 7662 Acked Read 2 Sent Done Done false 7663 Acked Read 9 Sent Done Done false 7664 Acked Read 5 Sent Done Done false 7665 Acked Read 8 Sent Done Done false 7666 Acked Write 8 Sent Done Done false 7667 Acked Write 3 Sent Done Done false 7668 Acked Write 6 Sent Done Done false 7669 Acked Read 7 Sent Done Done false 7670 Acked Write 4 Sent Done Done false 7671 Acked Write 5 Sent Done Done false 7672 Acked Write 6 Sent Done Done false 7673 Acked Flush 0 Sent Done Done false 7674 Acked Read 5 Sent Done Done false 7675 Acked Write 3 Sent Done Done false 7676 Acked Read 1 Sent Done Done false 7677 Acked Write 7 Sent Done Done false 7678 Acked Read 5 Sent Done Done false 7679 Acked Write 2 Sent Done Done false 7680 Acked Read 8 Sent Done Done false 7681 Acked Write 2 Sent Done Done false 7682 Acked Write 1 Sent Done Done false 7683 Acked Write 7 Sent Done Done false 7684 Acked Write 1 Sent Done Done false 7685 Acked Flush 0 Sent Done Done false 7686 Acked Write 6 Sent Done Done false 7687 Acked Read 2 Sent Done Done false 7688 Acked Read 3 Sent Done Done false 7689 Acked Read 4 Sent Done Done false 7690 Acked Read 10 Sent Done Done false 7691 Acked Read 2 Sent Done Done false 7692 Acked Write 5 Sent Done Done false 7693 Acked Read 2 Sent Done Done false 7694 Acked Read 8 Sent Done Done false 7695 Acked Write 10 Sent Done Done false 7696 Acked Read 8 Sent Done Done false 7697 Acked Read 1 Sent Done Done false 7698 Acked Write 3 Sent Done Done false 7699 Acked Read 6 Sent Done Done false 7700 Acked Read 4 Sent Done Done false 7701 Acked Write 1 Sent Done Done false 7702 Acked Read 9 Sent Done Done false 7703 Acked Read 10 Sent Done Done false 7704 Acked Write 3 Sent Done Done false 7705 Acked Read 7 Sent Done Done false 7706 Acked Read 9 Sent Done Done false 7707 Acked Read 7 Sent Done Done false 7708 Acked Read 8 Sent Done Done false 7709 Acked Write 7 Sent Done Done false 7710 Acked Read 5 Sent Done Done false 7711 Acked Read 6 Sent Done Done false 7712 Acked Write 5 Sent Done Done false 7713 Acked Flush 0 Sent Done Done false 7714 Acked Read 1 Sent Done Done false 7715 Acked Read 6 Sent Done Done false 7716 Acked Write 8 Sent Done Done false 7717 Acked Read 3 Sent Done Done false 7718 Acked Read 6 Sent Done Done false 7719 Acked Flush 0 Sent Done Done false 7720 Acked Read 4 Sent Done Done false 7721 Acked Write 6 Sent Done Done false 7722 Acked Read 6 Sent Done Done false 7723 Acked Write 1 Sent Done Done false 7724 Acked Write 7 Sent Done Done false 7725 Acked Read 9 Sent Done Done false 7726 Acked Write 3 Sent Done Done false 7727 Acked Write 10 Sent Done Done false 7728 Acked Write 5 Sent Done Done false 7729 Acked Read 1 Sent Done Done false 7730 Acked Read 3 Sent Done Done false 7731 Acked Read 5 Sent Done Done false 7732 Acked Read 1 Sent Done Done false 7733 Acked Flush 0 Sent Done Done false 7734 Acked Read 7 Sent Done Done false 7735 Acked Write 6 Sent Done Done false 7736 Acked Write 4 Sent Done Done false 7737 Acked Flush 0 Sent Done Done false 7738 Acked Read 9 Sent Done Done false 7739 Acked Read 10 Sent Done Done false 7740 Acked Flush 0 Sent Done Done false 7741 Acked Read 6 Sent Done Done false 7742 Acked Read 7 Sent Done Done false 7743 Acked Write 6 Sent Done Done false 7744 Acked Read 1 Sent Done Done false 7745 Acked Read 8 Sent Done Done false 7746 Acked Flush 0 Sent Done Done false 7747 Acked Read 1 Sent Done Done false 7748 Acked Read 5 Sent Done Done false 7749 Acked Write 3 Sent Done Done false 7750 Acked Write 10 Sent Done Done false 7751 Acked Read 6 Sent Done Done false 7752 Acked Read 1 Sent Done Done false 7753 Acked Read 2 Sent Done Done false 7754 Acked Flush 0 Sent Done Done false 7755 Acked Write 6 Sent Done Done false 7756 Acked Read 8 Sent Done Done false 7757 Acked Write 7 Sent Done Done false 7758 Acked Read 7 Sent Done Done false 7759 Acked Read 10 Sent Done Done false 7760 Acked Read 5 Sent Done Done false 7761 Acked Read 4 Sent Done Done false 7762 Acked Read 2 Sent Done Done false 7763 Acked Flush 0 Sent Done Done false 7764 Acked Write 2 Sent Done Done false 7765 Acked Read 7 Sent Done Done false 7766 Acked Write 8 Sent Done Done false 7767 Acked Write 5 Sent Done Done false 7768 Acked Write 3 Sent Done Done false 7769 Acked Write 8 Sent Done Done false 7770 Acked Read 6 Sent Done Done false 7771 Acked Read 1 Sent Done Done false 7772 Acked Read 5 Sent Done Done false 7773 Acked Write 10 Sent Done Done false 7774 Acked Read 6 Sent Done Done false 7775 Acked Write 5 Sent Done Done false 7776 Acked Read 3 Sent Done Done false 7777 Acked Write 5 Sent Done Done false 7778 Acked Write 8 Sent Done Done false 7779 Acked Read 3 Sent Done Done false 7780 Acked Write 6 Sent Done Done false 7781 Acked Write 7 Sent Done Done false 7782 Acked Write 10 Sent Done Done false 7783 Acked Write 8 Sent Done Done false 7784 Acked Write 10 Sent Done Done false 7785 Acked Read 2 Sent Done Done false 7786 Acked Flush 0 Sent Done Done false 7787 Acked Write 7 Sent Done Done false 7788 Acked Write 10 Sent Done Done false 7789 Acked Write 7 Sent Done Done false 7790 Acked Read 10 Sent Done Done false 7791 Acked Write 2 Sent Done Done false 7792 Acked Write 3 Sent Done Done false 7793 Acked Write 10 Sent Done Done false 7794 Acked Flush 0 Sent Done Done false 7795 Acked Read 6 Sent Done Done false 7796 Acked Read 7 Sent Done Done false 7797 Acked Read 1 Sent Done Done false 7798 Acked Write 7 Sent Done Done false 7799 Acked Flush 0 Sent Done Done false 7800 Acked Write 3 Sent Done Done false 7801 Acked Write 10 Sent Done Done false 7802 Acked Write 5 Sent Done Done false 7803 Acked Read 8 Sent Done Done false 7804 Acked Write 5 Sent Done Done false 7805 Acked Read 10 Sent Done Done false 7806 Acked Write 5 Sent Done Done false 7807 Acked Write 10 Sent Done Done false 7808 Acked Write 5 Sent Done Done false 7809 Acked Write 1 Sent Done Done false 7810 Acked Read 6 Sent Done Done false 7811 Acked Write 3 Sent Done Done false 7812 Acked Read 2 Sent Done Done false 7813 Acked Write 3 Sent Done Done false 7814 Acked Flush 0 Sent Done Done false 7815 Acked Read 6 Sent Done Done false 7816 Acked Flush 0 Sent Done Done false 7817 Acked Flush 0 Sent Done Done false 7818 Acked Read 6 Sent Done Done false 7819 Acked Write 5 Sent Done Done false 7820 Acked Read 3 Sent Done Done false 7821 Acked Write 7 Sent Done Done false 7822 Acked Read 5 Sent Done Done false 7823 Acked Write 10 Sent Done Done false 7824 Acked Read 10 Sent Done Done false 7825 Acked Read 3 Sent Done Done false 7826 Acked Read 6 Sent Done Done false 7827 Acked Write 1 Sent Done Done false 7828 Acked Read 4 Sent Done Done false 7829 Acked Write 3 Sent Done Done false 7830 Acked Read 3 Sent Done Done false 7831 Acked Write 6 Sent Done Done false 7832 Acked Read 8 Sent Done Done false 7833 Acked Read 9 Sent Done Done false 7834 Acked Read 7 Sent Done Done false 7835 Acked Write 10 Sent Done Done false 7836 Acked Write 9 Sent Done Done false 7837 Acked Write 6 Sent Done Done false 7838 Acked Read 3 Sent Done Done false 7839 Acked Flush 0 Sent Done Done false 7840 Acked Write 6 Sent Done Done false 7841 Acked Read 8 Sent Done Done false 7842 Acked Flush 0 Sent Done Done false 7843 Acked Write 3 Sent Done Done false 7844 Acked Read 8 Sent Done Done false 7845 Acked Read 5 Sent Done Done false 7846 Acked Write 5 Sent Done Done false 7847 Acked Read 4 Sent Done Done false 7848 Acked Write 7 Sent Done Done false 7849 Acked Read 5 Sent Done Done false 7850 Acked Read 7 Sent Done Done false 7851 Acked Flush 0 Sent Done Done false 7852 Acked Read 1 Sent Done Done false 7853 Acked Write 10 Sent Done Done false 7854 Acked Write 3 Sent Done Done false 7855 Acked Read 9 Sent Done Done false 7856 Acked Write 5 Sent Done Done false 7857 Acked Write 10 Sent Done Done false 7858 Acked Read 1 Sent Done Done false 7859 Acked Write 8 Sent Done Done false 7860 Acked Flush 0 Sent Done Done false 7861 Acked Write 7 Sent Done Done false 7862 Acked Flush 0 Sent Done Done false 7863 Acked Write 7 Sent Done Done false 7864 Acked Flush 0 Sent Done Done false 7865 Acked Write 1 Sent Done Done false 7866 Acked Read 1 Sent Done Done false 7867 Acked Read 3 Sent Done Done false 7868 Acked Read 1 Sent Done Done false 7869 Acked Flush 0 Sent Done Done false 7870 Acked Flush 0 Sent Done Done false 7871 Acked Read 2 Sent Done Done false 7872 Acked Write 3 Sent Done Done false 7873 Acked Read 5 Sent Done Done false 7874 Acked Write 2 Sent Done Done false 7875 Acked Write 5 Sent Done Done false 7876 Acked Flush 0 Sent Done Done false 7877 Acked Read 10 Sent Done Done false 7878 Acked Read 5 Sent Done Done false 7879 Acked Read 9 Sent Done Done false 7880 Acked Read 3 Sent Done Done false 7881 Acked Write 2 Sent Done Done false 7882 Acked Read 4 Sent Done Done false 7883 Acked Read 8 Sent Done Done false 7884 Acked Write 3 Sent Done Done false 7885 Acked Read 7 Sent Done Done false 7886 Acked Read 3 Sent Done Done false 7887 Acked Write 5 Sent Done Done false 7888 Acked Flush 0 Sent Done Done false 7889 Acked Read 9 Sent Done Done false 7890 Acked Read 8 Sent Done Done false 7891 Acked Read 2 Sent Done Done false 7892 Acked Write 9 Sent Done Done false 7893 Acked Read 3 Sent Done Done false 7894 Acked Read 1 Sent Done Done false 7895 Acked Flush 0 Sent Done Done false 7896 Acked Flush 0 Sent Done Done false 7897 Acked Write 8 Sent Done Done false 7898 Acked Read 5 Sent Done Done false 7899 Acked Write 10 Sent Done Done false 7900 Acked Flush 0 Sent Done Done false 7901 Acked Write 8 Sent Done Done false 7902 Acked Write 10 Sent Done Done false 7903 Acked Read 9 Sent Done Done false 7904 Acked Write 1 Sent Done Done false 7905 Acked Read 2 Sent Done Done false 7906 Acked Write 3 Sent Done Done false 7907 Acked Flush 0 Sent Done Done false 7908 Acked Read 7 Sent Done Done false 7909 Acked Flush 0 Sent Done Done false 7910 Acked Write 5 Sent Done Done false 7911 Acked Write 5 Sent Done Done false 7912 Acked Write 6 Sent Done Done false 7913 Acked Write 2 Sent Done Done false 7914 Acked Flush 0 Sent Done Done false 7915 Acked Read 10 Sent Done Done false 7916 Acked Flush 0 Sent Done Done false 7917 Acked Write 10 Sent Done Done false 7918 Acked Read 1 Sent Done Done false 7919 Acked Read 2 Sent Done Done false 7920 Acked Read 1 Sent Done Done false 7921 Acked Write 6 Sent Done Done false 7922 Acked Read 4 Sent Done Done false 7923 Acked Read 10 Sent Done Done false 7924 Acked Read 4 Sent Done Done false 7925 Acked Write 1 Sent Done Done false 7926 Acked Read 4 Sent Done Done false 7927 Acked Write 9 Sent Done Done false 7928 Acked Write 1 Sent Done Done false 7929 Acked Write 8 Sent Done Done false 7930 Acked Read 9 Sent Done Done false 7931 Acked Read 2 Sent Done Done false 7932 Acked Read 2 Sent Done Done false 7933 Acked Read 6 Sent Done Done false 7934 Acked Read 2 Sent Done Done false 7935 Acked Write 8 Sent Done Done false 7936 Acked Write 5 Sent Done Done false 7937 Acked Read 5 Sent Done Done false 7938 Acked Read 9 Sent Done Done false 7939 Acked Write 6 Sent Done Done false 7940 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7639 7916 7916 Downstairs last five completed: 7639 7638 7637 7636 7635 Upstairs last five completed: 7940 7939 7938 7937 7936 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7640 Acked Read 3 Sent Done Done false 7641 Acked Read 5 Sent Done Done false 7642 Acked Write 9 Sent Done Done false 7643 Acked Write 9 Sent Done Done false 7644 Acked Write 10 Sent Done Done false 7645 Acked Write 2 Sent Done Done false 7646 Acked Write 1 Sent Done Done false 7647 Acked Read 7 Sent Done Done false 7648 Acked Flush 0 Sent Done Done false 7649 Acked Flush 0 Sent Done Done false 7650 Acked Read 4 Sent Done Done false 7651 Acked Read 10 Sent Done Done false 7652 Acked Read 3 Sent Done Done false 7653 Acked Write 8 Sent Done Done false 7654 Acked Write 1 Sent Done Done false 7655 Acked Read 4 Sent Done Done false 7656 Acked Write 10 Sent Done Done false 7657 Acked Write 1 Sent Done Done false 7658 Acked Read 6 Sent Done Done false 7659 Acked Write 7 Sent Done Done false 7660 Acked Flush 0 Sent Done Done false 7661 Acked Read 8 Sent Done Done false 7662 Acked Read 2 Sent Done Done false 7663 Acked Read 9 Sent Done Done false 7664 Acked Read 5 Sent Done Done false 7665 Acked Read 8 Sent Done Done false 7666 Acked Write 8 Sent Done Done false 7667 Acked Write 3 Sent Done Done false 7668 Acked Write 6 Sent Done Done false 7669 Acked Read 7 Sent Done Done false 7670 Acked Write 4 Sent Done Done false 7671 Acked Write 5 Sent Done Done false 7672 Acked Write 6 Sent Done Done false 7673 Acked Flush 0 Sent Done Done false 7674 Acked Read 5 Sent Done Done false 7675 Acked Write 3 Sent Done Done false 7676 Acked Read 1 Sent Done Done false 7677 Acked Write 7 Sent Done Done false 7678 Acked Read 5 Sent Done Done false 7679 Acked Write 2 Sent Done Done false 7680 Acked Read 8 Sent Done Done false 7681 Acked Write 2 Sent Done Done false 7682 Acked Write 1 Sent Done Done false 7683 Acked Write 7 Sent Done Done false 7684 Acked Write 1 Sent Done Done false 7685 Acked Flush 0 Sent Done Done false 7686 Acked Write 6 Sent Done Done false 7687 Acked Read 2 Sent Done Done false 7688 Acked Read 3 Sent Done Done false 7689 Acked Read 4 Sent Done Done false 7690 Acked Read 10 Sent Done Done false 7691 Acked Read 2 Sent Done Done false 7692 Acked Write 5 Sent Done Done false 7693 Acked Read 2 Sent Done Done false 7694 Acked Read 8 Sent Done Done false 7695 Acked Write 10 Sent Done Done false 7696 Acked Read 8 Sent Done Done false 7697 Acked Read 1 Sent Done Done false 7698 Acked Write 3 Sent Done Done false 7699 Acked Read 6 Sent Done Done false 7700 Acked Read 4 Sent Done Done false 7701 Acked Write 1 Sent Done Done false 7702 Acked Read 9 Sent Done Done false 7703 Acked Read 10 Sent Done Done false 7704 Acked Write 3 Sent Done Done false 7705 Acked Read 7 Sent Done Done false 7706 Acked Read 9 Sent Done Done false 7707 Acked Read 7 Sent Done Done false 7708 Acked Read 8 Sent Done Done false 7709 Acked Write 7 Sent Done Done false 7710 Acked Read 5 Sent Done Done false 7711 Acked Read 6 Sent Done Done false 7712 Acked Write 5 Sent Done Done false 7713 Acked Flush 0 Sent Done Done false 7714 Acked Read 1 Sent Done Done false 7715 Acked Read 6 Sent Done Done false 7716 Acked Write 8 Sent Done Done false 7717 Acked Read 3 Sent Done Done false 7718 Acked Read 6 Sent Done Done false 7719 Acked Flush 0 Sent Done Done false 7720 Acked Read 4 Sent Done Done false 7721 Acked Write 6 Sent Done Done false 7722 Acked Read 6 Sent Done Done false 7723 Acked Write 1 Sent Done Done false 7724 Acked Write 7 Sent Done Done false 7725 Acked Read 9 Sent Done Done false 7726 Acked Write 3 Sent Done Done false 7727 Acked Write 10 Sent Done Done false 7728 Acked Write 5 Sent Done Done false 7729 Acked Read 1 Sent Done Done false 7730 Acked Read 3 Sent Done Done false 7731 Acked Read 5 Sent Done Done false 7732 Acked Read 1 Sent Done Done false 7733 Acked Flush 0 Sent Done Done false 7734 Acked Read 7 Sent Done Done false 7735 Acked Write 6 Sent Done Done false 7736 Acked Write 4 Sent Done Done false 7737 Acked Flush 0 Sent Done Done false 7738 Acked Read 9 Sent Done Done false 7739 Acked Read 10 Sent Done Done false 7740 Acked Flush 0 Sent Done Done false 7741 Acked Read 6 Sent Done Done false 7742 Acked Read 7 Sent Done Done false 7743 Acked Write 6 Sent Done Done false 7744 Acked Read 1 Sent Done Done false 7745 Acked Read 8 Sent Done Done false 7746 Acked Flush 0 Sent Done Done false 7747 Acked Read 1 Sent Done Done false 7748 Acked Read 5 Sent Done Done false 7749 Acked Write 3 Sent Done Done false 7750 Acked Write 10 Sent Done Done false 7751 Acked Read 6 Sent Done Done false 7752 Acked Read 1 Sent Done Done false 7753 Acked Read 2 Sent Done Done false 7754 Acked Flush 0 Sent Done Done false 7755 Acked Write 6 Sent Done Done false 7756 Acked Read 8 Sent Done Done false 7757 Acked Write 7 Sent Done Done false 7758 Acked Read 7 Sent Done Done false 7759 Acked Read 10 Sent Done Done false 7760 Acked Read 5 Sent Done Done false 7761 Acked Read 4 Sent Done Done false 7762 Acked Read 2 Sent Done Done false 7763 Acked Flush 0 Sent Done Done false 7764 Acked Write 2 Sent Done Done false 7765 Acked Read 7 Sent Done Done false 7766 Acked Write 8 Sent Done Done false 7767 Acked Write 5 Sent Done Done false 7768 Acked Write 3 Sent Done Done false 7769 Acked Write 8 Sent Done Done false 7770 Acked Read 6 Sent Done Done false 7771 Acked Read 1 Sent Done Done false 7772 Acked Read 5 Sent Done Done false 7773 Acked Write 10 Sent Done Done false 7774 Acked Read 6 Sent Done Done false 7775 Acked Write 5 Sent Done Done false 7776 Acked Read 3 Sent Done Done false 7777 Acked Write 5 Sent Done Done false 7778 Acked Write 8 Sent Done Done false 7779 Acked Read 3 Sent Done Done false 7780 Acked Write 6 Sent Done Done false 7781 Acked Write 7 Sent Done Done false 7782 Acked Write 10 Sent Done Done false 7783 Acked Write 8 Sent Done Done false 7784 Acked Write 10 Sent Done Done false 7785 Acked Read 2 Sent Done Done false 7786 Acked Flush 0 Sent Done Done false 7787 Acked Write 7 Sent Done Done false 7788 Acked Write 10 Sent Done Done false 7789 Acked Write 7 Sent Done Done false 7790 Acked Read 10 Sent Done Done false 7791 Acked Write 2 Sent Done Done false 7792 Acked Write 3 Sent Done Done false 7793 Acked Write 10 Sent Done Done false 7794 Acked Flush 0 Sent Done Done false 7795 Acked Read 6 Sent Done Done false 7796 Acked Read 7 Sent Done Done false 7797 Acked Read 1 Sent Done Done false 7798 Acked Write 7 Sent Done Done false 7799 Acked Flush 0 Sent Done Done false 7800 Acked Write 3 Sent Done Done false 7801 Acked Write 10 Sent Done Done false 7802 Acked Write 5 Sent Done Done false 7803 Acked Read 8 Sent Done Done false 7804 Acked Write 5 Sent Done Done false 7805 Acked Read 10 Sent Done Done false 7806 Acked Write 5 Sent Done Done false 7807 Acked Write 10 Sent Done Done false 7808 Acked Write 5 Sent Done Done false 7809 Acked Write 1 Sent Done Done false 7810 Acked Read 6 Sent Done Done false 7811 Acked Write 3 Sent Done Done false 7812 Acked Read 2 Sent Done Done false 7813 Acked Write 3 Sent Done Done false 7814 Acked Flush 0 Sent Done Done false 7815 Acked Read 6 Sent Done Done false 7816 Acked Flush 0 Sent Done Done false 7817 Acked Flush 0 Sent Done Done false 7818 Acked Read 6 Sent Done Done false 7819 Acked Write 5 Sent Done Done false 7820 Acked Read 3 Sent Done Done false 7821 Acked Write 7 Sent Done Done false 7822 Acked Read 5 Sent Done Done false 7823 Acked Write 10 Sent Done Done false 7824 Acked Read 10 Sent Done Done false 7825 Acked Read 3 Sent Done Done false 7826 Acked Read 6 Sent Done Done false 7827 Acked Write 1 Sent Done Done false 7828 Acked Read 4 Sent Done Done false 7829 Acked Write 3 Sent Done Done false 7830 Acked Read 3 Sent Done Done false 7831 Acked Write 6 Sent Done Done false 7832 Acked Read 8 Sent Done Done false 7833 Acked Read 9 Sent Done Done false 7834 Acked Read 7 Sent Done Done false 7835 Acked Write 10 Sent Done Done false 7836 Acked Write 9 Sent Done Done false 7837 Acked Write 6 Sent Done Done false 7838 Acked Read 3 Sent Done Done false 7839 Acked Flush 0 Sent Done Done false 7840 Acked Write 6 Sent Done Done false 7841 Acked Read 8 Sent Done Done false 7842 Acked Flush 0 Sent Done Done false 7843 Acked Write 3 Sent Done Done false 7844 Acked Read 8 Sent Done Done false 7845 Acked Read 5 Sent Done Done false 7846 Acked Write 5 Sent Done Done false 7847 Acked Read 4 Sent Done Done false 7848 Acked Write 7 Sent Done Done false 7849 Acked Read 5 Sent Done Done false 7850 Acked Read 7 Sent Done Done false 7851 Acked Flush 0 Sent Done Done false 7852 Acked Read 1 Sent Done Done false 7853 Acked Write 10 Sent Done Done false 7854 Acked Write 3 Sent Done Done false 7855 Acked Read 9 Sent Done Done false 7856 Acked Write 5 Sent Done Done false 7857 Acked Write 10 Sent Done Done false 7858 Acked Read 1 Sent Done Done false 7859 Acked Write 8 Sent Done Done false 7860 Acked Flush 0 Sent Done Done false 7861 Acked Write 7 Sent Done Done false 7862 Acked Flush 0 Sent Done Done false 7863 Acked Write 7 Sent Done Done false 7864 Acked Flush 0 Sent Done Done false 7865 Acked Write 1 Sent Done Done false 7866 Acked Read 1 Sent Done Done false 7867 Acked Read 3 Sent Done Done false 7868 Acked Read 1 Sent Done Done false 7869 Acked Flush 0 Sent Done Done false 7870 Acked Flush 0 Sent Done Done false 7871 Acked Read 2 Sent Done Done false 7872 Acked Write 3 Sent Done Done false 7873 Acked Read 5 Sent Done Done false 7874 Acked Write 2 Sent Done Done false 7875 Acked Write 5 Sent Done Done false 7876 Acked Flush 0 Sent Done Done false 7877 Acked Read 10 Sent Done Done false 7878 Acked Read 5 Sent Done Done false 7879 Acked Read 9 Sent Done Done false 7880 Acked Read 3 Sent Done Done false 7881 Acked Write 2 Sent Done Done false 7882 Acked Read 4 Sent Done Done false 7883 Acked Read 8 Sent Done Done false 7884 Acked Write 3 Sent Done Done false 7885 Acked Read 7 Sent Done Done false 7886 Acked Read 3 Sent Done Done false 7887 Acked Write 5 Sent Done Done false 7888 Acked Flush 0 Sent Done Done false 7889 Acked Read 9 Sent Done Done false 7890 Acked Read 8 Sent Done Done false 7891 Acked Read 2 Sent Done Done false 7892 Acked Write 9 Sent Done Done false 7893 Acked Read 3 Sent Done Done false 7894 Acked Read 1 Sent Done Done false 7895 Acked Flush 0 Sent Done Done false 7896 Acked Flush 0 Sent Done Done false 7897 Acked Write 8 Sent Done Done false 7898 Acked Read 5 Sent Done Done false 7899 Acked Write 10 Sent Done Done false 7900 Acked Flush 0 Sent Done Done false 7901 Acked Write 8 Sent Done Done false 7902 Acked Write 10 Sent Done Done false 7903 Acked Read 9 Sent Done Done false 7904 Acked Write 1 Sent Done Done false 7905 Acked Read 2 Sent Done Done false 7906 Acked Write 3 Sent Done Done false 7907 Acked Flush 0 Sent Done Done false 7908 Acked Read 7 Sent Done Done false 7909 Acked Flush 0 Sent Done Done false 7910 Acked Write 5 Sent Done Done false 7911 Acked Write 5 Sent Done Done false 7912 Acked Write 6 Sent Done Done false 7913 Acked Write 2 Sent Done Done false 7914 Acked Flush 0 Sent Done Done false 7915 Acked Read 10 Sent Done Done false 7916 Acked Flush 0 Sent Done Done false 7917 Acked Write 10 Sent Done Done false 7918 Acked Read 1 Sent Done Done false 7919 Acked Read 2 Sent Done Done false 7920 Acked Read 1 Sent Done Done false 7921 Acked Write 6 Sent Done Done false 7922 Acked Read 4 Sent Done Done false 7923 Acked Read 10 Sent Done Done false 7924 Acked Read 4 Sent Done Done false 7925 Acked Write 1 Sent Done Done false 7926 Acked Read 4 Sent Done Done false 7927 Acked Write 9 Sent Done Done false 7928 Acked Write 1 Sent Done Done false 7929 Acked Write 8 Sent Done Done false 7930 Acked Read 9 Sent Done Done false 7931 Acked Read 2 Sent Done Done false 7932 Acked Read 2 Sent Done Done false 7933 Acked Read 6 Sent Done Done false 7934 Acked Read 2 Sent Done Done false 7935 Acked Write 8 Sent Done Done false 7936 Acked Write 5 Sent Done Done false 7937 Acked Read 5 Sent Done Done false 7938 Acked Read 9 Sent Done Done false 7939 Acked Write 6 Sent Done Done false 7940 Acked Read 4 Sent Done Done false 7941 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7639 7941 7941 Downstairs last five completed: 7639 7638 7637 7636 7635 Upstairs last five completed: 7941 7940 7939 7938 7937 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:31.022821944Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:31.022884268Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:31.02291242Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:31.022933274Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:31.02298667Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7640 Acked Read 3 Sent Done Done false 7641 Acked Read 5 Sent Done Done false 7642 Acked Write 9 Sent Done Done false 7643 Acked Write 9 Sent Done Done false 7644 Acked Write 10 Sent Done Done false 7645 Acked Write 2 Sent Done Done false 7646 Acked Write 1 Sent Done Done false 7647 Acked Read 7 Sent Done Done false 7648 Acked Flush 0 Sent Done Done false 7649 Acked Flush 0 Sent Done Done false 7650 Acked Read 4 Sent Done Done false 7651 Acked Read 10 Sent Done Done false 7652 Acked Read 3 Sent Done Done false 7653 Acked Write 8 Sent Done Done false 7654 Acked Write 1 Sent Done Done false 7655 Acked Read 4 Sent Done Done false 7656 Acked Write 10 Sent Done Done false 7657 Acked Write 1 Sent Done Done false 7658 Acked Read 6 Sent Done Done false 7659 Acked Write 7 Sent Done Done false 7660 Acked Flush 0 Sent Done Done false 7661 Acked Read 8 Sent Done Done false 7662 Acked Read 2 Sent Done Done false 7663 Acked Read 9 Sent Done Done false 7664 Acked Read 5 Sent Done Done false 7665 Acked Read 8 Sent Done Done false 7666 Acked Write 8 Sent Done Done false 7667 Acked Write 3 Sent Done Done false 7668 Acked Write 6 Sent Done Done false 7669 Acked Read 7 Sent Done Done false 7670 Acked Write 4 Sent Done Done false 7671 Acked Write 5 Sent Done Done false 7672 Acked Write 6 Sent Done Done false 7673 Acked Flush 0 Sent Done Done false 7674 Acked Read 5 Sent Done Done false 7675 Acked Write 3 Sent Done Done false 7676 Acked Read 1 Sent Done Done false 7677 Acked Write 7 Sent Done Done false 7678 Acked Read 5 Sent Done Done false 7679 Acked Write 2 Sent Done Done false 7680 Acked Read 8 Sent Done Done false 7681 Acked Write 2 Sent Done Done false 7682 Acked Write 1 Sent Done Done false 7683 Acked Write 7 Sent Done Done false 7684 Acked Write 1 Sent Done Done false 7685 Acked Flush 0 Sent Done Done false 7686 Acked Write 6 Sent Done Done false 7687 Acked Read 2 Sent Done Done false 7688 Acked Read 3 Sent Done Done false 7689 Acked Read 4 Sent Done Done false 7690 Acked Read 10 Sent Done Done false 7691 Acked Read 2 Sent Done Done false 7692 Acked Write 5 Sent Done Done false 7693 Acked Read 2 Sent Done Done false 7694 Acked Read 8 Sent Done Done false 7695 Acked Write 10 Sent Done Done false 7696 Acked Read 8 Sent Done Done false 7697 Acked Read 1 Sent Done Done false 7698 Acked Write 3 Sent Done Done false 7699 Acked Read 6 Sent Done Done false 7700 Acked Read 4 Sent Done Done false 7701 Acked Write 1 Sent Done Done false 7702 Acked Read 9 Sent Done Done false 7703 Acked Read 10 Sent Done Done false 7704 Acked Write 3 Sent Done Done false 7705 Acked Read 7 Sent Done Done false 7706 Acked Read 9 Sent Done Done false 7707 Acked Read 7 Sent Done Done false 7708 Acked Read 8 Sent Done Done false 7709 Acked Write 7 Sent Done Done false 7710 Acked Read 5 Sent Done Done false 7711 Acked Read 6 Sent Done Done false 7712 Acked Write 5 Sent Done Done false 7713 Acked Flush 0 Sent Done Done false 7714 Acked Read 1 Sent Done Done false 7715 Acked Read 6 Sent Done Done false 7716 Acked Write 8 Sent Done Done false 7717 Acked Read 3 Sent Done Done false 7718 Acked Read 6 Sent Done Done false 7719 Acked Flush 0 Sent Done Done false 7720 Acked Read 4 Sent Done Done false 7721 Acked Write 6 Sent Done Done false 7722 Acked Read 6 Sent Done Done false 7723 Acked Write 1 Sent Done Done false 7724 Acked Write 7 Sent Done Done false 7725 Acked Read 9 Sent Done Done false 7726 Acked Write 3 Sent Done Done false 7727 Acked Write 10 Sent Done Done false 7728 Acked Write 5 Sent Done Done false 7729 Acked Read 1 Sent Done Done false 7730 Acked Read 3 Sent Done Done false 7731 Acked Read 5 Sent Done Done false 7732 Acked Read 1 Sent Done Done false 7733 Acked Flush 0 Sent Done Done false 7734 Acked Read 7 Sent Done Done false 7735 Acked Write 6 Sent Done Done false 7736 Acked Write 4 Sent Done Done false 7737 Acked Flush 0 Sent Done Done false 7738 Acked Read 9 Sent Done Done false 7739 Acked Read 10 Sent Done Done false 7740 Acked Flush 0 Sent Done Done false 7741 Acked Read 6 Sent Done Done false 7742 Acked Read 7 Sent Done Done false 7743 Acked Write 6 Sent Done Done false 7744 Acked Read 1 Sent Done Done false 7745 Acked Read 8 Sent Done Done false 7746 Acked Flush 0 Sent Done Done false 7747 Acked Read 1 Sent Done Done false 7748 Acked Read 5 Sent Done Done false 7749 Acked Write 3 Sent Done Done false 7750 Acked Write 10 Sent Done Done false 7751 Acked Read 6 Sent Done Done false 7752 Acked Read 1 Sent Done Done false 7753 Acked Read 2 Sent Done Done false 7754 Acked Flush 0 Sent Done Done false 7755 Acked Write 6 Sent Done Done false 7756 Acked Read 8 Sent Done Done false 7757 Acked Write 7 Sent Done Done false 7758 Acked Read 7 Sent Done Done false 7759 Acked Read 10 Sent Done Done false 7760 Acked Read 5 Sent Done Done false 7761 Acked Read 4 Sent Done Done false 7762 Acked Read 2 Sent Done Done false 7763 Acked Flush 0 Sent Done Done false 7764 Acked Write 2 Sent Done Done false 7765 Acked Read 7 Sent Done Done false 7766 Acked Write 8 Sent Done Done false 7767 Acked Write 5 Sent Done Done false 7768 Acked Write 3 Sent Done Done false 7769 Acked Write 8 Sent Done Done false 7770 Acked Read 6 Sent Done Done false 7771 Acked Read 1 Sent Done Done false 7772 Acked Read 5 Sent Done Done false 7773 Acked Write 10 Sent Done Done false 7774 Acked Read 6 Sent Done Done false 7775 Acked Write 5 Sent Done Done false 7776 Acked Read 3 Sent Done Done false 7777 Acked Write 5 Sent Done Done false 7778 Acked Write 8 Sent Done Done false 7779 Acked Read 3 Sent Done Done false 7780 Acked Write 6 Sent Done Done false 7781 Acked Write 7 Sent Done Done false 7782 Acked Write 10 Sent Done Done false 7783 Acked Write 8 Sent Done Done false 7784 Acked Write 10 Sent Done Done false 7785 Acked Read 2 Sent Done Done false 7786 Acked Flush 0 Sent Done Done false 7787 Acked Write 7 Sent Done Done false 7788 Acked Write 10 Sent Done Done false 7789 Acked Write 7 Sent Done Done false 7790 Acked Read 10 Sent Done Done false 7791 Acked Write 2 Sent Done Done false 7792 Acked Write 3 Sent Done Done false 7793 Acked Write 10 Sent Done Done false 7794 Acked Flush 0 Sent Done Done false 7795 Acked Read 6 Sent Done Done false 7796 Acked Read 7 Sent Done Done false 7797 Acked Read 1 Sent Done Done false 7798 Acked Write 7 Sent Done Done false 7799 Acked Flush 0 Sent Done Done false 7800 Acked Write 3 Sent Done Done false 7801 Acked Write 10 Sent Done Done false 7802 Acked Write 5 Sent Done Done false 7803 Acked Read 8 Sent Done Done false 7804 Acked Write 5 Sent Done Done false 7805 Acked Read 10 Sent Done Done false 7806 Acked Write 5 Sent Done Done false 7807 Acked Write 10 Sent Done Done false 7808 Acked Write 5 Sent Done Done false 7809 Acked Write 1 Sent Done Done false 7810 Acked Read 6 Sent Done Done false 7811 Acked Write 3 Sent Done Done false 7812 Acked Read 2 Sent Done Done false 7813 Acked Write 3 Sent Done Done false 7814 Acked Flush 0 Sent Done Done false 7815 Acked Read 6 Sent Done Done false 7816 Acked Flush 0 Sent Done Done false 7817 Acked Flush 0 Sent Done Done false 7818 Acked Read 6 Sent Done Done false 7819 Acked Write 5 Sent Done Done false 7820 Acked Read 3 Sent Done Done false 7821 Acked Write 7 Sent Done Done false 7822 Acked Read 5 Sent Done Done false 7823 Acked Write 10 Sent Done Done false 7824 Acked Read 10 Sent Done Done false 7825 Acked Read 3 Sent Done Done false 7826 Acked Read 6 Sent Done Done false 7827 Acked Write 1 Sent Done Done false 7828 Acked Read 4 Sent Done Done false 7829 Acked Write 3 Sent Done Done false 7830 Acked Read 3 Sent Done Done false 7831 Acked Write 6 Sent Done Done false 7832 Acked Read 8 Sent Done Done false 7833 Acked Read 9 Sent Done Done false 7834 Acked Read 7 Sent Done Done false 7835 Acked Write 10 Sent Done Done false 7836 Acked Write 9 Sent Done Done false 7837 Acked Write 6 Sent Done Done false 7838 Acked Read 3 Sent Done Done false 7839 Acked Flush 0 Sent Done Done false 7840 Acked Write 6 Sent Done Done false 7841 Acked Read 8 Sent Done Done false 7842 Acked Flush 0 Sent Done Done false 7843 Acked Write 3 Sent Done Done false 7844 Acked Read 8 Sent Done Done false 7845 Acked Read 5 Sent Done Done false 7846 Acked Write 5 Sent Done Done false 7847 Acked Read 4 Sent Done Done false 7848 Acked Write 7 Sent Done Done false 7849 Acked Read 5 Sent Done Done false 7850 Acked Read 7 Sent Done Done false 7851 Acked Flush 0 Sent Done Done false 7852 Acked Read 1 Sent Done Done false 7853 Acked Write 10 Sent Done Done false 7854 Acked Write 3 Sent Done Done false 7855 Acked Read 9 Sent Done Done false 7856 Acked Write 5 Sent Done Done false 7857 Acked Write 10 Sent Done Done false 7858 Acked Read 1 Sent Done Done false 7859 Acked Write 8 Sent Done Done false 7860 Acked Flush 0 Sent Done Done false 7861 Acked Write 7 Sent Done Done false 7862 Acked Flush 0 Sent Done Done false 7863 Acked Write 7 Sent Done Done false 7864 Acked Flush 0 Sent Done Done false 7865 Acked Write 1 Sent Done Done false 7866 Acked Read 1 Sent Done Done false 7867 Acked Read 3 Sent Done Done false 7868 Acked Read 1 Sent Done Done false 7869 Acked Flush 0 Sent Done Done false 7870 Acked Flush 0 Sent Done Done false 7871 Acked Read 2 Sent Done Done false 7872 Acked Write 3 Sent Done Done false 7873 Acked Read 5 Sent Done Done false 7874 Acked Write 2 Sent Done Done false 7875 Acked Write 5 Sent Done Done false 7876 Acked Flush 0 Sent Done Done false 7877 Acked Read 10 Sent Done Done false 7878 Acked Read 5 Sent Done Done false 7879 Acked Read 9 Sent Done Done false 7880 Acked Read 3 Sent Done Done false 7881 Acked Write 2 Sent Done Done false 7882 Acked Read 4 Sent Done Done false 7883 Acked Read 8 Sent Done Done false 7884 Acked Write 3 Sent Done Done false 7885 Acked Read 7 Sent Done Done false 7886 Acked Read 3 Sent Done Done false 7887 Acked Write 5 Sent Done Done false 7888 Acked Flush 0 Sent Done Done false 7889 Acked Read 9 Sent Done Done false 7890 Acked Read 8 Sent Done Done false 7891 Acked Read 2 Sent Done Done false 7892 Acked Write 9 Sent Done Done false 7893 Acked Read 3 Sent Done Done false 7894 Acked Read 1 Sent Done Done false 7895 Acked Flush 0 Sent Done Done false 7896 Acked Flush 0 Sent Done Done false 7897 Acked Write 8 Sent Done Done false 7898 Acked Read 5 Sent Done Done false 7899 Acked Write 10 Sent Done Done false 7900 Acked Flush 0 Sent Done Done false 7901 Acked Write 8 Sent Done Done false 7902 Acked Write 10 Sent Done Done false 7903 Acked Read 9 Sent Done Done false 7904 Acked Write 1 Sent Done Done false 7905 Acked Read 2 Sent Done Done false 7906 Acked Write 3 Sent Done Done false 7907 Acked Flush 0 Sent Done Done false 7908 Acked Read 7 Sent Done Done false 7909 Acked Flush 0 Sent Done Done false 7910 Acked Write 5 Sent Done Done false 7911 Acked Write 5 Sent Done Done false 7912 Acked Write 6 Sent Done Done false 7913 Acked Write 2 Sent Done Done false 7914 Acked Flush 0 Sent Done Done false 7915 Acked Read 10 Sent Done Done false 7916 Acked Flush 0 Sent Done Done false 7917 Acked Write 10 Sent Done Done false 7918 Acked Read 1 Sent Done Done false 7919 Acked Read 2 Sent Done Done false 7920 Acked Read 1 Sent Done Done false 7921 Acked Write 6 Sent Done Done false 7922 Acked Read 4 Sent Done Done false 7923 Acked Read 10 Sent Done Done false 7924 Acked Read 4 Sent Done Done false 7925 Acked Write 1 Sent Done Done false 7926 Acked Read 4 Sent Done Done false 7927 Acked Write 9 Sent Done Done false 7928 Acked Write 1 Sent Done Done false 7929 Acked Write 8 Sent Done Done false 7930 Acked Read 9 Sent Done Done false 7931 Acked Read 2 Sent Done Done false 7932 Acked Read 2 Sent Done Done false 7933 Acked Read 6 Sent Done Done false 7934 Acked Read 2 Sent Done Done false 7935 Acked Write 8 Sent Done Done false 7936 Acked Write 5 Sent Done Done false 7937 Acked Read 5 Sent Done Done false 7938 Acked Read 9 Sent Done Done false 7939 Acked Write 6 Sent Done Done false 7940 Acked Read 4 Sent Done Done false 7941 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7639 7941 7941 Downstairs last five completed: 7639 7638 7637 7636 7635 Upstairs last five completed: 7941 7940 7939 7938 7937 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7640 Acked Read 3 Sent Done Done false 7641 Acked Read 5 Sent Done Done false 7642 Acked Write 9 Sent Done Done false 7643 Acked Write 9 Sent Done Done false 7644 Acked Write 10 Sent Done Done false 7645 Acked Write 2 Sent Done Done false 7646 Acked Write 1 Sent Done Done false 7647 Acked Read 7 Sent Done Done false 7648 Acked Flush 0 Sent Done Done false 7649 Acked Flush 0 Sent Done Done false 7650 Acked Read 4 Sent Done Done false 7651 Acked Read 10 Sent Done Done false 7652 Acked Read 3 Sent Done Done false 7653 Acked Write 8 Sent Done Done false 7654 Acked Write 1 Sent Done Done false 7655 Acked Read 4 Sent Done Done false 7656 Acked Write 10 Sent Done Done false 7657 Acked Write 1 Sent Done Done false 7658 Acked Read 6 Sent Done Done false 7659 Acked Write 7 Sent Done Done false 7660 Acked Flush 0 Sent Done Done false 7661 Acked Read 8 Sent Done Done false 7662 Acked Read 2 Sent Done Done false 7663 Acked Read 9 Sent Done Done false 7664 Acked Read 5 Sent Done Done false 7665 Acked Read 8 Sent Done Done false 7666 Acked Write 8 Sent Done Done false 7667 Acked Write 3 Sent Done Done false 7668 Acked Write 6 Sent Done Done false 7669 Acked Read 7 Sent Done Done false 7670 Acked Write 4 Sent Done Done false 7671 Acked Write 5 Sent Done Done false 7672 Acked Write 6 Sent Done Done false 7673 Acked Flush 0 Sent Done Done false 7674 Acked Read 5 Sent Done Done false 7675 Acked Write 3 Sent Done Done false 7676 Acked Read 1 Sent Done Done false 7677 Acked Write 7 Sent Done Done false 7678 Acked Read 5 Sent Done Done false 7679 Acked Write 2 Sent Done Done false 7680 Acked Read 8 Sent Done Done false 7681 Acked Write 2 Sent Done Done false 7682 Acked Write 1 Sent Done Done false 7683 Acked Write 7 Sent Done Done false 7684 Acked Write 1 Sent Done Done false 7685 Acked Flush 0 Sent Done Done false 7686 Acked Write 6 Sent Done Done false 7687 Acked Read 2 Sent Done Done false 7688 Acked Read 3 Sent Done Done false 7689 Acked Read 4 Sent Done Done false 7690 Acked Read 10 Sent Done Done false 7691 Acked Read 2 Sent Done Done false 7692 Acked Write 5 Sent Done Done false 7693 Acked Read 2 Sent Done Done false 7694 Acked Read 8 Sent Done Done false 7695 Acked Write 10 Sent Done Done false 7696 Acked Read 8 Sent Done Done false 7697 Acked Read 1 Sent Done Done false 7698 Acked Write 3 Sent Done Done false 7699 Acked Read 6 Sent Done Done false 7700 Acked Read 4 Sent Done Done false 7701 Acked Write 1 Sent Done Done false 7702 Acked Read 9 Sent Done Done false 7703 Acked Read 10 Sent Done Done false 7704 Acked Write 3 Sent Done Done false 7705 Acked Read 7 Sent Done Done false 7706 Acked Read 9 Sent Done Done false 7707 Acked Read 7 Sent Done Done false 7708 Acked Read 8 Sent Done Done false 7709 Acked Write 7 Sent Done Done false 7710 Acked Read 5 Sent Done Done false 7711 Acked Read 6 Sent Done Done false 7712 Acked Write 5 Sent Done Done false 7713 Acked Flush 0 Sent Done Done false 7714 Acked Read 1 Sent Done Done false 7715 Acked Read 6 Sent Done Done false 7716 Acked Write 8 Sent Done Done false 7717 Acked Read 3 Sent Done Done false 7718 Acked Read 6 Sent Done Done false 7719 Acked Flush 0 Sent Done Done false 7720 Acked Read 4 Sent Done Done false 7721 Acked Write 6 Sent Done Done false 7722 Acked Read 6 Sent Done Done false 7723 Acked Write 1 Sent Done Done false 7724 Acked Write 7 Sent Done Done false 7725 Acked Read 9 Sent Done Done false 7726 Acked Write 3 Sent Done Done false 7727 Acked Write 10 Sent Done Done false 7728 Acked Write 5 Sent Done Done false 7729 Acked Read 1 Sent Done Done false 7730 Acked Read 3 Sent Done Done false 7731 Acked Read 5 Sent Done Done false 7732 Acked Read 1 Sent Done Done false 7733 Acked Flush 0 Sent Done Done false 7734 Acked Read 7 Sent Done Done false 7735 Acked Write 6 Sent Done Done false 7736 Acked Write 4 Sent Done Done false 7737 Acked Flush 0 Sent Done Done false 7738 Acked Read 9 Sent Done Done false 7739 Acked Read 10 Sent Done Done false 7740 Acked Flush 0 Sent Done Done false 7741 Acked Read 6 Sent Done Done false 7742 Acked Read 7 Sent Done Done false 7743 Acked Write 6 Sent Done Done false 7744 Acked Read 1 Sent Done Done false 7745 Acked Read 8 Sent Done Done false 7746 Acked Flush 0 Sent Done Done false 7747 Acked Read 1 Sent Done Done false 7748 Acked Read 5 Sent Done Done false 7749 Acked Write 3 Sent Done Done false 7750 Acked Write 10 Sent Done Done false 7751 Acked Read 6 Sent Done Done false 7752 Acked Read 1 Sent Done Done false 7753 Acked Read 2 Sent Done Done false 7754 Acked Flush 0 Sent Done Done false 7755 Acked Write 6 Sent Done Done false 7756 Acked Read 8 Sent Done Done false 7757 Acked Write 7 Sent Done Done false 7758 Acked Read 7 Sent Done Done false 7759 Acked Read 10 Sent Done Done false 7760 Acked Read 5 Sent Done Done false 7761 Acked Read 4 Sent Done Done false 7762 Acked Read 2 Sent Done Done false 7763 Acked Flush 0 Sent Done Done false 7764 Acked Write 2 Sent Done Done false 7765 Acked Read 7 Sent Done Done false 7766 Acked Write 8 Sent Done Done false 7767 Acked Write 5 Sent Done Done false 7768 Acked Write 3 Sent Done Done false 7769 Acked Write 8 Sent Done Done false 7770 Acked Read 6 Sent Done Done false 7771 Acked Read 1 Sent Done Done false 7772 Acked Read 5 Sent Done Done false 7773 Acked Write 10 Sent Done Done false 7774 Acked Read 6 Sent Done Done false 7775 Acked Write 5 Sent Done Done false 7776 Acked Read 3 Sent Done Done false 7777 Acked Write 5 Sent Done Done false 7778 Acked Write 8 Sent Done Done false 7779 Acked Read 3 Sent Done Done false 7780 Acked Write 6 Sent Done Done false 7781 Acked Write 7 Sent Done Done false 7782 Acked Write 10 Sent Done Done false 7783 Acked Write 8 Sent Done Done false 7784 Acked Write 10 Sent Done Done false 7785 Acked Read 2 Sent Done Done false 7786 Acked Flush 0 Sent Done Done false 7787 Acked Write 7 Sent Done Done false 7788 Acked Write 10 Sent Done Done false 7789 Acked Write 7 Sent Done Done false 7790 Acked Read 10 Sent Done Done false 7791 Acked Write 2 Sent Done Done false 7792 Acked Write 3 Sent Done Done false 7793 Acked Write 10 Sent Done Done false 7794 Acked Flush 0 Sent Done Done false 7795 Acked Read 6 Sent Done Done false 7796 Acked Read 7 Sent Done Done false 7797 Acked Read 1 Sent Done Done false 7798 Acked Write 7 Sent Done Done false 7799 Acked Flush 0 Sent Done Done false 7800 Acked Write 3 Sent Done Done false 7801 Acked Write 10 Sent Done Done false 7802 Acked Write 5 Sent Done Done false 7803 Acked Read 8 Sent Done Done false 7804 Acked Write 5 Sent Done Done false 7805 Acked Read 10 Sent Done Done false 7806 Acked Write 5 Sent Done Done false 7807 Acked Write 10 Sent Done Done false 7808 Acked Write 5 Sent Done Done false 7809 Acked Write 1 Sent Done Done false 7810 Acked Read 6 Sent Done Done false 7811 Acked Write 3 Sent Done Done false 7812 Acked Read 2 Sent Done Done false 7813 Acked Write 3 Sent Done Done false 7814 Acked Flush 0 Sent Done Done false 7815 Acked Read 6 Sent Done Done false 7816 Acked Flush 0 Sent Done Done false 7817 Acked Flush 0 Sent Done Done false 7818 Acked Read 6 Sent Done Done false 7819 Acked Write 5 Sent Done Done false 7820 Acked Read 3 Sent Done Done false 7821 Acked Write 7 Sent Done Done false 7822 Acked Read 5 Sent Done Done false 7823 Acked Write 10 Sent Done Done false 7824 Acked Read 10 Sent Done Done false 7825 Acked Read 3 Sent Done Done false 7826 Acked Read 6 Sent Done Done false 7827 Acked Write 1 Sent Done Done false 7828 Acked Read 4 Sent Done Done false 7829 Acked Write 3 Sent Done Done false 7830 Acked Read 3 Sent Done Done false 7831 Acked Write 6 Sent Done Done false 7832 Acked Read 8 Sent Done Done false 7833 Acked Read 9 Sent Done Done false 7834 Acked Read 7 Sent Done Done false 7835 Acked Write 10 Sent Done Done false 7836 Acked Write 9 Sent Done Done false 7837 Acked Write 6 Sent Done Done false 7838 Acked Read 3 Sent Done Done false 7839 Acked Flush 0 Sent Done Done false 7840 Acked Write 6 Sent Done Done false 7841 Acked Read 8 Sent Done Done false 7842 Acked Flush 0 Sent Done Done false 7843 Acked Write 3 Sent Done Done false 7844 Acked Read 8 Sent Done Done false 7845 Acked Read 5 Sent Done Done false 7846 Acked Write 5 Sent Done Done false 7847 Acked Read 4 Sent Done Done false 7848 Acked Write 7 Sent Done Done false 7849 Acked Read 5 Sent Done Done false 7850 Acked Read 7 Sent Done Done false 7851 Acked Flush 0 Sent Done Done false 7852 Acked Read 1 Sent Done Done false 7853 Acked Write 10 Sent Done Done false 7854 Acked Write 3 Sent Done Done false 7855 Acked Read 9 Sent Done Done false 7856 Acked Write 5 Sent Done Done false 7857 Acked Write 10 Sent Done Done false 7858 Acked Read 1 Sent Done Done false 7859 Acked Write 8 Sent Done Done false 7860 Acked Flush 0 Sent Done Done false 7861 Acked Write 7 Sent Done Done false 7862 Acked Flush 0 Sent Done Done false 7863 Acked Write 7 Sent Done Done false 7864 Acked Flush 0 Sent Done Done false 7865 Acked Write 1 Sent Done Done false 7866 Acked Read 1 Sent Done Done false 7867 Acked Read 3 Sent Done Done false 7868 Acked Read 1 Sent Done Done false 7869 Acked Flush 0 Sent Done Done false 7870 Acked Flush 0 Sent Done Done false 7871 Acked Read 2 Sent Done Done false 7872 Acked Write 3 Sent Done Done false 7873 Acked Read 5 Sent Done Done false 7874 Acked Write 2 Sent Done Done false 7875 Acked Write 5 Sent Done Done false 7876 Acked Flush 0 Sent Done Done false 7877 Acked Read 10 Sent Done Done false 7878 Acked Read 5 Sent Done Done false 7879 Acked Read 9 Sent Done Done false 7880 Acked Read 3 Sent Done Done false 7881 Acked Write 2 Sent Done Done false 7882 Acked Read 4 Sent Done Done false 7883 Acked Read 8 Sent Done Done false 7884 Acked Write 3 Sent Done Done false 7885 Acked Read 7 Sent Done Done false 7886 Acked Read 3 Sent Done Done false 7887 Acked Write 5 Sent Done Done false 7888 Acked Flush 0 Sent Done Done false 7889 Acked Read 9 Sent Done Done false 7890 Acked Read 8 Sent Done Done false 7891 Acked Read 2 Sent Done Done false 7892 Acked Write 9 Sent Done Done false 7893 Acked Read 3 Sent Done Done false 7894 Acked Read 1 Sent Done Done false 7895 Acked Flush 0 Sent Done Done false 7896 Acked Flush 0 Sent Done Done false 7897 Acked Write 8 Sent Done Done false 7898 Acked Read 5 Sent Done Done false 7899 Acked Write 10 Sent Done Done false 7900 Acked Flush 0 Sent Done Done false 7901 Acked Write 8 Sent Done Done false 7902 Acked Write 10 Sent Done Done false 7903 Acked Read 9 Sent Done Done false 7904 Acked Write 1 Sent Done Done false 7905 Acked Read 2 Sent Done Done false 7906 Acked Write 3 Sent Done Done false 7907 Acked Flush 0 Sent Done Done false 7908 Acked Read 7 Sent Done Done false 7909 Acked Flush 0 Sent Done Done false 7910 Acked Write 5 Sent Done Done false 7911 Acked Write 5 Sent Done Done false 7912 Acked Write 6 Sent Done Done false 7913 Acked Write 2 Sent Done Done false 7914 Acked Flush 0 Sent Done Done false 7915 Acked Read 10 Sent Done Done false 7916 Acked Flush 0 Sent Done Done false 7917 Acked Write 10 Sent Done Done false 7918 Acked Read 1 Sent Done Done false 7919 Acked Read 2 Sent Done Done false 7920 Acked Read 1 Sent Done Done false 7921 Acked Write 6 Sent Done Done false 7922 Acked Read 4 Sent Done Done false 7923 Acked Read 10 Sent Done Done false 7924 Acked Read 4 Sent Done Done false 7925 Acked Write 1 Sent Done Done false 7926 Acked Read 4 Sent Done Done false 7927 Acked Write 9 Sent Done Done false 7928 Acked Write 1 Sent Done Done false 7929 Acked Write 8 Sent Done Done false 7930 Acked Read 9 Sent Done Done false 7931 Acked Read 2 Sent Done Done false 7932 Acked Read 2 Sent Done Done false 7933 Acked Read 6 Sent Done Done false 7934 Acked Read 2 Sent Done Done false 7935 Acked Write 8 Sent Done Done false 7936 Acked Write 5 Sent Done Done false 7937 Acked Read 5 Sent Done Done false 7938 Acked Read 9 Sent Done Done false 7939 Acked Write 6 Sent Done Done false 7940 Acked Read 4 Sent Done Done false 7941 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7639 7941 7941 Downstairs last five completed: 7639 7638 7637 7636 7635 Upstairs last five completed: 7941 7940 7939 7938 7937 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.024425363Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:55320)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.024491725Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:41.025632869Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.02614909Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.026171174Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 7639","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.0261863Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 7639","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.026401632Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.026421277Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.026435563Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 7639","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.026449329Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 7639","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.02666816Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7941 7940 7939 7938 7937 Upstairs last five completed: 7941 7940 7939 7938 7937 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:47:41.259[8820] Got stop action so:false kr:false INFO request completed, latency_us: 351, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 10b5062d-49c7-4281-b892-59484bf7fe15, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:41.260342339Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:41.260388007Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:41.260405442Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:47:41.260 INFO request completed, latency_us: 156, response_code:{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.260423757Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 200, uri: {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.260455319Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} /state/cid/1, method: GET, req_id: 700937e1-35ec-4bc3-a394-9e908c39d35b, remote_addr: 127.0.0.1:51788, local_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:41.260492039Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} .0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:47:45.261 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2bc983e8-e8c2-4cf3-85f0-f6accff89555, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 4076 len 12288 002/300 Write block 3797 len 16384 data: 3 4 5 5 003/300 Write block 1267 len 4096 data: 7 004/300 Read block 3517 len 40960 005/300 Write block 4888 len 40960 data: 6 7 6 5 5 4 4 3 3 4 006/300 Flush 007/300 Read block 4638 len 36864 008/300 Write block 2104 len 4096 data: 6 009/300 Read block 3793 len 12288 010/300 Read block 3585 len 36864 011/300 Write block 2245 len 36864 data: 9 10 10 10 8 5 5 4 2 012/300 Flush 013/300 Read block 2605 len 24576 014/300 Write block 2636 len 12288 data: 5 6 6 015/300 Write block 4038 len 24576 data: 5 5 4 5 4 3 016/300 Write block 1170 len 40960 data: 7 8 7 5 4 4 6 8 8 10 017/300 Write block 2969 len 20480 data: 6 7 7 7 6 018/300 Write block 1509 len 4096 data: 6 019/300 Write block 2975 len 12288 data: 7 7 5 020/300 Read block 2165 len 40960 021/300 Read block 1684 len 8192 022/300 Read block 4073 len 40960 023/300 Read block 2703 len 32768 024/300 Read block 816 len 4096 025/300 Write block 4088 len 12288 data: 5 5 7 026/300 Read block 2788 len 16384 027/300 Write block 2150 len 40960 data: 3 3 3 2 3 3 5 5 4 4 028/300 Write block 1620 len 40960 data: 4 4 4 4 4 4 5 6 7 7 029/300 Read block 1878 len 16384 030/300 Write block 2951 len 32768 data: 3 3 3 2 4 6 8 7 031/300 Read block 4801 len 32768 032/300 Read block 3167 len 24576 033/300 Read block 3741 len 16384 034/300 Write block 1079 len 20480 data: 4 3 4 6 6 035/300 Write block 2597 len 8192 data: 5 5 036/300 Write block 302 len 4096 data: 2 037/300 Write block 3435 len 16384 data: 3 3 4 4 038/300 Read block 4169 len 8192 039/300 Read block 4145 len 8192 040/300 Read block 2491 len 40960 041/300 Read block 240 len 4096 042/300 Write block 2626 len 36864 data: 4 4 4 3 4 4 3 3 4 043/300 Flush 044/300 Read block 4256 len 4096 045/300 Read block 1721 len 40960 046/300 Read block 4714 len 12288 047/300 Read block 3743 len 20480 048/300 Read block 4936 len 12288 049/300 Write block 368 len 20480 data: 6 6 8 7 8 050/300 Read block 1711 len 36864 051/300 Write block 3979 len 16384 data: 5 4 4 5 052/300 Flush 053/300 Read block 3039 len 12288 054/300 Write block 27 len 28672 data: 5 4 8 8 10 9 8 055/300 Write block 2252 len 24576 data: 5 3 3 3 5 5 056/300 Write block 2857 len 4096 data: 6 057/300 Read block 2894 len 36864 058/300 Read block 3869 len 28672 059/300 Read block 240 len 36864 060/300 Read block 1746 len 12288 061/300 Write block 4453 len 4096 data: 4 062/300 Write block 4482 len 28672 data: 5 5 6 6 3 2 2 063/300 Write block 1034 len 4096 data: 6 064/300 Read block 1975 len 8192 065/300 Read block 1296 len 40960 066/300 Flush 067/300 Write block 1065 len 28672 data: 4 3 3 3 2 3 3 068/300 Write block 4356 len 36864 data: 9 8 7 6 5 4 7 7 7 069/300 Read block 2080 len 8192 070/300 Read block 136 len 24576 071/300 Write block 2043 len 16384 data: 8 9 9 8 072/300 Read block 543 len 24576 073/300 Read block 292 len 36864 074/300 Read block 378 len 28672 075/300 Flush 076/300 Write block 3483 len 24576 data: 5 3 4 4 6 5 077/300 Read block 1768 len 36864 078/300 Write block 3125 len 4096 data: 5 079/300 Read block 2772 len 4096 080/300 Read block 4568 len 16384 081/300 Write block 1361 len 32768 data: 5 5 5 5 4 5 6 7 082/300 Read block 4591 len 32768 083/300 Write block 3046 len 28672 data: 5 5 4 4 4 4 4 084/300 Read block 4697 len 32768 085/300 Write block 3765 len 4096 data: 5 086/300 Flush 087/300 Read block 2131 len 20480 088/300 Flush 089/300 Read block 1585 len 28672 090/300 Write block 4019 len 12288 data: 3 3 3 091/300 Read block 419 len 12288 092/300 Read block 469 len 12288 093/300 Read block 4386 len 40960 094/300 Write block 1013 len 28672 data: 5 6 6 5 6 5 5 095/300 Write block 991 len 36864 data: 4 5 5 5 5 4 4 4 4 096/300 Write block 3861 len 20480 data: 6 4 4 3 3 097/300 Flush 098/300 Write block 1728 len 32768 data: 3 3 3 3 3 4 4 3 099/300 Write block 2573 len 20480 data: 6 5 5 5 6 100/300 Read block 2832 len 32768 101/300 Flush 102/300 Write block 4283 len 24576 data: 5 5 5 5 6 5 103/300 Read block 4134 len 8192 104/300 Write block 2134 len 28672 data: 4 3 4 5 6 6 4 105/300 Flush 106/300 Write block 1990 len 16384 data: 5 7 9 8 107/300 Read block 3709 len 36864 108/300 Write block 3032 len 20480 data: 6 5 6 6 5 109/300 Write block 453 len 20480 data: 3 5 6 4 3 110/300 Read block 2535 len 8192 111/300 Read block 4692 len 24576 112/300 Write block 3893 len 28672 data: 5 5 5 4 5 5 5 113/300 Read block 3190 len 20480 114/300 Read block 4871 len 32768 115/300 Read block 31 len 32768 116/300 Write block 825 len 40960 data: 11 13 12 12 13 12 12 10 8 7 117/300 Read block 2150 len 32768 118/300 Read block 882 len 16384 119/300 Write block 3803 len 4096 data: 6 120/300 Write block 1169 len 20480 data: 6 8 9 8 6 121/300 Read block 4782 len 40960 122/300 Write block 1230 len 32768 data: 7 5 6 8 7 6 4 4 123/300 Read block 2627 len 8192 124/300 Write block 2079 len 12288 data: 6 6 5 125/300 Write block 1516 len 32768 data: 3 4 4 5 4 3 4 3 126/300 Read block 273 len 20480 127/300 Read block 2001 len 12288 128/300 Flush 129/300 Write block 2897 len 12288 data: 4 3 4 130/300 Write block 395 len 40960 data: 2 2 2 2 2 2 3 3 3 3 131/300 Read block 4156 len 40960 132/300 Write block 203 len 16384 data: 5 5 4 6 133/300 Write block 2063 len 24576 data: 3 3 4 4 3 2 134/300 Write block 2225 len 16384 data: 5 5 5 5 135/300 Read block 2741 len 8192 136/300 Read block 561 len 36864 137/300 Read block 3478 len 4096 138/300 Read block 3722 len 36864 139/300 Read block 2271 len 20480 140/300 Read block 1651 len 12288 141/300 Flush 142/300 Read block 1571 len 24576 143/300 Read block 4140 len 16384 144/300 Write block 3664 len 36864 data: 4 4 3 6 6 6 6 6 4 145/300 Flush 146/300 Read block 3066 len 24576 147/300 Write block 4179 len 40960 data: 7 6 5 6 6 7 6 6 5 5 148/300 Read block 3709 len 40960 149/300 Read block 2249 len 16384 150/300 Read block 3736 len 16384 151/300 Write block 3260 len 4096 data: 5 152/300 Read block 15 len 28672 153/300 Read block 3579 len 12288 154/300 Write block 1165 len 8192 data: 5 5 155/300 Write block 1129 len 16384 data: 3 3 3 3 156/300 Read block 1481 len 36864 157/300 Write block 4577 len 20480 data: 5 5 5 4 5 158/300 Read block 1322 len 4096 159/300 Write block 4601 len 28672 data: 3 3 4 4 4 4 5 160/300 Write block 3196 len 28672 data: 7 9 7 5 4 5 5 161/300 Read block 1068 len 32768 162/300 Flush 163/300 Write block 3436 len 8192 data: 4 5 164/300 Write block 3475 len 28672 data: 5 6 6 5 6 6 5 165/300 Read block 3707 len 4096 166/300 Read block 1998 len 12288 167/300 Read block 3961 len 4096 168/300 Read block 3878 len 4096 169/300 Write block 1994 len 20480 data: 6 6 8 9 10 170/300 Read block 1176 len 16384 171/300 Read block 1776 len 28672 172/300 Write block 3608 len 28672 data: 3 4 4 5 5 5 6 173/300 Read block 2154 len 28672 174/300 Write block 3887 len 12288 data: 4 4 6 175/300 Read block 4939 len 4096 176/300 Write block 375 len 28672 data: 5 3 4 4 4 3 3 177/300 Write block 807 len 20480 data: 6 5 4 4 5 178/300 Read block 3062 len 4096 179/300 Write block 2182 len 4096 data: 6 180/300 Write block 988 len 40960 data: 4 5 5 5 6 6 6 6 5 5 181/300 Write block 642 len 36864 data: 4 3 2 2 2 2 3 3 4 182/300 Read block 3807 len 32768 183/300 Read block 3355 len 24576 184/300 Read block 2566 len 8192 185/300 Write block 1847 len 36864 data: 3 4 5 6 6 7 6 7 5 186/300 Write block 1677 len 4096 data: 4 187/300 Read block 765 len 4096 188/300 Write block 3892 len 28672 data: 6 6 6 6 5 6 6 189/300 Read block 4713 len 12288 190/300 Write block 1564 len 20480 data: 4 3 2 2 2 191/300 Read block 4587 len 20480 192/300 Read block 3499 len 12288 193/300 Read block 2915 len 40960 194/300 Read block 2516 len 36864 195/300 Read block 2503 len 8192 196/300 Write block 799 len 4096 data: 5 197/300 Write block 991 len 36864 data: 6 7 7 7 7 6 6 5 5 198/300 Write block 170 len 4096 data: 6 199/300 Read block 2329 len 36864 200/300 Write block 1256 len 40960 data: 5 3 4 4 4 4 4 6 7 8 201/300 Read block 4850 len 20480 202/300 Read block 3239 len 20480 203/300 Write block 1880 len 16384 data: 4 4 5 8 204/300 Write block 3074 len 28672 data: 3 3 4 4 4 3 3 205/300 Flush 206/300 Read block 4680 len 40960 207/300 Read block 749 len 20480 208/300 Flush 209/300 Read block 519 len 8192 210/300 Flush 211/300 Write block 3196 len 32768 data: 8 10 8 6 5 6 6 5 212/300 Write block 1572 len 16384 data: 6 9 8 9 213/300 Write block 2101 len 24576 data: 7 7 6 7 5 5 214/300 Read block 3784 len 20480 215/300 Write block 1817 len 40960 data: 11 10 8 9 8 9 8 5 5 7 216/300 Write block 4973 len 8192 data: 3 3 217/300 Read block 928 len 24576 218/300 Read block 3944 len 20480 219/300 Read block 2652 len 12288 220/300 Read block 4428 len 28672 221/300 Read block 4369 len 4096 222/300 Read block 604 len 20480 223/300 Read block 1878 len 32768 224/300 Read block 4870 len 24576 225/300 Write block 22 len 4096 data: 4 226/300 Read block 820 len 16384 227/300 Read block 1721 len 28672 228/300 Write block 1610 len 16384 data: 6 7 8 6 229/300 Write block 1957 len 4096 data: 7 230/300 Read block 2268 len 12288 231/300 Read block 4975 len 12288 232/300 Flush 233/300 Read block 3489 len 36864 234/300 Read block 30 len 12288 235/300 Write block 2916 len 8192 data: 4 6 236/300 Write block 2894 len 40960 data: 5 5 5 5 4 5 4 5 6 5 237/300 Write block 311 len 24576 data: 6 5 5 5 5 5 238/300 Write block 2988 len 24576 data: 5 5 6 7 7 6 239/300 Read block 4622 len 36864 240/300 Flush 241/300 Write block 3097 len 4096 data: 7 242/300 Read block 1572 len 8192 243/300 Read block 3948 len 24576 244/300 Read block 3938 len 24576 245/300 Read block 1347 len 12288 246/300 Write block 1107 len 16384 data: 7 10 9 8 247/300 Write block 2829 len 8192 data: 4 6 248/300 Read block 2782 len 24576 249/300 Write block 3660 len 20480 data: 6 6 6 4 5 250/300 Read block 1199 len 24576 251/300 Write block 2640 len 8192 data: 4 5 252/300 Write block 662 len 32768 data: 6 6 5 4 4 4 4 4 253/300 Read block 1044 len 4096 254/300 Write block 313 len 16384 data: 6 6 6 6 255/300 Write block 2863 len 40960 data: 5 4 4 4 5 5 5 5 5 3 256/300 Flush 257/300 Read block 434 len 28672 258/300 Read block 4656 len 40960 259/300 Write block 2142 len 40960 data: 6 6 7 4 4 4 3 3 4 4 260/300 Read block 2846 len 32768 261/300 Write block 239 len 4096 data: 5 262/300 Write block 4470 len 40960 data: 5 6 7 7 7 4 5 4 5 5 263/300 Write block 666 len 28672 data: 5 5 5 5 3 6 4 264/300 Write block 2240 len 16384 data: 3 4 4 4 265/300 Read block 2948 len 28672 266/300 Write block 4508 len 40960 data: 4 4 4 3 5 5 5 5 5 5 267/300 Write block 1629 len 24576 data: 8 7 7 7 6 5 268/300 Write block 943 len 4096 data: 5 269/300 Write block 516 len 28672 data: 5 4 5 4 5 5 5 270/300 Read block 2037 len 28672 271/300 Flush 272/300 Read block 1136 len 36864 273/300 Write block 2944 len 32768 data: 4 2 4 4 4 4 3 4 274/300 Write block 2391 len 24576 data: 8 6 6 5 7 6 275/300 Write block 1152 len 4096 data: 4 276/300 Read block 4704 len 32768 277/300 Read block 2629 len 8192 278/300 Write block 1051 len 12288 data: 2 4 4 279/300 Write block 3394 len 12288 data: 4 4 6 280/300 Read block 4169 len 36864 281/300 Write block 3788 len 28672 data: 5 5 5 5 4 5 6 282/300 Write block 4549 len 4096 data: 6 283/300 Write block 4911 len 16384 data: 4 5 5 3 284/300 Read block 2479 len 12288 285/300 Write block 3869 len 4096 data: 2 286/300 Read block 2108 len 16384 287/300 Write block 1466 len 40960 data: 5 8 9 8 8 8 8 6 5 3 288/300 Write block 769 len 28672 data: 4 5 5 6 7 6 5 289/300 Write block 4333 len 36864 data: 6 5 5 5 3 3 3 3 2 290/300 Read block 1665 len 4096 291/300 Read block 4089 len 16384 292/300 Write block 3221 len 8192 data: 6 6 293/300 Write block 3663 len 32768 data: 5 6 5 4 7 7 7 7 294/300 Write block 3907 len 12288 data: 4 4 3 295/300 Read block 2582 len 28672 296/300 Read block 2498 len 40960 297/300 Write block 1534 len 12288 data: 2 4 5 298/300 Read block 1211 len 36864 299/300 Read block 1369 len 20480 300/300 Read block 2943 len 16384 301/300 Flush Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:47:45.489 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/1, method: POST, req_id: 57b3b724-e281-41ee-8329-c989f1c3d3b6, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7942 Acked Read 3 Done Sent Done false 7943 Acked Write 4 Done Sent Done false 7944 Acked Write 1 Done Sent Done false 7945 Acked Read 10 Done Sent Done false 7946 Acked Write 10 Done Sent Done false 7947 Acked Flush 0 Done Sent Done false 7948 Acked Read 9 Done Sent Done false 7949 Acked Write 1 Done Sent Done false 7950 Acked Read 3 Done Sent Done false 7951 Acked Read 9 Done Sent Done false 7952 Acked Write 9 Done Sent Done false 7953 Acked Flush 0 Done Sent Done false 7954 Acked Read 6 Done Sent Done false 7955 Acked Write 3 Done Sent Done false 7956 Acked Write 6 Done Sent Done false 7957 Acked Write 10 Done Sent Done false 7958 Acked Write 5 Done Sent Done false 7959 Acked Write 1 Done Sent Done false 7960 Acked Write 3 Done Sent Done false 7961 Acked Read 10 Done Sent Done false 7962 Acked Read 2 Done Sent Done false 7963 Acked Read 10 Done Sent Done false 7964 Acked Read 8 Done Sent Done false 7965 Acked Read 1 Done Sent Done false 7966 Acked Write 3 Done Sent Done false 7967 Acked Read 4 Done Sent Done false 7968 Acked Write 10 Done Sent Done false 7969 Acked Write 10 Done Sent Done false 7970 Acked Read 4 Done Sent Done false 7971 Acked Write 8 Done Sent Done false 7972 Acked Read 8 Done Sent Done false 7973 Acked Read 6 Done Sent Done false 7974 Acked Read 4 Done Sent Done false 7975 Acked Write 5 Done Sent Done false 7976 Acked Write 2 Done Sent Done false 7977 Acked Write 1 Done Sent Done false 7978 Acked Write 4 Done Sent Done false 7979 Acked Read 2 Done Sent Done false 7980 Acked Read 2 Done Sent Done false 7981 Acked Read 10 Done Sent Done false 7982 Acked Read 1 Done Sent Done false 7983 Acked Write 9 Done Sent Done false 7984 Acked Flush 0 Done Sent Done false 7985 Acked Read 1 Done Sent Done false 7986 Acked Read 10 Done Sent Done false 7987 Acked Read 3 Done Sent Done false 7988 Acked Read 5 Done Sent Done false 7989 Acked Read 3 Done Sent Done false 7990 Acked Write 5 Done Sent Done false 7991 Acked Read 9 Done Sent Done false 7992 Acked Write 4 Done Sent Done false 7993 Acked Flush 0 Done Sent Done false 7994 Acked Read 3 Done Sent Done false 7995 Acked Write 7 Done Sent Done false 7996 Acked Write 6 Done Sent Done false 7997 Acked Write 1 Done Sent Done false 7998 Acked Read 9 Done Sent Done false 7999 Acked Read 7 Done Sent Done false 8000 Acked Read 9 Done Sent Done false 8001 Acked Read 3 Done Sent Done false 8002 Acked Write 1 Done Sent Done false 8003 Acked Write 7 Done Sent Done false 8004 Acked Write 1 Done Sent Done false 8005 Acked Read 2 Done Sent Done false 8006 Acked Read 10 Done Sent Done false 8007 Acked Flush 0 Done Sent Done false 8008 Acked Write 7 Done Sent Done false 8009 Acked Write 9 Done Sent Done false 8010 Acked Read 2 Done Sent Done false 8011 Acked Read 6 Done Sent Done false 8012 Acked Write 4 Done Sent Done false 8013 Acked Read 6 Done Sent Done false 8014 Acked Read 9 Done Sent Done false 8015 Acked Read 7 Done Sent Done false 8016 Acked Flush 0 Done Sent Done false 8017 Acked Write 6 Done Sent Done false 8018 Acked Read 9 Done Sent Done false 8019 Acked Write 1 Done Sent Done false 8020 Acked Read 1 Done Sent Done false 8021 Acked Read 4 Done Sent Done false 8022 Acked Write 8 Done Sent Done false 8023 Acked Read 8 Done Sent Done false 8024 Acked Write 7 Done Sent Done false 8025 Acked Read 8 Done Sent Done false 8026 Acked Write 1 Done Sent Done false 8027 Acked Flush 0 Done Sent Done false 8028 Acked Read 5 Done Sent Done false 8029 Acked Flush 0 Done Sent Done false 8030 Acked Read 7 Done Sent Done false 8031 Acked Write 3 Done Sent Done false 8032 Acked Read 3 Done Sent Done false 8033 Acked Read 3 Done Sent Done false 8034 Acked Read 10 Done Sent Done false 8035 Acked Write 7 Done Sent Done false 8036 Acked Write 9 Done Sent Done false 8037 Acked Write 5 Done Sent Done false 8038 Acked Flush 0 Done Sent Done false 8039 Acked Write 8 Done Sent Done false 8040 Acked Write 5 Done Sent Done false 8041 Acked Read 8 Done Sent Done false 8042 Acked Flush 0 Done Sent Done false 8043 Acked Write 6 Done Sent Done false 8044 Acked Read 2 Done Sent Done false 8045 Acked Write 7 Done Sent Done false 8046 Acked Flush 0 Done Sent Done false 8047 Acked Write 4 Done Sent Done false 8048 Acked Read 9 Done Sent Done false 8049 Acked Write 5 Done Sent Done false 8050 Acked Write 5 Done Sent Done false 8051 Acked Read 2 Done Sent Done false 8052 Acked Read 6 Done Sent Done false 8053 Acked Write 7 Done Sent Done false 8054 Acked Read 5 Done Sent Done false 8055 Acked Read 8 Done Sent Done false 8056 Acked Read 8 Done Sent Done false 8057 Acked Write 10 Done Sent Done false 8058 Acked Read 8 Done Sent Done false 8059 Acked Read 4 Done Sent Done false 8060 Acked Write 1 Done Sent Done false 8061 Acked Write 5 Done Sent Done false 8062 Acked Read 10 Done Sent Done false 8063 Acked Write 8 Done Sent Done false 8064 Acked Read 2 Done Sent Done false 8065 Acked Write 3 Done Sent Done false 8066 Acked Write 8 Done Sent Done false 8067 Acked Read 5 Done Sent Done false 8068 Acked Read 3 Done Sent Done false 8069 Acked Flush 0 Done Sent Done false 8070 Acked Write 3 Done Sent Done false 8071 Acked Write 10 Done Sent Done false 8072 Acked Read 10 Done Sent Done false 8073 Acked Write 4 Done Sent Done false 8074 Acked Write 6 Done Sent Done false 8075 Acked Write 4 Done Sent Done false 8076 Acked Read 2 Done Sent Done false 8077 Acked Read 9 Done Sent Done false 8078 Acked Read 1 Done Sent Done false 8079 Acked Read 9 Done Sent Done false 8080 Acked Read 5 Done Sent Done false 8081 Acked Read 3 Done Sent Done false 8082 Acked Flush 0 Done Sent Done false 8083 Acked Read 6 Done Sent Done false 8084 Acked Read 4 Done Sent Done false 8085 Acked Write 9 Done Sent Done false 8086 Acked Flush 0 Done Sent Done false 8087 Acked Read 6 Done Sent Done false 8088 Acked Write 10 Done Sent Done false 8089 Acked Read 10 Done Sent Done false 8090 Acked Read 4 Done Sent Done false 8091 Acked Read 4 Done Sent Done false 8092 Acked Write 1 Done Sent Done false 8093 Acked Read 7 Done Sent Done false 8094 Acked Read 3 Done Sent Done false 8095 Acked Write 2 Done Sent Done false 8096 Acked Write 4 Done Sent Done false 8097 Acked Read 9 Done Sent Done false 8098 Acked Write 5 Done Sent Done false 8099 Acked Read 1 Done Sent Done false 8100 Acked Write 7 Done Sent Done false 8101 Acked Write 7 Done Sent Done false 8102 Acked Read 8 Done Sent Done false 8103 Acked Flush 0 Done Sent Done false 8104 Acked Write 2 Done Sent Done false 8105 Acked Write 7 Done Sent Done false 8106 Acked Read 1 Done Sent Done false 8107 Acked Read 3 Done Sent Done false 8108 Acked Read 1 Done Sent Done false 8109 Acked Read 1 Done Sent Done false 8110 Acked Write 5 Done Sent Done false 8111 Acked Read 4 Done Sent Done false 8112 Acked Read 7 Done Sent Done false 8113 Acked Write 7 Done Sent Done false 8114 Acked Read 7 Done Sent Done false 8115 Acked Write 3 Done Sent Done false 8116 Acked Read 1 Done Sent Done false 8117 Acked Write 7 Done Sent Done false 8118 Acked Write 5 Done Sent Done false 8119 Acked Read 1 Done Sent Done false 8120 Acked Write 1 Done Sent Done false 8121 Acked Write 10 Done Sent Done false 8122 Acked Write 9 Done Sent Done false 8123 Acked Read 8 Done Sent Done false 8124 Acked Read 6 Done Sent Done false 8125 Acked Read 2 Done Sent Done false 8126 Acked Write 9 Done Sent Done false 8127 Acked Write 1 Done Sent Done false 8128 Acked Read 1 Done Sent Done false 8129 Acked Write 7 Done Sent Done false 8130 Acked Read 3 Done Sent Done false 8131 Acked Write 5 Done Sent Done false 8132 Acked Read 5 Done Sent Done false 8133 Acked Read 3 Done Sent Done false 8134 Acked Read 10 Done Sent Done false 8135 Acked Read 9 Done Sent Done false 8136 Acked Read 2 Done Sent Done false 8137 Acked Write 1 Done Sent Done false 8138 Acked Write 9 Done Sent Done false 8139 Acked Write 1 Done Sent Done false 8140 Acked Read 9 Done Sent Done false 8141 Acked Write 10 Done Sent Done false 8142 Acked Read 5 Done Sent Done false 8143 Acked Read 5 Done Sent Done false 8144 Acked Write 4 Done Sent Done false 8145 Acked Write 7 Done Sent Done false 8146 Acked Flush 0 Done Sent Done false 8147 Acked Read 10 Done Sent Done false 8148 Acked Read 5 Done Sent Done false 8149 Acked Flush 0 Done Sent Done false 8150 Acked Read 2 Done Sent Done false 8151 Acked Flush 0 Done Sent Done false 8152 Acked Write 8 Done Sent Done false 8153 Acked Write 4 Done Sent Done false 8154 Acked Write 6 Done Sent Done false 8155 Acked Read 5 Done Sent Done false 8156 Acked Write 10 Done Sent Done false 8157 Acked Write 2 Done Sent Done false 8158 Acked Read 6 Done Sent Done false 8159 Acked Read 5 Done Sent Done false 8160 Acked Read 3 Done Sent Done false 8161 Acked Read 7 Done Sent Done false 8162 Acked Read 1 Done Sent Done false 8163 Acked Read 5 Done Sent Done false 8164 Acked Read 8 Done Sent Done false 8165 Acked Read 6 Done Sent Done false 8166 Acked Write 1 Done Sent Done false 8167 Acked Read 4 Done Sent Done false 8168 Acked Read 7 Done Sent Done false 8169 Acked Write 4 Done Sent Done false 8170 Acked Write 1 Done Sent Done false 8171 Acked Read 3 Done Sent Done false 8172 Acked Read 3 Done Sent Done false 8173 Acked Flush 0 Done Sent Done false 8174 Acked Read 9 Done Sent Done false 8175 Acked Read 3 Done Sent Done false 8176 Acked Write 2 Done Sent Done false 8177 Acked Write 10 Done Sent Done false 8178 Acked Write 6 Done Sent Done false 8179 Acked Write 6 Done Sent Done false 8180 Acked Read 9 Done Sent Done false 8181 Acked Flush 0 Done Sent Done false 8182 Acked Write 1 Done Sent Done false 8183 Acked Read 2 Done Sent Done false 8184 Acked Read 6 Done Sent Done false 8185 Acked Read 6 Done Sent Done false 8186 Acked Read 3 Done Sent Done false 8187 Acked Write 4 Done Sent Done false 8188 Acked Write 2 Done Sent Done false 8189 Acked Read 6 Done Sent Done false 8190 Acked Write 5 Done Sent Done false 8191 Acked Read 6 Done Sent Done false 8192 Acked Write 2 Done Sent Done false 8193 Acked Write 8 Done Sent Done false 8194 Acked Read 1 Done Sent Done false 8195 Acked Write 4 Done Sent Done false 8196 Acked Write 10 Done Sent Done false 8197 Acked Flush 0 Done Sent Done false 8198 Acked Read 7 Done Sent Done false 8199 Acked Read 10 Done Sent Done false 8200 Acked Write 10 Done Sent Done false 8201 Acked Read 8 Done Sent Done false 8202 Acked Write 1 Done Sent Done false 8203 Acked Write 10 Done Sent Done false 8204 Acked Write 7 Done Sent Done false 8205 Acked Write 4 Done Sent Done false 8206 Acked Read 7 Done Sent Done false 8207 Acked Write 10 Done Sent Done false 8208 Acked Write 6 Done Sent Done false 8209 Acked Write 1 Done Sent Done false 8210 Acked Write 7 Done Sent Done false 8211 Acked Read 7 Done Sent Done false 8212 Acked Flush 0 Done Sent Done false 8213 Acked Read 9 Done Sent Done false 8214 Acked Write 8 Done Sent Done false 8215 Acked Write 6 Done Sent Done false 8216 Acked Write 1 Done Sent Done false 8217 Acked Read 8 Done Sent Done false 8218 Acked Read 2 Done Sent Done false 8219 Acked Write 3 Done Sent Done false 8220 Acked Write 3 Done Sent Done false 8221 Acked Read 9 Done Sent Done false 8222 Acked Write 7 Done Sent Done false 8223 Acked Write 1 Done Sent Done false 8224 Acked Write 4 Done Sent Done false 8225 Acked Read 3 Done Sent Done false 8226 Acked Write 1 Done Sent Done false 8227 Acked Read 4 Done Sent Done false 8228 Acked Write 10 Done Sent Done false 8229 Acked Write 7 Done Sent Done false 8230 Acked Write 9 Done Sent Done false 8231 Acked Read 1 Done Sent Done false 8232 Acked Read 4 Done Sent Done false 8233 Acked Write 2 Done Sent Done false 8234 Acked Write 8 Done Sent Done false 8235 Acked Write 3 Done Sent Done false 8236 Acked Read 7 Done Sent Done false 8237 Acked Read 10 Done Sent Done false 8238 Acked Write 3 Done Sent Done false 8239 Acked Read 9 Done Sent Done false 8240 Acked Read 5 Done Sent Done false 8241 Acked Read 4 Done Sent Done false 8242 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8242 7941 8242 Downstairs last five completed: 7941 7940 7939 7938 7937 Upstairs last five completed: 8242 8241 8240 8239 8238 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1226) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 7942 Acked Read 3 Done Sent Done false 7943 Acked Write 4 Done Sent Done false 7944 Acked Write 1 Done Sent Done false 7945 Acked Read 10 Done Sent Done false 7946 Acked Write 10 Done Sent Done false 7947 Acked Flush 0 Done Sent Done false 7948 Acked Read 9 Done Sent Done false 7949 Acked Write 1 Done Sent Done false 7950 Acked Read 3 Done Sent Done false 7951 Acked Read 9 Done Sent Done false 7952 Acked Write 9 Done Sent Done false 7953 Acked Flush 0 Done Sent Done false 7954 Acked Read 6 Done Sent Done false 7955 Acked Write 3 Done Sent Done false 7956 Acked Write 6 Done Sent Done false 7957 Acked Write 10 Done Sent Done false 7958 Acked Write 5 Done Sent Done false 7959 Acked Write 1 Done Sent Done false 7960 Acked Write 3 Done Sent Done false 7961 Acked Read 10 Done Sent Done false 7962 Acked Read 2 Done Sent Done false 7963 Acked Read 10 Done Sent Done false 7964 Acked Read 8 Done Sent Done false 7965 Acked Read 1 Done Sent Done false 7966 Acked Write 3 Done Sent Done false 7967 Acked Read 4 Done Sent Done false 7968 Acked Write 10 Done Sent Done false 7969 Acked Write 10 Done Sent Done false 7970 Acked Read 4 Done Sent Done false 7971 Acked Write 8 Done Sent Done false 7972 Acked Read 8 Done Sent Done false 7973 Acked Read 6 Done Sent Done false 7974 Acked Read 4 Done Sent Done false 7975 Acked Write 5 Done Sent Done false 7976 Acked Write 2 Done Sent Done false 7977 Acked Write 1 Done Sent Done false 7978 Acked Write 4 Done Sent Done false 7979 Acked Read 2 Done Sent Done false 7980 Acked Read 2 Done Sent Done false 7981 Acked Read 10 Done Sent Done false 7982 Acked Read 1 Done Sent Done false 7983 Acked Write 9 Done Sent Done false 7984 Acked Flush 0 Done Sent Done false 7985 Acked Read 1 Done Sent Done false 7986 Acked Read 10 Done Sent Done false 7987 Acked Read 3 Done Sent Done false 7988 Acked Read 5 Done Sent Done false 7989 Acked Read 3 Done Sent Done false 7990 Acked Write 5 Done Sent Done false 7991 Acked Read 9 Done Sent Done false 7992 Acked Write 4 Done Sent Done false 7993 Acked Flush 0 Done Sent Done false 7994 Acked Read 3 Done Sent Done false 7995 Acked Write 7 Done Sent Done false 7996 Acked Write 6 Done Sent Done false 7997 Acked Write 1 Done Sent Done false 7998 Acked Read 9 Done Sent Done false 7999 Acked Read 7 Done Sent Done false 8000 Acked Read 9 Done Sent Done false 8001 Acked Read 3 Done Sent Done false 8002 Acked Write 1 Done Sent Done false 8003 Acked Write 7 Done Sent Done false 8004 Acked Write 1 Done Sent Done false 8005 Acked Read 2 Done Sent Done false 8006 Acked Read 10 Done Sent Done false 8007 Acked Flush 0 Done Sent Done false 8008 Acked Write 7 Done Sent Done false 8009 Acked Write 9 Done Sent Done false 8010 Acked Read 2 Done Sent Done false 8011 Acked Read 6 Done Sent Done false 8012 Acked Write 4 Done Sent Done false 8013 Acked Read 6 Done Sent Done false 8014 Acked Read 9 Done Sent Done false 8015 Acked Read 7 Done Sent Done false 8016 Acked Flush 0 Done Sent Done false 8017 Acked Write 6 Done Sent Done false 8018 Acked Read 9 Done Sent Done false 8019 Acked Write 1 Done Sent Done false 8020 Acked Read 1 Done Sent Done false 8021 Acked Read 4 Done Sent Done false 8022 Acked Write 8 Done Sent Done false 8023 Acked Read 8 Done Sent Done false 8024 Acked Write 7 Done Sent Done false 8025 Acked Read 8 Done Sent Done false 8026 Acked Write 1 Done Sent Done false 8027 Acked Flush 0 Done Sent Done false 8028 Acked Read 5 Done Sent Done false 8029 Acked Flush 0 Done Sent Done false 8030 Acked Read 7 Done Sent Done false 8031 Acked Write 3 Done Sent Done false 8032 Acked Read 3 Done Sent Done false 8033 Acked Read 3 Done Sent Done false 8034 Acked Read 10 Done Sent Done false 8035 Acked Write 7 Done Sent Done false 8036 Acked Write 9 Done Sent Done false 8037 Acked Write 5 Done Sent Done false 8038 Acked Flush 0 Done Sent Done false 8039 Acked Write 8 Done Sent Done false 8040 Acked Write 5 Done Sent Done false 8041 Acked Read 8 Done Sent Done false 8042 Acked Flush 0 Done Sent Done false 8043 Acked Write 6 Done Sent Done false 8044 Acked Read 2 Done Sent Done false 8045 Acked Write 7 Done Sent Done false 8046 Acked Flush 0 Done Sent Done false 8047 Acked Write 4 Done Sent Done false 8048 Acked Read 9 Done Sent Done false 8049 Acked Write 5 Done Sent Done false 8050 Acked Write 5 Done Sent Done false 8051 Acked Read 2 Done Sent Done false 8052 Acked Read 6 Done Sent Done false 8053 Acked Write 7 Done Sent Done false 8054 Acked Read 5 Done Sent Done false 8055 Acked Read 8 Done Sent Done false 8056 Acked Read 8 Done Sent Done false 8057 Acked Write 10 Done Sent Done false 8058 Acked Read 8 Done Sent Done false 8059 Acked Read 4 Done Sent Done false 8060 Acked Write 1 Done Sent Done false 8061 Acked Write 5 Done Sent Done false 8062 Acked Read 10 Done Sent Done false 8063 Acked Write 8 Done Sent Done false 8064 Acked Read 2 Done Sent Done false 8065 Acked Write 3 Done Sent Done false 8066 Acked Write 8 Done Sent Done false 8067 Acked Read 5 Done Sent Done false 8068 Acked Read 3 Done Sent Done false 8069 Acked Flush 0 Done Sent Done false 8070 Acked Write 3 Done Sent Done false 8071 Acked Write 10 Done Sent Done false 8072 Acked Read 10 Done Sent Done false 8073 Acked Write 4 Done Sent Done false 8074 Acked Write 6 Done Sent Done false 8075 Acked Write 4 Done Sent Done false 8076 Acked Read 2 Done Sent Done false 8077 Acked Read 9 Done Sent Done false 8078 Acked Read 1 Done Sent Done false 8079 Acked Read 9 Done Sent Done false 8080 Acked Read 5 Done Sent Done false 8081 Acked Read 3 Done Sent Done false 8082 Acked Flush 0 Done Sent Done false 8083 Acked Read 6 Done Sent Done false 8084 Acked Read 4 Done Sent Done false 8085 Acked Write 9 Done Sent Done false 8086 Acked Flush 0 Done Sent Done false 8087 Acked Read 6 Done Sent Done false 8088 Acked Write 10 Done Sent Done false 8089 Acked Read 10 Done Sent Done false 8090 Acked Read 4 Done Sent Done false 8091 Acked Read 4 Done Sent Done false 8092 Acked Write 1 Done Sent Done false 8093 Acked Read 7 Done Sent Done false 8094 Acked Read 3 Done Sent Done false 8095 Acked Write 2 Done Sent Done false 8096 Acked Write 4 Done Sent Done false 8097 Acked Read 9 Done Sent Done false 8098 Acked Write 5 Done Sent Done false 8099 Acked Read 1 Done Sent Done false 8100 Acked Write 7 Done Sent Done false 8101 Acked Write 7 Done Sent Done false 8102 Acked Read 8 Done Sent Done false 8103 Acked Flush 0 Done Sent Done false 8104 Acked Write 2 Done Sent Done false 8105 Acked Write 7 Done Sent Done false 8106 Acked Read 1 Done Sent Done false 8107 Acked Read 3 Done Sent Done false 8108 Acked Read 1 Done Sent Done false 8109 Acked Read 1 Done Sent Done false 8110 Acked Write 5 Done Sent Done false 8111 Acked Read 4 Done Sent Done false 8112 Acked Read 7 Done Sent Done false 8113 Acked Write 7 Done Sent Done false 8114 Acked Read 7 Done Sent Done false 8115 Acked Write 3 Done Sent Done false 8116 Acked Read 1 Done Sent Done false 8117 Acked Write 7 Done Sent Done false 8118 Acked Write 5 Done Sent Done false 8119 Acked Read 1 Done Sent Done false 8120 Acked Write 1 Done Sent Done false 8121 Acked Write 10 Done Sent Done false 8122 Acked Write 9 Done Sent Done false 8123 Acked Read 8 Done Sent Done false 8124 Acked Read 6 Done Sent Done false 8125 Acked Read 2 Done Sent Done false 8126 Acked Write 9 Done Sent Done false 8127 Acked Write 1 Done Sent Done false 8128 Acked Read 1 Done Sent Done false 8129 Acked Write 7 Done Sent Done false 8130 Acked Read 3 Done Sent Done false 8131 Acked Write 5 Done Sent Done false 8132 Acked Read 5 Done Sent Done false 8133 Acked Read 3 Done Sent Done false 8134 Acked Read 10 Done Sent Done false 8135 Acked Read 9 Done Sent Done false 8136 Acked Read 2 Done Sent Done false 8137 Acked Write 1 Done Sent Done false 8138 Acked Write 9 Done Sent Done false 8139 Acked Write 1 Done Sent Done false 8140 Acked Read 9 Done Sent Done false 8141 Acked Write 10 Done Sent Done false 8142 Acked Read 5 Done Sent Done false 8143 Acked Read 5 Done Sent Done false 8144 Acked Write 4 Done Sent Done false 8145 Acked Write 7 Done Sent Done false 8146 Acked Flush 0 Done Sent Done false 8147 Acked Read 10 Done Sent Done false 8148 Acked Read 5 Done Sent Done false 8149 Acked Flush 0 Done Sent Done false 8150 Acked Read 2 Done Sent Done false 8151 Acked Flush 0 Done Sent Done false 8152 Acked Write 8 Done Sent Done false 8153 Acked Write 4 Done Sent Done false 8154 Acked Write 6 Done Sent Done false 8155 Acked Read 5 Done Sent Done false 8156 Acked Write 10 Done Sent Done false 8157 Acked Write 2 Done Sent Done false 8158 Acked Read 6 Done Sent Done false 8159 Acked Read 5 Done Sent Done false 8160 Acked Read 3 Done Sent Done false 8161 Acked Read 7 Done Sent Done false 8162 Acked Read 1 Done Sent Done false 8163 Acked Read 5 Done Sent Done false 8164 Acked Read 8 Done Sent Done false 8165 Acked Read 6 Done Sent Done false 8166 Acked Write 1 Done Sent Done false 8167 Acked Read 4 Done Sent Done false 8168 Acked Read 7 Done Sent Done false 8169 Acked Write 4 Done Sent Done false 8170 Acked Write 1 Done Sent Done false 8171 Acked Read 3 Done Sent Done false 8172 Acked Read 3 Done Sent Done false 8173 Acked Flush 0 Done Sent Done false 8174 Acked Read 9 Done Sent Done false 8175 Acked Read 3 Done Sent Done false 8176 Acked Write 2 Done Sent Done false 8177 Acked Write 10 Done Sent Done false 8178 Acked Write 6 Done Sent Done false 8179 Acked Write 6 Done Sent Done false 8180 Acked Read 9 Done Sent Done false 8181 Acked Flush 0 Done Sent Done false 8182 Acked Write 1 Done Sent Done false 8183 Acked Read 2 Done Sent Done false 8184 Acked Read 6 Done Sent Done false 8185 Acked Read 6 Done Sent Done false 8186 Acked Read 3 Done Sent Done false 8187 Acked Write 4 Done Sent Done false 8188 Acked Write 2 Done Sent Done false 8189 Acked Read 6 Done Sent Done false 8190 Acked Write 5 Done Sent Done false 8191 Acked Read 6 Done Sent Done false 8192 Acked Write 2 Done Sent Done false 8193 Acked Write 8 Done Sent Done false 8194 Acked Read 1 Done Sent Done false 8195 Acked Write 4 Done Sent Done false 8196 Acked Write 10 Done Sent Done false 8197 Acked Flush 0 Done Sent Done false 8198 Acked Read 7 Done Sent Done false 8199 Acked Read 10 Done Sent Done false 8200 Acked Write 10 Done Sent Done false 8201 Acked Read 8 Done Sent Done false 8202 Acked Write 1 Done Sent Done false 8203 Acked Write 10 Done Sent Done false 8204 Acked Write 7 Done Sent Done false 8205 Acked Write 4 Done Sent Done false 8206 Acked Read 7 Done Sent Done false 8207 Acked Write 10 Done Sent Done false 8208 Acked Write 6 Done Sent Done false 8209 Acked Write 1 Done Sent Done false 8210 Acked Write 7 Done Sent Done false 8211 Acked Read 7 Done Sent Done false 8212 Acked Flush 0 Done Sent Done false 8213 Acked Read 9 Done Sent Done false 8214 Acked Write 8 Done Sent Done false 8215 Acked Write 6 Done Sent Done false 8216 Acked Write 1 Done Sent Done false 8217 Acked Read 8 Done Sent Done false 8218 Acked Read 2 Done Sent Done false 8219 Acked Write 3 Done Sent Done false 8220 Acked Write 3 Done Sent Done false 8221 Acked Read 9 Done Sent Done false 8222 Acked Write 7 Done Sent Done false 8223 Acked Write 1 Done Sent Done false 8224 Acked Write 4 Done Sent Done false 8225 Acked Read 3 Done Sent Done false 8226 Acked Write 1 Done Sent Done false 8227 Acked Read 4 Done Sent Done false 8228 Acked Write 10 Done Sent Done false 8229 Acked Write 7 Done Sent Done false 8230 Acked Write 9 Done Sent Done false 8231 Acked Read 1 Done Sent Done false 8232 Acked Read 4 Done Sent Done false 8233 Acked Write 2 Done Sent Done false 8234 Acked Write 8 Done Sent Done false 8235 Acked Write 3 Done Sent Done false 8236 Acked Read 7 Done Sent Done false 8237 Acked Read 10 Done Sent Done false 8238 Acked Write 3 Done Sent Done false 8239 Acked Read 9 Done Sent Done false 8240 Acked Read 5 Done Sent Done false 8241 Acked Read 4 Done Sent Done false 8242 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8242 7941 8242 Downstairs last five completed: 7941 7940 7939 7938 7937 Upstairs last five completed: 8242 8241 8240 8239 8238 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.261688407Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:62224)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.261759818Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:51.262677551Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263230323Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263252517Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 7941","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263267483Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 7941","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263485564Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263509238Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263523534Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 301 jobs since flush 7941","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.26353749Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 301 jobs since flush: 7941","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:51.263788963Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8242 8241 8240 8239 8238 Upstairs last five completed: 8242 8241 8240 8239 8238 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:47:53.493 INFO request completed, latency_us: 362, response_code[8830] Got stop action so:false kr:false : 204, uri: /stop/cid/2, method: POST, req_id: 7350eac6-9b58-432a-99c0-1903635d4257, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:47:53.494 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8384d9d3-20e5-43ce-843a-0ff9bf5d35d9, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:53.494468259Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:53.49454069Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:47:53.494560034Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:53.494573971Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:53.494588017Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:47:53.494657648Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:47:57.497 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: 66dd8d26-16e7-494e-b148-cef5899e4003, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 813 len 4096 002/300 Read block 4164 len 40960 003/300 Write block 510 len 32768 data: 8 7 8 5 5 5 6 5 004/300 Read block 2375 len 40960 005/300 Read block 2258 len 8192 006/300 Write block 3523 len 20480 data: 6 7 5 4 4 007/300 Read block 791 len 16384 008/300 Read block 4346 len 28672 009/300 Read block 1179 len 40960 010/300 Write block 4239 len 20480 data: 4 3 4 4 4 011/300 Flush 012/300 Flush 013/300 Read block 1583 len 8192 014/300 Write block 1630 len 4096 data: 8 015/300 Flush 016/300 Write block 2255 len 20480 data: 4 6 6 6 7 017/300 Read block 4885 len 40960 018/300 Read block 406 len 36864 019/300 Read block 4647 len 8192 020/300 Flush 021/300 Write block 2705 len 8192 data: 4 3 022/300 Read block 3640 len 16384 023/300 Write block 2738 len 12288 data: 8 8 9 024/300 Read block 3420 len 12288 025/300 Write block 4200 len 8192 data: 7 5 026/300 Read block 2098 len 36864 027/300 Write block 2109 len 36864 data: 5 5 4 6 7 8 7 6 6 028/300 Write block 760 len 12288 data: 4 4 4 029/300 Write block 3335 len 4096 data: 5 030/300 Write block 1791 len 28672 data: 3 5 7 7 5 4 4 031/300 Read block 2595 len 20480 032/300 Read block 1953 len 40960 033/300 Read block 2510 len 16384 034/300 Write block 1291 len 8192 data: 4 6 035/300 Read block 2562 len 8192 036/300 Read block 560 len 32768 037/300 Write block 1181 len 36864 data: 10 8 6 5 5 3 3 3 3 038/300 Read block 4649 len 40960 039/300 Write block 374 len 40960 data: 5 6 4 5 5 5 4 4 2 2 040/300 Flush 041/300 Read block 439 len 8192 042/300 Write block 1190 len 12288 data: 4 4 4 043/300 Read block 1185 len 28672 044/300 Read block 3945 len 40960 045/300 Read block 2993 len 4096 046/300 Read block 2251 len 16384 047/300 Write block 4922 len 4096 data: 6 048/300 Read block 2411 len 16384 049/300 Read block 3424 len 20480 050/300 Write block 734 len 4096 data: 3 051/300 Read block 478 len 12288 052/300 Flush 053/300 Read block 1719 len 36864 054/300 Read block 3584 len 32768 055/300 Write block 378 len 40960 data: 6 6 5 5 3 3 3 3 3 3 056/300 Read block 4346 len 8192 057/300 Write block 561 len 40960 data: 3 3 3 4 4 4 4 4 5 5 058/300 Read block 4760 len 8192 059/300 Flush 060/300 Write block 2386 len 16384 data: 8 8 9 9 061/300 Read block 1052 len 28672 062/300 Read block 3315 len 28672 063/300 Read block 760 len 16384 064/300 Write block 119 len 12288 data: 6 4 5 065/300 Write block 2899 len 8192 data: 6 5 066/300 Write block 945 len 20480 data: 5 6 5 5 4 067/300 Write block 1122 len 40960 data: 4 5 4 3 3 3 3 4 4 4 068/300 Write block 1724 len 28672 data: 4 5 5 5 4 4 4 069/300 Write block 2471 len 12288 data: 7 4 4 070/300 Write block 468 len 32768 data: 7 9 9 7 6 5 3 4 071/300 Flush 072/300 Write block 4332 len 36864 data: 6 7 6 6 6 4 4 4 4 073/300 Write block 4192 len 4096 data: 5 074/300 Read block 3985 len 8192 075/300 Read block 1017 len 16384 076/300 Flush 077/300 Write block 1145 len 8192 data: 4 3 078/300 Read block 104 len 4096 079/300 Flush 080/300 Read block 4777 len 40960 081/300 Write block 4275 len 20480 data: 6 6 5 6 6 082/300 Write block 1492 len 8192 data: 3 5 083/300 Flush 084/300 Write block 2945 len 16384 data: 3 5 5 5 085/300 Read block 3376 len 16384 086/300 Read block 2496 len 8192 087/300 Write block 4801 len 4096 data: 8 088/300 Read block 4395 len 4096 089/300 Write block 4382 len 32768 data: 4 4 4 7 7 7 9 12 090/300 Write block 4792 len 32768 data: 6 8 9 10 10 9 11 10 091/300 Read block 2003 len 8192 092/300 Read block 2993 len 4096 093/300 Read block 1467 len 20480 094/300 Read block 3057 len 16384 095/300 Read block 2568 len 32768 096/300 Write block 2427 len 28672 data: 4 5 4 4 3 3 3 097/300 Read block 3713 len 32768 098/300 Flush 099/300 Read block 95 len 4096 100/300 Read block 3418 len 36864 101/300 Write block 4027 len 28672 data: 4 4 5 7 8 8 6 102/300 Write block 3726 len 8192 data: 6 5 103/300 Read block 2616 len 16384 104/300 Write block 197 len 12288 data: 4 4 3 105/300 Read block 1943 len 24576 106/300 Write block 1360 len 4096 data: 3 107/300 Read block 4178 len 4096 108/300 Read block 2519 len 12288 109/300 Read block 3324 len 28672 110/300 Read block 1707 len 12288 111/300 Read block 3511 len 12288 112/300 Write block 4328 len 12288 data: 5 6 4 113/300 Read block 2463 len 8192 114/300 Read block 3081 len 36864 115/300 Read block 826 len 40960 116/300 Read block 320 len 4096 117/300 Read block 2854 len 36864 118/300 Write block 4917 len 24576 data: 2 2 2 3 5 7 119/300 Write block 3176 len 12288 data: 4 4 5 120/300 Read block 3563 len 16384 121/300 Write block 4652 len 32768 data: 3 4 5 7 7 8 8 8 122/300 Write block 3831 len 24576 data: 7 7 6 5 5 5 123/300 Write block 579 len 8192 data: 9 9 124/300 Write block 1510 len 32768 data: 5 4 4 4 5 4 4 5 125/300 Flush 126/300 Read block 4224 len 8192 127/300 Write block 4394 len 16384 data: 6 6 6 5 128/300 Read block 940 len 20480 129/300 Read block 694 len 32768 130/300 Write block 3674 len 4096 data: 5 131/300 Write block 4393 len 36864 data: 8 7 7 7 6 5 4 5 4 132/300 Read block 2790 len 16384 133/300 Read block 4424 len 8192 134/300 Write block 1483 len 28672 data: 6 6 4 6 6 5 5 135/300 Read block 2853 len 8192 136/300 Read block 1810 len 20480 137/300 Read block 3840 len 28672 138/300 Read block 275 len 28672 139/300 Read block 842 len 20480 140/300 Read block 163 len 40960 141/300 Flush 142/300 Read block 2546 len 24576 143/300 Read block 1234 len 36864 144/300 Write block 60 len 16384 data: 2 2 2 2 145/300 Write block 2261 len 4096 data: 7 146/300 Flush 147/300 Write block 3220 len 4096 data: 6 148/300 Flush 149/300 Read block 3890 len 16384 150/300 Read block 4417 len 20480 151/300 Read block 4492 len 4096 152/300 Write block 4936 len 40960 data: 5 6 9 7 7 9 9 10 7 6 153/300 Write block 4667 len 20480 data: 7 7 7 8 6 154/300 Read block 1967 len 32768 155/300 Write block 679 len 36864 data: 9 7 7 6 5 5 5 4 4 156/300 Write block 3400 len 16384 data: 6 5 6 6 157/300 Write block 1226 len 20480 data: 7 5 5 7 8 158/300 Write block 4247 len 16384 data: 2 2 3 6 159/300 Write block 887 len 40960 data: 7 6 8 9 8 8 8 6 5 4 160/300 Read block 1902 len 24576 161/300 Read block 3940 len 20480 162/300 Read block 4575 len 20480 163/300 Read block 21 len 12288 164/300 Write block 3814 len 4096 data: 6 165/300 Write block 4744 len 16384 data: 8 8 8 8 166/300 Flush 167/300 Write block 3807 len 20480 data: 7 7 8 6 6 168/300 Read block 4380 len 4096 169/300 Write block 2945 len 32768 data: 4 6 6 6 5 4 5 4 170/300 Write block 3541 len 24576 data: 9 7 6 6 6 4 171/300 Read block 2722 len 24576 172/300 Read block 4033 len 28672 173/300 Write block 2710 len 16384 data: 4 4 4 5 174/300 Read block 3471 len 16384 175/300 Read block 3668 len 4096 176/300 Flush 177/300 Write block 4930 len 36864 data: 8 9 9 8 6 6 6 7 10 178/300 Read block 1217 len 32768 179/300 Write block 2719 len 4096 data: 6 180/300 Read block 3149 len 16384 181/300 Read block 2214 len 12288 182/300 Write block 2464 len 20480 data: 6 6 6 6 6 183/300 Write block 38 len 36864 data: 4 3 3 3 3 3 3 3 3 184/300 Read block 4734 len 36864 185/300 Read block 2110 len 36864 186/300 Write block 3799 len 4096 data: 6 187/300 Write block 851 len 32768 data: 3 3 3 4 5 3 4 4 188/300 Read block 2545 len 16384 189/300 Read block 2010 len 8192 190/300 Write block 308 len 40960 data: 4 5 5 7 6 7 7 7 7 6 191/300 Read block 3930 len 20480 192/300 Read block 1203 len 24576 193/300 Write block 1202 len 28672 data: 7 8 8 7 6 7 6 194/300 Read block 4795 len 20480 195/300 Read block 4220 len 4096 196/300 Write block 2231 len 32768 data: 2 3 3 2 2 3 4 4 197/300 Read block 4717 len 40960 198/300 Flush 199/300 Read block 2071 len 16384 200/300 Write block 1187 len 40960 data: 4 4 4 5 5 5 4 4 5 6 201/300 Read block 2630 len 12288 202/300 Read block 1572 len 12288 203/300 Read block 2252 len 4096 204/300 Flush 205/300 Read block 3095 len 32768 206/300 Read block 4592 len 12288 207/300 Read block 3825 len 4096 208/300 Read block 707 len 32768 209/300 Write block 760 len 32768 data: 5 5 5 4 4 3 3 3 210/300 Flush 211/300 Read block 4465 len 4096 212/300 Flush 213/300 Read block 4008 len 16384 214/300 Write block 4126 len 16384 data: 7 7 8 8 215/300 Read block 383 len 36864 216/300 Read block 864 len 8192 217/300 Read block 3630 len 20480 218/300 Read block 2335 len 12288 219/300 Write block 4263 len 16384 data: 4 5 6 5 220/300 Flush 221/300 Read block 1991 len 36864 222/300 Read block 4458 len 12288 223/300 Read block 2488 len 40960 224/300 Read block 3107 len 40960 225/300 Read block 3768 len 12288 226/300 Read block 1421 len 4096 227/300 Read block 564 len 28672 228/300 Read block 3969 len 20480 229/300 Write block 4357 len 28672 data: 9 8 7 6 5 8 8 230/300 Flush 231/300 Read block 4250 len 20480 232/300 Read block 3734 len 20480 233/300 Write block 4775 len 20480 data: 5 3 3 4 5 234/300 Read block 216 len 24576 235/300 Write block 1869 len 24576 data: 7 7 8 8 8 5 236/300 Read block 2823 len 4096 237/300 Flush 238/300 Write block 4532 len 8192 data: 7 5 239/300 Write block 1333 len 4096 data: 4 240/300 Read block 4656 len 16384 241/300 Read block 1977 len 16384 242/300 Read block 1400 len 40960 243/300 Write block 3663 len 16384 data: 6 7 6 5 244/300 Read block 4710 len 8192 245/300 Write block 248 len 24576 data: 3 3 3 4 4 5 246/300 Write block 4836 len 40960 data: 6 6 5 6 8 7 8 7 8 10 247/300 Write block 4108 len 24576 data: 3 3 3 4 3 3 248/300 Write block 3310 len 16384 data: 3 2 3 5 249/300 Write block 368 len 28672 data: 7 7 9 8 9 6 6 250/300 Write block 1807 len 28672 data: 6 6 6 5 5 5 5 251/300 Read block 229 len 12288 252/300 Write block 3012 len 28672 data: 5 5 5 5 5 4 4 253/300 Flush 254/300 Read block 3336 len 8192 255/300 Write block 2019 len 4096 data: 5 256/300 Read block 2021 len 36864 257/300 Read block 3926 len 40960 258/300 Read block 61 len 4096 259/300 Write block 1126 len 36864 data: 4 4 4 5 5 5 4 4 4 260/300 Read block 4834 len 36864 261/300 Read block 3170 len 20480 262/300 Write block 1316 len 28672 data: 8 7 7 4 6 4 3 263/300 Flush 264/300 Flush 265/300 Flush 266/300 Write block 844 len 32768 data: 3 3 3 4 3 4 3 4 267/300 Read block 2529 len 28672 268/300 Read block 1715 len 8192 269/300 Read block 4186 len 12288 270/300 Read block 4962 len 4096 271/300 Read block 4516 len 12288 272/300 Write block 3673 len 28672 data: 5 6 5 5 6 6 7 273/300 Write block 4607 len 16384 data: 6 4 5 5 274/300 Write block 2048 len 24576 data: 4 4 4 6 7 7 275/300 Write block 4658 len 20480 data: 9 9 7 5 7 276/300 Write block 4594 len 20480 data: 6 7 5 4 4 277/300 Write block 3307 len 20480 data: 4 4 4 4 3 278/300 Write block 4830 len 24576 data: 7 5 5 6 5 4 279/300 Read block 4017 len 32768 280/300 Read block 844 len 20480 281/300 Write block 4888 len 24576 data: 7 8 7 6 6 5 282/300 Read block 4552 len 12288 283/300 Read block 4931 len 4096 284/300 Flush 285/300 Read block 2544 len 28672 286/300 Write block 1587 len 12288 data: 8 8 7 287/300 Read block 1616 len 28672 288/300 Read block 1352 len 40960 289/300 Read block 3807 len 24576 290/300 Read block 866 len 32768 291/300 Read block 1130 len 4096 292/300 Read block 2348 len 12288 293/300 Read block 884 len 40960 294/300 Read block 4506 len 40960 295/300 Read block 4863 len 24576 296/300 Read block 2698 len 8192 297/300 Write block 3546 len 36864 data: 5 5 5 7 6 7 8 7 8 298/300 Write block 1380 len 32768 data: 4 3 2 2 2 4 4 6 299/300 Write block 4917 len 20480 data: 3 3 3 4 6 300/300 Read block 1700 len 24576 301/300 Read block 3975 len 20480 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:47:57.726 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6d3719d7-42dd-42ae-9f11-7963805a600d, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8243 Acked Read 1 Done Done Sent false 8244 Acked Read 10 Done Done Sent false 8245 Acked Write 8 Done Done Sent false 8246 Acked Read 10 Done Done Sent false 8247 Acked Read 2 Done Done Sent false 8248 Acked Write 5 Done Done Sent false 8249 Acked Read 4 Done Done Sent false 8250 Acked Read 7 Done Done Sent false 8251 Acked Read 10 Done Done Sent false 8252 Acked Write 5 Done Done Sent false 8253 Acked Flush 0 Done Done Sent false 8254 Acked Flush 0 Done Done Sent false 8255 Acked Read 2 Done Done Sent false 8256 Acked Write 1 Done Done Sent false 8257 Acked Flush 0 Done Done Sent false 8258 Acked Write 5 Done Done Sent false 8259 Acked Read 10 Done Done Sent false 8260 Acked Read 9 Done Done Sent false 8261 Acked Read 2 Done Done Sent false 8262 Acked Flush 0 Done Done Sent false 8263 Acked Write 2 Done Done Sent false 8264 Acked Read 4 Done Done Sent false 8265 Acked Write 3 Done Done Sent false 8266 Acked Read 3 Done Done Sent false 8267 Acked Write 2 Done Done Sent false 8268 Acked Read 9 Done Done Sent false 8269 Acked Write 9 Done Done Sent false 8270 Acked Write 3 Done Done Sent false 8271 Acked Write 1 Done Done Sent false 8272 Acked Write 7 Done Done Sent false 8273 Acked Read 5 Done Done Sent false 8274 Acked Read 10 Done Done Sent false 8275 Acked Read 4 Done Done Sent false 8276 Acked Write 2 Done Done Sent false 8277 Acked Read 2 Done Done Sent false 8278 Acked Read 8 Done Done Sent false 8279 Acked Write 9 Done Done Sent false 8280 Acked Read 10 Done Done Sent false 8281 Acked Write 10 Done Done Sent false 8282 Acked Flush 0 Done Done Sent false 8283 Acked Read 2 Done Done Sent false 8284 Acked Write 3 Done Done Sent false 8285 Acked Read 7 Done Done Sent false 8286 Acked Read 10 Done Done Sent false 8287 Acked Read 1 Done Done Sent false 8288 Acked Read 4 Done Done Sent false 8289 Acked Write 1 Done Done Sent false 8290 Acked Read 4 Done Done Sent false 8291 Acked Read 5 Done Done Sent false 8292 Acked Write 1 Done Done Sent false 8293 Acked Read 3 Done Done Sent false 8294 Acked Flush 0 Done Done Sent false 8295 Acked Read 9 Done Done Sent false 8296 Acked Read 8 Done Done Sent false 8297 Acked Write 10 Done Done Sent false 8298 Acked Read 2 Done Done Sent false 8299 Acked Write 10 Done Done Sent false 8300 Acked Read 2 Done Done Sent false 8301 Acked Flush 0 Done Done Sent false 8302 Acked Write 4 Done Done Sent false 8303 Acked Read 7 Done Done Sent false 8304 Acked Read 7 Done Done Sent false 8305 Acked Read 4 Done Done Sent false 8306 Acked Write 3 Done Done Sent false 8307 Acked Write 2 Done Done Sent false 8308 Acked Write 5 Done Done Sent false 8309 Acked Write 10 Done Done Sent false 8310 Acked Write 7 Done Done Sent false 8311 Acked Write 3 Done Done Sent false 8312 Acked Write 8 Done Done Sent false 8313 Acked Flush 0 Done Done Sent false 8314 Acked Write 9 Done Done Sent false 8315 Acked Write 1 Done Done Sent false 8316 Acked Read 2 Done Done Sent false 8317 Acked Read 4 Done Done Sent false 8318 Acked Flush 0 Done Done Sent false 8319 Acked Write 2 Done Done Sent false 8320 Acked Read 1 Done Done Sent false 8321 Acked Flush 0 Done Done Sent false 8322 Acked Read 10 Done Done Sent false 8323 Acked Write 5 Done Done Sent false 8324 Acked Write 2 Done Done Sent false 8325 Acked Flush 0 Done Done Sent false 8326 Acked Write 4 Done Done Sent false 8327 Acked Read 4 Done Done Sent false 8328 Acked Read 2 Done Done Sent false 8329 Acked Write 1 Done Done Sent false 8330 Acked Read 1 Done Done Sent false 8331 Acked Write 8 Done Done Sent false 8332 Acked Write 8 Done Done Sent false 8333 Acked Read 2 Done Done Sent false 8334 Acked Read 1 Done Done Sent false 8335 Acked Read 5 Done Done Sent false 8336 Acked Read 4 Done Done Sent false 8337 Acked Read 8 Done Done Sent false 8338 Acked Write 7 Done Done Sent false 8339 Acked Read 8 Done Done Sent false 8340 Acked Flush 0 Done Done Sent false 8341 Acked Read 1 Done Done Sent false 8342 Acked Read 9 Done Done Sent false 8343 Acked Write 7 Done Done Sent false 8344 Acked Write 2 Done Done Sent false 8345 Acked Read 4 Done Done Sent false 8346 Acked Write 3 Done Done Sent false 8347 Acked Read 6 Done Done Sent false 8348 Acked Write 1 Done Done Sent false 8349 Acked Read 1 Done Done Sent false 8350 Acked Read 3 Done Done Sent false 8351 Acked Read 7 Done Done Sent false 8352 Acked Read 3 Done Done Sent false 8353 Acked Read 3 Done Done Sent false 8354 Acked Write 3 Done Done Sent false 8355 Acked Read 2 Done Done Sent false 8356 Acked Read 9 Done Done Sent false 8357 Acked Read 10 Done Done Sent false 8358 Acked Read 1 Done Done Sent false 8359 Acked Read 9 Done Done Sent false 8360 Acked Write 6 Done Done Sent false 8361 Acked Write 3 Done Done Sent false 8362 Acked Read 4 Done Done Sent false 8363 Acked Write 8 Done Done Sent false 8364 Acked Write 6 Done Done Sent false 8365 Acked Write 2 Done Done Sent false 8366 Acked Write 8 Done Done Sent false 8367 Acked Flush 0 Done Done Sent false 8368 Acked Read 2 Done Done Sent false 8369 Acked Write 4 Done Done Sent false 8370 Acked Read 5 Done Done Sent false 8371 Acked Read 8 Done Done Sent false 8372 Acked Write 1 Done Done Sent false 8373 Acked Write 9 Done Done Sent false 8374 Acked Read 4 Done Done Sent false 8375 Acked Read 2 Done Done Sent false 8376 Acked Write 7 Done Done Sent false 8377 Acked Read 2 Done Done Sent false 8378 Acked Read 5 Done Done Sent false 8379 Acked Read 7 Done Done Sent false 8380 Acked Read 7 Done Done Sent false 8381 Acked Read 5 Done Done Sent false 8382 Acked Read 10 Done Done Sent false 8383 Acked Flush 0 Done Done Sent false 8384 Acked Read 6 Done Done Sent false 8385 Acked Read 9 Done Done Sent false 8386 Acked Write 4 Done Done Sent false 8387 Acked Write 1 Done Done Sent false 8388 Acked Flush 0 Done Done Sent false 8389 Acked Write 1 Done Done Sent false 8390 Acked Flush 0 Done Done Sent false 8391 Acked Read 4 Done Done Sent false 8392 Acked Read 5 Done Done Sent false 8393 Acked Read 1 Done Done Sent false 8394 Acked Write 10 Done Done Sent false 8395 Acked Write 5 Done Done Sent false 8396 Acked Read 8 Done Done Sent false 8397 Acked Write 9 Done Done Sent false 8398 Acked Write 4 Done Done Sent false 8399 Acked Write 5 Done Done Sent false 8400 Acked Write 4 Done Done Sent false 8401 Acked Write 10 Done Done Sent false 8402 Acked Read 6 Done Done Sent false 8403 Acked Read 5 Done Done Sent false 8404 Acked Read 5 Done Done Sent false 8405 Acked Read 3 Done Done Sent false 8406 Acked Write 1 Done Done Sent false 8407 Acked Write 4 Done Done Sent false 8408 Acked Flush 0 Done Done Sent false 8409 Acked Write 5 Done Done Sent false 8410 Acked Read 1 Done Done Sent false 8411 Acked Write 8 Done Done Sent false 8412 Acked Write 6 Done Done Sent false 8413 Acked Read 6 Done Done Sent false 8414 Acked Read 7 Done Done Sent false 8415 Acked Write 4 Done Done Sent false 8416 Acked Read 4 Done Done Sent false 8417 Acked Read 1 Done Done Sent false 8418 Acked Flush 0 Done Done Sent false 8419 Acked Write 9 Done Done Sent false 8420 Acked Read 8 Done Done Sent false 8421 Acked Write 1 Done Done Sent false 8422 Acked Read 4 Done Done Sent false 8423 Acked Read 3 Done Done Sent false 8424 Acked Write 5 Done Done Sent false 8425 Acked Write 9 Done Done Sent false 8426 Acked Read 9 Done Done Sent false 8427 Acked Read 9 Done Done Sent false 8428 Acked Write 1 Done Done Sent false 8429 Acked Write 8 Done Done Sent false 8430 Acked Read 4 Done Done Sent false 8431 Acked Read 2 Done Done Sent false 8432 Acked Write 10 Done Done Sent false 8433 Acked Read 5 Done Done Sent false 8434 Acked Read 6 Done Done Sent false 8435 Acked Write 7 Done Done Sent false 8436 Acked Read 5 Done Done Sent false 8437 Acked Read 1 Done Done Sent false 8438 Acked Write 8 Done Done Sent false 8439 Acked Read 10 Done Done Sent false 8440 Acked Flush 0 Done Done Sent false 8441 Acked Read 4 Done Done Sent false 8442 Acked Write 10 Done Done Sent false 8443 Acked Read 3 Done Done Sent false 8444 Acked Read 3 Done Done Sent false 8445 Acked Read 1 Done Done Sent false 8446 Acked Flush 0 Done Done Sent false 8447 Acked Read 8 Done Done Sent false 8448 Acked Read 3 Done Done Sent false 8449 Acked Read 1 Done Done Sent false 8450 Acked Read 8 Done Done Sent false 8451 Acked Write 8 Done Done Sent false 8452 Acked Flush 0 Done Done Sent false 8453 Acked Read 1 Done Done Sent false 8454 Acked Flush 0 Done Done Sent false 8455 Acked Read 4 Done Done Sent false 8456 Acked Write 4 Done Done Sent false 8457 Acked Read 9 Done Done Sent false 8458 Acked Read 2 Done Done Sent false 8459 Acked Read 5 Done Done Sent false 8460 Acked Read 3 Done Done Sent false 8461 Acked Write 4 Done Done Sent false 8462 Acked Flush 0 Done Done Sent false 8463 Acked Read 9 Done Done Sent false 8464 Acked Read 3 Done Done Sent false 8465 Acked Read 10 Done Done Sent false 8466 Acked Read 10 Done Done Sent false 8467 Acked Read 3 Done Done Sent false 8468 Acked Read 1 Done Done Sent false 8469 Acked Read 7 Done Done Sent false 8470 Acked Read 5 Done Done Sent false 8471 Acked Write 7 Done Done Sent false 8472 Acked Flush 0 Done Done Sent false 8473 Acked Read 5 Done Done Sent false 8474 Acked Read 5 Done Done Sent false 8475 Acked Write 5 Done Done Sent false 8476 Acked Read 6 Done Done Sent false 8477 Acked Write 6 Done Done Sent false 8478 Acked Read 1 Done Done Sent false 8479 Acked Flush 0 Done Done Sent false 8480 Acked Write 2 Done Done Sent false 8481 Acked Write 1 Done Done Sent false 8482 Acked Read 4 Done Done Sent false 8483 Acked Read 4 Done Done Sent false 8484 Acked Read 10 Done Done Sent false 8485 Acked Write 4 Done Done Sent false 8486 Acked Read 2 Done Done Sent false 8487 Acked Write 6 Done Done Sent false 8488 Acked Write 10 Done Done Sent false 8489 Acked Write 6 Done Done Sent false 8490 Acked Write 4 Done Done Sent false 8491 Acked Write 7 Done Done Sent false 8492 Acked Write 7 Done Done Sent false 8493 Acked Read 3 Done Done Sent false 8494 Acked Write 7 Done Done Sent false 8495 Acked Flush 0 Done Done Sent false 8496 Acked Read 2 Done Done Sent false 8497 Acked Write 1 Done Done Sent false 8498 Acked Read 9 Done Done Sent false 8499 Acked Read 10 Done Done Sent false 8500 Acked Read 1 Done Done Sent false 8501 Acked Write 9 Done Done Sent false 8502 Acked Read 9 Done Done Sent false 8503 Acked Read 5 Done Done Sent false 8504 Acked Write 7 Done Done Sent false 8505 Acked Flush 0 Done Done Sent false 8506 Acked Flush 0 Done Done Sent false 8507 Acked Flush 0 Done Done Sent false 8508 Acked Write 8 Done Done Sent false 8509 Acked Read 7 Done Done Sent false 8510 Acked Read 2 Done Done Sent false 8511 Acked Read 3 Done Done Sent false 8512 Acked Read 1 Done Done Sent false 8513 Acked Read 3 Done Done Sent false 8514 Acked Write 7 Done Done Sent false 8515 Acked Write 4 Done Done Sent false 8516 Acked Write 6 Done Done Sent false 8517 Acked Write 5 Done Done Sent false 8518 Acked Write 5 Done Done Sent false 8519 Acked Write 5 Done Done Sent false 8520 Acked Write 6 Done Done Sent false 8521 Acked Read 8 Done Done Sent false 8522 Acked Read 5 Done Done Sent false 8523 Acked Write 6 Done Done Sent false 8524 Acked Read 3 Done Done Sent false 8525 Acked Read 1 Done Done Sent false 8526 Acked Flush 0 Done Done Sent false 8527 Acked Read 7 Done Done Sent false 8528 Acked Write 3 Done Done Sent false 8529 Acked Read 7 Done Done Sent false 8530 Acked Read 10 Done Done Sent false 8531 Acked Read 6 Done Done Sent false 8532 Acked Read 8 Done Done Sent false 8533 Acked Read 1 Done Done Sent false 8534 Acked Read 3 Done Done Sent false 8535 Acked Read 10 Done Done Sent false 8536 Acked Read 10 Done Done Sent false 8537 Acked Read 6 Done Done Sent false 8538 Acked Read 2 Done Done Sent false 8539 Acked Write 9 Done Done Sent false 8540 Acked Write 8 Done Done Sent false 8541 Acked Write 5 Done Done Sent false 8542 Acked Read 6 Done Done Sent false 8543 Acked Read 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8526 8526 8242 Downstairs last five completed: 8242 8241 8240 8239 8238 Upstairs last five completed: 8543 8542 8541 8540 8539 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8243 Acked Read 1 Done Done Sent false 8244 Acked Read 10 Done Done Sent false 8245 Acked Write 8 Done Done Sent false 8246 Acked Read 10 Done Done Sent false 8247 Acked Read 2 Done Done Sent false 8248 Acked Write 5 Done Done Sent false 8249 Acked Read 4 Done Done Sent false 8250 Acked Read 7 Done Done Sent false 8251 Acked Read 10 Done Done Sent false 8252 Acked Write 5 Done Done Sent false 8253 Acked Flush 0 Done Done Sent false 8254 Acked Flush 0 Done Done Sent false 8255 Acked Read 2 Done Done Sent false 8256 Acked Write 1 Done Done Sent false 8257 Acked Flush 0 Done Done Sent false 8258 Acked Write 5 Done Done Sent false 8259 Acked Read 10 Done Done Sent false 8260 Acked Read 9 Done Done Sent false 8261 Acked Read 2 Done Done Sent false 8262 Acked Flush 0 Done Done Sent false 8263 Acked Write 2 Done Done Sent false 8264 Acked Read 4 Done Done Sent false 8265 Acked Write 3 Done Done Sent false 8266 Acked Read 3 Done Done Sent false 8267 Acked Write 2 Done Done Sent false 8268 Acked Read 9 Done Done Sent false 8269 Acked Write 9 Done Done Sent false 8270 Acked Write 3 Done Done Sent false 8271 Acked Write 1 Done Done Sent false 8272 Acked Write 7 Done Done Sent false 8273 Acked Read 5 Done Done Sent false 8274 Acked Read 10 Done Done Sent false 8275 Acked Read 4 Done Done Sent false 8276 Acked Write 2 Done Done Sent false 8277 Acked Read 2 Done Done Sent false 8278 Acked Read 8 Done Done Sent false 8279 Acked Write 9 Done Done Sent false 8280 Acked Read 10 Done Done Sent false 8281 Acked Write 10 Done Done Sent false 8282 Acked Flush 0 Done Done Sent false 8283 Acked Read 2 Done Done Sent false 8284 Acked Write 3 Done Done Sent false 8285 Acked Read 7 Done Done Sent false 8286 Acked Read 10 Done Done Sent false 8287 Acked Read 1 Done Done Sent false 8288 Acked Read 4 Done Done Sent false 8289 Acked Write 1 Done Done Sent false 8290 Acked Read 4 Done Done Sent false 8291 Acked Read 5 Done Done Sent false 8292 Acked Write 1 Done Done Sent false 8293 Acked Read 3 Done Done Sent false 8294 Acked Flush 0 Done Done Sent false 8295 Acked Read 9 Done Done Sent false 8296 Acked Read 8 Done Done Sent false 8297 Acked Write 10 Done Done Sent false 8298 Acked Read 2 Done Done Sent false 8299 Acked Write 10 Done Done Sent false 8300 Acked Read 2 Done Done Sent false 8301 Acked Flush 0 Done Done Sent false 8302 Acked Write 4 Done Done Sent false 8303 Acked Read 7 Done Done Sent false 8304 Acked Read 7 Done Done Sent false 8305 Acked Read 4 Done Done Sent false 8306 Acked Write 3 Done Done Sent false 8307 Acked Write 2 Done Done Sent false 8308 Acked Write 5 Done Done Sent false 8309 Acked Write 10 Done Done Sent false 8310 Acked Write 7 Done Done Sent false 8311 Acked Write 3 Done Done Sent false 8312 Acked Write 8 Done Done Sent false 8313 Acked Flush 0 Done Done Sent false 8314 Acked Write 9 Done Done Sent false 8315 Acked Write 1 Done Done Sent false 8316 Acked Read 2 Done Done Sent false 8317 Acked Read 4 Done Done Sent false 8318 Acked Flush 0 Done Done Sent false 8319 Acked Write 2 Done Done Sent false 8320 Acked Read 1 Done Done Sent false 8321 Acked Flush 0 Done Done Sent false 8322 Acked Read 10 Done Done Sent false 8323 Acked Write 5 Done Done Sent false 8324 Acked Write 2 Done Done Sent false 8325 Acked Flush 0 Done Done Sent false 8326 Acked Write 4 Done Done Sent false 8327 Acked Read 4 Done Done Sent false 8328 Acked Read 2 Done Done Sent false 8329 Acked Write 1 Done Done Sent false 8330 Acked Read 1 Done Done Sent false 8331 Acked Write 8 Done Done Sent false 8332 Acked Write 8 Done Done Sent false 8333 Acked Read 2 Done Done Sent false 8334 Acked Read 1 Done Done Sent false 8335 Acked Read 5 Done Done Sent false 8336 Acked Read 4 Done Done Sent false 8337 Acked Read 8 Done Done Sent false 8338 Acked Write 7 Done Done Sent false 8339 Acked Read 8 Done Done Sent false 8340 Acked Flush 0 Done Done Sent false 8341 Acked Read 1 Done Done Sent false 8342 Acked Read 9 Done Done Sent false 8343 Acked Write 7 Done Done Sent false 8344 Acked Write 2 Done Done Sent false 8345 Acked Read 4 Done Done Sent false 8346 Acked Write 3 Done Done Sent false 8347 Acked Read 6 Done Done Sent false 8348 Acked Write 1 Done Done Sent false 8349 Acked Read 1 Done Done Sent false 8350 Acked Read 3 Done Done Sent false 8351 Acked Read 7 Done Done Sent false 8352 Acked Read 3 Done Done Sent false 8353 Acked Read 3 Done Done Sent false 8354 Acked Write 3 Done Done Sent false 8355 Acked Read 2 Done Done Sent false 8356 Acked Read 9 Done Done Sent false 8357 Acked Read 10 Done Done Sent false 8358 Acked Read 1 Done Done Sent false 8359 Acked Read 9 Done Done Sent false 8360 Acked Write 6 Done Done Sent false 8361 Acked Write 3 Done Done Sent false 8362 Acked Read 4 Done Done Sent false 8363 Acked Write 8 Done Done Sent false 8364 Acked Write 6 Done Done Sent false 8365 Acked Write 2 Done Done Sent false 8366 Acked Write 8 Done Done Sent false 8367 Acked Flush 0 Done Done Sent false 8368 Acked Read 2 Done Done Sent false 8369 Acked Write 4 Done Done Sent false 8370 Acked Read 5 Done Done Sent false 8371 Acked Read 8 Done Done Sent false 8372 Acked Write 1 Done Done Sent false 8373 Acked Write 9 Done Done Sent false 8374 Acked Read 4 Done Done Sent false 8375 Acked Read 2 Done Done Sent false 8376 Acked Write 7 Done Done Sent false 8377 Acked Read 2 Done Done Sent false 8378 Acked Read 5 Done Done Sent false 8379 Acked Read 7 Done Done Sent false 8380 Acked Read 7 Done Done Sent false 8381 Acked Read 5 Done Done Sent false 8382 Acked Read 10 Done Done Sent false 8383 Acked Flush 0 Done Done Sent false 8384 Acked Read 6 Done Done Sent false 8385 Acked Read 9 Done Done Sent false 8386 Acked Write 4 Done Done Sent false 8387 Acked Write 1 Done Done Sent false 8388 Acked Flush 0 Done Done Sent false 8389 Acked Write 1 Done Done Sent false 8390 Acked Flush 0 Done Done Sent false 8391 Acked Read 4 Done Done Sent false 8392 Acked Read 5 Done Done Sent false 8393 Acked Read 1 Done Done Sent false 8394 Acked Write 10 Done Done Sent false 8395 Acked Write 5 Done Done Sent false 8396 Acked Read 8 Done Done Sent false 8397 Acked Write 9 Done Done Sent false 8398 Acked Write 4 Done Done Sent false 8399 Acked Write 5 Done Done Sent false 8400 Acked Write 4 Done Done Sent false 8401 Acked Write 10 Done Done Sent false 8402 Acked Read 6 Done Done Sent false 8403 Acked Read 5 Done Done Sent false 8404 Acked Read 5 Done Done Sent false 8405 Acked Read 3 Done Done Sent false 8406 Acked Write 1 Done Done Sent false 8407 Acked Write 4 Done Done Sent false 8408 Acked Flush 0 Done Done Sent false 8409 Acked Write 5 Done Done Sent false 8410 Acked Read 1 Done Done Sent false 8411 Acked Write 8 Done Done Sent false 8412 Acked Write 6 Done Done Sent false 8413 Acked Read 6 Done Done Sent false 8414 Acked Read 7 Done Done Sent false 8415 Acked Write 4 Done Done Sent false 8416 Acked Read 4 Done Done Sent false 8417 Acked Read 1 Done Done Sent false 8418 Acked Flush 0 Done Done Sent false 8419 Acked Write 9 Done Done Sent false 8420 Acked Read 8 Done Done Sent false 8421 Acked Write 1 Done Done Sent false 8422 Acked Read 4 Done Done Sent false 8423 Acked Read 3 Done Done Sent false 8424 Acked Write 5 Done Done Sent false 8425 Acked Write 9 Done Done Sent false 8426 Acked Read 9 Done Done Sent false 8427 Acked Read 9 Done Done Sent false 8428 Acked Write 1 Done Done Sent false 8429 Acked Write 8 Done Done Sent false 8430 Acked Read 4 Done Done Sent false 8431 Acked Read 2 Done Done Sent false 8432 Acked Write 10 Done Done Sent false 8433 Acked Read 5 Done Done Sent false 8434 Acked Read 6 Done Done Sent false 8435 Acked Write 7 Done Done Sent false 8436 Acked Read 5 Done Done Sent false 8437 Acked Read 1 Done Done Sent false 8438 Acked Write 8 Done Done Sent false 8439 Acked Read 10 Done Done Sent false 8440 Acked Flush 0 Done Done Sent false 8441 Acked Read 4 Done Done Sent false 8442 Acked Write 10 Done Done Sent false 8443 Acked Read 3 Done Done Sent false 8444 Acked Read 3 Done Done Sent false 8445 Acked Read 1 Done Done Sent false 8446 Acked Flush 0 Done Done Sent false 8447 Acked Read 8 Done Done Sent false 8448 Acked Read 3 Done Done Sent false 8449 Acked Read 1 Done Done Sent false 8450 Acked Read 8 Done Done Sent false 8451 Acked Write 8 Done Done Sent false 8452 Acked Flush 0 Done Done Sent false 8453 Acked Read 1 Done Done Sent false 8454 Acked Flush 0 Done Done Sent false 8455 Acked Read 4 Done Done Sent false 8456 Acked Write 4 Done Done Sent false 8457 Acked Read 9 Done Done Sent false 8458 Acked Read 2 Done Done Sent false 8459 Acked Read 5 Done Done Sent false 8460 Acked Read 3 Done Done Sent false 8461 Acked Write 4 Done Done Sent false 8462 Acked Flush 0 Done Done Sent false 8463 Acked Read 9 Done Done Sent false 8464 Acked Read 3 Done Done Sent false 8465 Acked Read 10 Done Done Sent false 8466 Acked Read 10 Done Done Sent false 8467 Acked Read 3 Done Done Sent false 8468 Acked Read 1 Done Done Sent false 8469 Acked Read 7 Done Done Sent false 8470 Acked Read 5 Done Done Sent false 8471 Acked Write 7 Done Done Sent false 8472 Acked Flush 0 Done Done Sent false 8473 Acked Read 5 Done Done Sent false 8474 Acked Read 5 Done Done Sent false 8475 Acked Write 5 Done Done Sent false 8476 Acked Read 6 Done Done Sent false 8477 Acked Write 6 Done Done Sent false 8478 Acked Read 1 Done Done Sent false 8479 Acked Flush 0 Done Done Sent false 8480 Acked Write 2 Done Done Sent false 8481 Acked Write 1 Done Done Sent false 8482 Acked Read 4 Done Done Sent false 8483 Acked Read 4 Done Done Sent false 8484 Acked Read 10 Done Done Sent false 8485 Acked Write 4 Done Done Sent false 8486 Acked Read 2 Done Done Sent false 8487 Acked Write 6 Done Done Sent false 8488 Acked Write 10 Done Done Sent false 8489 Acked Write 6 Done Done Sent false 8490 Acked Write 4 Done Done Sent false 8491 Acked Write 7 Done Done Sent false 8492 Acked Write 7 Done Done Sent false 8493 Acked Read 3 Done Done Sent false 8494 Acked Write 7 Done Done Sent false 8495 Acked Flush 0 Done Done Sent false 8496 Acked Read 2 Done Done Sent false 8497 Acked Write 1 Done Done Sent false 8498 Acked Read 9 Done Done Sent false 8499 Acked Read 10 Done Done Sent false 8500 Acked Read 1 Done Done Sent false 8501 Acked Write 9 Done Done Sent false 8502 Acked Read 9 Done Done Sent false 8503 Acked Read 5 Done Done Sent false 8504 Acked Write 7 Done Done Sent false 8505 Acked Flush 0 Done Done Sent false 8506 Acked Flush 0 Done Done Sent false 8507 Acked Flush 0 Done Done Sent false 8508 Acked Write 8 Done Done Sent false 8509 Acked Read 7 Done Done Sent false 8510 Acked Read 2 Done Done Sent false 8511 Acked Read 3 Done Done Sent false 8512 Acked Read 1 Done Done Sent false 8513 Acked Read 3 Done Done Sent false 8514 Acked Write 7 Done Done Sent false 8515 Acked Write 4 Done Done Sent false 8516 Acked Write 6 Done Done Sent false 8517 Acked Write 5 Done Done Sent false 8518 Acked Write 5 Done Done Sent false 8519 Acked Write 5 Done Done Sent false 8520 Acked Write 6 Done Done Sent false 8521 Acked Read 8 Done Done Sent false 8522 Acked Read 5 Done Done Sent false 8523 Acked Write 6 Done Done Sent false 8524 Acked Read 3 Done Done Sent false 8525 Acked Read 1 Done Done Sent false 8526 Acked Flush 0 Done Done Sent false 8527 Acked Read 7 Done Done Sent false 8528 Acked Write 3 Done Done Sent false 8529 Acked Read 7 Done Done Sent false 8530 Acked Read 10 Done Done Sent false 8531 Acked Read 6 Done Done Sent false 8532 Acked Read 8 Done Done Sent false 8533 Acked Read 1 Done Done Sent false 8534 Acked Read 3 Done Done Sent false 8535 Acked Read 10 Done Done Sent false 8536 Acked Read 10 Done Done Sent false 8537 Acked Read 6 Done Done Sent false 8538 Acked Read 2 Done Done Sent false 8539 Acked Write 9 Done Done Sent false 8540 Acked Write 8 Done Done Sent false 8541 Acked Write 5 Done Done Sent false 8542 Acked Read 6 Done Done Sent false 8543 Acked Read 5 Done Done Sent false 8544 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8544 8544 8242 Downstairs last five completed: 8242 8241 8240 8239 8238 Upstairs last five completed: 8544 8543 8542 8541 8540 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:03.495560977Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:03.495625669Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:03.495648493Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:03.495670097Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:03.49569623Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1227) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8243 Acked Read 1 Done Done Sent false 8244 Acked Read 10 Done Done Sent false 8245 Acked Write 8 Done Done Sent false 8246 Acked Read 10 Done Done Sent false 8247 Acked Read 2 Done Done Sent false 8248 Acked Write 5 Done Done Sent false 8249 Acked Read 4 Done Done Sent false 8250 Acked Read 7 Done Done Sent false 8251 Acked Read 10 Done Done Sent false 8252 Acked Write 5 Done Done Sent false 8253 Acked Flush 0 Done Done Sent false 8254 Acked Flush 0 Done Done Sent false 8255 Acked Read 2 Done Done Sent false 8256 Acked Write 1 Done Done Sent false 8257 Acked Flush 0 Done Done Sent false 8258 Acked Write 5 Done Done Sent false 8259 Acked Read 10 Done Done Sent false 8260 Acked Read 9 Done Done Sent false 8261 Acked Read 2 Done Done Sent false 8262 Acked Flush 0 Done Done Sent false 8263 Acked Write 2 Done Done Sent false 8264 Acked Read 4 Done Done Sent false 8265 Acked Write 3 Done Done Sent false 8266 Acked Read 3 Done Done Sent false 8267 Acked Write 2 Done Done Sent false 8268 Acked Read 9 Done Done Sent false 8269 Acked Write 9 Done Done Sent false 8270 Acked Write 3 Done Done Sent false 8271 Acked Write 1 Done Done Sent false 8272 Acked Write 7 Done Done Sent false 8273 Acked Read 5 Done Done Sent false 8274 Acked Read 10 Done Done Sent false 8275 Acked Read 4 Done Done Sent false 8276 Acked Write 2 Done Done Sent false 8277 Acked Read 2 Done Done Sent false 8278 Acked Read 8 Done Done Sent false 8279 Acked Write 9 Done Done Sent false 8280 Acked Read 10 Done Done Sent false 8281 Acked Write 10 Done Done Sent false 8282 Acked Flush 0 Done Done Sent false 8283 Acked Read 2 Done Done Sent false 8284 Acked Write 3 Done Done Sent false 8285 Acked Read 7 Done Done Sent false 8286 Acked Read 10 Done Done Sent false 8287 Acked Read 1 Done Done Sent false 8288 Acked Read 4 Done Done Sent false 8289 Acked Write 1 Done Done Sent false 8290 Acked Read 4 Done Done Sent false 8291 Acked Read 5 Done Done Sent false 8292 Acked Write 1 Done Done Sent false 8293 Acked Read 3 Done Done Sent false 8294 Acked Flush 0 Done Done Sent false 8295 Acked Read 9 Done Done Sent false 8296 Acked Read 8 Done Done Sent false 8297 Acked Write 10 Done Done Sent false 8298 Acked Read 2 Done Done Sent false 8299 Acked Write 10 Done Done Sent false 8300 Acked Read 2 Done Done Sent false 8301 Acked Flush 0 Done Done Sent false 8302 Acked Write 4 Done Done Sent false 8303 Acked Read 7 Done Done Sent false 8304 Acked Read 7 Done Done Sent false 8305 Acked Read 4 Done Done Sent false 8306 Acked Write 3 Done Done Sent false 8307 Acked Write 2 Done Done Sent false 8308 Acked Write 5 Done Done Sent false 8309 Acked Write 10 Done Done Sent false 8310 Acked Write 7 Done Done Sent false 8311 Acked Write 3 Done Done Sent false 8312 Acked Write 8 Done Done Sent false 8313 Acked Flush 0 Done Done Sent false 8314 Acked Write 9 Done Done Sent false 8315 Acked Write 1 Done Done Sent false 8316 Acked Read 2 Done Done Sent false 8317 Acked Read 4 Done Done Sent false 8318 Acked Flush 0 Done Done Sent false 8319 Acked Write 2 Done Done Sent false 8320 Acked Read 1 Done Done Sent false 8321 Acked Flush 0 Done Done Sent false 8322 Acked Read 10 Done Done Sent false 8323 Acked Write 5 Done Done Sent false 8324 Acked Write 2 Done Done Sent false 8325 Acked Flush 0 Done Done Sent false 8326 Acked Write 4 Done Done Sent false 8327 Acked Read 4 Done Done Sent false 8328 Acked Read 2 Done Done Sent false 8329 Acked Write 1 Done Done Sent false 8330 Acked Read 1 Done Done Sent false 8331 Acked Write 8 Done Done Sent false 8332 Acked Write 8 Done Done Sent false 8333 Acked Read 2 Done Done Sent false 8334 Acked Read 1 Done Done Sent false 8335 Acked Read 5 Done Done Sent false 8336 Acked Read 4 Done Done Sent false 8337 Acked Read 8 Done Done Sent false 8338 Acked Write 7 Done Done Sent false 8339 Acked Read 8 Done Done Sent false 8340 Acked Flush 0 Done Done Sent false 8341 Acked Read 1 Done Done Sent false 8342 Acked Read 9 Done Done Sent false 8343 Acked Write 7 Done Done Sent false 8344 Acked Write 2 Done Done Sent false 8345 Acked Read 4 Done Done Sent false 8346 Acked Write 3 Done Done Sent false 8347 Acked Read 6 Done Done Sent false 8348 Acked Write 1 Done Done Sent false 8349 Acked Read 1 Done Done Sent false 8350 Acked Read 3 Done Done Sent false 8351 Acked Read 7 Done Done Sent false 8352 Acked Read 3 Done Done Sent false 8353 Acked Read 3 Done Done Sent false 8354 Acked Write 3 Done Done Sent false 8355 Acked Read 2 Done Done Sent false 8356 Acked Read 9 Done Done Sent false 8357 Acked Read 10 Done Done Sent false 8358 Acked Read 1 Done Done Sent false 8359 Acked Read 9 Done Done Sent false 8360 Acked Write 6 Done Done Sent false 8361 Acked Write 3 Done Done Sent false 8362 Acked Read 4 Done Done Sent false 8363 Acked Write 8 Done Done Sent false 8364 Acked Write 6 Done Done Sent false 8365 Acked Write 2 Done Done Sent false 8366 Acked Write 8 Done Done Sent false 8367 Acked Flush 0 Done Done Sent false 8368 Acked Read 2 Done Done Sent false 8369 Acked Write 4 Done Done Sent false 8370 Acked Read 5 Done Done Sent false 8371 Acked Read 8 Done Done Sent false 8372 Acked Write 1 Done Done Sent false 8373 Acked Write 9 Done Done Sent false 8374 Acked Read 4 Done Done Sent false 8375 Acked Read 2 Done Done Sent false 8376 Acked Write 7 Done Done Sent false 8377 Acked Read 2 Done Done Sent false 8378 Acked Read 5 Done Done Sent false 8379 Acked Read 7 Done Done Sent false 8380 Acked Read 7 Done Done Sent false 8381 Acked Read 5 Done Done Sent false 8382 Acked Read 10 Done Done Sent false 8383 Acked Flush 0 Done Done Sent false 8384 Acked Read 6 Done Done Sent false 8385 Acked Read 9 Done Done Sent false 8386 Acked Write 4 Done Done Sent false 8387 Acked Write 1 Done Done Sent false 8388 Acked Flush 0 Done Done Sent false 8389 Acked Write 1 Done Done Sent false 8390 Acked Flush 0 Done Done Sent false 8391 Acked Read 4 Done Done Sent false 8392 Acked Read 5 Done Done Sent false 8393 Acked Read 1 Done Done Sent false 8394 Acked Write 10 Done Done Sent false 8395 Acked Write 5 Done Done Sent false 8396 Acked Read 8 Done Done Sent false 8397 Acked Write 9 Done Done Sent false 8398 Acked Write 4 Done Done Sent false 8399 Acked Write 5 Done Done Sent false 8400 Acked Write 4 Done Done Sent false 8401 Acked Write 10 Done Done Sent false 8402 Acked Read 6 Done Done Sent false 8403 Acked Read 5 Done Done Sent false 8404 Acked Read 5 Done Done Sent false 8405 Acked Read 3 Done Done Sent false 8406 Acked Write 1 Done Done Sent false 8407 Acked Write 4 Done Done Sent false 8408 Acked Flush 0 Done Done Sent false 8409 Acked Write 5 Done Done Sent false 8410 Acked Read 1 Done Done Sent false 8411 Acked Write 8 Done Done Sent false 8412 Acked Write 6 Done Done Sent false 8413 Acked Read 6 Done Done Sent false 8414 Acked Read 7 Done Done Sent false 8415 Acked Write 4 Done Done Sent false 8416 Acked Read 4 Done Done Sent false 8417 Acked Read 1 Done Done Sent false 8418 Acked Flush 0 Done Done Sent false 8419 Acked Write 9 Done Done Sent false 8420 Acked Read 8 Done Done Sent false 8421 Acked Write 1 Done Done Sent false 8422 Acked Read 4 Done Done Sent false 8423 Acked Read 3 Done Done Sent false 8424 Acked Write 5 Done Done Sent false 8425 Acked Write 9 Done Done Sent false 8426 Acked Read 9 Done Done Sent false 8427 Acked Read 9 Done Done Sent false 8428 Acked Write 1 Done Done Sent false 8429 Acked Write 8 Done Done Sent false 8430 Acked Read 4 Done Done Sent false 8431 Acked Read 2 Done Done Sent false 8432 Acked Write 10 Done Done Sent false 8433 Acked Read 5 Done Done Sent false 8434 Acked Read 6 Done Done Sent false 8435 Acked Write 7 Done Done Sent false 8436 Acked Read 5 Done Done Sent false 8437 Acked Read 1 Done Done Sent false 8438 Acked Write 8 Done Done Sent false 8439 Acked Read 10 Done Done Sent false 8440 Acked Flush 0 Done Done Sent false 8441 Acked Read 4 Done Done Sent false 8442 Acked Write 10 Done Done Sent false 8443 Acked Read 3 Done Done Sent false 8444 Acked Read 3 Done Done Sent false 8445 Acked Read 1 Done Done Sent false 8446 Acked Flush 0 Done Done Sent false 8447 Acked Read 8 Done Done Sent false 8448 Acked Read 3 Done Done Sent false 8449 Acked Read 1 Done Done Sent false 8450 Acked Read 8 Done Done Sent false 8451 Acked Write 8 Done Done Sent false 8452 Acked Flush 0 Done Done Sent false 8453 Acked Read 1 Done Done Sent false 8454 Acked Flush 0 Done Done Sent false 8455 Acked Read 4 Done Done Sent false 8456 Acked Write 4 Done Done Sent false 8457 Acked Read 9 Done Done Sent false 8458 Acked Read 2 Done Done Sent false 8459 Acked Read 5 Done Done Sent false 8460 Acked Read 3 Done Done Sent false 8461 Acked Write 4 Done Done Sent false 8462 Acked Flush 0 Done Done Sent false 8463 Acked Read 9 Done Done Sent false 8464 Acked Read 3 Done Done Sent false 8465 Acked Read 10 Done Done Sent false 8466 Acked Read 10 Done Done Sent false 8467 Acked Read 3 Done Done Sent false 8468 Acked Read 1 Done Done Sent false 8469 Acked Read 7 Done Done Sent false 8470 Acked Read 5 Done Done Sent false 8471 Acked Write 7 Done Done Sent false 8472 Acked Flush 0 Done Done Sent false 8473 Acked Read 5 Done Done Sent false 8474 Acked Read 5 Done Done Sent false 8475 Acked Write 5 Done Done Sent false 8476 Acked Read 6 Done Done Sent false 8477 Acked Write 6 Done Done Sent false 8478 Acked Read 1 Done Done Sent false 8479 Acked Flush 0 Done Done Sent false 8480 Acked Write 2 Done Done Sent false 8481 Acked Write 1 Done Done Sent false 8482 Acked Read 4 Done Done Sent false 8483 Acked Read 4 Done Done Sent false 8484 Acked Read 10 Done Done Sent false 8485 Acked Write 4 Done Done Sent false 8486 Acked Read 2 Done Done Sent false 8487 Acked Write 6 Done Done Sent false 8488 Acked Write 10 Done Done Sent false 8489 Acked Write 6 Done Done Sent false 8490 Acked Write 4 Done Done Sent false 8491 Acked Write 7 Done Done Sent false 8492 Acked Write 7 Done Done Sent false 8493 Acked Read 3 Done Done Sent false 8494 Acked Write 7 Done Done Sent false 8495 Acked Flush 0 Done Done Sent false 8496 Acked Read 2 Done Done Sent false 8497 Acked Write 1 Done Done Sent false 8498 Acked Read 9 Done Done Sent false 8499 Acked Read 10 Done Done Sent false 8500 Acked Read 1 Done Done Sent false 8501 Acked Write 9 Done Done Sent false 8502 Acked Read 9 Done Done Sent false 8503 Acked Read 5 Done Done Sent false 8504 Acked Write 7 Done Done Sent false 8505 Acked Flush 0 Done Done Sent false 8506 Acked Flush 0 Done Done Sent false 8507 Acked Flush 0 Done Done Sent false 8508 Acked Write 8 Done Done Sent false 8509 Acked Read 7 Done Done Sent false 8510 Acked Read 2 Done Done Sent false 8511 Acked Read 3 Done Done Sent false 8512 Acked Read 1 Done Done Sent false 8513 Acked Read 3 Done Done Sent false 8514 Acked Write 7 Done Done Sent false 8515 Acked Write 4 Done Done Sent false 8516 Acked Write 6 Done Done Sent false 8517 Acked Write 5 Done Done Sent false 8518 Acked Write 5 Done Done Sent false 8519 Acked Write 5 Done Done Sent false 8520 Acked Write 6 Done Done Sent false 8521 Acked Read 8 Done Done Sent false 8522 Acked Read 5 Done Done Sent false 8523 Acked Write 6 Done Done Sent false 8524 Acked Read 3 Done Done Sent false 8525 Acked Read 1 Done Done Sent false 8526 Acked Flush 0 Done Done Sent false 8527 Acked Read 7 Done Done Sent false 8528 Acked Write 3 Done Done Sent false 8529 Acked Read 7 Done Done Sent false 8530 Acked Read 10 Done Done Sent false 8531 Acked Read 6 Done Done Sent false 8532 Acked Read 8 Done Done Sent false 8533 Acked Read 1 Done Done Sent false 8534 Acked Read 3 Done Done Sent false 8535 Acked Read 10 Done Done Sent false 8536 Acked Read 10 Done Done Sent false 8537 Acked Read 6 Done Done Sent false 8538 Acked Read 2 Done Done Sent false 8539 Acked Write 9 Done Done Sent false 8540 Acked Write 8 Done Done Sent false 8541 Acked Write 5 Done Done Sent false 8542 Acked Read 6 Done Done Sent false 8543 Acked Read 5 Done Done Sent false 8544 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8544 8544 8242 Downstairs last five completed: 8242 8241 8240 8239 8238 Upstairs last five completed: 8544 8543 8542 8541 8540 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8243 Acked Read 1 Done Done Sent false 8244 Acked Read 10 Done Done Sent false 8245 Acked Write 8 Done Done Sent false 8246 Acked Read 10 Done Done Sent false 8247 Acked Read 2 Done Done Sent false 8248 Acked Write 5 Done Done Sent false 8249 Acked Read 4 Done Done Sent false 8250 Acked Read 7 Done Done Sent false 8251 Acked Read 10 Done Done Sent false 8252 Acked Write 5 Done Done Sent false 8253 Acked Flush 0 Done Done Sent false 8254 Acked Flush 0 Done Done Sent false 8255 Acked Read 2 Done Done Sent false 8256 Acked Write 1 Done Done Sent false 8257 Acked Flush 0 Done Done Sent false 8258 Acked Write 5 Done Done Sent false 8259 Acked Read 10 Done Done Sent false 8260 Acked Read 9 Done Done Sent false 8261 Acked Read 2 Done Done Sent false 8262 Acked Flush 0 Done Done Sent false 8263 Acked Write 2 Done Done Sent false 8264 Acked Read 4 Done Done Sent false 8265 Acked Write 3 Done Done Sent false 8266 Acked Read 3 Done Done Sent false 8267 Acked Write 2 Done Done Sent false 8268 Acked Read 9 Done Done Sent false 8269 Acked Write 9 Done Done Sent false 8270 Acked Write 3 Done Done Sent false 8271 Acked Write 1 Done Done Sent false 8272 Acked Write 7 Done Done Sent false 8273 Acked Read 5 Done Done Sent false 8274 Acked Read 10 Done Done Sent false 8275 Acked Read 4 Done Done Sent false 8276 Acked Write 2 Done Done Sent false 8277 Acked Read 2 Done Done Sent false 8278 Acked Read 8 Done Done Sent false 8279 Acked Write 9 Done Done Sent false 8280 Acked Read 10 Done Done Sent false 8281 Acked Write 10 Done Done Sent false 8282 Acked Flush 0 Done Done Sent false 8283 Acked Read 2 Done Done Sent false 8284 Acked Write 3 Done Done Sent false 8285 Acked Read 7 Done Done Sent false 8286 Acked Read 10 Done Done Sent false 8287 Acked Read 1 Done Done Sent false 8288 Acked Read 4 Done Done Sent false 8289 Acked Write 1 Done Done Sent false 8290 Acked Read 4 Done Done Sent false 8291 Acked Read 5 Done Done Sent false 8292 Acked Write 1 Done Done Sent false 8293 Acked Read 3 Done Done Sent false 8294 Acked Flush 0 Done Done Sent false 8295 Acked Read 9 Done Done Sent false 8296 Acked Read 8 Done Done Sent false 8297 Acked Write 10 Done Done Sent false 8298 Acked Read 2 Done Done Sent false 8299 Acked Write 10 Done Done Sent false 8300 Acked Read 2 Done Done Sent false 8301 Acked Flush 0 Done Done Sent false 8302 Acked Write 4 Done Done Sent false 8303 Acked Read 7 Done Done Sent false 8304 Acked Read 7 Done Done Sent false 8305 Acked Read 4 Done Done Sent false 8306 Acked Write 3 Done Done Sent false 8307 Acked Write 2 Done Done Sent false 8308 Acked Write 5 Done Done Sent false 8309 Acked Write 10 Done Done Sent false 8310 Acked Write 7 Done Done Sent false 8311 Acked Write 3 Done Done Sent false 8312 Acked Write 8 Done Done Sent false 8313 Acked Flush 0 Done Done Sent false 8314 Acked Write 9 Done Done Sent false 8315 Acked Write 1 Done Done Sent false 8316 Acked Read 2 Done Done Sent false 8317 Acked Read 4 Done Done Sent false 8318 Acked Flush 0 Done Done Sent false 8319 Acked Write 2 Done Done Sent false 8320 Acked Read 1 Done Done Sent false 8321 Acked Flush 0 Done Done Sent false 8322 Acked Read 10 Done Done Sent false 8323 Acked Write 5 Done Done Sent false 8324 Acked Write 2 Done Done Sent false 8325 Acked Flush 0 Done Done Sent false 8326 Acked Write 4 Done Done Sent false 8327 Acked Read 4 Done Done Sent false 8328 Acked Read 2 Done Done Sent false 8329 Acked Write 1 Done Done Sent false 8330 Acked Read 1 Done Done Sent false 8331 Acked Write 8 Done Done Sent false 8332 Acked Write 8 Done Done Sent false 8333 Acked Read 2 Done Done Sent false 8334 Acked Read 1 Done Done Sent false 8335 Acked Read 5 Done Done Sent false 8336 Acked Read 4 Done Done Sent false 8337 Acked Read 8 Done Done Sent false 8338 Acked Write 7 Done Done Sent false 8339 Acked Read 8 Done Done Sent false 8340 Acked Flush 0 Done Done Sent false 8341 Acked Read 1 Done Done Sent false 8342 Acked Read 9 Done Done Sent false 8343 Acked Write 7 Done Done Sent false 8344 Acked Write 2 Done Done Sent false 8345 Acked Read 4 Done Done Sent false 8346 Acked Write 3 Done Done Sent false 8347 Acked Read 6 Done Done Sent false 8348 Acked Write 1 Done Done Sent false 8349 Acked Read 1 Done Done Sent false 8350 Acked Read 3 Done Done Sent false 8351 Acked Read 7 Done Done Sent false 8352 Acked Read 3 Done Done Sent false 8353 Acked Read 3 Done Done Sent false 8354 Acked Write 3 Done Done Sent false 8355 Acked Read 2 Done Done Sent false 8356 Acked Read 9 Done Done Sent false 8357 Acked Read 10 Done Done Sent false 8358 Acked Read 1 Done Done Sent false 8359 Acked Read 9 Done Done Sent false 8360 Acked Write 6 Done Done Sent false 8361 Acked Write 3 Done Done Sent false 8362 Acked Read 4 Done Done Sent false 8363 Acked Write 8 Done Done Sent false 8364 Acked Write 6 Done Done Sent false 8365 Acked Write 2 Done Done Sent false 8366 Acked Write 8 Done Done Sent false 8367 Acked Flush 0 Done Done Sent false 8368 Acked Read 2 Done Done Sent false 8369 Acked Write 4 Done Done Sent false 8370 Acked Read 5 Done Done Sent false 8371 Acked Read 8 Done Done Sent false 8372 Acked Write 1 Done Done Sent false 8373 Acked Write 9 Done Done Sent false 8374 Acked Read 4 Done Done Sent false 8375 Acked Read 2 Done Done Sent false 8376 Acked Write 7 Done Done Sent false 8377 Acked Read 2 Done Done Sent false 8378 Acked Read 5 Done Done Sent false 8379 Acked Read 7 Done Done Sent false 8380 Acked Read 7 Done Done Sent false 8381 Acked Read 5 Done Done Sent false 8382 Acked Read 10 Done Done Sent false 8383 Acked Flush 0 Done Done Sent false 8384 Acked Read 6 Done Done Sent false 8385 Acked Read 9 Done Done Sent false 8386 Acked Write 4 Done Done Sent false 8387 Acked Write 1 Done Done Sent false 8388 Acked Flush 0 Done Done Sent false 8389 Acked Write 1 Done Done Sent false 8390 Acked Flush 0 Done Done Sent false 8391 Acked Read 4 Done Done Sent false 8392 Acked Read 5 Done Done Sent false 8393 Acked Read 1 Done Done Sent false 8394 Acked Write 10 Done Done Sent false 8395 Acked Write 5 Done Done Sent false 8396 Acked Read 8 Done Done Sent false 8397 Acked Write 9 Done Done Sent false 8398 Acked Write 4 Done Done Sent false 8399 Acked Write 5 Done Done Sent false 8400 Acked Write 4 Done Done Sent false 8401 Acked Write 10 Done Done Sent false 8402 Acked Read 6 Done Done Sent false 8403 Acked Read 5 Done Done Sent false 8404 Acked Read 5 Done Done Sent false 8405 Acked Read 3 Done Done Sent false 8406 Acked Write 1 Done Done Sent false 8407 Acked Write 4 Done Done Sent false 8408 Acked Flush 0 Done Done Sent false 8409 Acked Write 5 Done Done Sent false 8410 Acked Read 1 Done Done Sent false 8411 Acked Write 8 Done Done Sent false 8412 Acked Write 6 Done Done Sent false 8413 Acked Read 6 Done Done Sent false 8414 Acked Read 7 Done Done Sent false 8415 Acked Write 4 Done Done Sent false 8416 Acked Read 4 Done Done Sent false 8417 Acked Read 1 Done Done Sent false 8418 Acked Flush 0 Done Done Sent false 8419 Acked Write 9 Done Done Sent false 8420 Acked Read 8 Done Done Sent false 8421 Acked Write 1 Done Done Sent false 8422 Acked Read 4 Done Done Sent false 8423 Acked Read 3 Done Done Sent false 8424 Acked Write 5 Done Done Sent false 8425 Acked Write 9 Done Done Sent false 8426 Acked Read 9 Done Done Sent false 8427 Acked Read 9 Done Done Sent false 8428 Acked Write 1 Done Done Sent false 8429 Acked Write 8 Done Done Sent false 8430 Acked Read 4 Done Done Sent false 8431 Acked Read 2 Done Done Sent false 8432 Acked Write 10 Done Done Sent false 8433 Acked Read 5 Done Done Sent false 8434 Acked Read 6 Done Done Sent false 8435 Acked Write 7 Done Done Sent false 8436 Acked Read 5 Done Done Sent false 8437 Acked Read 1 Done Done Sent false 8438 Acked Write 8 Done Done Sent false 8439 Acked Read 10 Done Done Sent false 8440 Acked Flush 0 Done Done Sent false 8441 Acked Read 4 Done Done Sent false 8442 Acked Write 10 Done Done Sent false 8443 Acked Read 3 Done Done Sent false 8444 Acked Read 3 Done Done Sent false 8445 Acked Read 1 Done Done Sent false 8446 Acked Flush 0 Done Done Sent false 8447 Acked Read 8 Done Done Sent false 8448 Acked Read 3 Done Done Sent false 8449 Acked Read 1 Done Done Sent false 8450 Acked Read 8 Done Done Sent false 8451 Acked Write 8 Done Done Sent false 8452 Acked Flush 0 Done Done Sent false 8453 Acked Read 1 Done Done Sent false 8454 Acked Flush 0 Done Done Sent false 8455 Acked Read 4 Done Done Sent false 8456 Acked Write 4 Done Done Sent false 8457 Acked Read 9 Done Done Sent false 8458 Acked Read 2 Done Done Sent false 8459 Acked Read 5 Done Done Sent false 8460 Acked Read 3 Done Done Sent false 8461 Acked Write 4 Done Done Sent false 8462 Acked Flush 0 Done Done Sent false 8463 Acked Read 9 Done Done Sent false 8464 Acked Read 3 Done Done Sent false 8465 Acked Read 10 Done Done Sent false 8466 Acked Read 10 Done Done Sent false 8467 Acked Read 3 Done Done Sent false 8468 Acked Read 1 Done Done Sent false 8469 Acked Read 7 Done Done Sent false 8470 Acked Read 5 Done Done Sent false 8471 Acked Write 7 Done Done Sent false 8472 Acked Flush 0 Done Done Sent false 8473 Acked Read 5 Done Done Sent false 8474 Acked Read 5 Done Done Sent false 8475 Acked Write 5 Done Done Sent false 8476 Acked Read 6 Done Done Sent false 8477 Acked Write 6 Done Done Sent false 8478 Acked Read 1 Done Done Sent false 8479 Acked Flush 0 Done Done Sent false 8480 Acked Write 2 Done Done Sent false 8481 Acked Write 1 Done Done Sent false 8482 Acked Read 4 Done Done Sent false 8483 Acked Read 4 Done Done Sent false 8484 Acked Read 10 Done Done Sent false 8485 Acked Write 4 Done Done Sent false 8486 Acked Read 2 Done Done Sent false 8487 Acked Write 6 Done Done Sent false 8488 Acked Write 10 Done Done Sent false 8489 Acked Write 6 Done Done Sent false 8490 Acked Write 4 Done Done Sent false 8491 Acked Write 7 Done Done Sent false 8492 Acked Write 7 Done Done Sent false 8493 Acked Read 3 Done Done Sent false 8494 Acked Write 7 Done Done Sent false 8495 Acked Flush 0 Done Done Sent false 8496 Acked Read 2 Done Done Sent false 8497 Acked Write 1 Done Done Sent false 8498 Acked Read 9 Done Done Sent false 8499 Acked Read 10 Done Done Sent false 8500 Acked Read 1 Done Done Sent false 8501 Acked Write 9 Done Done Sent false 8502 Acked Read 9 Done Done Sent false 8503 Acked Read 5 Done Done Sent false 8504 Acked Write 7 Done Done Sent false 8505 Acked Flush 0 Done Done Sent false 8506 Acked Flush 0 Done Done Sent false 8507 Acked Flush 0 Done Done Sent false 8508 Acked Write 8 Done Done Sent false 8509 Acked Read 7 Done Done Sent false 8510 Acked Read 2 Done Done Sent false 8511 Acked Read 3 Done Done Sent false 8512 Acked Read 1 Done Done Sent false 8513 Acked Read 3 Done Done Sent false 8514 Acked Write 7 Done Done Sent false 8515 Acked Write 4 Done Done Sent false 8516 Acked Write 6 Done Done Sent false 8517 Acked Write 5 Done Done Sent false 8518 Acked Write 5 Done Done Sent false 8519 Acked Write 5 Done Done Sent false 8520 Acked Write 6 Done Done Sent false 8521 Acked Read 8 Done Done Sent false 8522 Acked Read 5 Done Done Sent false 8523 Acked Write 6 Done Done Sent false 8524 Acked Read 3 Done Done Sent false 8525 Acked Read 1 Done Done Sent false 8526 Acked Flush 0 Done Done Sent false 8527 Acked Read 7 Done Done Sent false 8528 Acked Write 3 Done Done Sent false 8529 Acked Read 7 Done Done Sent false 8530 Acked Read 10 Done Done Sent false 8531 Acked Read 6 Done Done Sent false 8532 Acked Read 8 Done Done Sent false 8533 Acked Read 1 Done Done Sent false 8534 Acked Read 3 Done Done Sent false 8535 Acked Read 10 Done Done Sent false 8536 Acked Read 10 Done Done Sent false 8537 Acked Read 6 Done Done Sent false 8538 Acked Read 2 Done Done Sent false 8539 Acked Write 9 Done Done Sent false 8540 Acked Write 8 Done Done Sent false 8541 Acked Write 5 Done Done Sent false 8542 Acked Read 6 Done Done Sent false 8543 Acked Read 5 Done Done Sent false 8544 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8544 8544 8242 Downstairs last five completed: 8242 8241 8240 8239 8238 Upstairs last five completed: 8544 8543 8542 8541 8540 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.496838562Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:52875)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.496906304Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:13.497970028Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498501615Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498532797Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 8242","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498549533Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 8242","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498751258Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498770813Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498790108Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 8242","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.498804664Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 8242","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.499036312Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8544 8543 8542 8541 8540 Upstairs last five completed: 8544 8543 8542 8541 8540 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Nov 27 01:48:13.735 INFO request completed, latency_us: 309, response_code: 204, uri: /stop/cid/0, method: POST, req_id: fd042ec9-2a7f-4dbe-81f6-25330d3f8ebb, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:48:13.736{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:13.736387213Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} INFO request completed, latency_us: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:13.736447897Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 125{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:13.736477979Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} , response_code: 200, uri: /state/cid/0, {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.736492535Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} method{"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.736514849Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} : GET, req_id: 7d5580a2-2536-48e8-9b5a-c1bc7f1ae793, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:13.736553828Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:48:17.738 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/0, method: GET, req_id: a18adf0c-c563-49d3-bf98-c1f301f8929c, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 2256 len 4096 002/300 Read block 234 len 20480 003/300 Write block 1821 len 4096 data: 9 004/300 Write block 3703 len 40960 data: 8 7 7 6 5 4 4 5 4 5 005/300 Write block 2271 len 24576 data: 7 5 6 6 6 5 006/300 Flush 007/300 Flush 008/300 Read block 3992 len 24576 009/300 Write block 2068 len 32768 data: 3 3 3 4 4 4 5 5 010/300 Write block 2228 len 24576 data: 6 3 2 3 4 4 011/300 Write block 1604 len 28672 data: 5 4 5 7 6 7 7 012/300 Flush 013/300 Write block 2161 len 32768 data: 4 4 6 6 5 4 4 4 014/300 Write block 3407 len 4096 data: 7 015/300 Write block 3040 len 8192 data: 3 3 016/300 Read block 2442 len 28672 017/300 Read block 1278 len 12288 018/300 Write block 770 len 24576 data: 6 6 7 8 7 6 019/300 Read block 268 len 32768 020/300 Read block 2564 len 36864 021/300 Read block 834 len 24576 022/300 Read block 4631 len 20480 023/300 Read block 4727 len 16384 024/300 Read block 4977 len 36864 025/300 Read block 4333 len 16384 026/300 Read block 3429 len 20480 027/300 Write block 3956 len 12288 data: 5 4 4 028/300 Read block 2578 len 8192 029/300 Write block 596 len 32768 data: 5 5 3 2 2 4 3 3 030/300 Read block 3538 len 8192 031/300 Write block 2853 len 28672 data: 8 8 7 7 7 7 8 032/300 Write block 4267 len 36864 data: 5 2 3 4 4 5 6 5 7 033/300 Read block 1282 len 16384 034/300 Write block 1804 len 16384 data: 6 6 5 7 035/300 Read block 3551 len 12288 036/300 Flush 037/300 Read block 2511 len 20480 038/300 Read block 3046 len 28672 039/300 Write block 2863 len 12288 data: 6 5 5 040/300 Read block 2591 len 24576 041/300 Read block 4026 len 16384 042/300 Flush 043/300 Read block 3717 len 24576 044/300 Write block 2254 len 24576 data: 4 5 7 7 7 8 045/300 Read block 2162 len 28672 046/300 Read block 302 len 36864 047/300 Write block 2140 len 12288 data: 5 4 7 048/300 Read block 2254 len 20480 049/300 Read block 598 len 4096 050/300 Flush 051/300 Flush 052/300 Write block 3653 len 36864 data: 7 8 9 9 9 10 9 7 7 053/300 Read block 4942 len 40960 054/300 Read block 3364 len 32768 055/300 Write block 1947 len 36864 data: 4 4 4 6 6 6 7 8 8 056/300 Write block 1693 len 12288 data: 6 6 6 057/300 Write block 961 len 36864 data: 6 5 5 5 4 4 5 4 4 058/300 Write block 160 len 20480 data: 3 3 3 4 4 059/300 Read block 4255 len 4096 060/300 Flush 061/300 Read block 3258 len 8192 062/300 Read block 1019 len 16384 063/300 Write block 2074 len 32768 data: 6 6 5 4 6 7 7 6 064/300 Read block 375 len 20480 065/300 Flush 066/300 Flush 067/300 Read block 2005 len 32768 068/300 Write block 4421 len 36864 data: 7 8 7 6 9 7 6 6 4 069/300 Write block 3508 len 36864 data: 3 3 3 4 4 3 4 6 6 070/300 Write block 1428 len 16384 data: 5 5 5 4 071/300 Write block 1030 len 20480 data: 4 4 6 7 7 072/300 Write block 2722 len 24576 data: 5 5 4 4 5 4 073/300 Write block 4820 len 20480 data: 4 4 4 4 4 074/300 Write block 4916 len 16384 data: 3 4 4 4 075/300 Read block 171 len 36864 076/300 Write block 2701 len 20480 data: 4 4 4 4 5 077/300 Write block 1067 len 24576 data: 4 4 3 4 4 3 078/300 Read block 4250 len 8192 079/300 Write block 1465 len 40960 data: 5 6 9 10 9 9 9 9 7 6 080/300 Write block 3283 len 20480 data: 2 3 3 3 4 081/300 Write block 3154 len 12288 data: 4 4 5 082/300 Write block 3490 len 32768 data: 8 8 6 6 5 4 4 3 083/300 Read block 3451 len 8192 084/300 Read block 3838 len 16384 085/300 Read block 2599 len 40960 086/300 Write block 1657 len 12288 data: 3 3 3 087/300 Flush 088/300 Write block 806 len 16384 data: 6 7 6 5 089/300 Flush 090/300 Read block 2618 len 8192 091/300 Read block 2602 len 36864 092/300 Write block 777 len 32768 data: 5 4 4 4 3 4 5 5 093/300 Read block 2778 len 4096 094/300 Read block 4860 len 16384 095/300 Flush 096/300 Read block 291 len 20480 097/300 Read block 2079 len 12288 098/300 Read block 4035 len 8192 099/300 Read block 3120 len 4096 100/300 Write block 889 len 8192 data: 9 10 101/300 Write block 1949 len 36864 data: 5 7 7 7 8 9 9 5 8 102/300 Write block 2610 len 4096 data: 6 103/300 Write block 2148 len 8192 data: 4 4 104/300 Flush 105/300 Flush 106/300 Read block 1238 len 28672 107/300 Read block 408 len 12288 108/300 Write block 1961 len 20480 data: 5 6 6 5 4 109/300 Write block 4667 len 24576 data: 8 8 8 9 7 5 110/300 Write block 521 len 4096 data: 6 111/300 Flush 112/300 Read block 4415 len 16384 113/300 Read block 1744 len 36864 114/300 Read block 4126 len 28672 115/300 Read block 3143 len 32768 116/300 Read block 2332 len 24576 117/300 Read block 429 len 20480 118/300 Read block 2716 len 36864 119/300 Write block 1585 len 36864 data: 6 7 9 9 8 6 4 4 3 120/300 Write block 4368 len 32768 data: 5 5 4 4 3 3 4 3 121/300 Write block 4391 len 12288 data: 9 7 9 122/300 Write block 4661 len 32768 data: 6 8 9 9 8 8 9 9 123/300 Read block 1399 len 28672 124/300 Write block 2455 len 20480 data: 6 7 8 8 8 125/300 Write block 3671 len 20480 data: 7 5 6 7 6 126/300 Read block 1477 len 4096 127/300 Write block 3354 len 40960 data: 5 6 6 6 5 7 7 8 8 9 128/300 Write block 3076 len 28672 data: 5 5 5 4 4 3 4 129/300 Write block 4865 len 16384 data: 5 5 2 2 130/300 Write block 406 len 40960 data: 4 5 4 4 3 3 3 3 5 3 131/300 Read block 1024 len 32768 132/300 Read block 4760 len 16384 133/300 Read block 3162 len 16384 134/300 Write block 930 len 32768 data: 3 5 5 5 5 5 4 6 135/300 Write block 793 len 12288 data: 4 2 2 136/300 Read block 1543 len 20480 137/300 Read block 3530 len 16384 138/300 Write block 4504 len 12288 data: 5 6 6 139/300 Read block 1896 len 20480 140/300 Write block 1582 len 36864 data: 5 6 6 7 8 10 10 9 7 141/300 Read block 2678 len 20480 142/300 Write block 3422 len 32768 data: 5 4 4 3 3 2 3 3 143/300 Read block 2977 len 20480 144/300 Write block 4455 len 24576 data: 3 3 3 2 3 3 145/300 Read block 932 len 32768 146/300 Read block 300 len 4096 147/300 Read block 4437 len 40960 148/300 Write block 4832 len 40960 data: 6 7 6 5 7 7 6 7 9 8 149/300 Write block 3313 len 36864 data: 6 5 5 6 6 7 8 7 7 150/300 Write block 1074 len 28672 data: 5 5 5 3 4 5 4 151/300 Flush 152/300 Read block 4142 len 12288 153/300 Read block 657 len 4096 154/300 Write block 3838 len 36864 data: 5 4 3 2 2 2 2 2 3 155/300 Write block 1009 len 40960 data: 5 5 5 7 6 7 7 6 7 6 156/300 Read block 2236 len 8192 157/300 Write block 2542 len 20480 data: 10 9 9 8 6 158/300 Flush 159/300 Write block 2395 len 36864 data: 8 7 6 5 6 6 6 6 7 160/300 Read block 2748 len 8192 161/300 Write block 3548 len 8192 data: 6 8 162/300 Write block 4151 len 12288 data: 6 7 8 163/300 Read block 2031 len 16384 164/300 Write block 1942 len 40960 data: 3 4 3 4 4 5 5 6 8 8 165/300 Flush 166/300 Read block 2240 len 24576 167/300 Write block 3436 len 40960 data: 5 6 5 5 5 4 4 5 5 5 168/300 Write block 4265 len 12288 data: 7 6 6 169/300 Write block 3620 len 24576 data: 6 5 5 5 5 5 170/300 Read block 3144 len 40960 171/300 Read block 2466 len 28672 172/300 Read block 4793 len 4096 173/300 Read block 1975 len 28672 174/300 Write block 375 len 16384 data: 7 5 6 7 175/300 Write block 3610 len 32768 data: 5 6 6 6 7 7 7 5 176/300 Write block 563 len 12288 data: 4 5 5 177/300 Write block 639 len 16384 data: 5 4 4 5 178/300 Read block 4065 len 8192 179/300 Read block 4059 len 8192 180/300 Read block 297 len 16384 181/300 Read block 4499 len 24576 182/300 Read block 3290 len 8192 183/300 Write block 1545 len 24576 data: 7 4 3 3 3 3 184/300 Read block 4193 len 12288 185/300 Flush 186/300 Write block 2653 len 20480 data: 5 5 5 4 3 187/300 Write block 4375 len 28672 data: 4 3 4 4 2 3 4 188/300 Read block 454 len 20480 189/300 Write block 4729 len 8192 data: 3 5 190/300 Read block 2767 len 12288 191/300 Read block 3668 len 28672 192/300 Write block 1059 len 16384 data: 4 4 5 4 193/300 Read block 4338 len 20480 194/300 Write block 1001 len 32768 data: 4 3 6 6 5 5 5 5 195/300 Write block 14 len 32768 data: 5 6 6 5 5 5 4 5 196/300 Read block 1806 len 8192 197/300 Write block 3810 len 36864 data: 7 7 7 6 7 6 3 3 3 198/300 Read block 693 len 16384 199/300 Write block 3887 len 20480 data: 5 5 7 7 7 200/300 Flush 201/300 Read block 565 len 32768 202/300 Read block 3175 len 20480 203/300 Read block 3217 len 20480 204/300 Flush 205/300 Write block 2214 len 40960 data: 4 5 4 5 5 6 6 6 7 4 206/300 Write block 2122 len 8192 data: 4 4 207/300 Write block 1789 len 4096 data: 3 208/300 Read block 4960 len 16384 209/300 Read block 105 len 8192 210/300 Flush 211/300 Read block 4915 len 16384 212/300 Read block 4275 len 40960 213/300 Write block 2876 len 28672 data: 7 7 6 5 4 3 3 214/300 Write block 3539 len 24576 data: 8 9 10 8 7 7 215/300 Write block 1751 len 24576 data: 7 8 7 6 6 5 216/300 Read block 4371 len 20480 217/300 Write block 422 len 8192 data: 4 4 218/300 Write block 4259 len 28672 data: 4 4 4 4 5 6 8 219/300 Read block 3548 len 4096 220/300 Flush 221/300 Read block 1995 len 36864 222/300 Write block 4549 len 40960 data: 7 5 5 4 4 5 5 3 4 4 223/300 Write block 25 len 40960 data: 5 5 6 5 9 9 11 10 9 7 224/300 Write block 1877 len 40960 data: 3 3 4 5 5 6 9 8 8 8 225/300 Read block 2734 len 40960 226/300 Read block 4555 len 36864 227/300 Read block 4537 len 28672 228/300 Write block 2349 len 40960 data: 7 8 8 5 4 6 4 3 2 3 229/300 Write block 501 len 24576 data: 2 2 2 4 3 5 230/300 Write block 3351 len 40960 data: 5 5 5 6 7 7 7 6 8 8 231/300 Write block 480 len 4096 data: 3 232/300 Write block 555 len 40960 data: 4 5 4 4 4 3 4 4 5 6 233/300 Write block 2562 len 4096 data: 6 234/300 Write block 4422 len 20480 data: 9 8 7 10 8 235/300 Flush 236/300 Read block 3089 len 32768 237/300 Flush 238/300 Flush 239/300 Write block 2308 len 12288 data: 10 10 9 240/300 Read block 3610 len 12288 241/300 Write block 744 len 36864 data: 6 6 6 6 7 7 7 8 10 242/300 Write block 926 len 24576 data: 2 3 3 3 4 6 243/300 Read block 452 len 12288 244/300 Read block 4039 len 12288 245/300 Read block 4501 len 12288 246/300 Read block 3254 len 20480 247/300 Write block 2039 len 24576 data: 9 7 8 8 9 10 248/300 Read block 1887 len 32768 249/300 Flush 250/300 Read block 1649 len 36864 251/300 Read block 3002 len 40960 252/300 Read block 3580 len 28672 253/300 Read block 4933 len 32768 254/300 Read block 2528 len 20480 255/300 Read block 3230 len 4096 256/300 Read block 2890 len 32768 257/300 Flush 258/300 Read block 4814 len 32768 259/300 Write block 2970 len 20480 data: 8 8 8 7 6 260/300 Read block 232 len 8192 261/300 Flush 262/300 Flush 263/300 Read block 4909 len 28672 264/300 Read block 3113 len 20480 265/300 Read block 4696 len 32768 266/300 Read block 449 len 24576 267/300 Write block 4324 len 20480 data: 4 4 4 5 6 268/300 Read block 1173 len 28672 269/300 Read block 2144 len 12288 270/300 Read block 1588 len 24576 271/300 Read block 1897 len 40960 272/300 Read block 4897 len 40960 273/300 Write block 2828 len 16384 data: 4 5 7 5 274/300 Write block 523 len 8192 data: 7 10 275/300 Write block 3468 len 20480 data: 5 6 6 6 5 276/300 Read block 1416 len 28672 277/300 Read block 2738 len 20480 278/300 Read block 219 len 36864 279/300 Write block 2462 len 32768 data: 5 6 7 7 7 7 7 6 280/300 Write block 1899 len 12288 data: 5 4 4 281/300 Write block 435 len 12288 data: 6 5 6 282/300 Write block 1161 len 4096 data: 6 283/300 Write block 4509 len 4096 data: 5 284/300 Write block 103 len 12288 data: 6 6 7 285/300 Read block 4638 len 16384 286/300 Read block 3796 len 24576 287/300 Read block 4523 len 24576 288/300 Read block 4118 len 28672 289/300 Write block 545 len 32768 data: 4 3 3 3 3 3 5 5 290/300 Read block 4341 len 16384 291/300 Write block 4256 len 16384 data: 6 5 4 5 292/300 Write block 3993 len 20480 data: 6 6 6 4 4 293/300 Read block 4234 len 40960 294/300 Write block 960 len 32768 data: 5 7 6 6 6 5 5 6 295/300 Read block 168 len 36864 296/300 Read block 2191 len 20480 297/300 Read block 3073 len 4096 298/300 Write block 3783 len 12288 data: 5 5 5 299/300 Write block 51 len 40960 data: 4 5 5 5 3 2 3 3 3 3 300/300 Write block 3104 len 16384 data: 9 9 9 9 301/300 Read block 1864 len 24576 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:48:17.962 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/0, method: POST, req_id: b0c01cb5-3a16-42e1-8e4f-4bb173d00398, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8545 Acked Read 1 Sent Done Done false 8546 Acked Read 5 Sent Done Done false 8547 Acked Write 1 Sent Done Done false 8548 Acked Write 10 Sent Done Done false 8549 Acked Write 6 Sent Done Done false 8550 Acked Flush 0 Sent Done Done false 8551 Acked Flush 0 Sent Done Done false 8552 Acked Read 6 Sent Done Done false 8553 Acked Write 8 Sent Done Done false 8554 Acked Write 6 Sent Done Done false 8555 Acked Write 7 Sent Done Done false 8556 Acked Flush 0 Sent Done Done false 8557 Acked Write 8 Sent Done Done false 8558 Acked Write 1 Sent Done Done false 8559 Acked Write 2 Sent Done Done false 8560 Acked Read 7 Sent Done Done false 8561 Acked Read 3 Sent Done Done false 8562 Acked Write 6 Sent Done Done false 8563 Acked Read 8 Sent Done Done false 8564 Acked Read 9 Sent Done Done false 8565 Acked Read 6 Sent Done Done false 8566 Acked Read 5 Sent Done Done false 8567 Acked Read 4 Sent Done Done false 8568 Acked Read 9 Sent Done Done false 8569 Acked Read 4 Sent Done Done false 8570 Acked Read 5 Sent Done Done false 8571 Acked Write 3 Sent Done Done false 8572 Acked Read 2 Sent Done Done false 8573 Acked Write 8 Sent Done Done false 8574 Acked Read 2 Sent Done Done false 8575 Acked Write 7 Sent Done Done false 8576 Acked Write 9 Sent Done Done false 8577 Acked Read 4 Sent Done Done false 8578 Acked Write 4 Sent Done Done false 8579 Acked Read 3 Sent Done Done false 8580 Acked Flush 0 Sent Done Done false 8581 Acked Read 5 Sent Done Done false 8582 Acked Read 7 Sent Done Done false 8583 Acked Write 3 Sent Done Done false 8584 Acked Read 6 Sent Done Done false 8585 Acked Read 4 Sent Done Done false 8586 Acked Flush 0 Sent Done Done false 8587 Acked Read 6 Sent Done Done false 8588 Acked Write 6 Sent Done Done false 8589 Acked Read 7 Sent Done Done false 8590 Acked Read 9 Sent Done Done false 8591 Acked Write 3 Sent Done Done false 8592 Acked Read 5 Sent Done Done false 8593 Acked Read 1 Sent Done Done false 8594 Acked Flush 0 Sent Done Done false 8595 Acked Flush 0 Sent Done Done false 8596 Acked Write 9 Sent Done Done false 8597 Acked Read 10 Sent Done Done false 8598 Acked Read 8 Sent Done Done false 8599 Acked Write 9 Sent Done Done false 8600 Acked Write 3 Sent Done Done false 8601 Acked Write 9 Sent Done Done false 8602 Acked Write 5 Sent Done Done false 8603 Acked Read 1 Sent Done Done false 8604 Acked Flush 0 Sent Done Done false 8605 Acked Read 2 Sent Done Done false 8606 Acked Read 4 Sent Done Done false 8607 Acked Write 8 Sent Done Done false 8608 Acked Read 5 Sent Done Done false 8609 Acked Flush 0 Sent Done Done false 8610 Acked Flush 0 Sent Done Done false 8611 Acked Read 8 Sent Done Done false 8612 Acked Write 9 Sent Done Done false 8613 Acked Write 9 Sent Done Done false 8614 Acked Write 4 Sent Done Done false 8615 Acked Write 5 Sent Done Done false 8616 Acked Write 6 Sent Done Done false 8617 Acked Write 5 Sent Done Done false 8618 Acked Write 4 Sent Done Done false 8619 Acked Read 9 Sent Done Done false 8620 Acked Write 5 Sent Done Done false 8621 Acked Write 6 Sent Done Done false 8622 Acked Read 2 Sent Done Done false 8623 Acked Write 10 Sent Done Done false 8624 Acked Write 5 Sent Done Done false 8625 Acked Write 3 Sent Done Done false 8626 Acked Write 8 Sent Done Done false 8627 Acked Read 2 Sent Done Done false 8628 Acked Read 4 Sent Done Done false 8629 Acked Read 10 Sent Done Done false 8630 Acked Write 3 Sent Done Done false 8631 Acked Flush 0 Sent Done Done false 8632 Acked Write 4 Sent Done Done false 8633 Acked Flush 0 Sent Done Done false 8634 Acked Read 2 Sent Done Done false 8635 Acked Read 9 Sent Done Done false 8636 Acked Write 8 Sent Done Done false 8637 Acked Read 1 Sent Done Done false 8638 Acked Read 4 Sent Done Done false 8639 Acked Flush 0 Sent Done Done false 8640 Acked Read 5 Sent Done Done false 8641 Acked Read 3 Sent Done Done false 8642 Acked Read 2 Sent Done Done false 8643 Acked Read 1 Sent Done Done false 8644 Acked Write 2 Sent Done Done false 8645 Acked Write 9 Sent Done Done false 8646 Acked Write 1 Sent Done Done false 8647 Acked Write 2 Sent Done Done false 8648 Acked Flush 0 Sent Done Done false 8649 Acked Flush 0 Sent Done Done false 8650 Acked Read 7 Sent Done Done false 8651 Acked Read 3 Sent Done Done false 8652 Acked Write 5 Sent Done Done false 8653 Acked Write 6 Sent Done Done false 8654 Acked Write 1 Sent Done Done false 8655 Acked Flush 0 Sent Done Done false 8656 Acked Read 4 Sent Done Done false 8657 Acked Read 9 Sent Done Done false 8658 Acked Read 7 Sent Done Done false 8659 Acked Read 8 Sent Done Done false 8660 Acked Read 6 Sent Done Done false 8661 Acked Read 5 Sent Done Done false 8662 Acked Read 9 Sent Done Done false 8663 Acked Write 9 Sent Done Done false 8664 Acked Write 8 Sent Done Done false 8665 Acked Write 3 Sent Done Done false 8666 Acked Write 8 Sent Done Done false 8667 Acked Read 7 Sent Done Done false 8668 Acked Write 5 Sent Done Done false 8669 Acked Write 5 Sent Done Done false 8670 Acked Read 1 Sent Done Done false 8671 Acked Write 10 Sent Done Done false 8672 Acked Write 7 Sent Done Done false 8673 Acked Write 4 Sent Done Done false 8674 Acked Write 10 Sent Done Done false 8675 Acked Read 8 Sent Done Done false 8676 Acked Read 4 Sent Done Done false 8677 Acked Read 4 Sent Done Done false 8678 Acked Write 8 Sent Done Done false 8679 Acked Write 3 Sent Done Done false 8680 Acked Read 5 Sent Done Done false 8681 Acked Read 4 Sent Done Done false 8682 Acked Write 3 Sent Done Done false 8683 Acked Read 5 Sent Done Done false 8684 Acked Write 9 Sent Done Done false 8685 Acked Read 5 Sent Done Done false 8686 Acked Write 8 Sent Done Done false 8687 Acked Read 5 Sent Done Done false 8688 Acked Write 6 Sent Done Done false 8689 Acked Read 8 Sent Done Done false 8690 Acked Read 1 Sent Done Done false 8691 Acked Read 10 Sent Done Done false 8692 Acked Write 10 Sent Done Done false 8693 Acked Write 9 Sent Done Done false 8694 Acked Write 7 Sent Done Done false 8695 Acked Flush 0 Sent Done Done false 8696 Acked Read 3 Sent Done Done false 8697 Acked Read 1 Sent Done Done false 8698 Acked Write 9 Sent Done Done false 8699 Acked Write 10 Sent Done Done false 8700 Acked Read 2 Sent Done Done false 8701 Acked Write 5 Sent Done Done false 8702 Acked Flush 0 Sent Done Done false 8703 Acked Write 9 Sent Done Done false 8704 Acked Read 2 Sent Done Done false 8705 Acked Write 2 Sent Done Done false 8706 Acked Write 3 Sent Done Done false 8707 Acked Read 4 Sent Done Done false 8708 Acked Write 10 Sent Done Done false 8709 Acked Flush 0 Sent Done Done false 8710 Acked Read 6 Sent Done Done false 8711 Acked Write 10 Sent Done Done false 8712 Acked Write 3 Sent Done Done false 8713 Acked Write 6 Sent Done Done false 8714 Acked Read 10 Sent Done Done false 8715 Acked Read 7 Sent Done Done false 8716 Acked Read 1 Sent Done Done false 8717 Acked Read 7 Sent Done Done false 8718 Acked Write 4 Sent Done Done false 8719 Acked Write 8 Sent Done Done false 8720 Acked Write 3 Sent Done Done false 8721 Acked Write 4 Sent Done Done false 8722 Acked Read 2 Sent Done Done false 8723 Acked Read 2 Sent Done Done false 8724 Acked Read 4 Sent Done Done false 8725 Acked Read 6 Sent Done Done false 8726 Acked Read 2 Sent Done Done false 8727 Acked Write 6 Sent Done Done false 8728 Acked Read 3 Sent Done Done false 8729 Acked Flush 0 Sent Done Done false 8730 Acked Write 5 Sent Done Done false 8731 Acked Write 7 Sent Done Done false 8732 Acked Read 5 Sent Done Done false 8733 Acked Write 2 Sent Done Done false 8734 Acked Read 3 Sent Done Done false 8735 Acked Read 7 Sent Done Done false 8736 Acked Write 4 Sent Done Done false 8737 Acked Read 5 Sent Done Done false 8738 Acked Write 8 Sent Done Done false 8739 Acked Write 8 Sent Done Done false 8740 Acked Read 2 Sent Done Done false 8741 Acked Write 9 Sent Done Done false 8742 Acked Read 4 Sent Done Done false 8743 Acked Write 5 Sent Done Done false 8744 Acked Flush 0 Sent Done Done false 8745 Acked Read 8 Sent Done Done false 8746 Acked Read 5 Sent Done Done false 8747 Acked Read 5 Sent Done Done false 8748 Acked Flush 0 Sent Done Done false 8749 Acked Write 10 Sent Done Done false 8750 Acked Write 2 Sent Done Done false 8751 Acked Write 1 Sent Done Done false 8752 Acked Read 4 Sent Done Done false 8753 Acked Read 2 Sent Done Done false 8754 Acked Flush 0 Sent Done Done false 8755 Acked Read 4 Sent Done Done false 8756 Acked Read 10 Sent Done Done false 8757 Acked Write 7 Sent Done Done false 8758 Acked Write 6 Sent Done Done false 8759 Acked Write 6 Sent Done Done false 8760 Acked Read 5 Sent Done Done false 8761 Acked Write 2 Sent Done Done false 8762 Acked Write 7 Sent Done Done false 8763 Acked Read 1 Sent Done Done false 8764 Acked Flush 0 Sent Done Done false 8765 Acked Read 9 Sent Done Done false 8766 Acked Write 10 Sent Done Done false 8767 Acked Write 10 Sent Done Done false 8768 Acked Write 10 Sent Done Done false 8769 Acked Read 10 Sent Done Done false 8770 Acked Read 9 Sent Done Done false 8771 Acked Read 7 Sent Done Done false 8772 Acked Write 10 Sent Done Done false 8773 Acked Write 6 Sent Done Done false 8774 Acked Write 10 Sent Done Done false 8775 Acked Write 1 Sent Done Done false 8776 Acked Write 10 Sent Done Done false 8777 Acked Write 1 Sent Done Done false 8778 Acked Write 5 Sent Done Done false 8779 Acked Flush 0 Sent Done Done false 8780 Acked Read 8 Sent Done Done false 8781 Acked Flush 0 Sent Done Done false 8782 Acked Flush 0 Sent Done Done false 8783 Acked Write 3 Sent Done Done false 8784 Acked Read 3 Sent Done Done false 8785 Acked Write 9 Sent Done Done false 8786 Acked Write 6 Sent Done Done false 8787 Acked Read 3 Sent Done Done false 8788 Acked Read 3 Sent Done Done false 8789 Acked Read 3 Sent Done Done false 8790 Acked Read 5 Sent Done Done false 8791 Acked Write 6 Sent Done Done false 8792 Acked Read 8 Sent Done Done false 8793 Acked Flush 0 Sent Done Done false 8794 Acked Read 9 Sent Done Done false 8795 Acked Read 10 Sent Done Done false 8796 Acked Read 7 Sent Done Done false 8797 Acked Read 8 Sent Done Done false 8798 Acked Read 5 Sent Done Done false 8799 Acked Read 1 Sent Done Done false 8800 Acked Read 8 Sent Done Done false 8801 Acked Flush 0 Sent Done Done false 8802 Acked Read 8 Sent Done Done false 8803 Acked Write 5 Sent Done Done false 8804 Acked Read 2 Sent Done Done false 8805 Acked Flush 0 Sent Done Done false 8806 Acked Flush 0 Sent Done Done false 8807 Acked Read 7 Sent Done Done false 8808 Acked Read 5 Sent Done Done false 8809 Acked Read 8 Sent Done Done false 8810 Acked Read 6 Sent Done Done false 8811 Acked Write 5 Sent Done Done false 8812 Acked Read 7 Sent Done Done false 8813 Acked Read 3 Sent Done Done false 8814 Acked Read 6 Sent Done Done false 8815 Acked Read 10 Sent Done Done false 8816 Acked Read 10 Sent Done Done false 8817 Acked Write 4 Sent Done Done false 8818 Acked Write 2 Sent Done Done false 8819 Acked Write 5 Sent Done Done false 8820 Acked Read 7 Sent Done Done false 8821 Acked Read 5 Sent Done Done false 8822 Acked Read 9 Sent Done Done false 8823 Acked Write 8 Sent Done Done false 8824 Acked Write 3 Sent Done Done false 8825 Acked Write 3 Sent Done Done false 8826 Acked Write 1 Sent Done Done false 8827 Acked Write 1 Sent Done Done false 8828 Acked Write 3 Sent Done Done false 8829 Acked Read 4 Sent Done Done false 8830 Acked Read 6 Sent Done Done false 8831 Acked Read 6 Sent Done Done false 8832 Acked Read 7 Sent Done Done false 8833 Acked Write 8 Sent Done Done false 8834 Acked Read 4 Sent Done Done false 8835 Acked Write 4 Sent Done Done false 8836 Acked Write 5 Sent Done Done false 8837 Acked Read 10 Sent Done Done false 8838 Acked Write 8 Sent Done Done false 8839 Acked Read 9 Sent Done Done false 8840 Acked Read 5 Sent Done Done false 8841 Acked Read 1 Sent Done Done false 8842 Acked Write 3 Sent Done Done false 8843 Acked Write 10 Sent Done Done false 8844 Acked Write 4 Sent Done Done false 8845 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 0 0 301 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8544 8806 8806 Downstairs last five completed: 8544 8543 8542 8541 8540 Upstairs last five completed: 8845 8844 8843 8842 8841 CLIENT: Up:0 ds:301 act:2 [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1228) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8545 Acked Read 1 Sent Done Done false 8546 Acked Read 5 Sent Done Done false 8547 Acked Write 1 Sent Done Done false 8548 Acked Write 10 Sent Done Done false 8549 Acked Write 6 Sent Done Done false 8550 Acked Flush 0 Sent Done Done false 8551 Acked Flush 0 Sent Done Done false 8552 Acked Read 6 Sent Done Done false 8553 Acked Write 8 Sent Done Done false 8554 Acked Write 6 Sent Done Done false 8555 Acked Write 7 Sent Done Done false 8556 Acked Flush 0 Sent Done Done false 8557 Acked Write 8 Sent Done Done false 8558 Acked Write 1 Sent Done Done false 8559 Acked Write 2 Sent Done Done false 8560 Acked Read 7 Sent Done Done false 8561 Acked Read 3 Sent Done Done false 8562 Acked Write 6 Sent Done Done false 8563 Acked Read 8 Sent Done Done false 8564 Acked Read 9 Sent Done Done false 8565 Acked Read 6 Sent Done Done false 8566 Acked Read 5 Sent Done Done false 8567 Acked Read 4 Sent Done Done false 8568 Acked Read 9 Sent Done Done false 8569 Acked Read 4 Sent Done Done false 8570 Acked Read 5 Sent Done Done false 8571 Acked Write 3 Sent Done Done false 8572 Acked Read 2 Sent Done Done false 8573 Acked Write 8 Sent Done Done false 8574 Acked Read 2 Sent Done Done false 8575 Acked Write 7 Sent Done Done false 8576 Acked Write 9 Sent Done Done false 8577 Acked Read 4 Sent Done Done false 8578 Acked Write 4 Sent Done Done false 8579 Acked Read 3 Sent Done Done false 8580 Acked Flush 0 Sent Done Done false 8581 Acked Read 5 Sent Done Done false 8582 Acked Read 7 Sent Done Done false 8583 Acked Write 3 Sent Done Done false 8584 Acked Read 6 Sent Done Done false 8585 Acked Read 4 Sent Done Done false 8586 Acked Flush 0 Sent Done Done false 8587 Acked Read 6 Sent Done Done false 8588 Acked Write 6 Sent Done Done false 8589 Acked Read 7 Sent Done Done false 8590 Acked Read 9 Sent Done Done false 8591 Acked Write 3 Sent Done Done false 8592 Acked Read 5 Sent Done Done false 8593 Acked Read 1 Sent Done Done false 8594 Acked Flush 0 Sent Done Done false 8595 Acked Flush 0 Sent Done Done false 8596 Acked Write 9 Sent Done Done false 8597 Acked Read 10 Sent Done Done false 8598 Acked Read 8 Sent Done Done false 8599 Acked Write 9 Sent Done Done false 8600 Acked Write 3 Sent Done Done false 8601 Acked Write 9 Sent Done Done false 8602 Acked Write 5 Sent Done Done false 8603 Acked Read 1 Sent Done Done false 8604 Acked Flush 0 Sent Done Done false 8605 Acked Read 2 Sent Done Done false 8606 Acked Read 4 Sent Done Done false 8607 Acked Write 8 Sent Done Done false 8608 Acked Read 5 Sent Done Done false 8609 Acked Flush 0 Sent Done Done false 8610 Acked Flush 0 Sent Done Done false 8611 Acked Read 8 Sent Done Done false 8612 Acked Write 9 Sent Done Done false 8613 Acked Write 9 Sent Done Done false 8614 Acked Write 4 Sent Done Done false 8615 Acked Write 5 Sent Done Done false 8616 Acked Write 6 Sent Done Done false 8617 Acked Write 5 Sent Done Done false 8618 Acked Write 4 Sent Done Done false 8619 Acked Read 9 Sent Done Done false 8620 Acked Write 5 Sent Done Done false 8621 Acked Write 6 Sent Done Done false 8622 Acked Read 2 Sent Done Done false 8623 Acked Write 10 Sent Done Done false 8624 Acked Write 5 Sent Done Done false 8625 Acked Write 3 Sent Done Done false 8626 Acked Write 8 Sent Done Done false 8627 Acked Read 2 Sent Done Done false 8628 Acked Read 4 Sent Done Done false 8629 Acked Read 10 Sent Done Done false 8630 Acked Write 3 Sent Done Done false 8631 Acked Flush 0 Sent Done Done false 8632 Acked Write 4 Sent Done Done false 8633 Acked Flush 0 Sent Done Done false 8634 Acked Read 2 Sent Done Done false 8635 Acked Read 9 Sent Done Done false 8636 Acked Write 8 Sent Done Done false 8637 Acked Read 1 Sent Done Done false 8638 Acked Read 4 Sent Done Done false 8639 Acked Flush 0 Sent Done Done false 8640 Acked Read 5 Sent Done Done false 8641 Acked Read 3 Sent Done Done false 8642 Acked Read 2 Sent Done Done false 8643 Acked Read 1 Sent Done Done false 8644 Acked Write 2 Sent Done Done false 8645 Acked Write 9 Sent Done Done false 8646 Acked Write 1 Sent Done Done false 8647 Acked Write 2 Sent Done Done false 8648 Acked Flush 0 Sent Done Done false 8649 Acked Flush 0 Sent Done Done false 8650 Acked Read 7 Sent Done Done false 8651 Acked Read 3 Sent Done Done false 8652 Acked Write 5 Sent Done Done false 8653 Acked Write 6 Sent Done Done false 8654 Acked Write 1 Sent Done Done false 8655 Acked Flush 0 Sent Done Done false 8656 Acked Read 4 Sent Done Done false 8657 Acked Read 9 Sent Done Done false 8658 Acked Read 7 Sent Done Done false 8659 Acked Read 8 Sent Done Done false 8660 Acked Read 6 Sent Done Done false 8661 Acked Read 5 Sent Done Done false 8662 Acked Read 9 Sent Done Done false 8663 Acked Write 9 Sent Done Done false 8664 Acked Write 8 Sent Done Done false 8665 Acked Write 3 Sent Done Done false 8666 Acked Write 8 Sent Done Done false 8667 Acked Read 7 Sent Done Done false 8668 Acked Write 5 Sent Done Done false 8669 Acked Write 5 Sent Done Done false 8670 Acked Read 1 Sent Done Done false 8671 Acked Write 10 Sent Done Done false 8672 Acked Write 7 Sent Done Done false 8673 Acked Write 4 Sent Done Done false 8674 Acked Write 10 Sent Done Done false 8675 Acked Read 8 Sent Done Done false 8676 Acked Read 4 Sent Done Done false 8677 Acked Read 4 Sent Done Done false 8678 Acked Write 8 Sent Done Done false 8679 Acked Write 3 Sent Done Done false 8680 Acked Read 5 Sent Done Done false 8681 Acked Read 4 Sent Done Done false 8682 Acked Write 3 Sent Done Done false 8683 Acked Read 5 Sent Done Done false 8684 Acked Write 9 Sent Done Done false 8685 Acked Read 5 Sent Done Done false 8686 Acked Write 8 Sent Done Done false 8687 Acked Read 5 Sent Done Done false 8688 Acked Write 6 Sent Done Done false 8689 Acked Read 8 Sent Done Done false 8690 Acked Read 1 Sent Done Done false 8691 Acked Read 10 Sent Done Done false 8692 Acked Write 10 Sent Done Done false 8693 Acked Write 9 Sent Done Done false 8694 Acked Write 7 Sent Done Done false 8695 Acked Flush 0 Sent Done Done false 8696 Acked Read 3 Sent Done Done false 8697 Acked Read 1 Sent Done Done false 8698 Acked Write 9 Sent Done Done false 8699 Acked Write 10 Sent Done Done false 8700 Acked Read 2 Sent Done Done false 8701 Acked Write 5 Sent Done Done false 8702 Acked Flush 0 Sent Done Done false 8703 Acked Write 9 Sent Done Done false 8704 Acked Read 2 Sent Done Done false 8705 Acked Write 2 Sent Done Done false 8706 Acked Write 3 Sent Done Done false 8707 Acked Read 4 Sent Done Done false 8708 Acked Write 10 Sent Done Done false 8709 Acked Flush 0 Sent Done Done false 8710 Acked Read 6 Sent Done Done false 8711 Acked Write 10 Sent Done Done false 8712 Acked Write 3 Sent Done Done false 8713 Acked Write 6 Sent Done Done false 8714 Acked Read 10 Sent Done Done false 8715 Acked Read 7 Sent Done Done false 8716 Acked Read 1 Sent Done Done false 8717 Acked Read 7 Sent Done Done false 8718 Acked Write 4 Sent Done Done false 8719 Acked Write 8 Sent Done Done false 8720 Acked Write 3 Sent Done Done false 8721 Acked Write 4 Sent Done Done false 8722 Acked Read 2 Sent Done Done false 8723 Acked Read 2 Sent Done Done false 8724 Acked Read 4 Sent Done Done false 8725 Acked Read 6 Sent Done Done false 8726 Acked Read 2 Sent Done Done false 8727 Acked Write 6 Sent Done Done false 8728 Acked Read 3 Sent Done Done false 8729 Acked Flush 0 Sent Done Done false 8730 Acked Write 5 Sent Done Done false 8731 Acked Write 7 Sent Done Done false 8732 Acked Read 5 Sent Done Done false 8733 Acked Write 2 Sent Done Done false 8734 Acked Read 3 Sent Done Done false 8735 Acked Read 7 Sent Done Done false 8736 Acked Write 4 Sent Done Done false 8737 Acked Read 5 Sent Done Done false 8738 Acked Write 8 Sent Done Done false 8739 Acked Write 8 Sent Done Done false 8740 Acked Read 2 Sent Done Done false 8741 Acked Write 9 Sent Done Done false 8742 Acked Read 4 Sent Done Done false 8743 Acked Write 5 Sent Done Done false 8744 Acked Flush 0 Sent Done Done false 8745 Acked Read 8 Sent Done Done false 8746 Acked Read 5 Sent Done Done false 8747 Acked Read 5 Sent Done Done false 8748 Acked Flush 0 Sent Done Done false 8749 Acked Write 10 Sent Done Done false 8750 Acked Write 2 Sent Done Done false 8751 Acked Write 1 Sent Done Done false 8752 Acked Read 4 Sent Done Done false 8753 Acked Read 2 Sent Done Done false 8754 Acked Flush 0 Sent Done Done false 8755 Acked Read 4 Sent Done Done false 8756 Acked Read 10 Sent Done Done false 8757 Acked Write 7 Sent Done Done false 8758 Acked Write 6 Sent Done Done false 8759 Acked Write 6 Sent Done Done false 8760 Acked Read 5 Sent Done Done false 8761 Acked Write 2 Sent Done Done false 8762 Acked Write 7 Sent Done Done false 8763 Acked Read 1 Sent Done Done false 8764 Acked Flush 0 Sent Done Done false 8765 Acked Read 9 Sent Done Done false 8766 Acked Write 10 Sent Done Done false 8767 Acked Write 10 Sent Done Done false 8768 Acked Write 10 Sent Done Done false 8769 Acked Read 10 Sent Done Done false 8770 Acked Read 9 Sent Done Done false 8771 Acked Read 7 Sent Done Done false 8772 Acked Write 10 Sent Done Done false 8773 Acked Write 6 Sent Done Done false 8774 Acked Write 10 Sent Done Done false 8775 Acked Write 1 Sent Done Done false 8776 Acked Write 10 Sent Done Done false 8777 Acked Write 1 Sent Done Done false 8778 Acked Write 5 Sent Done Done false 8779 Acked Flush 0 Sent Done Done false 8780 Acked Read 8 Sent Done Done false 8781 Acked Flush 0 Sent Done Done false 8782 Acked Flush 0 Sent Done Done false 8783 Acked Write 3 Sent Done Done false 8784 Acked Read 3 Sent Done Done false 8785 Acked Write 9 Sent Done Done false 8786 Acked Write 6 Sent Done Done false 8787 Acked Read 3 Sent Done Done false 8788 Acked Read 3 Sent Done Done false 8789 Acked Read 3 Sent Done Done false 8790 Acked Read 5 Sent Done Done false 8791 Acked Write 6 Sent Done Done false 8792 Acked Read 8 Sent Done Done false 8793 Acked Flush 0 Sent Done Done false 8794 Acked Read 9 Sent Done Done false 8795 Acked Read 10 Sent Done Done false 8796 Acked Read 7 Sent Done Done false 8797 Acked Read 8 Sent Done Done false 8798 Acked Read 5 Sent Done Done false 8799 Acked Read 1 Sent Done Done false 8800 Acked Read 8 Sent Done Done false 8801 Acked Flush 0 Sent Done Done false 8802 Acked Read 8 Sent Done Done false 8803 Acked Write 5 Sent Done Done false 8804 Acked Read 2 Sent Done Done false 8805 Acked Flush 0 Sent Done Done false 8806 Acked Flush 0 Sent Done Done false 8807 Acked Read 7 Sent Done Done false 8808 Acked Read 5 Sent Done Done false 8809 Acked Read 8 Sent Done Done false 8810 Acked Read 6 Sent Done Done false 8811 Acked Write 5 Sent Done Done false 8812 Acked Read 7 Sent Done Done false 8813 Acked Read 3 Sent Done Done false 8814 Acked Read 6 Sent Done Done false 8815 Acked Read 10 Sent Done Done false 8816 Acked Read 10 Sent Done Done false 8817 Acked Write 4 Sent Done Done false 8818 Acked Write 2 Sent Done Done false 8819 Acked Write 5 Sent Done Done false 8820 Acked Read 7 Sent Done Done false 8821 Acked Read 5 Sent Done Done false 8822 Acked Read 9 Sent Done Done false 8823 Acked Write 8 Sent Done Done false 8824 Acked Write 3 Sent Done Done false 8825 Acked Write 3 Sent Done Done false 8826 Acked Write 1 Sent Done Done false 8827 Acked Write 1 Sent Done Done false 8828 Acked Write 3 Sent Done Done false 8829 Acked Read 4 Sent Done Done false 8830 Acked Read 6 Sent Done Done false 8831 Acked Read 6 Sent Done Done false 8832 Acked Read 7 Sent Done Done false 8833 Acked Write 8 Sent Done Done false 8834 Acked Read 4 Sent Done Done false 8835 Acked Write 4 Sent Done Done false 8836 Acked Write 5 Sent Done Done false 8837 Acked Read 10 Sent Done Done false 8838 Acked Write 8 Sent Done Done false 8839 Acked Read 9 Sent Done Done false 8840 Acked Read 5 Sent Done Done false 8841 Acked Read 1 Sent Done Done false 8842 Acked Write 3 Sent Done Done false 8843 Acked Write 10 Sent Done Done false 8844 Acked Write 4 Sent Done Done false 8845 Acked Read 6 Sent Done Done false 8846 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8544 8846 8846 Downstairs last five completed: 8544 8543 8542 8541 8540 Upstairs last five completed: 8846 8845 8844 8843 8842 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.73703241Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:53661)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.737103001Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:23.738068622Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738577955Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738600749Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 8544","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738615875Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 8544","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738834356Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.73885833Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738872726Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 8544","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.738886312Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 8544","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:23.739110982Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8846 8845 8844 8843 8842 Upstairs last five completed: 8846 8845 8844 8843 8842 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:48:25.967 INFO request completed, latency_us: 331, response_code: 204, uri: /stop/cid/1, method: POST, req_id: eda3b973-cd91-4625-afa0-c752cb2c0515, remote_addr:[8820] Got stop action so:false kr:false 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:25.968156237Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:25.968243673Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:25.968266687Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:48:25.968 INFO request completed, latency_us:{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:25.968297099Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 137, response_code: 200, uri: {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:25.968340887Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} /state/cid/1, method: GET, req_id: 6c57c9d3-1f3c-4f36-8226-2fb616f67ad3, remote_addr: 127.0.0.1:51788, local_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:25.968370329Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} 127.0.0.1:9998 [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:48:29.969 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/1, method: GET, req_id: 94e4928e-00ad-43bf-a096-9db44ddd8ab1, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 4753 len 40960 data: 5 6 6 5 5 7 6 6 6 7 002/300 Read block 252 len 20480 003/300 Write block 1596 len 24576 data: 2 3 5 6 5 5 004/300 Read block 2786 len 16384 005/300 Read block 4600 len 36864 006/300 Write block 4414 len 4096 data: 7 007/300 Write block 4699 len 8192 data: 6 6 008/300 Flush 009/300 Read block 2834 len 32768 010/300 Read block 3435 len 16384 011/300 Flush 012/300 Read block 3305 len 12288 013/300 Read block 3964 len 4096 014/300 Read block 2800 len 40960 015/300 Read block 4604 len 32768 016/300 Flush 017/300 Write block 1955 len 20480 data: 10 6 9 8 8 018/300 Write block 3812 len 20480 data: 8 7 8 7 4 019/300 Read block 2344 len 8192 020/300 Read block 2813 len 8192 021/300 Write block 4788 len 4096 data: 4 022/300 Read block 1449 len 4096 023/300 Write block 2914 len 36864 data: 5 3 5 7 8 7 7 7 8 024/300 Read block 2573 len 24576 025/300 Write block 4480 len 36864 data: 6 6 6 6 7 7 4 3 3 026/300 Write block 3431 len 8192 data: 3 3 027/300 Read block 3573 len 24576 028/300 Read block 3073 len 28672 029/300 Write block 1505 len 28672 data: 4 3 5 6 7 6 5 030/300 Write block 143 len 32768 data: 3 4 4 6 5 5 6 5 031/300 Read block 3896 len 28672 032/300 Read block 145 len 28672 033/300 Write block 2269 len 12288 data: 8 7 8 034/300 Write block 800 len 20480 data: 5 5 5 5 6 035/300 Read block 3734 len 36864 036/300 Write block 3635 len 40960 data: 5 4 4 4 3 4 5 5 4 4 037/300 Read block 2242 len 36864 038/300 Write block 2966 len 16384 data: 3 4 3 7 039/300 Write block 2215 len 16384 data: 6 5 6 6 040/300 Read block 4876 len 4096 041/300 Read block 777 len 36864 042/300 Read block 2507 len 28672 043/300 Flush 044/300 Read block 3101 len 40960 045/300 Read block 2112 len 8192 046/300 Flush 047/300 Write block 2382 len 24576 data: 3 3 5 8 9 9 048/300 Write block 4681 len 8192 data: 8 9 049/300 Write block 1107 len 8192 data: 8 11 050/300 Flush 051/300 Read block 4224 len 8192 052/300 Write block 3964 len 20480 data: 5 6 5 3 2 053/300 Read block 667 len 12288 054/300 Write block 1962 len 36864 data: 7 7 6 5 4 4 4 3 4 055/300 Read block 3225 len 20480 056/300 Write block 3595 len 8192 data: 6 8 057/300 Write block 651 len 40960 data: 5 5 5 5 6 5 4 5 5 6 058/300 Write block 3416 len 36864 data: 2 2 3 3 3 4 6 5 5 059/300 Read block 1358 len 28672 060/300 Write block 1562 len 4096 data: 5 061/300 Flush 062/300 Read block 3814 len 8192 063/300 Write block 526 len 8192 data: 10 7 064/300 Read block 1307 len 20480 065/300 Flush 066/300 Write block 1492 len 4096 data: 4 067/300 Read block 1528 len 16384 068/300 Write block 2817 len 28672 data: 5 5 5 4 5 6 7 069/300 Read block 4227 len 40960 070/300 Read block 2860 len 20480 071/300 Read block 1361 len 4096 072/300 Read block 1112 len 8192 073/300 Read block 23 len 36864 074/300 Write block 704 len 20480 data: 5 5 5 6 5 075/300 Write block 2305 len 12288 data: 10 11 11 076/300 Write block 4394 len 20480 data: 8 8 8 7 6 077/300 Write block 4706 len 36864 data: 6 7 9 8 7 8 7 6 6 078/300 Read block 3065 len 28672 079/300 Read block 1343 len 36864 080/300 Read block 3394 len 40960 081/300 Read block 1074 len 24576 082/300 Write block 1505 len 20480 data: 5 4 6 7 8 083/300 Read block 3134 len 4096 084/300 Write block 2548 len 4096 data: 4 085/300 Read block 4446 len 16384 086/300 Read block 1730 len 24576 087/300 Read block 1155 len 24576 088/300 Flush 089/300 Read block 4218 len 16384 090/300 Read block 2703 len 20480 091/300 Write block 134 len 4096 data: 3 092/300 Write block 17 len 36864 data: 6 6 6 5 6 5 4 4 6 093/300 Read block 1535 len 20480 094/300 Write block 2927 len 20480 data: 4 3 3 5 4 095/300 Read block 1955 len 40960 096/300 Read block 3428 len 24576 097/300 Write block 609 len 28672 data: 5 4 4 4 3 3 4 098/300 Write block 1674 len 20480 data: 5 6 5 5 4 099/300 Flush 100/300 Read block 4862 len 16384 101/300 Write block 4408 len 4096 data: 4 102/300 Write block 4662 len 4096 data: 9 103/300 Write block 1971 len 20480 data: 5 3 5 6 6 104/300 Write block 2035 len 28672 data: 6 6 6 6 10 8 9 105/300 Read block 1966 len 28672 106/300 Write block 4050 len 28672 data: 7 5 5 5 4 4 4 107/300 Write block 2278 len 20480 data: 4 5 5 5 5 108/300 Write block 1605 len 36864 data: 5 6 8 7 8 8 8 9 7 109/300 Write block 0 len 8192 data: 3 3 110/300 Flush 111/300 Read block 381 len 32768 112/300 Read block 1219 len 4096 113/300 Read block 867 len 40960 114/300 Read block 1479 len 4096 115/300 Flush 116/300 Read block 4011 len 8192 117/300 Write block 1798 len 4096 data: 5 118/300 Write block 802 len 20480 data: 6 6 7 5 7 119/300 Write block 393 len 36864 data: 4 3 3 3 3 3 3 3 4 120/300 Read block 3256 len 32768 121/300 Write block 4407 len 4096 data: 3 122/300 Read block 1458 len 12288 123/300 Read block 1581 len 4096 124/300 Read block 3013 len 36864 125/300 Read block 814 len 24576 126/300 Read block 292 len 24576 127/300 Read block 3919 len 8192 128/300 Read block 41 len 12288 129/300 Read block 2146 len 12288 130/300 Write block 4855 len 40960 data: 5 4 5 5 6 7 6 7 5 4 131/300 Write block 1209 len 28672 data: 7 6 6 7 7 4 5 132/300 Write block 2236 len 40960 data: 4 5 5 4 4 5 5 5 7 10 133/300 Flush 134/300 Read block 1043 len 36864 135/300 Read block 4640 len 36864 136/300 Write block 1547 len 4096 data: 4 137/300 Read block 3373 len 36864 138/300 Read block 4152 len 32768 139/300 Write block 3605 len 20480 data: 4 3 2 4 5 140/300 Read block 2938 len 32768 141/300 Write block 4731 len 40960 data: 5 5 5 4 4 4 3 3 3 3 142/300 Read block 923 len 24576 143/300 Write block 2910 len 24576 data: 4 4 4 4 6 4 144/300 Write block 4284 len 4096 data: 6 145/300 Read block 3962 len 36864 146/300 Read block 1061 len 12288 147/300 Read block 1859 len 20480 148/300 Flush 149/300 Write block 1560 len 16384 data: 6 6 6 5 150/300 Write block 2961 len 12288 data: 7 6 5 151/300 Read block 2788 len 28672 152/300 Write block 3852 len 28672 data: 8 7 5 4 4 4 6 153/300 Read block 1441 len 8192 154/300 Write block 1194 len 40960 data: 5 6 7 6 6 6 9 8 8 9 155/300 Read block 3734 len 4096 156/300 Write block 4115 len 20480 data: 4 4 4 5 4 157/300 Read block 1336 len 24576 158/300 Read block 3597 len 32768 159/300 Write block 2071 len 36864 data: 5 5 5 7 7 6 5 7 8 160/300 Write block 20 len 20480 data: 6 7 6 5 5 161/300 Read block 3327 len 4096 162/300 Read block 4182 len 8192 163/300 Read block 2908 len 12288 164/300 Read block 1534 len 12288 165/300 Write block 3646 len 32768 data: 4 3 5 5 4 3 4 8 166/300 Read block 108 len 36864 167/300 Read block 904 len 32768 168/300 Read block 2377 len 40960 169/300 Write block 1118 len 40960 data: 8 7 5 4 5 6 5 4 5 5 170/300 Write block 186 len 20480 data: 5 7 7 6 6 171/300 Write block 1862 len 28672 data: 2 4 5 5 7 7 7 172/300 Write block 2108 len 24576 data: 5 6 6 5 7 8 173/300 Read block 307 len 20480 174/300 Write block 2480 len 16384 data: 7 6 4 5 175/300 Read block 4219 len 40960 176/300 Write block 4266 len 4096 data: 7 177/300 Read block 4664 len 16384 178/300 Flush 179/300 Read block 3747 len 24576 180/300 Read block 2746 len 8192 181/300 Write block 3808 len 16384 data: 8 9 8 8 182/300 Write block 3511 len 16384 data: 5 5 4 5 183/300 Read block 4197 len 8192 184/300 Read block 4890 len 8192 185/300 Read block 3033 len 24576 186/300 Read block 1786 len 12288 187/300 Write block 25 len 24576 data: 7 6 7 6 10 10 188/300 Read block 1535 len 36864 189/300 Read block 412 len 24576 190/300 Flush 191/300 Read block 3732 len 36864 192/300 Read block 3447 len 32768 193/300 Read block 1930 len 28672 194/300 Write block 2956 len 20480 data: 7 9 8 7 7 195/300 Read block 25 len 28672 196/300 Read block 2558 len 28672 197/300 Read block 662 len 8192 198/300 Read block 1198 len 24576 199/300 Flush 200/300 Flush 201/300 Read block 4598 len 24576 202/300 Write block 1893 len 24576 data: 6 5 3 3 4 3 203/300 Write block 4763 len 36864 data: 7 4 4 4 4 4 3 2 3 204/300 Write block 4428 len 28672 data: 7 5 4 4 4 6 7 205/300 Write block 2385 len 24576 data: 9 10 10 10 10 8 206/300 Write block 1443 len 24576 data: 4 4 5 4 4 5 207/300 Read block 3464 len 32768 208/300 Read block 1827 len 40960 209/300 Read block 2506 len 20480 210/300 Read block 1981 len 12288 211/300 Read block 1116 len 16384 212/300 Write block 1627 len 16384 data: 7 8 9 9 213/300 Flush 214/300 Flush 215/300 Read block 215 len 24576 216/300 Read block 4043 len 12288 217/300 Read block 218 len 12288 218/300 Write block 115 len 20480 data: 7 6 6 5 7 219/300 Read block 3028 len 8192 220/300 Flush 221/300 Write block 3257 len 40960 data: 9 7 6 6 4 3 3 2 3 3 222/300 Flush 223/300 Read block 3392 len 32768 224/300 Write block 1745 len 4096 data: 4 225/300 Write block 1402 len 24576 data: 7 8 7 8 6 6 226/300 Write block 3663 len 4096 data: 7 227/300 Read block 430 len 20480 228/300 Write block 4678 len 16384 data: 9 10 9 9 229/300 Write block 3020 len 32768 data: 5 6 5 5 4 5 5 5 230/300 Write block 4705 len 16384 data: 7 7 8 10 231/300 Read block 3191 len 28672 232/300 Read block 4568 len 40960 233/300 Read block 4935 len 28672 234/300 Read block 468 len 28672 235/300 Read block 4539 len 28672 236/300 Read block 2931 len 24576 237/300 Read block 3116 len 16384 238/300 Read block 2380 len 28672 239/300 Write block 2040 len 32768 data: 9 10 9 10 11 10 9 5 240/300 Read block 3047 len 40960 241/300 Read block 4040 len 36864 242/300 Read block 4688 len 24576 243/300 Read block 369 len 12288 244/300 Read block 1502 len 4096 245/300 Write block 3581 len 8192 data: 5 4 246/300 Read block 4281 len 24576 247/300 Flush 248/300 Read block 1793 len 40960 249/300 Write block 1374 len 12288 data: 5 6 6 250/300 Flush 251/300 Read block 1177 len 28672 252/300 Flush 253/300 Write block 3130 len 4096 data: 7 254/300 Write block 4236 len 36864 data: 5 6 6 5 4 5 5 5 4 255/300 Read block 3549 len 4096 256/300 Read block 3946 len 32768 257/300 Read block 1183 len 36864 258/300 Read block 185 len 12288 259/300 Read block 2758 len 4096 260/300 Flush 261/300 Read block 3378 len 20480 262/300 Read block 815 len 36864 263/300 Flush 264/300 Write block 4137 len 4096 data: 4 265/300 Write block 1847 len 36864 data: 4 5 6 7 7 8 7 8 6 266/300 Write block 3370 len 12288 data: 3 3 3 267/300 Write block 4939 len 32768 data: 8 8 10 10 11 8 7 5 268/300 Read block 2656 len 28672 269/300 Read block 3347 len 32768 270/300 Write block 4100 len 24576 data: 5 5 4 4 3 3 271/300 Write block 4084 len 40960 data: 6 6 5 5 6 6 8 6 6 7 272/300 Write block 2292 len 12288 data: 3 3 3 273/300 Write block 3685 len 8192 data: 8 9 274/300 Read block 523 len 4096 275/300 Read block 4117 len 8192 276/300 Read block 1214 len 40960 277/300 Flush 278/300 Write block 2757 len 32768 data: 4 5 4 5 4 5 6 6 279/300 Read block 714 len 12288 280/300 Read block 944 len 20480 281/300 Flush 282/300 Flush 283/300 Read block 2637 len 40960 284/300 Read block 3931 len 12288 285/300 Read block 4814 len 20480 286/300 Read block 4604 len 12288 287/300 Read block 1570 len 32768 288/300 Flush 289/300 Write block 470 len 16384 data: 10 8 7 6 290/300 Flush 291/300 Write block 3221 len 40960 data: 7 7 6 6 6 4 4 5 5 6 292/300 Write block 3978 len 32768 data: 4 6 5 5 6 5 6 6 293/300 Read block 3444 len 28672 294/300 Write block 3807 len 20480 data: 8 9 10 9 9 295/300 Flush 296/300 Read block 2393 len 4096 297/300 Read block 807 len 28672 298/300 Write block 3501 len 28672 data: 4 3 3 3 2 2 2 299/300 Read block 1776 len 24576 300/300 Read block 3865 len 8192 301/300 Flush Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:48:30.204 INFO request completed, latency_us: 171, response_code: 204, uri: /start/cid/1, method: POST, req_id: b9e9a16c-41ef-4723-bf20-7d6818271df1, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8847 Acked Write 10 Done Sent Done false 8848 Acked Read 5 Done Sent Done false 8849 Acked Write 6 Done Sent Done false 8850 Acked Read 4 Done Sent Done false 8851 Acked Read 9 Done Sent Done false 8852 Acked Write 1 Done Sent Done false 8853 Acked Write 2 Done Sent Done false 8854 Acked Flush 0 Done Sent Done false 8855 Acked Read 8 Done Sent Done false 8856 Acked Read 4 Done Sent Done false 8857 Acked Flush 0 Done Sent Done false 8858 Acked Read 3 Done Sent Done false 8859 Acked Read 1 Done Sent Done false 8860 Acked Read 10 Done Sent Done false 8861 Acked Read 8 Done Sent Done false 8862 Acked Flush 0 Done Sent Done false 8863 Acked Write 5 Done Sent Done false 8864 Acked Write 5 Done Sent Done false 8865 Acked Read 2 Done Sent Done false 8866 Acked Read 2 Done Sent Done false 8867 Acked Write 1 Done Sent Done false 8868 Acked Read 1 Done Sent Done false 8869 Acked Write 9 Done Sent Done false 8870 Acked Read 6 Done Sent Done false 8871 Acked Write 9 Done Sent Done false 8872 Acked Write 2 Done Sent Done false 8873 Acked Read 6 Done Sent Done false 8874 Acked Read 7 Done Sent Done false 8875 Acked Write 7 Done Sent Done false 8876 Acked Write 8 Done Sent Done false 8877 Acked Read 7 Done Sent Done false 8878 Acked Read 7 Done Sent Done false 8879 Acked Write 3 Done Sent Done false 8880 Acked Write 5 Done Sent Done false 8881 Acked Read 9 Done Sent Done false 8882 Acked Write 10 Done Sent Done false 8883 Acked Read 9 Done Sent Done false 8884 Acked Write 4 Done Sent Done false 8885 Acked Write 4 Done Sent Done false 8886 Acked Read 1 Done Sent Done false 8887 Acked Read 9 Done Sent Done false 8888 Acked Read 7 Done Sent Done false 8889 Acked Flush 0 Done Sent Done false 8890 Acked Read 10 Done Sent Done false 8891 Acked Read 2 Done Sent Done false 8892 Acked Flush 0 Done Sent Done false 8893 Acked Write 6 Done Sent Done false 8894 Acked Write 2 Done Sent Done false 8895 Acked Write 2 Done Sent Done false 8896 Acked Flush 0 Done Sent Done false 8897 Acked Read 2 Done Sent Done false 8898 Acked Write 5 Done Sent Done false 8899 Acked Read 3 Done Sent Done false 8900 Acked Write 9 Done Sent Done false 8901 Acked Read 5 Done Sent Done false 8902 Acked Write 2 Done Sent Done false 8903 Acked Write 10 Done Sent Done false 8904 Acked Write 9 Done Sent Done false 8905 Acked Read 7 Done Sent Done false 8906 Acked Write 1 Done Sent Done false 8907 Acked Flush 0 Done Sent Done false 8908 Acked Read 2 Done Sent Done false 8909 Acked Write 2 Done Sent Done false 8910 Acked Read 5 Done Sent Done false 8911 Acked Flush 0 Done Sent Done false 8912 Acked Write 1 Done Sent Done false 8913 Acked Read 4 Done Sent Done false 8914 Acked Write 7 Done Sent Done false 8915 Acked Read 10 Done Sent Done false 8916 Acked Read 5 Done Sent Done false 8917 Acked Read 1 Done Sent Done false 8918 Acked Read 2 Done Sent Done false 8919 Acked Read 9 Done Sent Done false 8920 Acked Write 5 Done Sent Done false 8921 Acked Write 3 Done Sent Done false 8922 Acked Write 5 Done Sent Done false 8923 Acked Write 9 Done Sent Done false 8924 Acked Read 7 Done Sent Done false 8925 Acked Read 9 Done Sent Done false 8926 Acked Read 10 Done Sent Done false 8927 Acked Read 6 Done Sent Done false 8928 Acked Write 5 Done Sent Done false 8929 Acked Read 1 Done Sent Done false 8930 Acked Write 1 Done Sent Done false 8931 Acked Read 4 Done Sent Done false 8932 Acked Read 6 Done Sent Done false 8933 Acked Read 6 Done Sent Done false 8934 Acked Flush 0 Done Sent Done false 8935 Acked Read 4 Done Sent Done false 8936 Acked Read 5 Done Sent Done false 8937 Acked Write 1 Done Sent Done false 8938 Acked Write 9 Done Sent Done false 8939 Acked Read 5 Done Sent Done false 8940 Acked Write 5 Done Sent Done false 8941 Acked Read 10 Done Sent Done false 8942 Acked Read 6 Done Sent Done false 8943 Acked Write 7 Done Sent Done false 8944 Acked Write 5 Done Sent Done false 8945 Acked Flush 0 Done Sent Done false 8946 Acked Read 4 Done Sent Done false 8947 Acked Write 1 Done Sent Done false 8948 Acked Write 1 Done Sent Done false 8949 Acked Write 5 Done Sent Done false 8950 Acked Write 7 Done Sent Done false 8951 Acked Read 7 Done Sent Done false 8952 Acked Write 7 Done Sent Done false 8953 Acked Write 5 Done Sent Done false 8954 Acked Write 9 Done Sent Done false 8955 Acked Write 2 Done Sent Done false 8956 Acked Flush 0 Done Sent Done false 8957 Acked Read 8 Done Sent Done false 8958 Acked Read 1 Done Sent Done false 8959 Acked Read 10 Done Sent Done false 8960 Acked Read 1 Done Sent Done false 8961 Acked Flush 0 Done Sent Done false 8962 Acked Read 2 Done Sent Done false 8963 Acked Write 1 Done Sent Done false 8964 Acked Write 5 Done Sent Done false 8965 Acked Write 9 Done Sent Done false 8966 Acked Read 8 Done Sent Done false 8967 Acked Write 1 Done Sent Done false 8968 Acked Read 3 Done Sent Done false 8969 Acked Read 1 Done Sent Done false 8970 Acked Read 9 Done Sent Done false 8971 Acked Read 6 Done Sent Done false 8972 Acked Read 6 Done Sent Done false 8973 Acked Read 2 Done Sent Done false 8974 Acked Read 3 Done Sent Done false 8975 Acked Read 3 Done Sent Done false 8976 Acked Write 10 Done Sent Done false 8977 Acked Write 7 Done Sent Done false 8978 Acked Write 10 Done Sent Done false 8979 Acked Flush 0 Done Sent Done false 8980 Acked Read 9 Done Sent Done false 8981 Acked Read 9 Done Sent Done false 8982 Acked Write 1 Done Sent Done false 8983 Acked Read 9 Done Sent Done false 8984 Acked Read 8 Done Sent Done false 8985 Acked Write 5 Done Sent Done false 8986 Acked Read 8 Done Sent Done false 8987 Acked Write 10 Done Sent Done false 8988 Acked Read 6 Done Sent Done false 8989 Acked Write 6 Done Sent Done false 8990 Acked Write 1 Done Sent Done false 8991 Acked Read 9 Done Sent Done false 8992 Acked Read 3 Done Sent Done false 8993 Acked Read 5 Done Sent Done false 8994 Acked Flush 0 Done Sent Done false 8995 Acked Write 4 Done Sent Done false 8996 Acked Write 3 Done Sent Done false 8997 Acked Read 7 Done Sent Done false 8998 Acked Write 7 Done Sent Done false 8999 Acked Read 2 Done Sent Done false 9000 Acked Write 10 Done Sent Done false 9001 Acked Read 1 Done Sent Done false 9002 Acked Write 5 Done Sent Done false 9003 Acked Read 6 Done Sent Done false 9004 Acked Read 8 Done Sent Done false 9005 Acked Write 9 Done Sent Done false 9006 Acked Write 5 Done Sent Done false 9007 Acked Read 1 Done Sent Done false 9008 Acked Read 2 Done Sent Done false 9009 Acked Read 3 Done Sent Done false 9010 Acked Read 3 Done Sent Done false 9011 Acked Write 8 Done Sent Done false 9012 Acked Read 9 Done Sent Done false 9013 Acked Read 8 Done Sent Done false 9014 Acked Read 10 Done Sent Done false 9015 Acked Write 10 Done Sent Done false 9016 Acked Write 5 Done Sent Done false 9017 Acked Write 7 Done Sent Done false 9018 Acked Write 6 Done Sent Done false 9019 Acked Read 5 Done Sent Done false 9020 Acked Write 4 Done Sent Done false 9021 Acked Read 10 Done Sent Done false 9022 Acked Write 1 Done Sent Done false 9023 Acked Read 4 Done Sent Done false 9024 Acked Flush 0 Done Sent Done false 9025 Acked Read 6 Done Sent Done false 9026 Acked Read 2 Done Sent Done false 9027 Acked Write 4 Done Sent Done false 9028 Acked Write 4 Done Sent Done false 9029 Acked Read 2 Done Sent Done false 9030 Acked Read 2 Done Sent Done false 9031 Acked Read 6 Done Sent Done false 9032 Acked Read 3 Done Sent Done false 9033 Acked Write 6 Done Sent Done false 9034 Acked Read 9 Done Sent Done false 9035 Acked Read 6 Done Sent Done false 9036 Acked Flush 0 Done Sent Done false 9037 Acked Read 9 Done Sent Done false 9038 Acked Read 8 Done Sent Done false 9039 Acked Read 7 Done Sent Done false 9040 Acked Write 5 Done Sent Done false 9041 Acked Read 7 Done Sent Done false 9042 Acked Read 7 Done Sent Done false 9043 Acked Read 2 Done Sent Done false 9044 Acked Read 6 Done Sent Done false 9045 Acked Flush 0 Done Sent Done false 9046 Acked Flush 0 Done Sent Done false 9047 Acked Read 6 Done Sent Done false 9048 Acked Write 6 Done Sent Done false 9049 Acked Write 9 Done Sent Done false 9050 Acked Write 7 Done Sent Done false 9051 Acked Write 6 Done Sent Done false 9052 Acked Write 6 Done Sent Done false 9053 Acked Read 8 Done Sent Done false 9054 Acked Read 10 Done Sent Done false 9055 Acked Read 5 Done Sent Done false 9056 Acked Read 3 Done Sent Done false 9057 Acked Read 4 Done Sent Done false 9058 Acked Write 4 Done Sent Done false 9059 Acked Flush 0 Done Sent Done false 9060 Acked Flush 0 Done Sent Done false 9061 Acked Read 6 Done Sent Done false 9062 Acked Read 3 Done Sent Done false 9063 Acked Read 3 Done Sent Done false 9064 Acked Write 5 Done Sent Done false 9065 Acked Read 2 Done Sent Done false 9066 Acked Flush 0 Done Sent Done false 9067 Acked Write 10 Done Sent Done false 9068 Acked Flush 0 Done Sent Done false 9069 Acked Read 8 Done Sent Done false 9070 Acked Write 1 Done Sent Done false 9071 Acked Write 6 Done Sent Done false 9072 Acked Write 1 Done Sent Done false 9073 Acked Read 5 Done Sent Done false 9074 Acked Write 4 Done Sent Done false 9075 Acked Write 8 Done Sent Done false 9076 Acked Write 4 Done Sent Done false 9077 Acked Read 7 Done Sent Done false 9078 Acked Read 10 Done Sent Done false 9079 Acked Read 7 Done Sent Done false 9080 Acked Read 7 Done Sent Done false 9081 Acked Read 7 Done Sent Done false 9082 Acked Read 6 Done Sent Done false 9083 Acked Read 4 Done Sent Done false 9084 Acked Read 7 Done Sent Done false 9085 Acked Write 8 Done Sent Done false 9086 Acked Read 10 Done Sent Done false 9087 Acked Read 9 Done Sent Done false 9088 Acked Read 6 Done Sent Done false 9089 Acked Read 3 Done Sent Done false 9090 Acked Read 1 Done Sent Done false 9091 Acked Write 2 Done Sent Done false 9092 Acked Read 6 Done Sent Done false 9093 Acked Flush 0 Done Sent Done false 9094 Acked Read 10 Done Sent Done false 9095 Acked Write 3 Done Sent Done false 9096 Acked Flush 0 Done Sent Done false 9097 Acked Read 7 Done Sent Done false 9098 Acked Flush 0 Done Sent Done false 9099 Acked Write 1 Done Sent Done false 9100 Acked Write 9 Done Sent Done false 9101 Acked Read 1 Done Sent Done false 9102 Acked Read 8 Done Sent Done false 9103 Acked Read 9 Done Sent Done false 9104 Acked Read 3 Done Sent Done false 9105 Acked Read 1 Done Sent Done false 9106 Acked Flush 0 Done Sent Done false 9107 Acked Read 5 Done Sent Done false 9108 Acked Read 9 Done Sent Done false 9109 Acked Flush 0 Done Sent Done false 9110 Acked Write 1 Done Sent Done false 9111 Acked Write 9 Done Sent Done false 9112 Acked Write 3 Done Sent Done false 9113 Acked Write 8 Done Sent Done false 9114 Acked Read 7 Done Sent Done false 9115 Acked Read 8 Done Sent Done false 9116 Acked Write 6 Done Sent Done false 9117 Acked Write 10 Done Sent Done false 9118 Acked Write 3 Done Sent Done false 9119 Acked Write 2 Done Sent Done false 9120 Acked Read 1 Done Sent Done false 9121 Acked Read 2 Done Sent Done false 9122 Acked Read 10 Done Sent Done false 9123 Acked Flush 0 Done Sent Done false 9124 Acked Write 8 Done Sent Done false 9125 Acked Read 3 Done Sent Done false 9126 Acked Read 5 Done Sent Done false 9127 Acked Flush 0 Done Sent Done false 9128 Acked Flush 0 Done Sent Done false 9129 Acked Read 10 Done Sent Done false 9130 Acked Read 3 Done Sent Done false 9131 Acked Read 5 Done Sent Done false 9132 Acked Read 3 Done Sent Done false 9133 Acked Read 8 Done Sent Done false 9134 Acked Flush 0 Done Sent Done false 9135 Acked Write 4 Done Sent Done false 9136 Acked Flush 0 Done Sent Done false 9137 Acked Write 10 Done Sent Done false 9138 Acked Write 8 Done Sent Done false 9139 Acked Read 7 Done Sent Done false 9140 Acked Write 5 Done Sent Done false 9141 Acked Flush 0 Done Sent Done false 9142 Acked Read 1 Done Sent Done false 9143 Acked Read 7 Done Sent Done false 9144 Acked Write 7 Done Sent Done false 9145 Acked Read 6 Done Sent Done false 9146 Acked Read 2 Done Sent Done false 9147 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 8846 9147 Downstairs last five completed: 8846 8845 8844 8843 8842 Upstairs last five completed: 9147 9146 9145 9144 9143 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8847 Acked Write 10 Done Sent Done false 8848 Acked Read 5 Done Sent Done false 8849 Acked Write 6 Done Sent Done false 8850 Acked Read 4 Done Sent Done false 8851 Acked Read 9 Done Sent Done false 8852 Acked Write 1 Done Sent Done false 8853 Acked Write 2 Done Sent Done false 8854 Acked Flush 0 Done Sent Done false 8855 Acked Read 8 Done Sent Done false 8856 Acked Read 4 Done Sent Done false 8857 Acked Flush 0 Done Sent Done false 8858 Acked Read 3 Done Sent Done false 8859 Acked Read 1 Done Sent Done false 8860 Acked Read 10 Done Sent Done false 8861 Acked Read 8 Done Sent Done false 8862 Acked Flush 0 Done Sent Done false 8863 Acked Write 5 Done Sent Done false 8864 Acked Write 5 Done Sent Done false 8865 Acked Read 2 Done Sent Done false 8866 Acked Read 2 Done Sent Done false 8867 Acked Write 1 Done Sent Done false 8868 Acked Read 1 Done Sent Done false 8869 Acked Write 9 Done Sent Done false 8870 Acked Read 6 Done Sent Done false 8871 Acked Write 9 Done Sent Done false 8872 Acked Write 2 Done Sent Done false 8873 Acked Read 6 Done Sent Done false 8874 Acked Read 7 Done Sent Done false 8875 Acked Write 7 Done Sent Done false 8876 Acked Write 8 Done Sent Done false 8877 Acked Read 7 Done Sent Done false 8878 Acked Read 7 Done Sent Done false 8879 Acked Write 3 Done Sent Done false 8880 Acked Write 5 Done Sent Done false 8881 Acked Read 9 Done Sent Done false 8882 Acked Write 10 Done Sent Done false 8883 Acked Read 9 Done Sent Done false 8884 Acked Write 4 Done Sent Done false 8885 Acked Write 4 Done Sent Done false 8886 Acked Read 1 Done Sent Done false 8887 Acked Read 9 Done Sent Done false 8888 Acked Read 7 Done Sent Done false 8889 Acked Flush 0 Done Sent Done false 8890 Acked Read 10 Done Sent Done false 8891 Acked Read 2 Done Sent Done false 8892 Acked Flush 0 Done Sent Done false 8893 Acked Write 6 Done Sent Done false 8894 Acked Write 2 Done Sent Done false 8895 Acked Write 2 Done Sent Done false 8896 Acked Flush 0 Done Sent Done false 8897 Acked Read 2 Done Sent Done false 8898 Acked Write 5 Done Sent Done false 8899 Acked Read 3 Done Sent Done false 8900 Acked Write 9 Done Sent Done false 8901 Acked Read 5 Done Sent Done false 8902 Acked Write 2 Done Sent Done false 8903 Acked Write 10 Done Sent Done false 8904 Acked Write 9 Done Sent Done false 8905 Acked Read 7 Done Sent Done false 8906 Acked Write 1 Done Sent Done false 8907 Acked Flush 0 Done Sent Done false 8908 Acked Read 2 Done Sent Done false 8909 Acked Write 2 Done Sent Done false 8910 Acked Read 5 Done Sent Done false 8911 Acked Flush 0 Done Sent Done false 8912 Acked Write 1 Done Sent Done false 8913 Acked Read 4 Done Sent Done false 8914 Acked Write 7 Done Sent Done false 8915 Acked Read 10 Done Sent Done false 8916 Acked Read 5 Done Sent Done false 8917 Acked Read 1 Done Sent Done false 8918 Acked Read 2 Done Sent Done false 8919 Acked Read 9 Done Sent Done false 8920 Acked Write 5 Done Sent Done false 8921 Acked Write 3 Done Sent Done false 8922 Acked Write 5 Done Sent Done false 8923 Acked Write 9 Done Sent Done false 8924 Acked Read 7 Done Sent Done false 8925 Acked Read 9 Done Sent Done false 8926 Acked Read 10 Done Sent Done false 8927 Acked Read 6 Done Sent Done false 8928 Acked Write 5 Done Sent Done false 8929 Acked Read 1 Done Sent Done false 8930 Acked Write 1 Done Sent Done false 8931 Acked Read 4 Done Sent Done false 8932 Acked Read 6 Done Sent Done false 8933 Acked Read 6 Done Sent Done false 8934 Acked Flush 0 Done Sent Done false 8935 Acked Read 4 Done Sent Done false 8936 Acked Read 5 Done Sent Done false 8937 Acked Write 1 Done Sent Done false 8938 Acked Write 9 Done Sent Done false 8939 Acked Read 5 Done Sent Done false 8940 Acked Write 5 Done Sent Done false 8941 Acked Read 10 Done Sent Done false 8942 Acked Read 6 Done Sent Done false 8943 Acked Write 7 Done Sent Done false 8944 Acked Write 5 Done Sent Done false 8945 Acked Flush 0 Done Sent Done false 8946 Acked Read 4 Done Sent Done false 8947 Acked Write 1 Done Sent Done false 8948 Acked Write 1 Done Sent Done false 8949 Acked Write 5 Done Sent Done false 8950 Acked Write 7 Done Sent Done false 8951 Acked Read 7 Done Sent Done false 8952 Acked Write 7 Done Sent Done false 8953 Acked Write 5 Done Sent Done false 8954 Acked Write 9 Done Sent Done false 8955 Acked Write 2 Done Sent Done false 8956 Acked Flush 0 Done Sent Done false 8957 Acked Read 8 Done Sent Done false 8958 Acked Read 1 Done Sent Done false 8959 Acked Read 10 Done Sent Done false 8960 Acked Read 1 Done Sent Done false 8961 Acked Flush 0 Done Sent Done false 8962 Acked Read 2 Done Sent Done false 8963 Acked Write 1 Done Sent Done false 8964 Acked Write 5 Done Sent Done false 8965 Acked Write 9 Done Sent Done false 8966 Acked Read 8 Done Sent Done false 8967 Acked Write 1 Done Sent Done false 8968 Acked Read 3 Done Sent Done false 8969 Acked Read 1 Done Sent Done false 8970 Acked Read 9 Done Sent Done false 8971 Acked Read 6 Done Sent Done false 8972 Acked Read 6 Done Sent Done false 8973 Acked Read 2 Done Sent Done false 8974 Acked Read 3 Done Sent Done false 8975 Acked Read 3 Done Sent Done false 8976 Acked Write 10 Done Sent Done false 8977 Acked Write 7 Done Sent Done false 8978 Acked Write 10 Done Sent Done false 8979 Acked Flush 0 Done Sent Done false 8980 Acked Read 9 Done Sent Done false 8981 Acked Read 9 Done Sent Done false 8982 Acked Write 1 Done Sent Done false 8983 Acked Read 9 Done Sent Done false 8984 Acked Read 8 Done Sent Done false 8985 Acked Write 5 Done Sent Done false 8986 Acked Read 8 Done Sent Done false 8987 Acked Write 10 Done Sent Done false 8988 Acked Read 6 Done Sent Done false 8989 Acked Write 6 Done Sent Done false 8990 Acked Write 1 Done Sent Done false 8991 Acked Read 9 Done Sent Done false 8992 Acked Read 3 Done Sent Done false 8993 Acked Read 5 Done Sent Done false 8994 Acked Flush 0 Done Sent Done false 8995 Acked Write 4 Done Sent Done false 8996 Acked Write 3 Done Sent Done false 8997 Acked Read 7 Done Sent Done false 8998 Acked Write 7 Done Sent Done false 8999 Acked Read 2 Done Sent Done false 9000 Acked Write 10 Done Sent Done false 9001 Acked Read 1 Done Sent Done false 9002 Acked Write 5 Done Sent Done false 9003 Acked Read 6 Done Sent Done false 9004 Acked Read 8 Done Sent Done false 9005 Acked Write 9 Done Sent Done false 9006 Acked Write 5 Done Sent Done false 9007 Acked Read 1 Done Sent Done false 9008 Acked Read 2 Done Sent Done false 9009 Acked Read 3 Done Sent Done false 9010 Acked Read 3 Done Sent Done false 9011 Acked Write 8 Done Sent Done false 9012 Acked Read 9 Done Sent Done false 9013 Acked Read 8 Done Sent Done false 9014 Acked Read 10 Done Sent Done false 9015 Acked Write 10 Done Sent Done false 9016 Acked Write 5 Done Sent Done false 9017 Acked Write 7 Done Sent Done false 9018 Acked Write 6 Done Sent Done false 9019 Acked Read 5 Done Sent Done false 9020 Acked Write 4 Done Sent Done false 9021 Acked Read 10 Done Sent Done false 9022 Acked Write 1 Done Sent Done false 9023 Acked Read 4 Done Sent Done false 9024 Acked Flush 0 Done Sent Done false 9025 Acked Read 6 Done Sent Done false 9026 Acked Read 2 Done Sent Done false 9027 Acked Write 4 Done Sent Done false 9028 Acked Write 4 Done Sent Done false 9029 Acked Read 2 Done Sent Done false 9030 Acked Read 2 Done Sent Done false 9031 Acked Read 6 Done Sent Done false 9032 Acked Read 3 Done Sent Done false 9033 Acked Write 6 Done Sent Done false 9034 Acked Read 9 Done Sent Done false 9035 Acked Read 6 Done Sent Done false 9036 Acked Flush 0 Done Sent Done false 9037 Acked Read 9 Done Sent Done false 9038 Acked Read 8 Done Sent Done false 9039 Acked Read 7 Done Sent Done false 9040 Acked Write 5 Done Sent Done false 9041 Acked Read 7 Done Sent Done false 9042 Acked Read 7 Done Sent Done false 9043 Acked Read 2 Done Sent Done false 9044 Acked Read 6 Done Sent Done false 9045 Acked Flush 0 Done Sent Done false 9046 Acked Flush 0 Done Sent Done false 9047 Acked Read 6 Done Sent Done false 9048 Acked Write 6 Done Sent Done false 9049 Acked Write 9 Done Sent Done false 9050 Acked Write 7 Done Sent Done false 9051 Acked Write 6 Done Sent Done false 9052 Acked Write 6 Done Sent Done false 9053 Acked Read 8 Done Sent Done false 9054 Acked Read 10 Done Sent Done false 9055 Acked Read 5 Done Sent Done false 9056 Acked Read 3 Done Sent Done false 9057 Acked Read 4 Done Sent Done false 9058 Acked Write 4 Done Sent Done false 9059 Acked Flush 0 Done Sent Done false 9060 Acked Flush 0 Done Sent Done false 9061 Acked Read 6 Done Sent Done false 9062 Acked Read 3 Done Sent Done false 9063 Acked Read 3 Done Sent Done false 9064 Acked Write 5 Done Sent Done false 9065 Acked Read 2 Done Sent Done false 9066 Acked Flush 0 Done Sent Done false 9067 Acked Write 10 Done Sent Done false 9068 Acked Flush 0 Done Sent Done false 9069 Acked Read 8 Done Sent Done false 9070 Acked Write 1 Done Sent Done false 9071 Acked Write 6 Done Sent Done false 9072 Acked Write 1 Done Sent Done false 9073 Acked Read 5 Done Sent Done false 9074 Acked Write 4 Done Sent Done false 9075 Acked Write 8 Done Sent Done false 9076 Acked Write 4 Done Sent Done false 9077 Acked Read 7 Done Sent Done false 9078 Acked Read 10 Done Sent Done false 9079 Acked Read 7 Done Sent Done false 9080 Acked Read 7 Done Sent Done false 9081 Acked Read 7 Done Sent Done false 9082 Acked Read 6 Done Sent Done false 9083 Acked Read 4 Done Sent Done false 9084 Acked Read 7 Done Sent Done false 9085 Acked Write 8 Done Sent Done false 9086 Acked Read 10 Done Sent Done false 9087 Acked Read 9 Done Sent Done false 9088 Acked Read 6 Done Sent Done false 9089 Acked Read 3 Done Sent Done false 9090 Acked Read 1 Done Sent Done false 9091 Acked Write 2 Done Sent Done false 9092 Acked Read 6 Done Sent Done false 9093 Acked Flush 0 Done Sent Done false 9094 Acked Read 10 Done Sent Done false 9095 Acked Write 3 Done Sent Done false 9096 Acked Flush 0 Done Sent Done false 9097 Acked Read 7 Done Sent Done false 9098 Acked Flush 0 Done Sent Done false 9099 Acked Write 1 Done Sent Done false 9100 Acked Write 9 Done Sent Done false 9101 Acked Read 1 Done Sent Done false 9102 Acked Read 8 Done Sent Done false 9103 Acked Read 9 Done Sent Done false 9104 Acked Read 3 Done Sent Done false 9105 Acked Read 1 Done Sent Done false 9106 Acked Flush 0 Done Sent Done false 9107 Acked Read 5 Done Sent Done false 9108 Acked Read 9 Done Sent Done false 9109 Acked Flush 0 Done Sent Done false 9110 Acked Write 1 Done Sent Done false 9111 Acked Write 9 Done Sent Done false 9112 Acked Write 3 Done Sent Done false 9113 Acked Write 8 Done Sent Done false 9114 Acked Read 7 Done Sent Done false 9115 Acked Read 8 Done Sent Done false 9116 Acked Write 6 Done Sent Done false 9117 Acked Write 10 Done Sent Done false 9118 Acked Write 3 Done Sent Done false 9119 Acked Write 2 Done Sent Done false 9120 Acked Read 1 Done Sent Done false 9121 Acked Read 2 Done Sent Done false 9122 Acked Read 10 Done Sent Done false 9123 Acked Flush 0 Done Sent Done false 9124 Acked Write 8 Done Sent Done false 9125 Acked Read 3 Done Sent Done false 9126 Acked Read 5 Done Sent Done false 9127 Acked Flush 0 Done Sent Done false 9128 Acked Flush 0 Done Sent Done false 9129 Acked Read 10 Done Sent Done false 9130 Acked Read 3 Done Sent Done false 9131 Acked Read 5 Done Sent Done false 9132 Acked Read 3 Done Sent Done false 9133 Acked Read 8 Done Sent Done false 9134 Acked Flush 0 Done Sent Done false 9135 Acked Write 4 Done Sent Done false 9136 Acked Flush 0 Done Sent Done false 9137 Acked Write 10 Done Sent Done false 9138 Acked Write 8 Done Sent Done false 9139 Acked Read 7 Done Sent Done false 9140 Acked Write 5 Done Sent Done false 9141 Acked Flush 0 Done Sent Done false 9142 Acked Read 1 Done Sent Done false 9143 Acked Read 7 Done Sent Done false 9144 Acked Write 7 Done Sent Done false 9145 Acked Read 6 Done Sent Done false 9146 Acked Read 2 Done Sent Done false 9147 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 8846 9147 Downstairs last five completed: 8846 8845 8844 8843 8842 Upstairs last five completed: 9147 9146 9145 9144 9143 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:35.969649195Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:35.969723955Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:35.969747719Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:35.969769643Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:35.969881233Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8847 Acked Write 10 Done Sent Done false 8848 Acked Read 5 Done Sent Done false 8849 Acked Write 6 Done Sent Done false 8850 Acked Read 4 Done Sent Done false 8851 Acked Read 9 Done Sent Done false 8852 Acked Write 1 Done Sent Done false 8853 Acked Write 2 Done Sent Done false 8854 Acked Flush 0 Done Sent Done false 8855 Acked Read 8 Done Sent Done false 8856 Acked Read 4 Done Sent Done false 8857 Acked Flush 0 Done Sent Done false 8858 Acked Read 3 Done Sent Done false 8859 Acked Read 1 Done Sent Done false 8860 Acked Read 10 Done Sent Done false 8861 Acked Read 8 Done Sent Done false 8862 Acked Flush 0 Done Sent Done false 8863 Acked Write 5 Done Sent Done false 8864 Acked Write 5 Done Sent Done false 8865 Acked Read 2 Done Sent Done false 8866 Acked Read 2 Done Sent Done false 8867 Acked Write 1 Done Sent Done false 8868 Acked Read 1 Done Sent Done false 8869 Acked Write 9 Done Sent Done false 8870 Acked Read 6 Done Sent Done false 8871 Acked Write 9 Done Sent Done false 8872 Acked Write 2 Done Sent Done false 8873 Acked Read 6 Done Sent Done false 8874 Acked Read 7 Done Sent Done false 8875 Acked Write 7 Done Sent Done false 8876 Acked Write 8 Done Sent Done false 8877 Acked Read 7 Done Sent Done false 8878 Acked Read 7 Done Sent Done false 8879 Acked Write 3 Done Sent Done false 8880 Acked Write 5 Done Sent Done false 8881 Acked Read 9 Done Sent Done false 8882 Acked Write 10 Done Sent Done false 8883 Acked Read 9 Done Sent Done false 8884 Acked Write 4 Done Sent Done false 8885 Acked Write 4 Done Sent Done false 8886 Acked Read 1 Done Sent Done false 8887 Acked Read 9 Done Sent Done false 8888 Acked Read 7 Done Sent Done false 8889 Acked Flush 0 Done Sent Done false 8890 Acked Read 10 Done Sent Done false 8891 Acked Read 2 Done Sent Done false 8892 Acked Flush 0 Done Sent Done false 8893 Acked Write 6 Done Sent Done false 8894 Acked Write 2 Done Sent Done false 8895 Acked Write 2 Done Sent Done false 8896 Acked Flush 0 Done Sent Done false 8897 Acked Read 2 Done Sent Done false 8898 Acked Write 5 Done Sent Done false 8899 Acked Read 3 Done Sent Done false 8900 Acked Write 9 Done Sent Done false 8901 Acked Read 5 Done Sent Done false 8902 Acked Write 2 Done Sent Done false 8903 Acked Write 10 Done Sent Done false 8904 Acked Write 9 Done Sent Done false 8905 Acked Read 7 Done Sent Done false 8906 Acked Write 1 Done Sent Done false 8907 Acked Flush 0 Done Sent Done false 8908 Acked Read 2 Done Sent Done false 8909 Acked Write 2 Done Sent Done false 8910 Acked Read 5 Done Sent Done false 8911 Acked Flush 0 Done Sent Done false 8912 Acked Write 1 Done Sent Done false 8913 Acked Read 4 Done Sent Done false 8914 Acked Write 7 Done Sent Done false 8915 Acked Read 10 Done Sent Done false 8916 Acked Read 5 Done Sent Done false 8917 Acked Read 1 Done Sent Done false 8918 Acked Read 2 Done Sent Done false 8919 Acked Read 9 Done Sent Done false 8920 Acked Write 5 Done Sent Done false 8921 Acked Write 3 Done Sent Done false 8922 Acked Write 5 Done Sent Done false 8923 Acked Write 9 Done Sent Done false 8924 Acked Read 7 Done Sent Done false 8925 Acked Read 9 Done Sent Done false 8926 Acked Read 10 Done Sent Done false 8927 Acked Read 6 Done Sent Done false 8928 Acked Write 5 Done Sent Done false 8929 Acked Read 1 Done Sent Done false 8930 Acked Write 1 Done Sent Done false 8931 Acked Read 4 Done Sent Done false 8932 Acked Read 6 Done Sent Done false 8933 Acked Read 6 Done Sent Done false 8934 Acked Flush 0 Done Sent Done false 8935 Acked Read 4 Done Sent Done false 8936 Acked Read 5 Done Sent Done false 8937 Acked Write 1 Done Sent Done false 8938 Acked Write 9 Done Sent Done false 8939 Acked Read 5 Done Sent Done false 8940 Acked Write 5 Done Sent Done false 8941 Acked Read 10 Done Sent Done false 8942 Acked Read 6 Done Sent Done false 8943 Acked Write 7 Done Sent Done false 8944 Acked Write 5 Done Sent Done false 8945 Acked Flush 0 Done Sent Done false 8946 Acked Read 4 Done Sent Done false 8947 Acked Write 1 Done Sent Done false 8948 Acked Write 1 Done Sent Done false 8949 Acked Write 5 Done Sent Done false 8950 Acked Write 7 Done Sent Done false 8951 Acked Read 7 Done Sent Done false 8952 Acked Write 7 Done Sent Done false 8953 Acked Write 5 Done Sent Done false 8954 Acked Write 9 Done Sent Done false 8955 Acked Write 2 Done Sent Done false 8956 Acked Flush 0 Done Sent Done false 8957 Acked Read 8 Done Sent Done false 8958 Acked Read 1 Done Sent Done false 8959 Acked Read 10 Done Sent Done false 8960 Acked Read 1 Done Sent Done false 8961 Acked Flush 0 Done Sent Done false 8962 Acked Read 2 Done Sent Done false 8963 Acked Write 1 Done Sent Done false 8964 Acked Write 5 Done Sent Done false 8965 Acked Write 9 Done Sent Done false 8966 Acked Read 8 Done Sent Done false 8967 Acked Write 1 Done Sent Done false 8968 Acked Read 3 Done Sent Done false 8969 Acked Read 1 Done Sent Done false 8970 Acked Read 9 Done Sent Done false 8971 Acked Read 6 Done Sent Done false 8972 Acked Read 6 Done Sent Done false 8973 Acked Read 2 Done Sent Done false 8974 Acked Read 3 Done Sent Done false 8975 Acked Read 3 Done Sent Done false 8976 Acked Write 10 Done Sent Done false 8977 Acked Write 7 Done Sent Done false 8978 Acked Write 10 Done Sent Done false 8979 Acked Flush 0 Done Sent Done false 8980 Acked Read 9 Done Sent Done false 8981 Acked Read 9 Done Sent Done false 8982 Acked Write 1 Done Sent Done false 8983 Acked Read 9 Done Sent Done false 8984 Acked Read 8 Done Sent Done false 8985 Acked Write 5 Done Sent Done false 8986 Acked Read 8 Done Sent Done false 8987 Acked Write 10 Done Sent Done false 8988 Acked Read 6 Done Sent Done false 8989 Acked Write 6 Done Sent Done false 8990 Acked Write 1 Done Sent Done false 8991 Acked Read 9 Done Sent Done false 8992 Acked Read 3 Done Sent Done false 8993 Acked Read 5 Done Sent Done false 8994 Acked Flush 0 Done Sent Done false 8995 Acked Write 4 Done Sent Done false 8996 Acked Write 3 Done Sent Done false 8997 Acked Read 7 Done Sent Done false 8998 Acked Write 7 Done Sent Done false 8999 Acked Read 2 Done Sent Done false 9000 Acked Write 10 Done Sent Done false 9001 Acked Read 1 Done Sent Done false 9002 Acked Write 5 Done Sent Done false 9003 Acked Read 6 Done Sent Done false 9004 Acked Read 8 Done Sent Done false 9005 Acked Write 9 Done Sent Done false 9006 Acked Write 5 Done Sent Done false 9007 Acked Read 1 Done Sent Done false 9008 Acked Read 2 Done Sent Done false 9009 Acked Read 3 Done Sent Done false 9010 Acked Read 3 Done Sent Done false 9011 Acked Write 8 Done Sent Done false 9012 Acked Read 9 Done Sent Done false 9013 Acked Read 8 Done Sent Done false 9014 Acked Read 10 Done Sent Done false 9015 Acked Write 10 Done Sent Done false 9016 Acked Write 5 Done Sent Done false 9017 Acked Write 7 Done Sent Done false 9018 Acked Write 6 Done Sent Done false 9019 Acked Read 5 Done Sent Done false 9020 Acked Write 4 Done Sent Done false 9021 Acked Read 10 Done Sent Done false 9022 Acked Write 1 Done Sent Done false 9023 Acked Read 4 Done Sent Done false 9024 Acked Flush 0 Done Sent Done false 9025 Acked Read 6 Done Sent Done false 9026 Acked Read 2 Done Sent Done false 9027 Acked Write 4 Done Sent Done false 9028 Acked Write 4 Done Sent Done false 9029 Acked Read 2 Done Sent Done false 9030 Acked Read 2 Done Sent Done false 9031 Acked Read 6 Done Sent Done false 9032 Acked Read 3 Done Sent Done false 9033 Acked Write 6 Done Sent Done false 9034 Acked Read 9 Done Sent Done false 9035 Acked Read 6 Done Sent Done false 9036 Acked Flush 0 Done Sent Done false 9037 Acked Read 9 Done Sent Done false 9038 Acked Read 8 Done Sent Done false 9039 Acked Read 7 Done Sent Done false 9040 Acked Write 5 Done Sent Done false 9041 Acked Read 7 Done Sent Done false 9042 Acked Read 7 Done Sent Done false 9043 Acked Read 2 Done Sent Done false 9044 Acked Read 6 Done Sent Done false 9045 Acked Flush 0 Done Sent Done false 9046 Acked Flush 0 Done Sent Done false 9047 Acked Read 6 Done Sent Done false 9048 Acked Write 6 Done Sent Done false 9049 Acked Write 9 Done Sent Done false 9050 Acked Write 7 Done Sent Done false 9051 Acked Write 6 Done Sent Done false 9052 Acked Write 6 Done Sent Done false 9053 Acked Read 8 Done Sent Done false 9054 Acked Read 10 Done Sent Done false 9055 Acked Read 5 Done Sent Done false 9056 Acked Read 3 Done Sent Done false 9057 Acked Read 4 Done Sent Done false 9058 Acked Write 4 Done Sent Done false 9059 Acked Flush 0 Done Sent Done false 9060 Acked Flush 0 Done Sent Done false 9061 Acked Read 6 Done Sent Done false 9062 Acked Read 3 Done Sent Done false 9063 Acked Read 3 Done Sent Done false 9064 Acked Write 5 Done Sent Done false 9065 Acked Read 2 Done Sent Done false 9066 Acked Flush 0 Done Sent Done false 9067 Acked Write 10 Done Sent Done false 9068 Acked Flush 0 Done Sent Done false 9069 Acked Read 8 Done Sent Done false 9070 Acked Write 1 Done Sent Done false 9071 Acked Write 6 Done Sent Done false 9072 Acked Write 1 Done Sent Done false 9073 Acked Read 5 Done Sent Done false 9074 Acked Write 4 Done Sent Done false 9075 Acked Write 8 Done Sent Done false 9076 Acked Write 4 Done Sent Done false 9077 Acked Read 7 Done Sent Done false 9078 Acked Read 10 Done Sent Done false 9079 Acked Read 7 Done Sent Done false 9080 Acked Read 7 Done Sent Done false 9081 Acked Read 7 Done Sent Done false 9082 Acked Read 6 Done Sent Done false 9083 Acked Read 4 Done Sent Done false 9084 Acked Read 7 Done Sent Done false 9085 Acked Write 8 Done Sent Done false 9086 Acked Read 10 Done Sent Done false 9087 Acked Read 9 Done Sent Done false 9088 Acked Read 6 Done Sent Done false 9089 Acked Read 3 Done Sent Done false 9090 Acked Read 1 Done Sent Done false 9091 Acked Write 2 Done Sent Done false 9092 Acked Read 6 Done Sent Done false 9093 Acked Flush 0 Done Sent Done false 9094 Acked Read 10 Done Sent Done false 9095 Acked Write 3 Done Sent Done false 9096 Acked Flush 0 Done Sent Done false 9097 Acked Read 7 Done Sent Done false 9098 Acked Flush 0 Done Sent Done false 9099 Acked Write 1 Done Sent Done false 9100 Acked Write 9 Done Sent Done false 9101 Acked Read 1 Done Sent Done false 9102 Acked Read 8 Done Sent Done false 9103 Acked Read 9 Done Sent Done false 9104 Acked Read 3 Done Sent Done false 9105 Acked Read 1 Done Sent Done false 9106 Acked Flush 0 Done Sent Done false 9107 Acked Read 5 Done Sent Done false 9108 Acked Read 9 Done Sent Done false 9109 Acked Flush 0 Done Sent Done false 9110 Acked Write 1 Done Sent Done false 9111 Acked Write 9 Done Sent Done false 9112 Acked Write 3 Done Sent Done false 9113 Acked Write 8 Done Sent Done false 9114 Acked Read 7 Done Sent Done false 9115 Acked Read 8 Done Sent Done false 9116 Acked Write 6 Done Sent Done false 9117 Acked Write 10 Done Sent Done false 9118 Acked Write 3 Done Sent Done false 9119 Acked Write 2 Done Sent Done false 9120 Acked Read 1 Done Sent Done false 9121 Acked Read 2 Done Sent Done false 9122 Acked Read 10 Done Sent Done false 9123 Acked Flush 0 Done Sent Done false 9124 Acked Write 8 Done Sent Done false 9125 Acked Read 3 Done Sent Done false 9126 Acked Read 5 Done Sent Done false 9127 Acked Flush 0 Done Sent Done false 9128 Acked Flush 0 Done Sent Done false 9129 Acked Read 10 Done Sent Done false 9130 Acked Read 3 Done Sent Done false 9131 Acked Read 5 Done Sent Done false 9132 Acked Read 3 Done Sent Done false 9133 Acked Read 8 Done Sent Done false 9134 Acked Flush 0 Done Sent Done false 9135 Acked Write 4 Done Sent Done false 9136 Acked Flush 0 Done Sent Done false 9137 Acked Write 10 Done Sent Done false 9138 Acked Write 8 Done Sent Done false 9139 Acked Read 7 Done Sent Done false 9140 Acked Write 5 Done Sent Done false 9141 Acked Flush 0 Done Sent Done false 9142 Acked Read 1 Done Sent Done false 9143 Acked Read 7 Done Sent Done false 9144 Acked Write 7 Done Sent Done false 9145 Acked Read 6 Done Sent Done false 9146 Acked Read 2 Done Sent Done false 9147 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 8846 9147 Downstairs last five completed: 8846 8845 8844 8843 8842 Upstairs last five completed: 9147 9146 9145 9144 9143 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 8847 Acked Write 10 Done Sent Done false 8848 Acked Read 5 Done Sent Done false 8849 Acked Write 6 Done Sent Done false 8850 Acked Read 4 Done Sent Done false 8851 Acked Read 9 Done Sent Done false 8852 Acked Write 1 Done Sent Done false 8853 Acked Write 2 Done Sent Done false 8854 Acked Flush 0 Done Sent Done false 8855 Acked Read 8 Done Sent Done false 8856 Acked Read 4 Done Sent Done false 8857 Acked Flush 0 Done Sent Done false 8858 Acked Read 3 Done Sent Done false 8859 Acked Read 1 Done Sent Done false 8860 Acked Read 10 Done Sent Done false 8861 Acked Read 8 Done Sent Done false 8862 Acked Flush 0 Done Sent Done false 8863 Acked Write 5 Done Sent Done false 8864 Acked Write 5 Done Sent Done false 8865 Acked Read 2 Done Sent Done false 8866 Acked Read 2 Done Sent Done false 8867 Acked Write 1 Done Sent Done false 8868 Acked Read 1 Done Sent Done false 8869 Acked Write 9 Done Sent Done false 8870 Acked Read 6 Done Sent Done false 8871 Acked Write 9 Done Sent Done false 8872 Acked Write 2 Done Sent Done false 8873 Acked Read 6 Done Sent Done false 8874 Acked Read 7 Done Sent Done false 8875 Acked Write 7 Done Sent Done false 8876 Acked Write 8 Done Sent Done false 8877 Acked Read 7 Done Sent Done false 8878 Acked Read 7 Done Sent Done false 8879 Acked Write 3 Done Sent Done false 8880 Acked Write 5 Done Sent Done false 8881 Acked Read 9 Done Sent Done false 8882 Acked Write 10 Done Sent Done false 8883 Acked Read 9 Done Sent Done false 8884 Acked Write 4 Done Sent Done false 8885 Acked Write 4 Done Sent Done false 8886 Acked Read 1 Done Sent Done false 8887 Acked Read 9 Done Sent Done false 8888 Acked Read 7 Done Sent Done false 8889 Acked Flush 0 Done Sent Done false 8890 Acked Read 10 Done Sent Done false 8891 Acked Read 2 Done Sent Done false 8892 Acked Flush 0 Done Sent Done false 8893 Acked Write 6 Done Sent Done false 8894 Acked Write 2 Done Sent Done false 8895 Acked Write 2 Done Sent Done false 8896 Acked Flush 0 Done Sent Done false 8897 Acked Read 2 Done Sent Done false 8898 Acked Write 5 Done Sent Done false 8899 Acked Read 3 Done Sent Done false 8900 Acked Write 9 Done Sent Done false 8901 Acked Read 5 Done Sent Done false 8902 Acked Write 2 Done Sent Done false 8903 Acked Write 10 Done Sent Done false 8904 Acked Write 9 Done Sent Done false 8905 Acked Read 7 Done Sent Done false 8906 Acked Write 1 Done Sent Done false 8907 Acked Flush 0 Done Sent Done false 8908 Acked Read 2 Done Sent Done false 8909 Acked Write 2 Done Sent Done false 8910 Acked Read 5 Done Sent Done false 8911 Acked Flush 0 Done Sent Done false 8912 Acked Write 1 Done Sent Done false 8913 Acked Read 4 Done Sent Done false 8914 Acked Write 7 Done Sent Done false 8915 Acked Read 10 Done Sent Done false 8916 Acked Read 5 Done Sent Done false 8917 Acked Read 1 Done Sent Done false 8918 Acked Read 2 Done Sent Done false 8919 Acked Read 9 Done Sent Done false 8920 Acked Write 5 Done Sent Done false 8921 Acked Write 3 Done Sent Done false 8922 Acked Write 5 Done Sent Done false 8923 Acked Write 9 Done Sent Done false 8924 Acked Read 7 Done Sent Done false 8925 Acked Read 9 Done Sent Done false 8926 Acked Read 10 Done Sent Done false 8927 Acked Read 6 Done Sent Done false 8928 Acked Write 5 Done Sent Done false 8929 Acked Read 1 Done Sent Done false 8930 Acked Write 1 Done Sent Done false 8931 Acked Read 4 Done Sent Done false 8932 Acked Read 6 Done Sent Done false 8933 Acked Read 6 Done Sent Done false 8934 Acked Flush 0 Done Sent Done false 8935 Acked Read 4 Done Sent Done false 8936 Acked Read 5 Done Sent Done false 8937 Acked Write 1 Done Sent Done false 8938 Acked Write 9 Done Sent Done false 8939 Acked Read 5 Done Sent Done false 8940 Acked Write 5 Done Sent Done false 8941 Acked Read 10 Done Sent Done false 8942 Acked Read 6 Done Sent Done false 8943 Acked Write 7 Done Sent Done false 8944 Acked Write 5 Done Sent Done false 8945 Acked Flush 0 Done Sent Done false 8946 Acked Read 4 Done Sent Done false 8947 Acked Write 1 Done Sent Done false 8948 Acked Write 1 Done Sent Done false 8949 Acked Write 5 Done Sent Done false 8950 Acked Write 7 Done Sent Done false 8951 Acked Read 7 Done Sent Done false 8952 Acked Write 7 Done Sent Done false 8953 Acked Write 5 Done Sent Done false 8954 Acked Write 9 Done Sent Done false 8955 Acked Write 2 Done Sent Done false 8956 Acked Flush 0 Done Sent Done false 8957 Acked Read 8 Done Sent Done false 8958 Acked Read 1 Done Sent Done false 8959 Acked Read 10 Done Sent Done false 8960 Acked Read 1 Done Sent Done false 8961 Acked Flush 0 Done Sent Done false 8962 Acked Read 2 Done Sent Done false 8963 Acked Write 1 Done Sent Done false 8964 Acked Write 5 Done Sent Done false 8965 Acked Write 9 Done Sent Done false 8966 Acked Read 8 Done Sent Done false 8967 Acked Write 1 Done Sent Done false 8968 Acked Read 3 Done Sent Done false 8969 Acked Read 1 Done Sent Done false 8970 Acked Read 9 Done Sent Done false 8971 Acked Read 6 Done Sent Done false 8972 Acked Read 6 Done Sent Done false 8973 Acked Read 2 Done Sent Done false 8974 Acked Read 3 Done Sent Done false 8975 Acked Read 3 Done Sent Done false 8976 Acked Write 10 Done Sent Done false 8977 Acked Write 7 Done Sent Done false 8978 Acked Write 10 Done Sent Done false 8979 Acked Flush 0 Done Sent Done false 8980 Acked Read 9 Done Sent Done false 8981 Acked Read 9 Done Sent Done false 8982 Acked Write 1 Done Sent Done false 8983 Acked Read 9 Done Sent Done false 8984 Acked Read 8 Done Sent Done false 8985 Acked Write 5 Done Sent Done false 8986 Acked Read 8 Done Sent Done false 8987 Acked Write 10 Done Sent Done false 8988 Acked Read 6 Done Sent Done false 8989 Acked Write 6 Done Sent Done false 8990 Acked Write 1 Done Sent Done false 8991 Acked Read 9 Done Sent Done false 8992 Acked Read 3 Done Sent Done false 8993 Acked Read 5 Done Sent Done false 8994 Acked Flush 0 Done Sent Done false 8995 Acked Write 4 Done Sent Done false 8996 Acked Write 3 Done Sent Done false 8997 Acked Read 7 Done Sent Done false 8998 Acked Write 7 Done Sent Done false 8999 Acked Read 2 Done Sent Done false 9000 Acked Write 10 Done Sent Done false 9001 Acked Read 1 Done Sent Done false 9002 Acked Write 5 Done Sent Done false 9003 Acked Read 6 Done Sent Done false 9004 Acked Read 8 Done Sent Done false 9005 Acked Write 9 Done Sent Done false 9006 Acked Write 5 Done Sent Done false 9007 Acked Read 1 Done Sent Done false 9008 Acked Read 2 Done Sent Done false 9009 Acked Read 3 Done Sent Done false 9010 Acked Read 3 Done Sent Done false 9011 Acked Write 8 Done Sent Done false 9012 Acked Read 9 Done Sent Done false 9013 Acked Read 8 Done Sent Done false 9014 Acked Read 10 Done Sent Done false 9015 Acked Write 10 Done Sent Done false 9016 Acked Write 5 Done Sent Done false 9017 Acked Write 7 Done Sent Done false 9018 Acked Write 6 Done Sent Done false 9019 Acked Read 5 Done Sent Done false 9020 Acked Write 4 Done Sent Done false 9021 Acked Read 10 Done Sent Done false 9022 Acked Write 1 Done Sent Done false 9023 Acked Read 4 Done Sent Done false 9024 Acked Flush 0 Done Sent Done false 9025 Acked Read 6 Done Sent Done false 9026 Acked Read 2 Done Sent Done false 9027 Acked Write 4 Done Sent Done false 9028 Acked Write 4 Done Sent Done false 9029 Acked Read 2 Done Sent Done false 9030 Acked Read 2 Done Sent Done false 9031 Acked Read 6 Done Sent Done false 9032 Acked Read 3 Done Sent Done false 9033 Acked Write 6 Done Sent Done false 9034 Acked Read 9 Done Sent Done false 9035 Acked Read 6 Done Sent Done false 9036 Acked Flush 0 Done Sent Done false 9037 Acked Read 9 Done Sent Done false 9038 Acked Read 8 Done Sent Done false 9039 Acked Read 7 Done Sent Done false 9040 Acked Write 5 Done Sent Done false 9041 Acked Read 7 Done Sent Done false 9042 Acked Read 7 Done Sent Done false 9043 Acked Read 2 Done Sent Done false 9044 Acked Read 6 Done Sent Done false 9045 Acked Flush 0 Done Sent Done false 9046 Acked Flush 0 Done Sent Done false 9047 Acked Read 6 Done Sent Done false 9048 Acked Write 6 Done Sent Done false 9049 Acked Write 9 Done Sent Done false 9050 Acked Write 7 Done Sent Done false 9051 Acked Write 6 Done Sent Done false 9052 Acked Write 6 Done Sent Done false 9053 Acked Read 8 Done Sent Done false 9054 Acked Read 10 Done Sent Done false 9055 Acked Read 5 Done Sent Done false 9056 Acked Read 3 Done Sent Done false 9057 Acked Read 4 Done Sent Done false 9058 Acked Write 4 Done Sent Done false 9059 Acked Flush 0 Done Sent Done false 9060 Acked Flush 0 Done Sent Done false 9061 Acked Read 6 Done Sent Done false 9062 Acked Read 3 Done Sent Done false 9063 Acked Read 3 Done Sent Done false 9064 Acked Write 5 Done Sent Done false 9065 Acked Read 2 Done Sent Done false 9066 Acked Flush 0 Done Sent Done false 9067 Acked Write 10 Done Sent Done false 9068 Acked Flush 0 Done Sent Done false 9069 Acked Read 8 Done Sent Done false 9070 Acked Write 1 Done Sent Done false 9071 Acked Write 6 Done Sent Done false 9072 Acked Write 1 Done Sent Done false 9073 Acked Read 5 Done Sent Done false 9074 Acked Write 4 Done Sent Done false 9075 Acked Write 8 Done Sent Done false 9076 Acked Write 4 Done Sent Done false 9077 Acked Read 7 Done Sent Done false 9078 Acked Read 10 Done Sent Done false 9079 Acked Read 7 Done Sent Done false 9080 Acked Read 7 Done Sent Done false 9081 Acked Read 7 Done Sent Done false 9082 Acked Read 6 Done Sent Done false 9083 Acked Read 4 Done Sent Done false 9084 Acked Read 7 Done Sent Done false 9085 Acked Write 8 Done Sent Done false 9086 Acked Read 10 Done Sent Done false 9087 Acked Read 9 Done Sent Done false 9088 Acked Read 6 Done Sent Done false 9089 Acked Read 3 Done Sent Done false 9090 Acked Read 1 Done Sent Done false 9091 Acked Write 2 Done Sent Done false 9092 Acked Read 6 Done Sent Done false 9093 Acked Flush 0 Done Sent Done false 9094 Acked Read 10 Done Sent Done false 9095 Acked Write 3 Done Sent Done false 9096 Acked Flush 0 Done Sent Done false 9097 Acked Read 7 Done Sent Done false 9098 Acked Flush 0 Done Sent Done false 9099 Acked Write 1 Done Sent Done false 9100 Acked Write 9 Done Sent Done false 9101 Acked Read 1 Done Sent Done false 9102 Acked Read 8 Done Sent Done false 9103 Acked Read 9 Done Sent Done false 9104 Acked Read 3 Done Sent Done false 9105 Acked Read 1 Done Sent Done false 9106 Acked Flush 0 Done Sent Done false 9107 Acked Read 5 Done Sent Done false 9108 Acked Read 9 Done Sent Done false 9109 Acked Flush 0 Done Sent Done false 9110 Acked Write 1 Done Sent Done false 9111 Acked Write 9 Done Sent Done false 9112 Acked Write 3 Done Sent Done false 9113 Acked Write 8 Done Sent Done false 9114 Acked Read 7 Done Sent Done false 9115 Acked Read 8 Done Sent Done false 9116 Acked Write 6 Done Sent Done false 9117 Acked Write 10 Done Sent Done false 9118 Acked Write 3 Done Sent Done false 9119 Acked Write 2 Done Sent Done false 9120 Acked Read 1 Done Sent Done false 9121 Acked Read 2 Done Sent Done false 9122 Acked Read 10 Done Sent Done false 9123 Acked Flush 0 Done Sent Done false 9124 Acked Write 8 Done Sent Done false 9125 Acked Read 3 Done Sent Done false 9126 Acked Read 5 Done Sent Done false 9127 Acked Flush 0 Done Sent Done false 9128 Acked Flush 0 Done Sent Done false 9129 Acked Read 10 Done Sent Done false 9130 Acked Read 3 Done Sent Done false 9131 Acked Read 5 Done Sent Done false 9132 Acked Read 3 Done Sent Done false 9133 Acked Read 8 Done Sent Done false 9134 Acked Flush 0 Done Sent Done false 9135 Acked Write 4 Done Sent Done false 9136 Acked Flush 0 Done Sent Done false 9137 Acked Write 10 Done Sent Done false 9138 Acked Write 8 Done Sent Done false 9139 Acked Read 7 Done Sent Done false 9140 Acked Write 5 Done Sent Done false 9141 Acked Flush 0 Done Sent Done false 9142 Acked Read 1 Done Sent Done false 9143 Acked Read 7 Done Sent Done false 9144 Acked Write 7 Done Sent Done false 9145 Acked Read 6 Done Sent Done false 9146 Acked Read 2 Done Sent Done false 9147 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 8846 9147 Downstairs last five completed: 8846 8845 8844 8843 8842 Upstairs last five completed: 9147 9146 9145 9144 9143 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.971783342Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:57982)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.971855623Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:45.972787963Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973288698Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973314461Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 8846","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973329487Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 8846","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.97354319Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973567183Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.97358211Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 301 jobs since flush 8846","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973595926Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 301 jobs since flush: 8846","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:45.973828313Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9147 9146 9145 9144 9143 Upstairs last five completed: 9147 9146 9145 9144 9143 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Nov 27 01:48:46.213 INFO request completed, latency_us: 412, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 38b91b4f-8e5c-4795-863f-bb17bf19702b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:46.214259928Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:46.214310494Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:46.214330029Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:46.214344505Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:46.214359741Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:46.214434311Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:48:46.214 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/0, method: GET, req_id: 78addab3-46ba-4481-a994-16e73d69a948, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit Nov 27 01:48:50.216 INFO request completed, latency_us: 242, response_code: 200, uri: /state/cid/0, method: GET, req_id: 08a9e383-5f19-47da-8132-1e22fe68fdd4, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 2293 len 20480 data: 4 4 2 2 4 002/300 Read block 3103 len 12288 003/300 Write block 1651 len 16384 data: 7 7 5 5 004/300 Write block 4242 len 32768 data: 6 6 5 4 4 3 3 4 005/300 Read block 1738 len 24576 006/300 Write block 3975 len 4096 data: 3 007/300 Write block 1750 len 32768 data: 8 8 9 8 7 7 6 5 008/300 Read block 3065 len 36864 009/300 Flush 010/300 Write block 2789 len 8192 data: 6 5 011/300 Write block 3074 len 20480 data: 4 4 6 6 6 012/300 Write block 1984 len 16384 data: 4 5 4 5 013/300 Write block 2092 len 36864 data: 5 4 4 4 5 5 6 5 6 014/300 Read block 934 len 36864 015/300 Write block 3301 len 40960 data: 5 5 5 6 6 4 5 5 5 5 016/300 Write block 3833 len 16384 data: 7 6 6 6 017/300 Read block 1965 len 20480 018/300 Write block 1998 len 36864 data: 11 9 8 8 6 8 7 9 8 019/300 Flush 020/300 Write block 4665 len 24576 data: 9 9 10 10 9 10 021/300 Read block 183 len 8192 022/300 Read block 4744 len 8192 023/300 Read block 4309 len 32768 024/300 Write block 3438 len 20480 data: 6 6 6 5 5 025/300 Read block 2049 len 28672 026/300 Read block 560 len 20480 027/300 Write block 798 len 28672 data: 5 6 6 6 7 7 8 028/300 Write block 189 len 36864 data: 7 7 7 7 6 5 4 3 5 029/300 Read block 1014 len 20480 030/300 Read block 1364 len 28672 031/300 Read block 2131 len 16384 032/300 Read block 1794 len 16384 033/300 Read block 3139 len 40960 034/300 Write block 3540 len 36864 data: 10 11 9 8 8 7 6 6 7 035/300 Read block 2687 len 28672 036/300 Flush 037/300 Write block 3208 len 40960 data: 6 8 8 9 8 9 8 8 7 7 038/300 Read block 612 len 20480 039/300 Read block 3875 len 32768 040/300 Write block 3587 len 36864 data: 3 3 4 5 6 7 7 7 7 041/300 Write block 4411 len 20480 data: 5 4 6 8 6 042/300 Read block 4629 len 20480 043/300 Read block 3753 len 40960 044/300 Read block 3714 len 4096 045/300 Read block 1997 len 8192 046/300 Write block 4870 len 20480 data: 2 2 2 2 3 047/300 Read block 1016 len 36864 048/300 Read block 3363 len 28672 049/300 Read block 2144 len 32768 050/300 Write block 4654 len 32768 data: 6 8 8 9 10 10 8 7 051/300 Read block 3287 len 24576 052/300 Read block 4340 len 36864 053/300 Write block 4429 len 12288 data: 6 5 5 054/300 Flush 055/300 Read block 2309 len 28672 056/300 Write block 3811 len 32768 data: 10 9 8 9 8 5 4 4 057/300 Read block 1180 len 16384 058/300 Read block 3075 len 4096 059/300 Read block 3911 len 40960 060/300 Read block 4969 len 28672 061/300 Write block 2058 len 20480 data: 5 3 3 3 3 062/300 Read block 4100 len 8192 063/300 Read block 1162 len 32768 064/300 Write block 44 len 40960 data: 4 4 4 4 3 4 4 5 6 6 065/300 Write block 4118 len 16384 data: 6 5 3 3 066/300 Write block 2443 len 40960 data: 7 6 7 7 5 5 5 4 4 4 067/300 Write block 4080 len 40960 data: 6 5 7 6 7 7 6 6 7 7 068/300 Read block 181 len 40960 069/300 Write block 1962 len 40960 data: 8 8 7 6 5 5 5 4 5 6 070/300 Write block 873 len 40960 data: 6 5 4 6 6 6 6 6 6 4 071/300 Write block 3067 len 12288 data: 2 3 4 072/300 Write block 969 len 12288 data: 5 4 4 073/300 Read block 2982 len 28672 074/300 Read block 264 len 16384 075/300 Read block 4438 len 40960 076/300 Read block 1130 len 32768 077/300 Read block 798 len 20480 078/300 Read block 4782 len 8192 079/300 Write block 3956 len 20480 data: 6 5 5 5 4 080/300 Read block 1644 len 28672 081/300 Read block 3883 len 40960 082/300 Write block 23 len 4096 data: 6 083/300 Read block 3092 len 36864 084/300 Read block 1763 len 24576 085/300 Read block 3465 len 40960 086/300 Write block 1584 len 24576 data: 7 8 9 11 11 10 087/300 Read block 1901 len 32768 088/300 Read block 3082 len 8192 089/300 Write block 2871 len 4096 data: 6 090/300 Read block 3431 len 16384 091/300 Write block 3037 len 12288 data: 5 4 4 092/300 Read block 26 len 32768 093/300 Read block 1867 len 32768 094/300 Flush 095/300 Read block 1793 len 28672 096/300 Read block 656 len 32768 097/300 Read block 4947 len 12288 098/300 Write block 2369 len 4096 data: 5 099/300 Read block 3497 len 16384 100/300 Read block 3510 len 28672 101/300 Write block 1190 len 20480 data: 6 6 6 5 6 102/300 Flush 103/300 Write block 1667 len 40960 data: 5 4 4 4 3 4 5 6 7 6 104/300 Flush 105/300 Write block 307 len 28672 data: 4 5 6 6 8 7 8 106/300 Read block 941 len 24576 107/300 Read block 16 len 12288 108/300 Read block 896 len 12288 109/300 Read block 962 len 12288 110/300 Read block 225 len 8192 111/300 Read block 2372 len 40960 112/300 Flush 113/300 Read block 3420 len 28672 114/300 Write block 3591 len 12288 data: 7 8 8 115/300 Write block 1597 len 24576 data: 4 6 7 6 6 5 116/300 Write block 1405 len 36864 data: 9 7 7 6 7 6 6 6 6 117/300 Read block 2798 len 32768 118/300 Write block 962 len 28672 data: 7 7 7 6 6 7 5 119/300 Write block 4404 len 36864 data: 3 3 3 4 5 4 5 6 5 120/300 Write block 4575 len 8192 data: 4 3 121/300 Read block 4720 len 40960 122/300 Read block 2366 len 8192 123/300 Read block 1385 len 24576 124/300 Write block 4950 len 16384 data: 3 4 4 4 125/300 Read block 1012 len 16384 126/300 Flush 127/300 Write block 310 len 8192 data: 7 9 128/300 Read block 2449 len 8192 129/300 Read block 3932 len 8192 130/300 Write block 4967 len 8192 data: 5 7 131/300 Write block 2210 len 40960 data: 5 4 4 4 5 7 6 7 7 7 132/300 Read block 3591 len 20480 133/300 Read block 468 len 4096 134/300 Read block 3635 len 28672 135/300 Write block 3841 len 40960 data: 3 3 3 3 3 4 2 4 5 9 136/300 Write block 1845 len 36864 data: 3 3 5 6 7 8 8 9 8 137/300 Read block 2651 len 36864 138/300 Read block 918 len 40960 139/300 Write block 3204 len 36864 data: 5 5 5 5 7 9 9 10 9 140/300 Write block 3415 len 36864 data: 3 3 3 4 4 4 5 7 6 141/300 Read block 4746 len 36864 142/300 Flush 143/300 Read block 665 len 24576 144/300 Read block 1406 len 12288 145/300 Flush 146/300 Read block 1446 len 4096 147/300 Write block 2975 len 40960 data: 8 8 6 5 5 5 4 4 4 4 148/300 Write block 2947 len 16384 data: 7 7 6 5 149/300 Read block 3013 len 40960 150/300 Read block 731 len 32768 151/300 Read block 2053 len 16384 152/300 Read block 1892 len 8192 153/300 Read block 2199 len 12288 154/300 Read block 3412 len 12288 155/300 Read block 3659 len 12288 156/300 Write block 2405 len 16384 data: 7 7 6 5 157/300 Read block 4574 len 36864 158/300 Read block 2269 len 20480 159/300 Write block 1184 len 12288 data: 6 6 4 160/300 Flush 161/300 Read block 1812 len 4096 162/300 Write block 229 len 8192 data: 5 6 163/300 Write block 683 len 40960 data: 6 6 6 5 5 5 4 4 4 5 164/300 Write block 906 len 8192 data: 5 5 165/300 Read block 2739 len 8192 166/300 Write block 2597 len 16384 data: 6 6 4 4 167/300 Read block 954 len 4096 168/300 Read block 4147 len 20480 169/300 Read block 1384 len 32768 170/300 Write block 3091 len 28672 data: 5 5 5 5 5 5 8 171/300 Write block 1260 len 24576 data: 5 5 5 7 8 9 172/300 Read block 4810 len 8192 173/300 Write block 362 len 32768 data: 3 3 2 3 3 3 8 8 174/300 Write block 4587 len 36864 data: 5 6 6 4 4 5 5 7 8 175/300 Read block 4881 len 24576 176/300 Read block 4638 len 24576 177/300 Read block 1674 len 20480 178/300 Read block 1991 len 20480 179/300 Read block 3501 len 16384 180/300 Write block 148 len 8192 data: 6 7 181/300 Read block 1332 len 16384 182/300 Write block 1238 len 4096 data: 4 183/300 Read block 2601 len 8192 184/300 Flush 185/300 Flush 186/300 Write block 1364 len 8192 data: 6 5 187/300 Read block 634 len 8192 188/300 Write block 4411 len 36864 data: 7 6 7 9 7 5 6 6 7 189/300 Flush 190/300 Read block 2917 len 36864 191/300 Read block 721 len 4096 192/300 Read block 3050 len 12288 193/300 Flush 194/300 Write block 3611 len 40960 data: 7 7 7 8 8 8 6 4 4 7 195/300 Read block 3706 len 32768 196/300 Write block 94 len 20480 data: 4 6 6 6 7 197/300 Read block 154 len 16384 198/300 Write block 4803 len 28672 data: 7 7 6 5 4 3 2 199/300 Write block 1366 len 12288 data: 6 7 8 200/300 Write block 1418 len 24576 data: 5 6 6 6 6 7 201/300 Read block 2047 len 40960 202/300 Write block 2781 len 4096 data: 4 203/300 Write block 1617 len 12288 data: 4 3 4 204/300 Read block 4301 len 16384 205/300 Write block 1794 len 8192 data: 8 6 206/300 Read block 70 len 12288 207/300 Read block 776 len 16384 208/300 Read block 2636 len 36864 209/300 Write block 1170 len 36864 data: 9 10 9 7 5 5 7 9 9 210/300 Write block 4429 len 32768 data: 7 6 6 5 7 8 7 6 211/300 Write block 3498 len 28672 data: 3 3 2 5 4 4 4 212/300 Read block 3093 len 4096 213/300 Write block 2737 len 36864 data: 8 9 9 10 8 7 7 4 5 214/300 Write block 3524 len 12288 data: 8 6 5 215/300 Read block 613 len 40960 216/300 Flush 217/300 Read block 1926 len 12288 218/300 Write block 1446 len 16384 data: 5 5 6 5 219/300 Read block 3424 len 16384 220/300 Read block 4540 len 40960 221/300 Write block 3673 len 24576 data: 7 8 7 6 7 7 222/300 Write block 4631 len 32768 data: 4 4 5 5 4 4 4 3 223/300 Write block 4954 len 32768 data: 5 5 6 5 5 6 6 6 224/300 Write block 2086 len 4096 data: 6 225/300 Read block 2662 len 8192 226/300 Flush 227/300 Write block 1952 len 40960 data: 8 9 10 11 7 10 9 9 6 6 228/300 Write block 1145 len 20480 data: 5 4 3 5 6 229/300 Read block 933 len 28672 230/300 Read block 259 len 16384 231/300 Write block 3332 len 8192 data: 9 8 232/300 Read block 4447 len 4096 233/300 Read block 3604 len 4096 234/300 Read block 137 len 36864 235/300 Write block 902 len 16384 data: 5 6 4 5 236/300 Write block 3188 len 36864 data: 5 4 4 3 5 6 7 7 9 237/300 Read block 1285 len 24576 238/300 Read block 4591 len 32768 239/300 Flush 240/300 Read block 2324 len 4096 241/300 Read block 1735 len 20480 242/300 Read block 1086 len 28672 243/300 Write block 1048 len 4096 data: 5 244/300 Read block 597 len 8192 245/300 Flush 246/300 Write block 71 len 32768 data: 5 5 4 4 4 4 3 3 247/300 Read block 883 len 32768 248/300 Read block 1890 len 4096 249/300 Read block 3715 len 24576 250/300 Write block 2975 len 16384 data: 9 9 7 6 251/300 Write block 3212 len 16384 data: 10 10 9 9 252/300 Flush 253/300 Write block 910 len 24576 data: 5 4 3 3 4 3 254/300 Read block 105 len 12288 255/300 Read block 1756 len 16384 256/300 Read block 3405 len 4096 257/300 Read block 4095 len 40960 258/300 Read block 3023 len 16384 259/300 Read block 4409 len 20480 260/300 Write block 19 len 8192 data: 7 7 261/300 Flush 262/300 Write block 2264 len 12288 data: 7 6 6 263/300 Flush 264/300 Read block 416 len 12288 265/300 Write block 3470 len 12288 data: 7 7 6 266/300 Read block 4207 len 20480 267/300 Write block 4323 len 12288 data: 3 5 5 268/300 Read block 4287 len 20480 269/300 Read block 198 len 28672 270/300 Write block 238 len 4096 data: 4 271/300 Flush 272/300 Write block 4651 len 16384 data: 3 4 5 7 273/300 Write block 4522 len 12288 data: 4 5 5 274/300 Write block 2502 len 12288 data: 8 7 9 275/300 Read block 3568 len 24576 276/300 Write block 1828 len 20480 data: 5 5 5 5 6 277/300 Write block 2388 len 16384 data: 11 11 9 9 278/300 Write block 2730 len 8192 data: 6 7 279/300 Read block 463 len 40960 280/300 Read block 1599 len 28672 281/300 Read block 3404 len 20480 282/300 Read block 442 len 8192 283/300 Read block 986 len 24576 284/300 Write block 4828 len 16384 data: 8 8 8 6 285/300 Read block 948 len 20480 286/300 Write block 1719 len 28672 data: 4 4 5 5 4 5 6 287/300 Read block 577 len 16384 288/300 Write block 297 len 4096 data: 4 289/300 Write block 1698 len 28672 data: 4 4 6 5 5 5 6 290/300 Read block 4643 len 36864 291/300 Read block 3156 len 32768 292/300 Write block 187 len 24576 data: 8 8 8 8 8 8 293/300 Write block 4042 len 12288 data: 5 4 3 294/300 Flush 295/300 Write block 225 len 20480 data: 5 6 6 6 6 296/300 Write block 2065 len 28672 data: 5 5 4 4 4 4 6 297/300 Flush 298/300 Write block 2335 len 28672 data: 5 6 5 3 3 4 3 299/300 Write block 825 len 16384 data: 12 14 13 13 300/300 Write block 413 len 32768 data: 4 6 4 3 3 4 4 4 301/300 Write block 3913 len 28672 data: 3 3 3 4 5 5 4 Main task has work to do, go find it got dsc Start(0) start 0 Nov 27 01:48:50.450 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/0, method: POST, req_id: d65e08b3-6a0f-4080-934f-b8ea9410c7f9, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9148 Acked Write 5 Sent Done Done false 9149 Acked Read 3 Sent Done Done false 9150 Acked Write 4 Sent Done Done false 9151 Acked Write 8 Sent Done Done false 9152 Acked Read 6 Sent Done Done false 9153 Acked Write 1 Sent Done Done false 9154 Acked Write 8 Sent Done Done false 9155 Acked Read 9 Sent Done Done false 9156 Acked Flush 0 Sent Done Done false 9157 Acked Write 2 Sent Done Done false 9158 Acked Write 5 Sent Done Done false 9159 Acked Write 4 Sent Done Done false 9160 Acked Write 9 Sent Done Done false 9161 Acked Read 9 Sent Done Done false 9162 Acked Write 10 Sent Done Done false 9163 Acked Write 4 Sent Done Done false 9164 Acked Read 5 Sent Done Done false 9165 Acked Write 9 Sent Done Done false 9166 Acked Flush 0 Sent Done Done false 9167 Acked Write 6 Sent Done Done false 9168 Acked Read 2 Sent Done Done false 9169 Acked Read 2 Sent Done Done false 9170 Acked Read 8 Sent Done Done false 9171 Acked Write 5 Sent Done Done false 9172 Acked Read 7 Sent Done Done false 9173 Acked Read 5 Sent Done Done false 9174 Acked Write 7 Sent Done Done false 9175 Acked Write 9 Sent Done Done false 9176 Acked Read 5 Sent Done Done false 9177 Acked Read 7 Sent Done Done false 9178 Acked Read 4 Sent Done Done false 9179 Acked Read 4 Sent Done Done false 9180 Acked Read 10 Sent Done Done false 9181 Acked Write 9 Sent Done Done false 9182 Acked Read 7 Sent Done Done false 9183 Acked Flush 0 Sent Done Done false 9184 Acked Write 10 Sent Done Done false 9185 Acked Read 5 Sent Done Done false 9186 Acked Read 8 Sent Done Done false 9187 Acked Write 9 Sent Done Done false 9188 Acked Write 5 Sent Done Done false 9189 Acked Read 5 Sent Done Done false 9190 Acked Read 10 Sent Done Done false 9191 Acked Read 1 Sent Done Done false 9192 Acked Read 2 Sent Done Done false 9193 Acked Write 5 Sent Done Done false 9194 Acked Read 9 Sent Done Done false 9195 Acked Read 7 Sent Done Done false 9196 Acked Read 8 Sent Done Done false 9197 Acked Write 8 Sent Done Done false 9198 Acked Read 6 Sent Done Done false 9199 Acked Read 9 Sent Done Done false 9200 Acked Write 3 Sent Done Done false 9201 Acked Flush 0 Sent Done Done false 9202 Acked Read 7 Sent Done Done false 9203 Acked Write 8 Sent Done Done false 9204 Acked Read 4 Sent Done Done false 9205 Acked Read 1 Sent Done Done false 9206 Acked Read 10 Sent Done Done false 9207 Acked Read 7 Sent Done Done false 9208 Acked Write 5 Sent Done Done false 9209 Acked Read 2 Sent Done Done false 9210 Acked Read 8 Sent Done Done false 9211 Acked Write 10 Sent Done Done false 9212 Acked Write 4 Sent Done Done false 9213 Acked Write 10 Sent Done Done false 9214 Acked Write 10 Sent Done Done false 9215 Acked Read 10 Sent Done Done false 9216 Acked Write 10 Sent Done Done false 9217 Acked Write 10 Sent Done Done false 9218 Acked Write 3 Sent Done Done false 9219 Acked Write 3 Sent Done Done false 9220 Acked Read 7 Sent Done Done false 9221 Acked Read 4 Sent Done Done false 9222 Acked Read 10 Sent Done Done false 9223 Acked Read 8 Sent Done Done false 9224 Acked Read 5 Sent Done Done false 9225 Acked Read 2 Sent Done Done false 9226 Acked Write 5 Sent Done Done false 9227 Acked Read 7 Sent Done Done false 9228 Acked Read 10 Sent Done Done false 9229 Acked Write 1 Sent Done Done false 9230 Acked Read 9 Sent Done Done false 9231 Acked Read 6 Sent Done Done false 9232 Acked Read 10 Sent Done Done false 9233 Acked Write 6 Sent Done Done false 9234 Acked Read 8 Sent Done Done false 9235 Acked Read 2 Sent Done Done false 9236 Acked Write 1 Sent Done Done false 9237 Acked Read 4 Sent Done Done false 9238 Acked Write 3 Sent Done Done false 9239 Acked Read 8 Sent Done Done false 9240 Acked Read 8 Sent Done Done false 9241 Acked Flush 0 Sent Done Done false 9242 Acked Read 7 Sent Done Done false 9243 Acked Read 8 Sent Done Done false 9244 Acked Read 3 Sent Done Done false 9245 Acked Write 1 Sent Done Done false 9246 Acked Read 4 Sent Done Done false 9247 Acked Read 7 Sent Done Done false 9248 Acked Write 5 Sent Done Done false 9249 Acked Flush 0 Sent Done Done false 9250 Acked Write 10 Sent Done Done false 9251 Acked Flush 0 Sent Done Done false 9252 Acked Write 7 Sent Done Done false 9253 Acked Read 6 Sent Done Done false 9254 Acked Read 3 Sent Done Done false 9255 Acked Read 3 Sent Done Done false 9256 Acked Read 3 Sent Done Done false 9257 Acked Read 2 Sent Done Done false 9258 Acked Read 10 Sent Done Done false 9259 Acked Flush 0 Sent Done Done false 9260 Acked Read 7 Sent Done Done false 9261 Acked Write 3 Sent Done Done false 9262 Acked Write 6 Sent Done Done false 9263 Acked Write 9 Sent Done Done false 9264 Acked Read 8 Sent Done Done false 9265 Acked Write 7 Sent Done Done false 9266 Acked Write 9 Sent Done Done false 9267 Acked Write 2 Sent Done Done false 9268 Acked Read 10 Sent Done Done false 9269 Acked Read 2 Sent Done Done false 9270 Acked Read 6 Sent Done Done false 9271 Acked Write 4 Sent Done Done false 9272 Acked Read 4 Sent Done Done false 9273 Acked Flush 0 Sent Done Done false 9274 Acked Write 2 Sent Done Done false 9275 Acked Read 2 Sent Done Done false 9276 Acked Read 2 Sent Done Done false 9277 Acked Write 2 Sent Done Done false 9278 Acked Write 10 Sent Done Done false 9279 Acked Read 5 Sent Done Done false 9280 Acked Read 1 Sent Done Done false 9281 Acked Read 7 Sent Done Done false 9282 Acked Write 10 Sent Done Done false 9283 Acked Write 9 Sent Done Done false 9284 Acked Read 9 Sent Done Done false 9285 Acked Read 10 Sent Done Done false 9286 Acked Write 9 Sent Done Done false 9287 Acked Write 9 Sent Done Done false 9288 Acked Read 9 Sent Done Done false 9289 Acked Flush 0 Sent Done Done false 9290 Acked Read 6 Sent Done Done false 9291 Acked Read 3 Sent Done Done false 9292 Acked Flush 0 Sent Done Done false 9293 Acked Read 1 Sent Done Done false 9294 Acked Write 10 Sent Done Done false 9295 Acked Write 4 Sent Done Done false 9296 Acked Read 10 Sent Done Done false 9297 Acked Read 8 Sent Done Done false 9298 Acked Read 4 Sent Done Done false 9299 Acked Read 2 Sent Done Done false 9300 Acked Read 3 Sent Done Done false 9301 Acked Read 3 Sent Done Done false 9302 Acked Read 3 Sent Done Done false 9303 Acked Write 4 Sent Done Done false 9304 Acked Read 9 Sent Done Done false 9305 Acked Read 5 Sent Done Done false 9306 Acked Write 3 Sent Done Done false 9307 Acked Flush 0 Sent Done Done false 9308 Acked Read 1 Sent Done Done false 9309 Acked Write 2 Sent Done Done false 9310 Acked Write 10 Sent Done Done false 9311 Acked Write 2 Sent Done Done false 9312 Acked Read 2 Sent Done Done false 9313 Acked Write 4 Sent Done Done false 9314 Acked Read 1 Sent Done Done false 9315 Acked Read 5 Sent Done Done false 9316 Acked Read 8 Sent Done Done false 9317 Acked Write 7 Sent Done Done false 9318 Acked Write 6 Sent Done Done false 9319 Acked Read 2 Sent Done Done false 9320 Acked Write 8 Sent Done Done false 9321 Acked Write 9 Sent Done Done false 9322 Acked Read 6 Sent Done Done false 9323 Acked Read 6 Sent Done Done false 9324 Acked Read 5 Sent Done Done false 9325 Acked Read 5 Sent Done Done false 9326 Acked Read 4 Sent Done Done false 9327 Acked Write 2 Sent Done Done false 9328 Acked Read 4 Sent Done Done false 9329 Acked Write 1 Sent Done Done false 9330 Acked Read 2 Sent Done Done false 9331 Acked Flush 0 Sent Done Done false 9332 Acked Flush 0 Sent Done Done false 9333 Acked Write 2 Sent Done Done false 9334 Acked Read 2 Sent Done Done false 9335 Acked Write 9 Sent Done Done false 9336 Acked Flush 0 Sent Done Done false 9337 Acked Read 9 Sent Done Done false 9338 Acked Read 1 Sent Done Done false 9339 Acked Read 3 Sent Done Done false 9340 Acked Flush 0 Sent Done Done false 9341 Acked Write 10 Sent Done Done false 9342 Acked Read 8 Sent Done Done false 9343 Acked Write 5 Sent Done Done false 9344 Acked Read 4 Sent Done Done false 9345 Acked Write 7 Sent Done Done false 9346 Acked Write 3 Sent Done Done false 9347 Acked Write 6 Sent Done Done false 9348 Acked Read 10 Sent Done Done false 9349 Acked Write 1 Sent Done Done false 9350 Acked Write 3 Sent Done Done false 9351 Acked Read 4 Sent Done Done false 9352 Acked Write 2 Sent Done Done false 9353 Acked Read 3 Sent Done Done false 9354 Acked Read 4 Sent Done Done false 9355 Acked Read 9 Sent Done Done false 9356 Acked Write 9 Sent Done Done false 9357 Acked Write 8 Sent Done Done false 9358 Acked Write 7 Sent Done Done false 9359 Acked Read 1 Sent Done Done false 9360 Acked Write 9 Sent Done Done false 9361 Acked Write 3 Sent Done Done false 9362 Acked Read 10 Sent Done Done false 9363 Acked Flush 0 Sent Done Done false 9364 Acked Read 3 Sent Done Done false 9365 Acked Write 4 Sent Done Done false 9366 Acked Read 4 Sent Done Done false 9367 Acked Read 10 Sent Done Done false 9368 Acked Write 6 Sent Done Done false 9369 Acked Write 8 Sent Done Done false 9370 Acked Write 8 Sent Done Done false 9371 Acked Write 1 Sent Done Done false 9372 Acked Read 2 Sent Done Done false 9373 Acked Flush 0 Sent Done Done false 9374 Acked Write 10 Sent Done Done false 9375 Acked Write 5 Sent Done Done false 9376 Acked Read 7 Sent Done Done false 9377 Acked Read 4 Sent Done Done false 9378 Acked Write 2 Sent Done Done false 9379 Acked Read 1 Sent Done Done false 9380 Acked Read 1 Sent Done Done false 9381 Acked Read 9 Sent Done Done false 9382 Acked Write 4 Sent Done Done false 9383 Acked Write 9 Sent Done Done false 9384 Acked Read 6 Sent Done Done false 9385 Acked Read 8 Sent Done Done false 9386 Acked Flush 0 Sent Done Done false 9387 Acked Read 1 Sent Done Done false 9388 Acked Read 5 Sent Done Done false 9389 Acked Read 7 Sent Done Done false 9390 Acked Write 1 Sent Done Done false 9391 Acked Read 2 Sent Done Done false 9392 Acked Flush 0 Sent Done Done false 9393 Acked Write 8 Sent Done Done false 9394 Acked Read 8 Sent Done Done false 9395 Acked Read 1 Sent Done Done false 9396 Acked Read 6 Sent Done Done false 9397 Acked Write 4 Sent Done Done false 9398 Acked Write 4 Sent Done Done false 9399 Acked Flush 0 Sent Done Done false 9400 Acked Write 6 Sent Done Done false 9401 Acked Read 3 Sent Done Done false 9402 Acked Read 4 Sent Done Done false 9403 Acked Read 1 Sent Done Done false 9404 Acked Read 10 Sent Done Done false 9405 Acked Read 4 Sent Done Done false 9406 Acked Read 5 Sent Done Done false 9407 Acked Write 2 Sent Done Done false 9408 Acked Flush 0 Sent Done Done false 9409 Acked Write 3 Sent Done Done false 9410 Acked Flush 0 Sent Done Done false 9411 Acked Read 3 Sent Done Done false 9412 Acked Write 3 Sent Done Done false 9413 Acked Read 5 Sent Done Done false 9414 Acked Write 3 Sent Done Done false 9415 Acked Read 5 Sent Done Done false 9416 Acked Read 7 Sent Done Done false 9417 Acked Write 1 Sent Done Done false 9418 Acked Flush 0 Sent Done Done false 9419 Acked Write 4 Sent Done Done false 9420 Acked Write 3 Sent Done Done false 9421 Acked Write 3 Sent Done Done false 9422 Acked Read 6 Sent Done Done false 9423 Acked Write 5 Sent Done Done false 9424 Acked Write 4 Sent Done Done false 9425 Acked Write 2 Sent Done Done false 9426 Acked Read 10 Sent Done Done false 9427 Acked Read 7 Sent Done Done false 9428 Acked Read 5 Sent Done Done false 9429 Acked Read 2 Sent Done Done false 9430 Acked Read 6 Sent Done Done false 9431 Acked Write 4 Sent Done Done false 9432 Acked Read 5 Sent Done Done false 9433 Acked Write 7 Sent Done Done false 9434 Acked Read 4 Sent Done Done false 9435 Acked Write 1 Sent Done Done false 9436 Acked Write 7 Sent Done Done false 9437 Acked Read 9 Sent Done Done false 9438 Acked Read 8 Sent Done Done false 9439 Acked Write 6 Sent Done Done false 9440 Acked Write 3 Sent Done Done false 9441 Acked Flush 0 Sent Done Done false 9442 Acked Write 5 Sent Done Done false 9443 Acked Write 7 Sent Done Done false 9444 Acked Flush 0 Sent Done Done false 9445 Acked Write 7 Sent Done Done false 9446 Acked Write 4 Sent Done Done false 9447 Acked Write 8 Sent Done Done false 9448 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 301 1 1 303 Done 0 300 300 600 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 9444 9444 Downstairs last five completed: 9147 9146 9145 9144 9143 Upstairs last five completed: 9448 9447 9446 9445 9444 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9148 Acked Write 5 Sent Done Done false 9149 Acked Read 3 Sent Done Done false 9150 Acked Write 4 Sent Done Done false 9151 Acked Write 8 Sent Done Done false 9152 Acked Read 6 Sent Done Done false 9153 Acked Write 1 Sent Done Done false 9154 Acked Write 8 Sent Done Done false 9155 Acked Read 9 Sent Done Done false 9156 Acked Flush 0 Sent Done Done false 9157 Acked Write 2 Sent Done Done false 9158 Acked Write 5 Sent Done Done false 9159 Acked Write 4 Sent Done Done false 9160 Acked Write 9 Sent Done Done false 9161 Acked Read 9 Sent Done Done false 9162 Acked Write 10 Sent Done Done false 9163 Acked Write 4 Sent Done Done false 9164 Acked Read 5 Sent Done Done false 9165 Acked Write 9 Sent Done Done false 9166 Acked Flush 0 Sent Done Done false 9167 Acked Write 6 Sent Done Done false 9168 Acked Read 2 Sent Done Done false 9169 Acked Read 2 Sent Done Done false 9170 Acked Read 8 Sent Done Done false 9171 Acked Write 5 Sent Done Done false 9172 Acked Read 7 Sent Done Done false 9173 Acked Read 5 Sent Done Done false 9174 Acked Write 7 Sent Done Done false 9175 Acked Write 9 Sent Done Done false 9176 Acked Read 5 Sent Done Done false 9177 Acked Read 7 Sent Done Done false 9178 Acked Read 4 Sent Done Done false 9179 Acked Read 4 Sent Done Done false 9180 Acked Read 10 Sent Done Done false 9181 Acked Write 9 Sent Done Done false 9182 Acked Read 7 Sent Done Done false 9183 Acked Flush 0 Sent Done Done false 9184 Acked Write 10 Sent Done Done false 9185 Acked Read 5 Sent Done Done false 9186 Acked Read 8 Sent Done Done false 9187 Acked Write 9 Sent Done Done false 9188 Acked Write 5 Sent Done Done false 9189 Acked Read 5 Sent Done Done false 9190 Acked Read 10 Sent Done Done false 9191 Acked Read 1 Sent Done Done false 9192 Acked Read 2 Sent Done Done false 9193 Acked Write 5 Sent Done Done false 9194 Acked Read 9 Sent Done Done false 9195 Acked Read 7 Sent Done Done false 9196 Acked Read 8 Sent Done Done false 9197 Acked Write 8 Sent Done Done false 9198 Acked Read 6 Sent Done Done false 9199 Acked Read 9 Sent Done Done false 9200 Acked Write 3 Sent Done Done false 9201 Acked Flush 0 Sent Done Done false 9202 Acked Read 7 Sent Done Done false 9203 Acked Write 8 Sent Done Done false 9204 Acked Read 4 Sent Done Done false 9205 Acked Read 1 Sent Done Done false 9206 Acked Read 10 Sent Done Done false 9207 Acked Read 7 Sent Done Done false 9208 Acked Write 5 Sent Done Done false 9209 Acked Read 2 Sent Done Done false 9210 Acked Read 8 Sent Done Done false 9211 Acked Write 10 Sent Done Done false 9212 Acked Write 4 Sent Done Done false 9213 Acked Write 10 Sent Done Done false 9214 Acked Write 10 Sent Done Done false 9215 Acked Read 10 Sent Done Done false 9216 Acked Write 10 Sent Done Done false 9217 Acked Write 10 Sent Done Done false 9218 Acked Write 3 Sent Done Done false 9219 Acked Write 3 Sent Done Done false 9220 Acked Read 7 Sent Done Done false 9221 Acked Read 4 Sent Done Done false 9222 Acked Read 10 Sent Done Done false 9223 Acked Read 8 Sent Done Done false 9224 Acked Read 5 Sent Done Done false 9225 Acked Read 2 Sent Done Done false 9226 Acked Write 5 Sent Done Done false 9227 Acked Read 7 Sent Done Done false 9228 Acked Read 10 Sent Done Done false 9229 Acked Write 1 Sent Done Done false 9230 Acked Read 9 Sent Done Done false 9231 Acked Read 6 Sent Done Done false 9232 Acked Read 10 Sent Done Done false 9233 Acked Write 6 Sent Done Done false 9234 Acked Read 8 Sent Done Done false 9235 Acked Read 2 Sent Done Done false 9236 Acked Write 1 Sent Done Done false 9237 Acked Read 4 Sent Done Done false 9238 Acked Write 3 Sent Done Done false 9239 Acked Read 8 Sent Done Done false 9240 Acked Read 8 Sent Done Done false 9241 Acked Flush 0 Sent Done Done false 9242 Acked Read 7 Sent Done Done false 9243 Acked Read 8 Sent Done Done false 9244 Acked Read 3 Sent Done Done false 9245 Acked Write 1 Sent Done Done false 9246 Acked Read 4 Sent Done Done false 9247 Acked Read 7 Sent Done Done false 9248 Acked Write 5 Sent Done Done false 9249 Acked Flush 0 Sent Done Done false 9250 Acked Write 10 Sent Done Done false 9251 Acked Flush 0 Sent Done Done false 9252 Acked Write 7 Sent Done Done false 9253 Acked Read 6 Sent Done Done false 9254 Acked Read 3 Sent Done Done false 9255 Acked Read 3 Sent Done Done false 9256 Acked Read 3 Sent Done Done false 9257 Acked Read 2 Sent Done Done false 9258 Acked Read 10 Sent Done Done false 9259 Acked Flush 0 Sent Done Done false 9260 Acked Read 7 Sent Done Done false 9261 Acked Write 3 Sent Done Done false 9262 Acked Write 6 Sent Done Done false 9263 Acked Write 9 Sent Done Done false 9264 Acked Read 8 Sent Done Done false 9265 Acked Write 7 Sent Done Done false 9266 Acked Write 9 Sent Done Done false 9267 Acked Write 2 Sent Done Done false 9268 Acked Read 10 Sent Done Done false 9269 Acked Read 2 Sent Done Done false 9270 Acked Read 6 Sent Done Done false 9271 Acked Write 4 Sent Done Done false 9272 Acked Read 4 Sent Done Done false 9273 Acked Flush 0 Sent Done Done false 9274 Acked Write 2 Sent Done Done false 9275 Acked Read 2 Sent Done Done false 9276 Acked Read 2 Sent Done Done false 9277 Acked Write 2 Sent Done Done false 9278 Acked Write 10 Sent Done Done false 9279 Acked Read 5 Sent Done Done false 9280 Acked Read 1 Sent Done Done false 9281 Acked Read 7 Sent Done Done false 9282 Acked Write 10 Sent Done Done false 9283 Acked Write 9 Sent Done Done false 9284 Acked Read 9 Sent Done Done false 9285 Acked Read 10 Sent Done Done false 9286 Acked Write 9 Sent Done Done false 9287 Acked Write 9 Sent Done Done false 9288 Acked Read 9 Sent Done Done false 9289 Acked Flush 0 Sent Done Done false 9290 Acked Read 6 Sent Done Done false 9291 Acked Read 3 Sent Done Done false 9292 Acked Flush 0 Sent Done Done false 9293 Acked Read 1 Sent Done Done false 9294 Acked Write 10 Sent Done Done false 9295 Acked Write 4 Sent Done Done false 9296 Acked Read 10 Sent Done Done false 9297 Acked Read 8 Sent Done Done false 9298 Acked Read 4 Sent Done Done false 9299 Acked Read 2 Sent Done Done false 9300 Acked Read 3 Sent Done Done false 9301 Acked Read 3 Sent Done Done false 9302 Acked Read 3 Sent Done Done false 9303 Acked Write 4 Sent Done Done false 9304 Acked Read 9 Sent Done Done false 9305 Acked Read 5 Sent Done Done false 9306 Acked Write 3 Sent Done Done false 9307 Acked Flush 0 Sent Done Done false 9308 Acked Read 1 Sent Done Done false 9309 Acked Write 2 Sent Done Done false 9310 Acked Write 10 Sent Done Done false 9311 Acked Write 2 Sent Done Done false 9312 Acked Read 2 Sent Done Done false 9313 Acked Write 4 Sent Done Done false 9314 Acked Read 1 Sent Done Done false 9315 Acked Read 5 Sent Done Done false 9316 Acked Read 8 Sent Done Done false 9317 Acked Write 7 Sent Done Done false 9318 Acked Write 6 Sent Done Done false 9319 Acked Read 2 Sent Done Done false 9320 Acked Write 8 Sent Done Done false 9321 Acked Write 9 Sent Done Done false 9322 Acked Read 6 Sent Done Done false 9323 Acked Read 6 Sent Done Done false 9324 Acked Read 5 Sent Done Done false 9325 Acked Read 5 Sent Done Done false 9326 Acked Read 4 Sent Done Done false 9327 Acked Write 2 Sent Done Done false 9328 Acked Read 4 Sent Done Done false 9329 Acked Write 1 Sent Done Done false 9330 Acked Read 2 Sent Done Done false 9331 Acked Flush 0 Sent Done Done false 9332 Acked Flush 0 Sent Done Done false 9333 Acked Write 2 Sent Done Done false 9334 Acked Read 2 Sent Done Done false 9335 Acked Write 9 Sent Done Done false 9336 Acked Flush 0 Sent Done Done false 9337 Acked Read 9 Sent Done Done false 9338 Acked Read 1 Sent Done Done false 9339 Acked Read 3 Sent Done Done false 9340 Acked Flush 0 Sent Done Done false 9341 Acked Write 10 Sent Done Done false 9342 Acked Read 8 Sent Done Done false 9343 Acked Write 5 Sent Done Done false 9344 Acked Read 4 Sent Done Done false 9345 Acked Write 7 Sent Done Done false 9346 Acked Write 3 Sent Done Done false 9347 Acked Write 6 Sent Done Done false 9348 Acked Read 10 Sent Done Done false 9349 Acked Write 1 Sent Done Done false 9350 Acked Write 3 Sent Done Done false 9351 Acked Read 4 Sent Done Done false 9352 Acked Write 2 Sent Done Done false 9353 Acked Read 3 Sent Done Done false 9354 Acked Read 4 Sent Done Done false 9355 Acked Read 9 Sent Done Done false 9356 Acked Write 9 Sent Done Done false 9357 Acked Write 8 Sent Done Done false 9358 Acked Write 7 Sent Done Done false 9359 Acked Read 1 Sent Done Done false 9360 Acked Write 9 Sent Done Done false 9361 Acked Write 3 Sent Done Done false 9362 Acked Read 10 Sent Done Done false 9363 Acked Flush 0 Sent Done Done false 9364 Acked Read 3 Sent Done Done false 9365 Acked Write 4 Sent Done Done false 9366 Acked Read 4 Sent Done Done false 9367 Acked Read 10 Sent Done Done false 9368 Acked Write 6 Sent Done Done false 9369 Acked Write 8 Sent Done Done false 9370 Acked Write 8 Sent Done Done false 9371 Acked Write 1 Sent Done Done false 9372 Acked Read 2 Sent Done Done false 9373 Acked Flush 0 Sent Done Done false 9374 Acked Write 10 Sent Done Done false 9375 Acked Write 5 Sent Done Done false 9376 Acked Read 7 Sent Done Done false 9377 Acked Read 4 Sent Done Done false 9378 Acked Write 2 Sent Done Done false 9379 Acked Read 1 Sent Done Done false 9380 Acked Read 1 Sent Done Done false 9381 Acked Read 9 Sent Done Done false 9382 Acked Write 4 Sent Done Done false 9383 Acked Write 9 Sent Done Done false 9384 Acked Read 6 Sent Done Done false 9385 Acked Read 8 Sent Done Done false 9386 Acked Flush 0 Sent Done Done false 9387 Acked Read 1 Sent Done Done false 9388 Acked Read 5 Sent Done Done false 9389 Acked Read 7 Sent Done Done false 9390 Acked Write 1 Sent Done Done false 9391 Acked Read 2 Sent Done Done false 9392 Acked Flush 0 Sent Done Done false 9393 Acked Write 8 Sent Done Done false 9394 Acked Read 8 Sent Done Done false 9395 Acked Read 1 Sent Done Done false 9396 Acked Read 6 Sent Done Done false 9397 Acked Write 4 Sent Done Done false 9398 Acked Write 4 Sent Done Done false 9399 Acked Flush 0 Sent Done Done false 9400 Acked Write 6 Sent Done Done false 9401 Acked Read 3 Sent Done Done false 9402 Acked Read 4 Sent Done Done false 9403 Acked Read 1 Sent Done Done false 9404 Acked Read 10 Sent Done Done false 9405 Acked Read 4 Sent Done Done false 9406 Acked Read 5 Sent Done Done false 9407 Acked Write 2 Sent Done Done false 9408 Acked Flush 0 Sent Done Done false 9409 Acked Write 3 Sent Done Done false 9410 Acked Flush 0 Sent Done Done false 9411 Acked Read 3 Sent Done Done false 9412 Acked Write 3 Sent Done Done false 9413 Acked Read 5 Sent Done Done false 9414 Acked Write 3 Sent Done Done false 9415 Acked Read 5 Sent Done Done false 9416 Acked Read 7 Sent Done Done false 9417 Acked Write 1 Sent Done Done false 9418 Acked Flush 0 Sent Done Done false 9419 Acked Write 4 Sent Done Done false 9420 Acked Write 3 Sent Done Done false 9421 Acked Write 3 Sent Done Done false 9422 Acked Read 6 Sent Done Done false 9423 Acked Write 5 Sent Done Done false 9424 Acked Write 4 Sent Done Done false 9425 Acked Write 2 Sent Done Done false 9426 Acked Read 10 Sent Done Done false 9427 Acked Read 7 Sent Done Done false 9428 Acked Read 5 Sent Done Done false 9429 Acked Read 2 Sent Done Done false 9430 Acked Read 6 Sent Done Done false 9431 Acked Write 4 Sent Done Done false 9432 Acked Read 5 Sent Done Done false 9433 Acked Write 7 Sent Done Done false 9434 Acked Read 4 Sent Done Done false 9435 Acked Write 1 Sent Done Done false 9436 Acked Write 7 Sent Done Done false 9437 Acked Read 9 Sent Done Done false 9438 Acked Read 8 Sent Done Done false 9439 Acked Write 6 Sent Done Done false 9440 Acked Write 3 Sent Done Done false 9441 Acked Flush 0 Sent Done Done false 9442 Acked Write 5 Sent Done Done false 9443 Acked Write 7 Sent Done Done false 9444 Acked Flush 0 Sent Done Done false 9445 Acked Write 7 Sent Done Done false 9446 Acked Write 4 Sent Done Done false 9447 Acked Write 8 Sent Done Done false 9448 Acked Write 7 Sent Done Done false 9449 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 9449 9449 Downstairs last five completed: 9147 9146 9145 9144 9143 Upstairs last five completed: 9449 9448 9447 9446 9445 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:56.21521475Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:56.215280833Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:56.215305406Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:48:56.21532622Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:48:56.215350304Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9148 Acked Write 5 Sent Done Done false 9149 Acked Read 3 Sent Done Done false 9150 Acked Write 4 Sent Done Done false 9151 Acked Write 8 Sent Done Done false 9152 Acked Read 6 Sent Done Done false 9153 Acked Write 1 Sent Done Done false 9154 Acked Write 8 Sent Done Done false 9155 Acked Read 9 Sent Done Done false 9156 Acked Flush 0 Sent Done Done false 9157 Acked Write 2 Sent Done Done false 9158 Acked Write 5 Sent Done Done false 9159 Acked Write 4 Sent Done Done false 9160 Acked Write 9 Sent Done Done false 9161 Acked Read 9 Sent Done Done false 9162 Acked Write 10 Sent Done Done false 9163 Acked Write 4 Sent Done Done false 9164 Acked Read 5 Sent Done Done false 9165 Acked Write 9 Sent Done Done false 9166 Acked Flush 0 Sent Done Done false 9167 Acked Write 6 Sent Done Done false 9168 Acked Read 2 Sent Done Done false 9169 Acked Read 2 Sent Done Done false 9170 Acked Read 8 Sent Done Done false 9171 Acked Write 5 Sent Done Done false 9172 Acked Read 7 Sent Done Done false 9173 Acked Read 5 Sent Done Done false 9174 Acked Write 7 Sent Done Done false 9175 Acked Write 9 Sent Done Done false 9176 Acked Read 5 Sent Done Done false 9177 Acked Read 7 Sent Done Done false 9178 Acked Read 4 Sent Done Done false 9179 Acked Read 4 Sent Done Done false 9180 Acked Read 10 Sent Done Done false 9181 Acked Write 9 Sent Done Done false 9182 Acked Read 7 Sent Done Done false 9183 Acked Flush 0 Sent Done Done false 9184 Acked Write 10 Sent Done Done false 9185 Acked Read 5 Sent Done Done false 9186 Acked Read 8 Sent Done Done false 9187 Acked Write 9 Sent Done Done false 9188 Acked Write 5 Sent Done Done false 9189 Acked Read 5 Sent Done Done false 9190 Acked Read 10 Sent Done Done false 9191 Acked Read 1 Sent Done Done false 9192 Acked Read 2 Sent Done Done false 9193 Acked Write 5 Sent Done Done false 9194 Acked Read 9 Sent Done Done false 9195 Acked Read 7 Sent Done Done false 9196 Acked Read 8 Sent Done Done false 9197 Acked Write 8 Sent Done Done false 9198 Acked Read 6 Sent Done Done false 9199 Acked Read 9 Sent Done Done false 9200 Acked Write 3 Sent Done Done false 9201 Acked Flush 0 Sent Done Done false 9202 Acked Read 7 Sent Done Done false 9203 Acked Write 8 Sent Done Done false 9204 Acked Read 4 Sent Done Done false 9205 Acked Read 1 Sent Done Done false 9206 Acked Read 10 Sent Done Done false 9207 Acked Read 7 Sent Done Done false 9208 Acked Write 5 Sent Done Done false 9209 Acked Read 2 Sent Done Done false 9210 Acked Read 8 Sent Done Done false 9211 Acked Write 10 Sent Done Done false 9212 Acked Write 4 Sent Done Done false 9213 Acked Write 10 Sent Done Done false 9214 Acked Write 10 Sent Done Done false 9215 Acked Read 10 Sent Done Done false 9216 Acked Write 10 Sent Done Done false 9217 Acked Write 10 Sent Done Done false 9218 Acked Write 3 Sent Done Done false 9219 Acked Write 3 Sent Done Done false 9220 Acked Read 7 Sent Done Done false 9221 Acked Read 4 Sent Done Done false 9222 Acked Read 10 Sent Done Done false 9223 Acked Read 8 Sent Done Done false 9224 Acked Read 5 Sent Done Done false 9225 Acked Read 2 Sent Done Done false 9226 Acked Write 5 Sent Done Done false 9227 Acked Read 7 Sent Done Done false 9228 Acked Read 10 Sent Done Done false 9229 Acked Write 1 Sent Done Done false 9230 Acked Read 9 Sent Done Done false 9231 Acked Read 6 Sent Done Done false 9232 Acked Read 10 Sent Done Done false 9233 Acked Write 6 Sent Done Done false 9234 Acked Read 8 Sent Done Done false 9235 Acked Read 2 Sent Done Done false 9236 Acked Write 1 Sent Done Done false 9237 Acked Read 4 Sent Done Done false 9238 Acked Write 3 Sent Done Done false 9239 Acked Read 8 Sent Done Done false 9240 Acked Read 8 Sent Done Done false 9241 Acked Flush 0 Sent Done Done false 9242 Acked Read 7 Sent Done Done false 9243 Acked Read 8 Sent Done Done false 9244 Acked Read 3 Sent Done Done false 9245 Acked Write 1 Sent Done Done false 9246 Acked Read 4 Sent Done Done false 9247 Acked Read 7 Sent Done Done false 9248 Acked Write 5 Sent Done Done false 9249 Acked Flush 0 Sent Done Done false 9250 Acked Write 10 Sent Done Done false 9251 Acked Flush 0 Sent Done Done false 9252 Acked Write 7 Sent Done Done false 9253 Acked Read 6 Sent Done Done false 9254 Acked Read 3 Sent Done Done false 9255 Acked Read 3 Sent Done Done false 9256 Acked Read 3 Sent Done Done false 9257 Acked Read 2 Sent Done Done false 9258 Acked Read 10 Sent Done Done false 9259 Acked Flush 0 Sent Done Done false 9260 Acked Read 7 Sent Done Done false 9261 Acked Write 3 Sent Done Done false 9262 Acked Write 6 Sent Done Done false 9263 Acked Write 9 Sent Done Done false 9264 Acked Read 8 Sent Done Done false 9265 Acked Write 7 Sent Done Done false 9266 Acked Write 9 Sent Done Done false 9267 Acked Write 2 Sent Done Done false 9268 Acked Read 10 Sent Done Done false 9269 Acked Read 2 Sent Done Done false 9270 Acked Read 6 Sent Done Done false 9271 Acked Write 4 Sent Done Done false 9272 Acked Read 4 Sent Done Done false 9273 Acked Flush 0 Sent Done Done false 9274 Acked Write 2 Sent Done Done false 9275 Acked Read 2 Sent Done Done false 9276 Acked Read 2 Sent Done Done false 9277 Acked Write 2 Sent Done Done false 9278 Acked Write 10 Sent Done Done false 9279 Acked Read 5 Sent Done Done false 9280 Acked Read 1 Sent Done Done false 9281 Acked Read 7 Sent Done Done false 9282 Acked Write 10 Sent Done Done false 9283 Acked Write 9 Sent Done Done false 9284 Acked Read 9 Sent Done Done false 9285 Acked Read 10 Sent Done Done false 9286 Acked Write 9 Sent Done Done false 9287 Acked Write 9 Sent Done Done false 9288 Acked Read 9 Sent Done Done false 9289 Acked Flush 0 Sent Done Done false 9290 Acked Read 6 Sent Done Done false 9291 Acked Read 3 Sent Done Done false 9292 Acked Flush 0 Sent Done Done false 9293 Acked Read 1 Sent Done Done false 9294 Acked Write 10 Sent Done Done false 9295 Acked Write 4 Sent Done Done false 9296 Acked Read 10 Sent Done Done false 9297 Acked Read 8 Sent Done Done false 9298 Acked Read 4 Sent Done Done false 9299 Acked Read 2 Sent Done Done false 9300 Acked Read 3 Sent Done Done false 9301 Acked Read 3 Sent Done Done false 9302 Acked Read 3 Sent Done Done false 9303 Acked Write 4 Sent Done Done false 9304 Acked Read 9 Sent Done Done false 9305 Acked Read 5 Sent Done Done false 9306 Acked Write 3 Sent Done Done false 9307 Acked Flush 0 Sent Done Done false 9308 Acked Read 1 Sent Done Done false 9309 Acked Write 2 Sent Done Done false 9310 Acked Write 10 Sent Done Done false 9311 Acked Write 2 Sent Done Done false 9312 Acked Read 2 Sent Done Done false 9313 Acked Write 4 Sent Done Done false 9314 Acked Read 1 Sent Done Done false 9315 Acked Read 5 Sent Done Done false 9316 Acked Read 8 Sent Done Done false 9317 Acked Write 7 Sent Done Done false 9318 Acked Write 6 Sent Done Done false 9319 Acked Read 2 Sent Done Done false 9320 Acked Write 8 Sent Done Done false 9321 Acked Write 9 Sent Done Done false 9322 Acked Read 6 Sent Done Done false 9323 Acked Read 6 Sent Done Done false 9324 Acked Read 5 Sent Done Done false 9325 Acked Read 5 Sent Done Done false 9326 Acked Read 4 Sent Done Done false 9327 Acked Write 2 Sent Done Done false 9328 Acked Read 4 Sent Done Done false 9329 Acked Write 1 Sent Done Done false 9330 Acked Read 2 Sent Done Done false 9331 Acked Flush 0 Sent Done Done false 9332 Acked Flush 0 Sent Done Done false 9333 Acked Write 2 Sent Done Done false 9334 Acked Read 2 Sent Done Done false 9335 Acked Write 9 Sent Done Done false 9336 Acked Flush 0 Sent Done Done false 9337 Acked Read 9 Sent Done Done false 9338 Acked Read 1 Sent Done Done false 9339 Acked Read 3 Sent Done Done false 9340 Acked Flush 0 Sent Done Done false 9341 Acked Write 10 Sent Done Done false 9342 Acked Read 8 Sent Done Done false 9343 Acked Write 5 Sent Done Done false 9344 Acked Read 4 Sent Done Done false 9345 Acked Write 7 Sent Done Done false 9346 Acked Write 3 Sent Done Done false 9347 Acked Write 6 Sent Done Done false 9348 Acked Read 10 Sent Done Done false 9349 Acked Write 1 Sent Done Done false 9350 Acked Write 3 Sent Done Done false 9351 Acked Read 4 Sent Done Done false 9352 Acked Write 2 Sent Done Done false 9353 Acked Read 3 Sent Done Done false 9354 Acked Read 4 Sent Done Done false 9355 Acked Read 9 Sent Done Done false 9356 Acked Write 9 Sent Done Done false 9357 Acked Write 8 Sent Done Done false 9358 Acked Write 7 Sent Done Done false 9359 Acked Read 1 Sent Done Done false 9360 Acked Write 9 Sent Done Done false 9361 Acked Write 3 Sent Done Done false 9362 Acked Read 10 Sent Done Done false 9363 Acked Flush 0 Sent Done Done false 9364 Acked Read 3 Sent Done Done false 9365 Acked Write 4 Sent Done Done false 9366 Acked Read 4 Sent Done Done false 9367 Acked Read 10 Sent Done Done false 9368 Acked Write 6 Sent Done Done false 9369 Acked Write 8 Sent Done Done false 9370 Acked Write 8 Sent Done Done false 9371 Acked Write 1 Sent Done Done false 9372 Acked Read 2 Sent Done Done false 9373 Acked Flush 0 Sent Done Done false 9374 Acked Write 10 Sent Done Done false 9375 Acked Write 5 Sent Done Done false 9376 Acked Read 7 Sent Done Done false 9377 Acked Read 4 Sent Done Done false 9378 Acked Write 2 Sent Done Done false 9379 Acked Read 1 Sent Done Done false 9380 Acked Read 1 Sent Done Done false 9381 Acked Read 9 Sent Done Done false 9382 Acked Write 4 Sent Done Done false 9383 Acked Write 9 Sent Done Done false 9384 Acked Read 6 Sent Done Done false 9385 Acked Read 8 Sent Done Done false 9386 Acked Flush 0 Sent Done Done false 9387 Acked Read 1 Sent Done Done false 9388 Acked Read 5 Sent Done Done false 9389 Acked Read 7 Sent Done Done false 9390 Acked Write 1 Sent Done Done false 9391 Acked Read 2 Sent Done Done false 9392 Acked Flush 0 Sent Done Done false 9393 Acked Write 8 Sent Done Done false 9394 Acked Read 8 Sent Done Done false 9395 Acked Read 1 Sent Done Done false 9396 Acked Read 6 Sent Done Done false 9397 Acked Write 4 Sent Done Done false 9398 Acked Write 4 Sent Done Done false 9399 Acked Flush 0 Sent Done Done false 9400 Acked Write 6 Sent Done Done false 9401 Acked Read 3 Sent Done Done false 9402 Acked Read 4 Sent Done Done false 9403 Acked Read 1 Sent Done Done false 9404 Acked Read 10 Sent Done Done false 9405 Acked Read 4 Sent Done Done false 9406 Acked Read 5 Sent Done Done false 9407 Acked Write 2 Sent Done Done false 9408 Acked Flush 0 Sent Done Done false 9409 Acked Write 3 Sent Done Done false 9410 Acked Flush 0 Sent Done Done false 9411 Acked Read 3 Sent Done Done false 9412 Acked Write 3 Sent Done Done false 9413 Acked Read 5 Sent Done Done false 9414 Acked Write 3 Sent Done Done false 9415 Acked Read 5 Sent Done Done false 9416 Acked Read 7 Sent Done Done false 9417 Acked Write 1 Sent Done Done false 9418 Acked Flush 0 Sent Done Done false 9419 Acked Write 4 Sent Done Done false 9420 Acked Write 3 Sent Done Done false 9421 Acked Write 3 Sent Done Done false 9422 Acked Read 6 Sent Done Done false 9423 Acked Write 5 Sent Done Done false 9424 Acked Write 4 Sent Done Done false 9425 Acked Write 2 Sent Done Done false 9426 Acked Read 10 Sent Done Done false 9427 Acked Read 7 Sent Done Done false 9428 Acked Read 5 Sent Done Done false 9429 Acked Read 2 Sent Done Done false 9430 Acked Read 6 Sent Done Done false 9431 Acked Write 4 Sent Done Done false 9432 Acked Read 5 Sent Done Done false 9433 Acked Write 7 Sent Done Done false 9434 Acked Read 4 Sent Done Done false 9435 Acked Write 1 Sent Done Done false 9436 Acked Write 7 Sent Done Done false 9437 Acked Read 9 Sent Done Done false 9438 Acked Read 8 Sent Done Done false 9439 Acked Write 6 Sent Done Done false 9440 Acked Write 3 Sent Done Done false 9441 Acked Flush 0 Sent Done Done false 9442 Acked Write 5 Sent Done Done false 9443 Acked Write 7 Sent Done Done false 9444 Acked Flush 0 Sent Done Done false 9445 Acked Write 7 Sent Done Done false 9446 Acked Write 4 Sent Done Done false 9447 Acked Write 8 Sent Done Done false 9448 Acked Write 7 Sent Done Done false 9449 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 9449 9449 Downstairs last five completed: 9147 9146 9145 9144 9143 Upstairs last five completed: 9449 9448 9447 9446 9445 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9148 Acked Write 5 Sent Done Done false 9149 Acked Read 3 Sent Done Done false 9150 Acked Write 4 Sent Done Done false 9151 Acked Write 8 Sent Done Done false 9152 Acked Read 6 Sent Done Done false 9153 Acked Write 1 Sent Done Done false 9154 Acked Write 8 Sent Done Done false 9155 Acked Read 9 Sent Done Done false 9156 Acked Flush 0 Sent Done Done false 9157 Acked Write 2 Sent Done Done false 9158 Acked Write 5 Sent Done Done false 9159 Acked Write 4 Sent Done Done false 9160 Acked Write 9 Sent Done Done false 9161 Acked Read 9 Sent Done Done false 9162 Acked Write 10 Sent Done Done false 9163 Acked Write 4 Sent Done Done false 9164 Acked Read 5 Sent Done Done false 9165 Acked Write 9 Sent Done Done false 9166 Acked Flush 0 Sent Done Done false 9167 Acked Write 6 Sent Done Done false 9168 Acked Read 2 Sent Done Done false 9169 Acked Read 2 Sent Done Done false 9170 Acked Read 8 Sent Done Done false 9171 Acked Write 5 Sent Done Done false 9172 Acked Read 7 Sent Done Done false 9173 Acked Read 5 Sent Done Done false 9174 Acked Write 7 Sent Done Done false 9175 Acked Write 9 Sent Done Done false 9176 Acked Read 5 Sent Done Done false 9177 Acked Read 7 Sent Done Done false 9178 Acked Read 4 Sent Done Done false 9179 Acked Read 4 Sent Done Done false 9180 Acked Read 10 Sent Done Done false 9181 Acked Write 9 Sent Done Done false 9182 Acked Read 7 Sent Done Done false 9183 Acked Flush 0 Sent Done Done false 9184 Acked Write 10 Sent Done Done false 9185 Acked Read 5 Sent Done Done false 9186 Acked Read 8 Sent Done Done false 9187 Acked Write 9 Sent Done Done false 9188 Acked Write 5 Sent Done Done false 9189 Acked Read 5 Sent Done Done false 9190 Acked Read 10 Sent Done Done false 9191 Acked Read 1 Sent Done Done false 9192 Acked Read 2 Sent Done Done false 9193 Acked Write 5 Sent Done Done false 9194 Acked Read 9 Sent Done Done false 9195 Acked Read 7 Sent Done Done false 9196 Acked Read 8 Sent Done Done false 9197 Acked Write 8 Sent Done Done false 9198 Acked Read 6 Sent Done Done false 9199 Acked Read 9 Sent Done Done false 9200 Acked Write 3 Sent Done Done false 9201 Acked Flush 0 Sent Done Done false 9202 Acked Read 7 Sent Done Done false 9203 Acked Write 8 Sent Done Done false 9204 Acked Read 4 Sent Done Done false 9205 Acked Read 1 Sent Done Done false 9206 Acked Read 10 Sent Done Done false 9207 Acked Read 7 Sent Done Done false 9208 Acked Write 5 Sent Done Done false 9209 Acked Read 2 Sent Done Done false 9210 Acked Read 8 Sent Done Done false 9211 Acked Write 10 Sent Done Done false 9212 Acked Write 4 Sent Done Done false 9213 Acked Write 10 Sent Done Done false 9214 Acked Write 10 Sent Done Done false 9215 Acked Read 10 Sent Done Done false 9216 Acked Write 10 Sent Done Done false 9217 Acked Write 10 Sent Done Done false 9218 Acked Write 3 Sent Done Done false 9219 Acked Write 3 Sent Done Done false 9220 Acked Read 7 Sent Done Done false 9221 Acked Read 4 Sent Done Done false 9222 Acked Read 10 Sent Done Done false 9223 Acked Read 8 Sent Done Done false 9224 Acked Read 5 Sent Done Done false 9225 Acked Read 2 Sent Done Done false 9226 Acked Write 5 Sent Done Done false 9227 Acked Read 7 Sent Done Done false 9228 Acked Read 10 Sent Done Done false 9229 Acked Write 1 Sent Done Done false 9230 Acked Read 9 Sent Done Done false 9231 Acked Read 6 Sent Done Done false 9232 Acked Read 10 Sent Done Done false 9233 Acked Write 6 Sent Done Done false 9234 Acked Read 8 Sent Done Done false 9235 Acked Read 2 Sent Done Done false 9236 Acked Write 1 Sent Done Done false 9237 Acked Read 4 Sent Done Done false 9238 Acked Write 3 Sent Done Done false 9239 Acked Read 8 Sent Done Done false 9240 Acked Read 8 Sent Done Done false 9241 Acked Flush 0 Sent Done Done false 9242 Acked Read 7 Sent Done Done false 9243 Acked Read 8 Sent Done Done false 9244 Acked Read 3 Sent Done Done false 9245 Acked Write 1 Sent Done Done false 9246 Acked Read 4 Sent Done Done false 9247 Acked Read 7 Sent Done Done false 9248 Acked Write 5 Sent Done Done false 9249 Acked Flush 0 Sent Done Done false 9250 Acked Write 10 Sent Done Done false 9251 Acked Flush 0 Sent Done Done false 9252 Acked Write 7 Sent Done Done false 9253 Acked Read 6 Sent Done Done false 9254 Acked Read 3 Sent Done Done false 9255 Acked Read 3 Sent Done Done false 9256 Acked Read 3 Sent Done Done false 9257 Acked Read 2 Sent Done Done false 9258 Acked Read 10 Sent Done Done false 9259 Acked Flush 0 Sent Done Done false 9260 Acked Read 7 Sent Done Done false 9261 Acked Write 3 Sent Done Done false 9262 Acked Write 6 Sent Done Done false 9263 Acked Write 9 Sent Done Done false 9264 Acked Read 8 Sent Done Done false 9265 Acked Write 7 Sent Done Done false 9266 Acked Write 9 Sent Done Done false 9267 Acked Write 2 Sent Done Done false 9268 Acked Read 10 Sent Done Done false 9269 Acked Read 2 Sent Done Done false 9270 Acked Read 6 Sent Done Done false 9271 Acked Write 4 Sent Done Done false 9272 Acked Read 4 Sent Done Done false 9273 Acked Flush 0 Sent Done Done false 9274 Acked Write 2 Sent Done Done false 9275 Acked Read 2 Sent Done Done false 9276 Acked Read 2 Sent Done Done false 9277 Acked Write 2 Sent Done Done false 9278 Acked Write 10 Sent Done Done false 9279 Acked Read 5 Sent Done Done false 9280 Acked Read 1 Sent Done Done false 9281 Acked Read 7 Sent Done Done false 9282 Acked Write 10 Sent Done Done false 9283 Acked Write 9 Sent Done Done false 9284 Acked Read 9 Sent Done Done false 9285 Acked Read 10 Sent Done Done false 9286 Acked Write 9 Sent Done Done false 9287 Acked Write 9 Sent Done Done false 9288 Acked Read 9 Sent Done Done false 9289 Acked Flush 0 Sent Done Done false 9290 Acked Read 6 Sent Done Done false 9291 Acked Read 3 Sent Done Done false 9292 Acked Flush 0 Sent Done Done false 9293 Acked Read 1 Sent Done Done false 9294 Acked Write 10 Sent Done Done false 9295 Acked Write 4 Sent Done Done false 9296 Acked Read 10 Sent Done Done false 9297 Acked Read 8 Sent Done Done false 9298 Acked Read 4 Sent Done Done false 9299 Acked Read 2 Sent Done Done false 9300 Acked Read 3 Sent Done Done false 9301 Acked Read 3 Sent Done Done false 9302 Acked Read 3 Sent Done Done false 9303 Acked Write 4 Sent Done Done false 9304 Acked Read 9 Sent Done Done false 9305 Acked Read 5 Sent Done Done false 9306 Acked Write 3 Sent Done Done false 9307 Acked Flush 0 Sent Done Done false 9308 Acked Read 1 Sent Done Done false 9309 Acked Write 2 Sent Done Done false 9310 Acked Write 10 Sent Done Done false 9311 Acked Write 2 Sent Done Done false 9312 Acked Read 2 Sent Done Done false 9313 Acked Write 4 Sent Done Done false 9314 Acked Read 1 Sent Done Done false 9315 Acked Read 5 Sent Done Done false 9316 Acked Read 8 Sent Done Done false 9317 Acked Write 7 Sent Done Done false 9318 Acked Write 6 Sent Done Done false 9319 Acked Read 2 Sent Done Done false 9320 Acked Write 8 Sent Done Done false 9321 Acked Write 9 Sent Done Done false 9322 Acked Read 6 Sent Done Done false 9323 Acked Read 6 Sent Done Done false 9324 Acked Read 5 Sent Done Done false 9325 Acked Read 5 Sent Done Done false 9326 Acked Read 4 Sent Done Done false 9327 Acked Write 2 Sent Done Done false 9328 Acked Read 4 Sent Done Done false 9329 Acked Write 1 Sent Done Done false 9330 Acked Read 2 Sent Done Done false 9331 Acked Flush 0 Sent Done Done false 9332 Acked Flush 0 Sent Done Done false 9333 Acked Write 2 Sent Done Done false 9334 Acked Read 2 Sent Done Done false 9335 Acked Write 9 Sent Done Done false 9336 Acked Flush 0 Sent Done Done false 9337 Acked Read 9 Sent Done Done false 9338 Acked Read 1 Sent Done Done false 9339 Acked Read 3 Sent Done Done false 9340 Acked Flush 0 Sent Done Done false 9341 Acked Write 10 Sent Done Done false 9342 Acked Read 8 Sent Done Done false 9343 Acked Write 5 Sent Done Done false 9344 Acked Read 4 Sent Done Done false 9345 Acked Write 7 Sent Done Done false 9346 Acked Write 3 Sent Done Done false 9347 Acked Write 6 Sent Done Done false 9348 Acked Read 10 Sent Done Done false 9349 Acked Write 1 Sent Done Done false 9350 Acked Write 3 Sent Done Done false 9351 Acked Read 4 Sent Done Done false 9352 Acked Write 2 Sent Done Done false 9353 Acked Read 3 Sent Done Done false 9354 Acked Read 4 Sent Done Done false 9355 Acked Read 9 Sent Done Done false 9356 Acked Write 9 Sent Done Done false 9357 Acked Write 8 Sent Done Done false 9358 Acked Write 7 Sent Done Done false 9359 Acked Read 1 Sent Done Done false 9360 Acked Write 9 Sent Done Done false 9361 Acked Write 3 Sent Done Done false 9362 Acked Read 10 Sent Done Done false 9363 Acked Flush 0 Sent Done Done false 9364 Acked Read 3 Sent Done Done false 9365 Acked Write 4 Sent Done Done false 9366 Acked Read 4 Sent Done Done false 9367 Acked Read 10 Sent Done Done false 9368 Acked Write 6 Sent Done Done false 9369 Acked Write 8 Sent Done Done false 9370 Acked Write 8 Sent Done Done false 9371 Acked Write 1 Sent Done Done false 9372 Acked Read 2 Sent Done Done false 9373 Acked Flush 0 Sent Done Done false 9374 Acked Write 10 Sent Done Done false 9375 Acked Write 5 Sent Done Done false 9376 Acked Read 7 Sent Done Done false 9377 Acked Read 4 Sent Done Done false 9378 Acked Write 2 Sent Done Done false 9379 Acked Read 1 Sent Done Done false 9380 Acked Read 1 Sent Done Done false 9381 Acked Read 9 Sent Done Done false 9382 Acked Write 4 Sent Done Done false 9383 Acked Write 9 Sent Done Done false 9384 Acked Read 6 Sent Done Done false 9385 Acked Read 8 Sent Done Done false 9386 Acked Flush 0 Sent Done Done false 9387 Acked Read 1 Sent Done Done false 9388 Acked Read 5 Sent Done Done false 9389 Acked Read 7 Sent Done Done false 9390 Acked Write 1 Sent Done Done false 9391 Acked Read 2 Sent Done Done false 9392 Acked Flush 0 Sent Done Done false 9393 Acked Write 8 Sent Done Done false 9394 Acked Read 8 Sent Done Done false 9395 Acked Read 1 Sent Done Done false 9396 Acked Read 6 Sent Done Done false 9397 Acked Write 4 Sent Done Done false 9398 Acked Write 4 Sent Done Done false 9399 Acked Flush 0 Sent Done Done false 9400 Acked Write 6 Sent Done Done false 9401 Acked Read 3 Sent Done Done false 9402 Acked Read 4 Sent Done Done false 9403 Acked Read 1 Sent Done Done false 9404 Acked Read 10 Sent Done Done false 9405 Acked Read 4 Sent Done Done false 9406 Acked Read 5 Sent Done Done false 9407 Acked Write 2 Sent Done Done false 9408 Acked Flush 0 Sent Done Done false 9409 Acked Write 3 Sent Done Done false 9410 Acked Flush 0 Sent Done Done false 9411 Acked Read 3 Sent Done Done false 9412 Acked Write 3 Sent Done Done false 9413 Acked Read 5 Sent Done Done false 9414 Acked Write 3 Sent Done Done false 9415 Acked Read 5 Sent Done Done false 9416 Acked Read 7 Sent Done Done false 9417 Acked Write 1 Sent Done Done false 9418 Acked Flush 0 Sent Done Done false 9419 Acked Write 4 Sent Done Done false 9420 Acked Write 3 Sent Done Done false 9421 Acked Write 3 Sent Done Done false 9422 Acked Read 6 Sent Done Done false 9423 Acked Write 5 Sent Done Done false 9424 Acked Write 4 Sent Done Done false 9425 Acked Write 2 Sent Done Done false 9426 Acked Read 10 Sent Done Done false 9427 Acked Read 7 Sent Done Done false 9428 Acked Read 5 Sent Done Done false 9429 Acked Read 2 Sent Done Done false 9430 Acked Read 6 Sent Done Done false 9431 Acked Write 4 Sent Done Done false 9432 Acked Read 5 Sent Done Done false 9433 Acked Write 7 Sent Done Done false 9434 Acked Read 4 Sent Done Done false 9435 Acked Write 1 Sent Done Done false 9436 Acked Write 7 Sent Done Done false 9437 Acked Read 9 Sent Done Done false 9438 Acked Read 8 Sent Done Done false 9439 Acked Write 6 Sent Done Done false 9440 Acked Write 3 Sent Done Done false 9441 Acked Flush 0 Sent Done Done false 9442 Acked Write 5 Sent Done Done false 9443 Acked Write 7 Sent Done Done false 9444 Acked Flush 0 Sent Done Done false 9445 Acked Write 7 Sent Done Done false 9446 Acked Write 4 Sent Done Done false 9447 Acked Write 8 Sent Done Done false 9448 Acked Write 7 Sent Done Done false 9449 Acked Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 302 0 0 302 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9147 9449 9449 Downstairs last five completed: 9147 9146 9145 9144 9143 Upstairs last five completed: 9449 9448 9447 9446 9445 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.216456447Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:48647)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.216523869Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:06.217393625Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.217910116Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.21793499Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 9147","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.217950595Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 9147","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.218172366Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.218196309Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.218215175Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"0","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] client re-new 302 jobs since flush 9147","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.218229371Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[0] Replayed 302 jobs since flush: 9147","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.218444753Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9449 9448 9447 9446 9445 Upstairs last five completed: 9449 9448 9447 9446 9445 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 Nov 27 01:49:06.458 INFO request completed[8820] Got stop action so:false kr:false , latency_us: 314, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 68696f2a-ce40-4428-92dd-bd2c3e8c623b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Nov 27 01:49:06.459 INFO accepted connection, remote_addr: 127.0.0.1:61022, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:06.45923936Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:06.459333245Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:06.459354559Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.459367745Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:49:06.459{"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.459382461Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/1, method: GET, req_id: b284a6dd-7f47-4b57-a90e-d2b7857714eb, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:06.459441276Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:49:10.461 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 250090df-8776-4dc1-aa24-cef76c4395ba, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Write block 2077 len 24576 data: 6 8 9 8 7 5 002/300 Write block 4441 len 20480 data: 8 8 10 9 8 003/300 Read block 3975 len 36864 004/300 Read block 1636 len 16384 005/300 Write block 1905 len 40960 data: 6 7 8 5 6 9 7 7 7 9 006/300 Write block 668 len 12288 data: 6 6 4 007/300 Write block 4971 len 24576 data: 4 3 4 4 4 4 008/300 Write block 4448 len 16384 data: 4 4 4 5 009/300 Read block 1460 len 28672 010/300 Read block 4919 len 8192 011/300 Write block 3923 len 8192 data: 4 4 012/300 Read block 1942 len 40960 013/300 Write block 3070 len 16384 data: 3 3 4 3 014/300 Write block 1919 len 24576 data: 3 2 2 3 3 4 015/300 Read block 2554 len 4096 016/300 Write block 3343 len 24576 data: 4 4 4 4 5 5 017/300 Write block 4986 len 40960 data: 6 6 7 6 5 4 5 5 6 5 018/300 Write block 2257 len 4096 data: 8 019/300 Flush 020/300 Write block 1493 len 40960 data: 6 7 8 8 9 7 6 6 6 4 021/300 Write block 3512 len 40960 data: 6 5 6 7 7 7 7 5 7 7 022/300 Read block 3532 len 4096 023/300 Write block 2451 len 24576 data: 5 5 5 6 7 8 024/300 Read block 54 len 32768 025/300 Flush 026/300 Write block 1705 len 28672 data: 5 4 4 5 6 5 4 027/300 Read block 3507 len 32768 028/300 Read block 3841 len 40960 029/300 Write block 2534 len 24576 data: 4 3 3 5 5 6 030/300 Write block 49 len 24576 data: 5 5 6 7 7 6 031/300 Write block 3838 len 36864 data: 6 5 4 4 4 4 4 4 5 032/300 Flush 033/300 Read block 804 len 40960 034/300 Read block 4425 len 32768 035/300 Write block 574 len 24576 data: 6 6 6 7 5 10 036/300 Read block 2274 len 8192 037/300 Read block 4005 len 36864 038/300 Write block 1509 len 4096 data: 9 039/300 Write block 774 len 24576 data: 8 7 5 6 5 5 040/300 Read block 4597 len 12288 041/300 Write block 512 len 36864 data: 9 6 6 6 7 6 6 5 6 042/300 Write block 4873 len 12288 data: 3 4 3 043/300 Flush 044/300 Flush 045/300 Write block 2420 len 20480 data: 3 2 2 3 4 046/300 Write block 332 len 36864 data: 3 6 6 6 5 3 3 3 5 047/300 Flush 048/300 Write block 1585 len 16384 data: 9 10 12 12 049/300 Flush 050/300 Write block 1325 len 24576 data: 3 3 4 3 3 3 051/300 Write block 1579 len 20480 data: 6 6 5 6 7 052/300 Flush 053/300 Read block 1586 len 4096 054/300 Read block 157 len 4096 055/300 Write block 3490 len 12288 data: 9 9 7 056/300 Read block 2022 len 16384 057/300 Read block 254 len 32768 058/300 Write block 928 len 24576 data: 4 4 5 7 6 6 059/300 Write block 4251 len 4096 data: 6 060/300 Read block 4828 len 16384 061/300 Write block 2554 len 8192 data: 5 6 062/300 Write block 469 len 32768 data: 10 11 9 8 7 4 5 6 063/300 Read block 3075 len 32768 064/300 Read block 4645 len 8192 065/300 Write block 646 len 24576 data: 3 3 4 4 5 6 066/300 Write block 478 len 32768 data: 5 3 4 4 4 4 5 4 067/300 Write block 4813 len 4096 data: 2 068/300 Read block 4569 len 28672 069/300 Write block 3508 len 16384 data: 4 4 4 6 070/300 Read block 4648 len 28672 071/300 Read block 4832 len 20480 072/300 Write block 1242 len 40960 data: 4 5 3 5 7 8 9 7 7 7 073/300 Read block 623 len 40960 074/300 Write block 3997 len 12288 data: 5 4 4 075/300 Write block 3325 len 28672 data: 3 4 6 7 8 8 8 076/300 Write block 2363 len 16384 data: 5 6 6 6 077/300 Write block 55 len 20480 data: 4 3 4 4 4 078/300 Read block 718 len 12288 079/300 Read block 4180 len 8192 080/300 Read block 2749 len 12288 081/300 Read block 224 len 8192 082/300 Read block 3638 len 20480 083/300 Write block 2868 len 40960 data: 6 6 6 7 4 3 6 6 8 8 084/300 Flush 085/300 Read block 4490 len 4096 086/300 Write block 1188 len 20480 data: 5 5 7 7 7 087/300 Write block 4433 len 40960 data: 8 9 8 7 6 6 6 6 9 9 088/300 Read block 2495 len 12288 089/300 Write block 1250 len 12288 data: 8 8 7 090/300 Read block 122 len 12288 091/300 Read block 3162 len 36864 092/300 Read block 3149 len 12288 093/300 Write block 2706 len 16384 data: 4 3 4 4 094/300 Write block 217 len 28672 data: 2 3 2 6 6 4 5 095/300 Write block 2530 len 20480 data: 5 3 3 4 5 096/300 Write block 1096 len 24576 data: 6 7 7 6 5 4 097/300 Read block 617 len 36864 098/300 Read block 3393 len 24576 099/300 Write block 1171 len 40960 data: 11 10 8 6 6 8 10 10 11 9 100/300 Read block 850 len 36864 101/300 Read block 805 len 4096 102/300 Flush 103/300 Read block 384 len 28672 104/300 Read block 1632 len 24576 105/300 Write block 2611 len 40960 data: 6 5 5 4 5 5 5 5 4 5 106/300 Read block 3444 len 12288 107/300 Write block 1376 len 24576 data: 7 6 6 4 5 4 108/300 Flush 109/300 Write block 335 len 40960 data: 7 6 4 4 4 6 7 7 7 6 110/300 Read block 3722 len 40960 111/300 Write block 593 len 20480 data: 6 5 5 6 6 112/300 Write block 3992 len 20480 data: 6 7 7 7 5 113/300 Write block 4253 len 4096 data: 6 114/300 Write block 3432 len 28672 data: 4 3 3 4 6 7 7 115/300 Write block 3785 len 20480 data: 6 6 5 6 6 116/300 Write block 2443 len 20480 data: 8 7 8 8 6 117/300 Write block 4736 len 4096 data: 5 118/300 Flush 119/300 Flush 120/300 Write block 1000 len 36864 data: 4 5 4 7 7 6 6 6 6 121/300 Flush 122/300 Read block 3093 len 4096 123/300 Read block 2675 len 4096 124/300 Flush 125/300 Read block 2533 len 32768 126/300 Write block 1679 len 32768 data: 4 5 6 6 6 5 6 8 127/300 Read block 3611 len 40960 128/300 Read block 2084 len 8192 129/300 Read block 2259 len 4096 130/300 Read block 3624 len 24576 131/300 Flush 132/300 Read block 1329 len 20480 133/300 Read block 4527 len 40960 134/300 Read block 1147 len 16384 135/300 Write block 459 len 24576 data: 3 3 3 4 5 6 136/300 Write block 561 len 24576 data: 5 5 6 7 6 5 137/300 Write block 1559 len 12288 data: 7 7 7 138/300 Write block 4608 len 4096 data: 5 139/300 Read block 2582 len 28672 140/300 Read block 4249 len 28672 141/300 Write block 3393 len 40960 data: 5 5 5 7 6 5 5 7 6 7 142/300 Write block 1153 len 28672 data: 5 6 6 5 7 7 7 143/300 Write block 1444 len 20480 data: 5 6 6 6 7 144/300 Read block 2947 len 32768 145/300 Write block 2576 len 40960 data: 6 7 5 5 4 4 4 4 4 4 146/300 Write block 1966 len 36864 data: 6 6 6 5 6 7 4 6 7 147/300 Read block 4718 len 12288 148/300 Read block 2966 len 28672 149/300 Write block 2132 len 32768 data: 5 4 5 4 5 6 7 7 150/300 Flush 151/300 Read block 2481 len 24576 152/300 Read block 2885 len 4096 153/300 Read block 3558 len 4096 154/300 Read block 3505 len 40960 155/300 Read block 3927 len 40960 156/300 Write block 1073 len 36864 data: 3 6 6 6 4 5 6 5 5 157/300 Write block 1810 len 28672 data: 6 6 6 6 6 9 12 158/300 Read block 4179 len 36864 159/300 Write block 3175 len 12288 data: 4 5 5 160/300 Write block 2126 len 28672 data: 5 4 4 4 4 4 6 161/300 Flush 162/300 Write block 4324 len 12288 data: 6 6 5 163/300 Read block 4536 len 32768 164/300 Read block 4600 len 40960 165/300 Read block 3791 len 20480 166/300 Read block 4352 len 32768 167/300 Read block 67 len 16384 168/300 Write block 1278 len 36864 data: 4 7 6 6 6 6 7 6 4 169/300 Write block 1212 len 12288 data: 8 8 5 170/300 Write block 3973 len 20480 data: 3 3 4 3 3 171/300 Write block 273 len 12288 data: 4 4 5 172/300 Write block 307 len 40960 data: 5 6 7 8 10 8 9 8 8 8 173/300 Write block 4098 len 32768 data: 7 5 6 6 5 5 4 4 174/300 Flush 175/300 Read block 3990 len 20480 176/300 Read block 1728 len 32768 177/300 Write block 3981 len 12288 data: 6 7 6 178/300 Write block 4561 len 20480 data: 8 7 9 10 10 179/300 Write block 1563 len 36864 data: 6 5 4 3 3 3 4 4 5 180/300 Flush 181/300 Write block 1995 len 40960 data: 7 9 10 12 10 9 9 7 9 8 182/300 Read block 4896 len 36864 183/300 Write block 2663 len 24576 data: 3 4 5 4 4 3 184/300 Write block 4261 len 12288 data: 5 5 6 185/300 Read block 312 len 36864 186/300 Read block 3316 len 8192 187/300 Read block 1402 len 8192 188/300 Write block 758 len 8192 data: 4 4 189/300 Read block 2566 len 36864 190/300 Write block 915 len 4096 data: 4 191/300 Write block 3578 len 28672 data: 6 6 5 6 5 4 5 192/300 Read block 4162 len 20480 193/300 Write block 1121 len 28672 data: 5 6 7 6 5 6 6 194/300 Read block 4351 len 32768 195/300 Write block 885 len 4096 data: 5 196/300 Read block 2897 len 40960 197/300 Read block 4359 len 36864 198/300 Read block 1454 len 40960 199/300 Read block 2238 len 20480 200/300 Read block 1949 len 24576 201/300 Read block 2594 len 24576 202/300 Write block 2573 len 36864 data: 7 6 6 7 8 6 6 5 5 203/300 Read block 842 len 32768 204/300 Write block 4373 len 36864 data: 4 5 5 4 5 5 3 4 5 205/300 Write block 2350 len 8192 data: 9 9 206/300 Read block 3054 len 4096 207/300 Read block 111 len 8192 208/300 Read block 4288 len 8192 209/300 Read block 5 len 8192 210/300 Read block 3884 len 40960 211/300 Read block 4868 len 20480 212/300 Read block 3877 len 4096 213/300 Flush 214/300 Read block 3687 len 36864 215/300 Read block 2257 len 24576 216/300 Read block 4809 len 36864 217/300 Read block 1939 len 32768 218/300 Write block 280 len 24576 data: 5 5 4 4 3 3 219/300 Read block 2042 len 4096 220/300 Read block 4960 len 8192 221/300 Flush 222/300 Flush 223/300 Read block 3569 len 12288 224/300 Write block 2578 len 16384 data: 7 7 6 6 225/300 Flush 226/300 Read block 3955 len 20480 227/300 Write block 642 len 12288 data: 6 4 3 228/300 Write block 4630 len 28672 data: 6 5 5 6 6 5 5 229/300 Write block 3176 len 20480 data: 6 6 6 6 6 230/300 Read block 3769 len 12288 231/300 Read block 2872 len 12288 232/300 Write block 3432 len 8192 data: 5 4 233/300 Write block 3264 len 20480 data: 3 4 4 3 4 234/300 Read block 4569 len 12288 235/300 Write block 709 len 4096 data: 5 236/300 Read block 2790 len 36864 237/300 Flush 238/300 Read block 261 len 40960 239/300 Write block 494 len 36864 data: 6 5 4 4 4 4 3 3 3 240/300 Write block 904 len 24576 data: 5 6 6 6 6 6 241/300 Flush 242/300 Write block 2678 len 16384 data: 4 2 2 3 243/300 Read block 4127 len 16384 244/300 Read block 3341 len 32768 245/300 Read block 3601 len 16384 246/300 Flush 247/300 Read block 3071 len 12288 248/300 Write block 1151 len 24576 data: 6 5 6 7 7 6 249/300 Write block 3078 len 40960 data: 7 5 5 4 5 4 4 7 8 8 250/300 Read block 2844 len 24576 251/300 Read block 1510 len 24576 252/300 Write block 1330 len 36864 data: 4 4 3 5 5 5 5 6 6 253/300 Read block 3245 len 4096 254/300 Read block 1247 len 36864 255/300 Write block 2872 len 28672 data: 5 4 7 7 9 9 7 256/300 Flush 257/300 Read block 4685 len 12288 258/300 Write block 483 len 32768 data: 5 6 5 4 4 4 6 6 259/300 Write block 4398 len 12288 data: 7 5 6 260/300 Write block 1999 len 24576 data: 11 10 10 8 10 9 261/300 Write block 3558 len 28672 data: 7 5 4 3 3 4 4 262/300 Read block 37 len 12288 263/300 Write block 4769 len 32768 data: 4 3 4 6 6 6 6 4 264/300 Read block 4633 len 12288 265/300 Write block 3706 len 40960 data: 7 6 5 5 6 5 6 4 3 3 266/300 Write block 2302 len 40960 data: 7 8 7 11 12 12 11 11 10 8 267/300 Read block 853 len 40960 268/300 Write block 800 len 36864 data: 7 7 8 8 9 6 8 8 7 269/300 Write block 2264 len 16384 data: 8 7 7 7 270/300 Read block 429 len 40960 271/300 Write block 3895 len 12288 data: 7 6 7 272/300 Read block 3025 len 16384 273/300 Flush 274/300 Write block 43 len 28672 data: 4 5 5 5 5 4 6 275/300 Read block 1097 len 8192 276/300 Write block 4658 len 16384 data: 11 11 9 8 277/300 Write block 4782 len 40960 data: 4 4 3 4 3 4 5 4 5 6 278/300 Read block 4499 len 16384 279/300 Read block 3447 len 20480 280/300 Read block 592 len 12288 281/300 Write block 3026 len 36864 data: 6 6 5 5 7 5 7 6 7 282/300 Write block 2365 len 16384 data: 7 7 5 5 283/300 Read block 3475 len 20480 284/300 Read block 1375 len 36864 285/300 Write block 670 len 24576 data: 5 7 5 5 6 6 286/300 Read block 1087 len 20480 287/300 Read block 4655 len 32768 288/300 Write block 3357 len 12288 data: 8 7 9 289/300 Read block 3969 len 28672 290/300 Write block 2690 len 20480 data: 4 3 5 5 5 291/300 Read block 4454 len 8192 292/300 Write block 1749 len 12288 data: 6 9 9 293/300 Read block 1486 len 16384 294/300 Read block 2458 len 20480 295/300 Write block 317 len 4096 data: 7 296/300 Write block 3777 len 24576 data: 4 4 4 4 5 5 297/300 Write block 3084 len 12288 data: 5 8 9 298/300 Read block 3037 len 28672 299/300 Flush 300/300 Read block 1713 len 4096 301/300 Write block 3033 len 28672 data: 7 8 7 6 6 5 5 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:49:10.684 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/1, method: POST, req_id: bd76a01d-f9c8-4d4e-b3db-ec21a7ddd81b, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9450 Acked Write 6 Done Sent Done false 9451 Acked Write 5 Done Sent Done false 9452 Acked Read 9 Done Sent Done false 9453 Acked Read 4 Done Sent Done false 9454 Acked Write 10 Done Sent Done false 9455 Acked Write 3 Done Sent Done false 9456 Acked Write 6 Done Sent Done false 9457 Acked Write 4 Done Sent Done false 9458 Acked Read 7 Done Sent Done false 9459 Acked Read 2 Done Sent Done false 9460 Acked Write 2 Done Sent Done false 9461 Acked Read 10 Done Sent Done false 9462 Acked Write 4 Done Sent Done false 9463 Acked Write 6 Done Sent Done false 9464 Acked Read 1 Done Sent Done false 9465 Acked Write 6 Done Sent Done false 9466 Acked Write 10 Done Sent Done false 9467 Acked Write 1 Done Sent Done false 9468 Acked Flush 0 Done Sent Done false 9469 Acked Write 10 Done Sent Done false 9470 Acked Write 10 Done Sent Done false 9471 Acked Read 1 Done Sent Done false 9472 Acked Write 6 Done Sent Done false 9473 Acked Read 8 Done Sent Done false 9474 Acked Flush 0 Done Sent Done false 9475 Acked Write 7 Done Sent Done false 9476 Acked Read 8 Done Sent Done false 9477 Acked Read 10 Done Sent Done false 9478 Acked Write 6 Done Sent Done false 9479 Acked Write 6 Done Sent Done false 9480 Acked Write 9 Done Sent Done false 9481 Acked Flush 0 Done Sent Done false 9482 Acked Read 10 Done Sent Done false 9483 Acked Read 8 Done Sent Done false 9484 Acked Write 6 Done Sent Done false 9485 Acked Read 2 Done Sent Done false 9486 Acked Read 9 Done Sent Done false 9487 Acked Write 1 Done Sent Done false 9488 Acked Write 6 Done Sent Done false 9489 Acked Read 3 Done Sent Done false 9490 Acked Write 9 Done Sent Done false 9491 Acked Write 3 Done Sent Done false 9492 Acked Flush 0 Done Sent Done false 9493 Acked Flush 0 Done Sent Done false 9494 Acked Write 5 Done Sent Done false 9495 Acked Write 9 Done Sent Done false 9496 Acked Flush 0 Done Sent Done false 9497 Acked Write 4 Done Sent Done false 9498 Acked Flush 0 Done Sent Done false 9499 Acked Write 6 Done Sent Done false 9500 Acked Write 5 Done Sent Done false 9501 Acked Flush 0 Done Sent Done false 9502 Acked Read 1 Done Sent Done false 9503 Acked Read 1 Done Sent Done false 9504 Acked Write 3 Done Sent Done false 9505 Acked Read 4 Done Sent Done false 9506 Acked Read 8 Done Sent Done false 9507 Acked Write 6 Done Sent Done false 9508 Acked Write 1 Done Sent Done false 9509 Acked Read 4 Done Sent Done false 9510 Acked Write 2 Done Sent Done false 9511 Acked Write 8 Done Sent Done false 9512 Acked Read 8 Done Sent Done false 9513 Acked Read 2 Done Sent Done false 9514 Acked Write 6 Done Sent Done false 9515 Acked Write 8 Done Sent Done false 9516 Acked Write 1 Done Sent Done false 9517 Acked Read 7 Done Sent Done false 9518 Acked Write 4 Done Sent Done false 9519 Acked Read 7 Done Sent Done false 9520 Acked Read 5 Done Sent Done false 9521 Acked Write 10 Done Sent Done false 9522 Acked Read 10 Done Sent Done false 9523 Acked Write 3 Done Sent Done false 9524 Acked Write 7 Done Sent Done false 9525 Acked Write 4 Done Sent Done false 9526 Acked Write 5 Done Sent Done false 9527 Acked Read 3 Done Sent Done false 9528 Acked Read 2 Done Sent Done false 9529 Acked Read 3 Done Sent Done false 9530 Acked Read 2 Done Sent Done false 9531 Acked Read 5 Done Sent Done false 9532 Acked Write 10 Done Sent Done false 9533 Acked Flush 0 Done Sent Done false 9534 Acked Read 1 Done Sent Done false 9535 Acked Write 5 Done Sent Done false 9536 Acked Write 10 Done Sent Done false 9537 Acked Read 3 Done Sent Done false 9538 Acked Write 3 Done Sent Done false 9539 Acked Read 3 Done Sent Done false 9540 Acked Read 9 Done Sent Done false 9541 Acked Read 3 Done Sent Done false 9542 Acked Write 4 Done Sent Done false 9543 Acked Write 7 Done Sent Done false 9544 Acked Write 5 Done Sent Done false 9545 Acked Write 6 Done Sent Done false 9546 Acked Read 9 Done Sent Done false 9547 Acked Read 6 Done Sent Done false 9548 Acked Write 10 Done Sent Done false 9549 Acked Read 9 Done Sent Done false 9550 Acked Read 1 Done Sent Done false 9551 Acked Flush 0 Done Sent Done false 9552 Acked Read 7 Done Sent Done false 9553 Acked Read 6 Done Sent Done false 9554 Acked Write 10 Done Sent Done false 9555 Acked Read 3 Done Sent Done false 9556 Acked Write 6 Done Sent Done false 9557 Acked Flush 0 Done Sent Done false 9558 Acked Write 10 Done Sent Done false 9559 Acked Read 10 Done Sent Done false 9560 Acked Write 5 Done Sent Done false 9561 Acked Write 5 Done Sent Done false 9562 Acked Write 1 Done Sent Done false 9563 Acked Write 7 Done Sent Done false 9564 Acked Write 5 Done Sent Done false 9565 Acked Write 5 Done Sent Done false 9566 Acked Write 1 Done Sent Done false 9567 Acked Flush 0 Done Sent Done false 9568 Acked Flush 0 Done Sent Done false 9569 Acked Write 9 Done Sent Done false 9570 Acked Flush 0 Done Sent Done false 9571 Acked Read 1 Done Sent Done false 9572 Acked Read 1 Done Sent Done false 9573 Acked Flush 0 Done Sent Done false 9574 Acked Read 8 Done Sent Done false 9575 Acked Write 8 Done Sent Done false 9576 Acked Read 10 Done Sent Done false 9577 Acked Read 2 Done Sent Done false 9578 Acked Read 1 Done Sent Done false 9579 Acked Read 6 Done Sent Done false 9580 Acked Flush 0 Done Sent Done false 9581 Acked Read 5 Done Sent Done false 9582 Acked Read 10 Done Sent Done false 9583 Acked Read 4 Done Sent Done false 9584 Acked Write 6 Done Sent Done false 9585 Acked Write 6 Done Sent Done false 9586 Acked Write 3 Done Sent Done false 9587 Acked Write 1 Done Sent Done false 9588 Acked Read 7 Done Sent Done false 9589 Acked Read 7 Done Sent Done false 9590 Acked Write 10 Done Sent Done false 9591 Acked Write 7 Done Sent Done false 9592 Acked Write 5 Done Sent Done false 9593 Acked Read 8 Done Sent Done false 9594 Acked Write 10 Done Sent Done false 9595 Acked Write 9 Done Sent Done false 9596 Acked Read 3 Done Sent Done false 9597 Acked Read 7 Done Sent Done false 9598 Acked Write 8 Done Sent Done false 9599 Acked Flush 0 Done Sent Done false 9600 Acked Read 6 Done Sent Done false 9601 Acked Read 1 Done Sent Done false 9602 Acked Read 1 Done Sent Done false 9603 Acked Read 10 Done Sent Done false 9604 Acked Read 10 Done Sent Done false 9605 Acked Write 9 Done Sent Done false 9606 Acked Write 7 Done Sent Done false 9607 Acked Read 9 Done Sent Done false 9608 Acked Write 3 Done Sent Done false 9609 Acked Write 7 Done Sent Done false 9610 Acked Flush 0 Done Sent Done false 9611 Acked Write 3 Done Sent Done false 9612 Acked Read 8 Done Sent Done false 9613 Acked Read 10 Done Sent Done false 9614 Acked Read 5 Done Sent Done false 9615 Acked Read 8 Done Sent Done false 9616 Acked Read 4 Done Sent Done false 9617 Acked Write 9 Done Sent Done false 9618 Acked Write 3 Done Sent Done false 9619 Acked Write 5 Done Sent Done false 9620 Acked Write 3 Done Sent Done false 9621 Acked Write 10 Done Sent Done false 9622 Acked Write 8 Done Sent Done false 9623 Acked Flush 0 Done Sent Done false 9624 Acked Read 5 Done Sent Done false 9625 Acked Read 8 Done Sent Done false 9626 Acked Write 3 Done Sent Done false 9627 Acked Write 5 Done Sent Done false 9628 Acked Write 9 Done Sent Done false 9629 Acked Flush 0 Done Sent Done false 9630 Acked Write 10 Done Sent Done false 9631 Acked Read 9 Done Sent Done false 9632 Acked Write 6 Done Sent Done false 9633 Acked Write 3 Done Sent Done false 9634 Acked Read 9 Done Sent Done false 9635 Acked Read 2 Done Sent Done false 9636 Acked Read 2 Done Sent Done false 9637 Acked Write 2 Done Sent Done false 9638 Acked Read 9 Done Sent Done false 9639 Acked Write 1 Done Sent Done false 9640 Acked Write 7 Done Sent Done false 9641 Acked Read 5 Done Sent Done false 9642 Acked Write 7 Done Sent Done false 9643 Acked Read 8 Done Sent Done false 9644 Acked Write 1 Done Sent Done false 9645 Acked Read 10 Done Sent Done false 9646 Acked Read 9 Done Sent Done false 9647 Acked Read 10 Done Sent Done false 9648 Acked Read 5 Done Sent Done false 9649 Acked Read 6 Done Sent Done false 9650 Acked Read 6 Done Sent Done false 9651 Acked Write 9 Done Sent Done false 9652 Acked Read 8 Done Sent Done false 9653 Acked Write 9 Done Sent Done false 9654 Acked Write 2 Done Sent Done false 9655 Acked Read 1 Done Sent Done false 9656 Acked Read 2 Done Sent Done false 9657 Acked Read 2 Done Sent Done false 9658 Acked Read 2 Done Sent Done false 9659 Acked Read 10 Done Sent Done false 9660 Acked Read 5 Done Sent Done false 9661 Acked Read 1 Done Sent Done false 9662 Acked Flush 0 Done Sent Done false 9663 Acked Read 9 Done Sent Done false 9664 Acked Read 6 Done Sent Done false 9665 Acked Read 9 Done Sent Done false 9666 Acked Read 8 Done Sent Done false 9667 Acked Write 6 Done Sent Done false 9668 Acked Read 1 Done Sent Done false 9669 Acked Read 2 Done Sent Done false 9670 Acked Flush 0 Done Sent Done false 9671 Acked Flush 0 Done Sent Done false 9672 Acked Read 3 Done Sent Done false 9673 Acked Write 4 Done Sent Done false 9674 Acked Flush 0 Done Sent Done false 9675 Acked Read 5 Done Sent Done false 9676 Acked Write 3 Done Sent Done false 9677 Acked Write 7 Done Sent Done false 9678 Acked Write 5 Done Sent Done false 9679 Acked Read 3 Done Sent Done false 9680 Acked Read 3 Done Sent Done false 9681 Acked Write 2 Done Sent Done false 9682 Acked Write 5 Done Sent Done false 9683 Acked Read 3 Done Sent Done false 9684 Acked Write 1 Done Sent Done false 9685 Acked Read 9 Done Sent Done false 9686 Acked Flush 0 Done Sent Done false 9687 Acked Read 10 Done Sent Done false 9688 Acked Write 9 Done Sent Done false 9689 Acked Write 6 Done Sent Done false 9690 Acked Flush 0 Done Sent Done false 9691 Acked Write 4 Done Sent Done false 9692 Acked Read 4 Done Sent Done false 9693 Acked Read 8 Done Sent Done false 9694 Acked Read 4 Done Sent Done false 9695 Acked Flush 0 Done Sent Done false 9696 Acked Read 3 Done Sent Done false 9697 Acked Write 6 Done Sent Done false 9698 Acked Write 10 Done Sent Done false 9699 Acked Read 6 Done Sent Done false 9700 Acked Read 6 Done Sent Done false 9701 Acked Write 9 Done Sent Done false 9702 Acked Read 1 Done Sent Done false 9703 Acked Read 9 Done Sent Done false 9704 Acked Write 7 Done Sent Done false 9705 Acked Flush 0 Done Sent Done false 9706 Acked Read 3 Done Sent Done false 9707 Acked Write 8 Done Sent Done false 9708 Acked Write 3 Done Sent Done false 9709 Acked Write 6 Done Sent Done false 9710 Acked Write 7 Done Sent Done false 9711 Acked Read 3 Done Sent Done false 9712 Acked Write 8 Done Sent Done false 9713 Acked Read 3 Done Sent Done false 9714 Acked Write 10 Done Sent Done false 9715 Acked Write 10 Done Sent Done false 9716 Acked Read 10 Done Sent Done false 9717 Acked Write 9 Done Sent Done false 9718 Acked Write 4 Done Sent Done false 9719 Acked Read 10 Done Sent Done false 9720 Acked Write 3 Done Sent Done false 9721 Acked Read 4 Done Sent Done false 9722 Acked Flush 0 Done Sent Done false 9723 Acked Write 7 Done Sent Done false 9724 Acked Read 2 Done Sent Done false 9725 Acked Write 4 Done Sent Done false 9726 Acked Write 10 Done Sent Done false 9727 Acked Read 4 Done Sent Done false 9728 Acked Read 5 Done Sent Done false 9729 Acked Read 3 Done Sent Done false 9730 Acked Write 9 Done Sent Done false 9731 Acked Write 4 Done Sent Done false 9732 Acked Read 5 Done Sent Done false 9733 Acked Read 9 Done Sent Done false 9734 Acked Write 6 Done Sent Done false 9735 Acked Read 5 Done Sent Done false 9736 Acked Read 8 Done Sent Done false 9737 Acked Write 3 Done Sent Done false 9738 Acked Read 7 Done Sent Done false 9739 Acked Write 5 Done Sent Done false 9740 Acked Read 2 Done Sent Done false 9741 Acked Write 3 Done Sent Done false 9742 Acked Read 4 Done Sent Done false 9743 Acked Read 5 Done Sent Done false 9744 Acked Write 1 Done Sent Done false 9745 Acked Write 6 Done Sent Done false 9746 Acked Write 3 Done Sent Done false 9747 Acked Read 7 Done Sent Done false 9748 Acked Flush 0 Done Sent Done false 9749 Acked Read 1 Done Sent Done false 9750 Acked Write 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9748 9449 9748 Downstairs last five completed: 9449 9448 9447 9446 9445 Upstairs last five completed: 9750 9749 9748 9747 9746 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1231) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9450 Acked Write 6 Done Sent Done false 9451 Acked Write 5 Done Sent Done false 9452 Acked Read 9 Done Sent Done false 9453 Acked Read 4 Done Sent Done false 9454 Acked Write 10 Done Sent Done false 9455 Acked Write 3 Done Sent Done false 9456 Acked Write 6 Done Sent Done false 9457 Acked Write 4 Done Sent Done false 9458 Acked Read 7 Done Sent Done false 9459 Acked Read 2 Done Sent Done false 9460 Acked Write 2 Done Sent Done false 9461 Acked Read 10 Done Sent Done false 9462 Acked Write 4 Done Sent Done false 9463 Acked Write 6 Done Sent Done false 9464 Acked Read 1 Done Sent Done false 9465 Acked Write 6 Done Sent Done false 9466 Acked Write 10 Done Sent Done false 9467 Acked Write 1 Done Sent Done false 9468 Acked Flush 0 Done Sent Done false 9469 Acked Write 10 Done Sent Done false 9470 Acked Write 10 Done Sent Done false 9471 Acked Read 1 Done Sent Done false 9472 Acked Write 6 Done Sent Done false 9473 Acked Read 8 Done Sent Done false 9474 Acked Flush 0 Done Sent Done false 9475 Acked Write 7 Done Sent Done false 9476 Acked Read 8 Done Sent Done false 9477 Acked Read 10 Done Sent Done false 9478 Acked Write 6 Done Sent Done false 9479 Acked Write 6 Done Sent Done false 9480 Acked Write 9 Done Sent Done false 9481 Acked Flush 0 Done Sent Done false 9482 Acked Read 10 Done Sent Done false 9483 Acked Read 8 Done Sent Done false 9484 Acked Write 6 Done Sent Done false 9485 Acked Read 2 Done Sent Done false 9486 Acked Read 9 Done Sent Done false 9487 Acked Write 1 Done Sent Done false 9488 Acked Write 6 Done Sent Done false 9489 Acked Read 3 Done Sent Done false 9490 Acked Write 9 Done Sent Done false 9491 Acked Write 3 Done Sent Done false 9492 Acked Flush 0 Done Sent Done false 9493 Acked Flush 0 Done Sent Done false 9494 Acked Write 5 Done Sent Done false 9495 Acked Write 9 Done Sent Done false 9496 Acked Flush 0 Done Sent Done false 9497 Acked Write 4 Done Sent Done false 9498 Acked Flush 0 Done Sent Done false 9499 Acked Write 6 Done Sent Done false 9500 Acked Write 5 Done Sent Done false 9501 Acked Flush 0 Done Sent Done false 9502 Acked Read 1 Done Sent Done false 9503 Acked Read 1 Done Sent Done false 9504 Acked Write 3 Done Sent Done false 9505 Acked Read 4 Done Sent Done false 9506 Acked Read 8 Done Sent Done false 9507 Acked Write 6 Done Sent Done false 9508 Acked Write 1 Done Sent Done false 9509 Acked Read 4 Done Sent Done false 9510 Acked Write 2 Done Sent Done false 9511 Acked Write 8 Done Sent Done false 9512 Acked Read 8 Done Sent Done false 9513 Acked Read 2 Done Sent Done false 9514 Acked Write 6 Done Sent Done false 9515 Acked Write 8 Done Sent Done false 9516 Acked Write 1 Done Sent Done false 9517 Acked Read 7 Done Sent Done false 9518 Acked Write 4 Done Sent Done false 9519 Acked Read 7 Done Sent Done false 9520 Acked Read 5 Done Sent Done false 9521 Acked Write 10 Done Sent Done false 9522 Acked Read 10 Done Sent Done false 9523 Acked Write 3 Done Sent Done false 9524 Acked Write 7 Done Sent Done false 9525 Acked Write 4 Done Sent Done false 9526 Acked Write 5 Done Sent Done false 9527 Acked Read 3 Done Sent Done false 9528 Acked Read 2 Done Sent Done false 9529 Acked Read 3 Done Sent Done false 9530 Acked Read 2 Done Sent Done false 9531 Acked Read 5 Done Sent Done false 9532 Acked Write 10 Done Sent Done false 9533 Acked Flush 0 Done Sent Done false 9534 Acked Read 1 Done Sent Done false 9535 Acked Write 5 Done Sent Done false 9536 Acked Write 10 Done Sent Done false 9537 Acked Read 3 Done Sent Done false 9538 Acked Write 3 Done Sent Done false 9539 Acked Read 3 Done Sent Done false 9540 Acked Read 9 Done Sent Done false 9541 Acked Read 3 Done Sent Done false 9542 Acked Write 4 Done Sent Done false 9543 Acked Write 7 Done Sent Done false 9544 Acked Write 5 Done Sent Done false 9545 Acked Write 6 Done Sent Done false 9546 Acked Read 9 Done Sent Done false 9547 Acked Read 6 Done Sent Done false 9548 Acked Write 10 Done Sent Done false 9549 Acked Read 9 Done Sent Done false 9550 Acked Read 1 Done Sent Done false 9551 Acked Flush 0 Done Sent Done false 9552 Acked Read 7 Done Sent Done false 9553 Acked Read 6 Done Sent Done false 9554 Acked Write 10 Done Sent Done false 9555 Acked Read 3 Done Sent Done false 9556 Acked Write 6 Done Sent Done false 9557 Acked Flush 0 Done Sent Done false 9558 Acked Write 10 Done Sent Done false 9559 Acked Read 10 Done Sent Done false 9560 Acked Write 5 Done Sent Done false 9561 Acked Write 5 Done Sent Done false 9562 Acked Write 1 Done Sent Done false 9563 Acked Write 7 Done Sent Done false 9564 Acked Write 5 Done Sent Done false 9565 Acked Write 5 Done Sent Done false 9566 Acked Write 1 Done Sent Done false 9567 Acked Flush 0 Done Sent Done false 9568 Acked Flush 0 Done Sent Done false 9569 Acked Write 9 Done Sent Done false 9570 Acked Flush 0 Done Sent Done false 9571 Acked Read 1 Done Sent Done false 9572 Acked Read 1 Done Sent Done false 9573 Acked Flush 0 Done Sent Done false 9574 Acked Read 8 Done Sent Done false 9575 Acked Write 8 Done Sent Done false 9576 Acked Read 10 Done Sent Done false 9577 Acked Read 2 Done Sent Done false 9578 Acked Read 1 Done Sent Done false 9579 Acked Read 6 Done Sent Done false 9580 Acked Flush 0 Done Sent Done false 9581 Acked Read 5 Done Sent Done false 9582 Acked Read 10 Done Sent Done false 9583 Acked Read 4 Done Sent Done false 9584 Acked Write 6 Done Sent Done false 9585 Acked Write 6 Done Sent Done false 9586 Acked Write 3 Done Sent Done false 9587 Acked Write 1 Done Sent Done false 9588 Acked Read 7 Done Sent Done false 9589 Acked Read 7 Done Sent Done false 9590 Acked Write 10 Done Sent Done false 9591 Acked Write 7 Done Sent Done false 9592 Acked Write 5 Done Sent Done false 9593 Acked Read 8 Done Sent Done false 9594 Acked Write 10 Done Sent Done false 9595 Acked Write 9 Done Sent Done false 9596 Acked Read 3 Done Sent Done false 9597 Acked Read 7 Done Sent Done false 9598 Acked Write 8 Done Sent Done false 9599 Acked Flush 0 Done Sent Done false 9600 Acked Read 6 Done Sent Done false 9601 Acked Read 1 Done Sent Done false 9602 Acked Read 1 Done Sent Done false 9603 Acked Read 10 Done Sent Done false 9604 Acked Read 10 Done Sent Done false 9605 Acked Write 9 Done Sent Done false 9606 Acked Write 7 Done Sent Done false 9607 Acked Read 9 Done Sent Done false 9608 Acked Write 3 Done Sent Done false 9609 Acked Write 7 Done Sent Done false 9610 Acked Flush 0 Done Sent Done false 9611 Acked Write 3 Done Sent Done false 9612 Acked Read 8 Done Sent Done false 9613 Acked Read 10 Done Sent Done false 9614 Acked Read 5 Done Sent Done false 9615 Acked Read 8 Done Sent Done false 9616 Acked Read 4 Done Sent Done false 9617 Acked Write 9 Done Sent Done false 9618 Acked Write 3 Done Sent Done false 9619 Acked Write 5 Done Sent Done false 9620 Acked Write 3 Done Sent Done false 9621 Acked Write 10 Done Sent Done false 9622 Acked Write 8 Done Sent Done false 9623 Acked Flush 0 Done Sent Done false 9624 Acked Read 5 Done Sent Done false 9625 Acked Read 8 Done Sent Done false 9626 Acked Write 3 Done Sent Done false 9627 Acked Write 5 Done Sent Done false 9628 Acked Write 9 Done Sent Done false 9629 Acked Flush 0 Done Sent Done false 9630 Acked Write 10 Done Sent Done false 9631 Acked Read 9 Done Sent Done false 9632 Acked Write 6 Done Sent Done false 9633 Acked Write 3 Done Sent Done false 9634 Acked Read 9 Done Sent Done false 9635 Acked Read 2 Done Sent Done false 9636 Acked Read 2 Done Sent Done false 9637 Acked Write 2 Done Sent Done false 9638 Acked Read 9 Done Sent Done false 9639 Acked Write 1 Done Sent Done false 9640 Acked Write 7 Done Sent Done false 9641 Acked Read 5 Done Sent Done false 9642 Acked Write 7 Done Sent Done false 9643 Acked Read 8 Done Sent Done false 9644 Acked Write 1 Done Sent Done false 9645 Acked Read 10 Done Sent Done false 9646 Acked Read 9 Done Sent Done false 9647 Acked Read 10 Done Sent Done false 9648 Acked Read 5 Done Sent Done false 9649 Acked Read 6 Done Sent Done false 9650 Acked Read 6 Done Sent Done false 9651 Acked Write 9 Done Sent Done false 9652 Acked Read 8 Done Sent Done false 9653 Acked Write 9 Done Sent Done false 9654 Acked Write 2 Done Sent Done false 9655 Acked Read 1 Done Sent Done false 9656 Acked Read 2 Done Sent Done false 9657 Acked Read 2 Done Sent Done false 9658 Acked Read 2 Done Sent Done false 9659 Acked Read 10 Done Sent Done false 9660 Acked Read 5 Done Sent Done false 9661 Acked Read 1 Done Sent Done false 9662 Acked Flush 0 Done Sent Done false 9663 Acked Read 9 Done Sent Done false 9664 Acked Read 6 Done Sent Done false 9665 Acked Read 9 Done Sent Done false 9666 Acked Read 8 Done Sent Done false 9667 Acked Write 6 Done Sent Done false 9668 Acked Read 1 Done Sent Done false 9669 Acked Read 2 Done Sent Done false 9670 Acked Flush 0 Done Sent Done false 9671 Acked Flush 0 Done Sent Done false 9672 Acked Read 3 Done Sent Done false 9673 Acked Write 4 Done Sent Done false 9674 Acked Flush 0 Done Sent Done false 9675 Acked Read 5 Done Sent Done false 9676 Acked Write 3 Done Sent Done false 9677 Acked Write 7 Done Sent Done false 9678 Acked Write 5 Done Sent Done false 9679 Acked Read 3 Done Sent Done false 9680 Acked Read 3 Done Sent Done false 9681 Acked Write 2 Done Sent Done false 9682 Acked Write 5 Done Sent Done false 9683 Acked Read 3 Done Sent Done false 9684 Acked Write 1 Done Sent Done false 9685 Acked Read 9 Done Sent Done false 9686 Acked Flush 0 Done Sent Done false 9687 Acked Read 10 Done Sent Done false 9688 Acked Write 9 Done Sent Done false 9689 Acked Write 6 Done Sent Done false 9690 Acked Flush 0 Done Sent Done false 9691 Acked Write 4 Done Sent Done false 9692 Acked Read 4 Done Sent Done false 9693 Acked Read 8 Done Sent Done false 9694 Acked Read 4 Done Sent Done false 9695 Acked Flush 0 Done Sent Done false 9696 Acked Read 3 Done Sent Done false 9697 Acked Write 6 Done Sent Done false 9698 Acked Write 10 Done Sent Done false 9699 Acked Read 6 Done Sent Done false 9700 Acked Read 6 Done Sent Done false 9701 Acked Write 9 Done Sent Done false 9702 Acked Read 1 Done Sent Done false 9703 Acked Read 9 Done Sent Done false 9704 Acked Write 7 Done Sent Done false 9705 Acked Flush 0 Done Sent Done false 9706 Acked Read 3 Done Sent Done false 9707 Acked Write 8 Done Sent Done false 9708 Acked Write 3 Done Sent Done false 9709 Acked Write 6 Done Sent Done false 9710 Acked Write 7 Done Sent Done false 9711 Acked Read 3 Done Sent Done false 9712 Acked Write 8 Done Sent Done false 9713 Acked Read 3 Done Sent Done false 9714 Acked Write 10 Done Sent Done false 9715 Acked Write 10 Done Sent Done false 9716 Acked Read 10 Done Sent Done false 9717 Acked Write 9 Done Sent Done false 9718 Acked Write 4 Done Sent Done false 9719 Acked Read 10 Done Sent Done false 9720 Acked Write 3 Done Sent Done false 9721 Acked Read 4 Done Sent Done false 9722 Acked Flush 0 Done Sent Done false 9723 Acked Write 7 Done Sent Done false 9724 Acked Read 2 Done Sent Done false 9725 Acked Write 4 Done Sent Done false 9726 Acked Write 10 Done Sent Done false 9727 Acked Read 4 Done Sent Done false 9728 Acked Read 5 Done Sent Done false 9729 Acked Read 3 Done Sent Done false 9730 Acked Write 9 Done Sent Done false 9731 Acked Write 4 Done Sent Done false 9732 Acked Read 5 Done Sent Done false 9733 Acked Read 9 Done Sent Done false 9734 Acked Write 6 Done Sent Done false 9735 Acked Read 5 Done Sent Done false 9736 Acked Read 8 Done Sent Done false 9737 Acked Write 3 Done Sent Done false 9738 Acked Read 7 Done Sent Done false 9739 Acked Write 5 Done Sent Done false 9740 Acked Read 2 Done Sent Done false 9741 Acked Write 3 Done Sent Done false 9742 Acked Read 4 Done Sent Done false 9743 Acked Read 5 Done Sent Done false 9744 Acked Write 1 Done Sent Done false 9745 Acked Write 6 Done Sent Done false 9746 Acked Write 3 Done Sent Done false 9747 Acked Read 7 Done Sent Done false 9748 Acked Flush 0 Done Sent Done false 9749 Acked Read 1 Done Sent Done false 9750 Acked Write 7 Done Sent Done false 9751 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9751 9449 9751 Downstairs last five completed: 9449 9448 9447 9446 9445 Upstairs last five completed: 9751 9750 9749 9748 9747 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.460889856Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:34576)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.460965886Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:16.461964418Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462482129Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462503743Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 9449","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462518779Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 9449","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462730442Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462750227Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462765223Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 9449","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.462779099Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 9449","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:16.463007468Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9751 9750 9749 9748 9747 Upstairs last five completed: 9751 9750 9749 9748 9747 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(1) stop 1 [8820] Got stop action so:false kr:false Nov 27 01:49:18.689 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7167af1e-ac57-48a5-b20e-9e8d9d7ebf49, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:18.689928583Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:49:18.689 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/1, method: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:18.690001224Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} GET, req_id: 4cba8e92-1999-4347-84e8-5889a55a2961, remote_addr: 127.0.0.1:51788, local_addr{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:18.690040593Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} : 127.0.0.1:9998 {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:18.690069626Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:18.690096518Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:18.690122911Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit Nov 27 01:49:22.691 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: b2f7491d-ded3-495d-aca1-35a3fa8f1553, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 1582 len 28672 002/300 Read block 4965 len 20480 003/300 Flush 004/300 Write block 4836 len 24576 data: 8 8 7 8 10 9 005/300 Write block 4042 len 40960 data: 6 5 4 4 4 4 4 6 8 6 006/300 Read block 87 len 32768 007/300 Read block 4794 len 20480 008/300 Flush 009/300 Read block 4292 len 32768 010/300 Read block 4201 len 16384 011/300 Read block 4961 len 16384 012/300 Read block 2646 len 12288 013/300 Write block 3557 len 12288 data: 6 8 6 014/300 Read block 2134 len 20480 015/300 Read block 2401 len 24576 016/300 Read block 3491 len 40960 017/300 Write block 1911 len 24576 data: 8 8 8 10 8 5 018/300 Flush 019/300 Write block 963 len 36864 data: 8 8 7 7 8 6 6 5 5 020/300 Write block 1140 len 24576 data: 4 3 3 2 4 6 021/300 Write block 1671 len 32768 data: 4 5 6 7 8 7 6 5 022/300 Write block 3820 len 8192 data: 4 4 023/300 Read block 1665 len 32768 024/300 Write block 4813 len 36864 data: 3 2 2 3 4 3 3 5 5 025/300 Read block 741 len 16384 026/300 Flush 027/300 Read block 877 len 8192 028/300 Read block 2867 len 16384 029/300 Write block 1233 len 32768 data: 9 8 7 5 5 5 3 3 030/300 Read block 3463 len 28672 031/300 Read block 4087 len 32768 032/300 Flush 033/300 Flush 034/300 Write block 1771 len 36864 data: 4 4 4 4 5 5 4 2 2 035/300 Write block 1543 len 20480 data: 10 8 8 5 5 036/300 Read block 944 len 32768 037/300 Write block 695 len 24576 data: 6 5 5 5 4 3 038/300 Read block 4632 len 12288 039/300 Read block 4750 len 4096 040/300 Read block 1104 len 28672 041/300 Read block 4566 len 28672 042/300 Write block 1885 len 28672 data: 9 9 9 9 7 7 7 043/300 Read block 1394 len 20480 044/300 Read block 1817 len 40960 045/300 Write block 1812 len 8192 data: 7 7 046/300 Write block 3016 len 40960 data: 6 5 5 5 6 7 6 6 5 6 047/300 Write block 301 len 16384 data: 2 3 2 2 048/300 Write block 4686 len 8192 data: 8 6 049/300 Write block 2016 len 36864 data: 6 5 6 6 6 6 7 6 7 050/300 Write block 828 len 28672 data: 14 14 13 13 11 9 8 051/300 Read block 3563 len 28672 052/300 Read block 596 len 16384 053/300 Read block 4017 len 20480 054/300 Write block 2999 len 28672 data: 3 3 6 6 6 5 5 055/300 Read block 2821 len 32768 056/300 Write block 1441 len 16384 data: 5 5 5 6 057/300 Write block 1046 len 36864 data: 6 5 6 5 3 3 5 5 4 058/300 Flush 059/300 Flush 060/300 Write block 1856 len 36864 data: 5 5 6 5 5 4 3 5 6 061/300 Write block 1435 len 20480 data: 5 2 2 4 7 062/300 Read block 3794 len 20480 063/300 Read block 795 len 16384 064/300 Read block 60 len 40960 065/300 Write block 722 len 8192 data: 5 5 066/300 Write block 4080 len 32768 data: 7 6 8 7 8 8 7 7 067/300 Read block 15 len 8192 068/300 Read block 4040 len 4096 069/300 Read block 4174 len 40960 070/300 Flush 071/300 Flush 072/300 Read block 3977 len 4096 073/300 Read block 4067 len 20480 074/300 Read block 2075 len 12288 075/300 Flush 076/300 Read block 41 len 40960 077/300 Read block 1335 len 32768 078/300 Read block 4889 len 8192 079/300 Read block 3010 len 12288 080/300 Write block 1646 len 4096 data: 6 081/300 Read block 4130 len 28672 082/300 Read block 3882 len 16384 083/300 Read block 3185 len 32768 084/300 Read block 2759 len 32768 085/300 Read block 1767 len 40960 086/300 Write block 2673 len 24576 data: 4 3 4 4 4 5 087/300 Write block 3277 len 28672 data: 5 3 3 3 3 3 3 088/300 Write block 4241 len 32768 data: 6 7 7 6 5 5 4 4 089/300 Write block 4062 len 20480 data: 4 5 5 5 6 090/300 Read block 3104 len 16384 091/300 Flush 092/300 Read block 2004 len 36864 093/300 Read block 4642 len 16384 094/300 Flush 095/300 Read block 1332 len 40960 096/300 Write block 2762 len 8192 data: 6 7 097/300 Write block 361 len 28672 data: 3 4 4 3 4 4 4 098/300 Write block 748 len 24576 data: 8 8 8 9 11 9 099/300 Read block 2457 len 4096 100/300 Read block 2776 len 40960 101/300 Read block 4082 len 28672 102/300 Write block 3213 len 36864 data: 11 10 10 8 8 6 7 7 8 103/300 Read block 3904 len 32768 104/300 Write block 2787 len 24576 data: 5 6 7 6 5 6 105/300 Write block 2524 len 24576 data: 4 5 6 7 8 8 106/300 Read block 3535 len 28672 107/300 Write block 2020 len 20480 data: 7 7 8 7 8 108/300 Write block 4560 len 32768 data: 7 9 8 10 11 11 9 9 109/300 Write block 4704 len 32768 data: 7 8 8 9 11 9 8 9 110/300 Read block 1175 len 40960 111/300 Read block 4105 len 32768 112/300 Write block 491 len 36864 data: 7 6 7 7 6 5 5 5 5 113/300 Write block 3930 len 40960 data: 4 4 3 2 3 2 2 3 6 5 114/300 Read block 3614 len 36864 115/300 Read block 1370 len 28672 116/300 Read block 4431 len 40960 117/300 Write block 4277 len 20480 data: 6 7 7 7 6 118/300 Read block 774 len 36864 119/300 Write block 3426 len 32768 data: 4 3 4 4 3 4 6 5 120/300 Write block 3350 len 16384 data: 5 6 6 6 121/300 Flush 122/300 Read block 436 len 36864 123/300 Read block 1024 len 20480 124/300 Write block 2285 len 28672 data: 8 7 7 6 7 4 4 125/300 Read block 4428 len 28672 126/300 Flush 127/300 Write block 464 len 28672 data: 7 7 6 6 8 11 12 128/300 Read block 281 len 12288 129/300 Write block 2300 len 36864 data: 5 6 8 9 8 12 13 13 12 130/300 Write block 4414 len 4096 data: 10 131/300 Flush 132/300 Read block 1788 len 32768 133/300 Write block 1889 len 20480 data: 8 8 8 7 7 134/300 Write block 3325 len 20480 data: 4 5 7 8 9 135/300 Read block 1701 len 12288 136/300 Read block 2273 len 4096 137/300 Write block 2512 len 40960 data: 4 5 4 4 4 6 7 7 6 3 138/300 Write block 1075 len 32768 data: 7 7 5 6 7 6 6 7 139/300 Write block 4236 len 36864 data: 6 7 7 6 5 7 8 8 7 140/300 Read block 4919 len 20480 141/300 Write block 2270 len 40960 data: 8 9 6 7 7 7 6 5 5 6 142/300 Read block 889 len 8192 143/300 Read block 4809 len 4096 144/300 Read block 3288 len 8192 145/300 Read block 1147 len 20480 146/300 Write block 3663 len 28672 data: 8 8 7 6 8 8 8 147/300 Read block 6 len 28672 148/300 Flush 149/300 Read block 2673 len 16384 150/300 Read block 3930 len 36864 151/300 Write block 3668 len 24576 data: 9 9 8 8 6 8 152/300 Read block 1623 len 16384 153/300 Flush 154/300 Read block 1025 len 16384 155/300 Write block 4252 len 40960 data: 6 7 5 7 7 6 5 6 5 6 156/300 Read block 2338 len 32768 157/300 Read block 2435 len 24576 158/300 Flush 159/300 Read block 336 len 24576 160/300 Read block 3369 len 28672 161/300 Read block 87 len 40960 162/300 Write block 3053 len 8192 data: 4 4 163/300 Read block 4331 len 16384 164/300 Read block 2204 len 24576 165/300 Write block 2138 len 8192 data: 8 8 166/300 Write block 1370 len 4096 data: 7 167/300 Write block 3777 len 40960 data: 5 5 5 5 6 6 6 6 7 7 168/300 Write block 3197 len 24576 data: 11 9 7 6 7 7 169/300 Read block 474 len 20480 170/300 Read block 869 len 36864 171/300 Write block 3593 len 20480 data: 9 8 8 9 8 172/300 Write block 1734 len 40960 data: 5 4 4 4 4 5 5 4 5 5 173/300 Write block 828 len 12288 data: 15 15 14 174/300 Read block 3237 len 20480 175/300 Write block 4109 len 24576 data: 4 4 5 4 4 3 176/300 Read block 4800 len 32768 177/300 Read block 3121 len 16384 178/300 Read block 838 len 24576 179/300 Read block 4323 len 24576 180/300 Flush 181/300 Write block 1974 len 40960 data: 8 7 6 5 5 6 6 7 6 4 182/300 Read block 4961 len 36864 183/300 Write block 4606 len 12288 data: 5 7 6 184/300 Read block 4622 len 28672 185/300 Read block 2163 len 24576 186/300 Write block 46 len 24576 data: 6 6 5 7 6 7 187/300 Write block 3784 len 20480 data: 7 8 8 6 7 188/300 Read block 3560 len 32768 189/300 Write block 108 len 4096 data: 9 190/300 Flush 191/300 Flush 192/300 Flush 193/300 Read block 3497 len 16384 194/300 Flush 195/300 Read block 1557 len 24576 196/300 Read block 221 len 24576 197/300 Read block 3891 len 16384 198/300 Read block 1914 len 12288 199/300 Read block 1125 len 20480 200/300 Read block 2793 len 36864 201/300 Read block 252 len 16384 202/300 Flush 203/300 Write block 4818 len 12288 data: 4 4 6 204/300 Flush 205/300 Read block 225 len 12288 206/300 Read block 1494 len 24576 207/300 Read block 3772 len 8192 208/300 Read block 1554 len 40960 209/300 Write block 510 len 16384 data: 9 8 10 7 210/300 Write block 1947 len 4096 data: 6 211/300 Write block 1943 len 20480 data: 5 4 5 5 7 212/300 Flush 213/300 Read block 2546 len 4096 214/300 Write block 917 len 36864 data: 5 5 5 5 5 5 5 4 2 215/300 Read block 2115 len 8192 216/300 Write block 3407 len 20480 data: 8 7 7 5 5 217/300 Write block 1021 len 28672 data: 4 4 4 4 3 2 3 218/300 Read block 440 len 36864 219/300 Write block 2528 len 8192 data: 9 9 220/300 Read block 487 len 16384 221/300 Flush 222/300 Read block 2083 len 24576 223/300 Read block 4561 len 4096 224/300 Read block 4455 len 24576 225/300 Write block 3719 len 4096 data: 3 226/300 Flush 227/300 Read block 2410 len 16384 228/300 Write block 3454 len 24576 data: 3 4 5 6 4 6 229/300 Write block 4347 len 40960 data: 2 2 2 2 2 4 4 6 6 10 230/300 Read block 3262 len 40960 231/300 Write block 54 len 12288 data: 7 5 4 232/300 Write block 1739 len 4096 data: 6 233/300 Read block 743 len 12288 234/300 Flush 235/300 Read block 1871 len 40960 236/300 Write block 4350 len 4096 data: 3 237/300 Write block 4679 len 4096 data: 11 238/300 Read block 2851 len 24576 239/300 Write block 1454 len 28672 data: 5 5 3 3 3 5 6 240/300 Read block 1397 len 16384 241/300 Read block 1373 len 20480 242/300 Read block 2221 len 28672 243/300 Read block 2133 len 36864 244/300 Write block 518 len 28672 data: 7 6 7 7 6 8 11 245/300 Write block 2712 len 12288 data: 5 6 5 246/300 Read block 269 len 12288 247/300 Write block 3938 len 12288 data: 7 6 6 248/300 Read block 1567 len 16384 249/300 Read block 2264 len 20480 250/300 Read block 3340 len 28672 251/300 Write block 3713 len 28672 data: 5 4 4 3 2 2 4 252/300 Read block 3361 len 12288 253/300 Write block 2173 len 40960 data: 2 2 2 2 2 2 2 5 6 7 254/300 Write block 2589 len 24576 data: 5 5 4 3 4 4 255/300 Read block 2211 len 32768 256/300 Write block 2447 len 12288 data: 7 6 6 257/300 Write block 4364 len 12288 data: 8 5 5 258/300 Write block 1001 len 36864 data: 6 5 8 8 7 7 7 7 6 259/300 Write block 1812 len 28672 data: 8 8 7 10 13 12 11 260/300 Write block 1682 len 20480 data: 7 7 6 7 9 261/300 Read block 3129 len 24576 262/300 Flush 263/300 Write block 4997 len 12288 data: 4 3 2 264/300 Write block 4162 len 24576 data: 6 6 6 6 6 4 265/300 Write block 4918 len 24576 data: 5 5 5 7 8 6 266/300 Read block 4864 len 4096 267/300 Flush 268/300 Flush 269/300 Read block 1062 len 12288 270/300 Write block 3022 len 8192 data: 7 7 271/300 Write block 1187 len 40960 data: 5 6 6 8 8 8 6 7 7 8 272/300 Write block 3466 len 16384 data: 6 6 6 7 273/300 Read block 678 len 28672 274/300 Write block 3638 len 32768 data: 5 4 5 6 6 5 5 4 275/300 Write block 3008 len 32768 data: 5 5 4 4 6 6 6 6 276/300 Write block 1477 len 16384 data: 5 3 4 3 277/300 Read block 2305 len 8192 278/300 Write block 2216 len 20480 data: 7 8 8 8 7 279/300 Read block 3844 len 16384 280/300 Flush 281/300 Read block 4314 len 4096 282/300 Write block 119 len 12288 data: 8 5 6 283/300 Read block 4247 len 20480 284/300 Read block 4411 len 12288 285/300 Read block 2556 len 32768 286/300 Write block 3880 len 24576 data: 4 4 3 2 2 3 287/300 Write block 4807 len 20480 data: 5 4 3 2 2 288/300 Read block 3263 len 24576 289/300 Read block 2040 len 20480 290/300 Read block 400 len 32768 291/300 Write block 2729 len 12288 data: 5 7 8 292/300 Write block 3332 len 8192 data: 10 9 293/300 Write block 730 len 40960 data: 5 5 5 4 4 3 3 2 3 3 294/300 Write block 3478 len 36864 data: 6 7 7 6 5 6 4 5 5 295/300 Read block 4753 len 4096 296/300 Read block 3816 len 28672 297/300 Read block 412 len 8192 298/300 Read block 1822 len 16384 299/300 Write block 3721 len 40960 data: 4 4 5 4 5 7 6 4 4 5 300/300 Flush 301/300 Read block 3767 len 36864 Main task has work to do, go find it got dsc Start(1) start 1 Nov 27 01:49:22.927 INFO request completed, latency_us: 189, response_code: 204, uri: /start/cid/1, method: POST, req_id: efc11876-929b-4913-b508-c55b6910ed63, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9752 Acked Read 7 Done Sent Done false 9753 Acked Read 5 Done Sent Done false 9754 Acked Flush 0 Done Sent Done false 9755 Acked Write 6 Done Sent Done false 9756 Acked Write 10 Done Sent Done false 9757 Acked Read 8 Done Sent Done false 9758 Acked Read 5 Done Sent Done false 9759 Acked Flush 0 Done Sent Done false 9760 Acked Read 8 Done Sent Done false 9761 Acked Read 4 Done Sent Done false 9762 Acked Read 4 Done Sent Done false 9763 Acked Read 3 Done Sent Done false 9764 Acked Write 3 Done Sent Done false 9765 Acked Read 5 Done Sent Done false 9766 Acked Read 6 Done Sent Done false 9767 Acked Read 10 Done Sent Done false 9768 Acked Write 6 Done Sent Done false 9769 Acked Flush 0 Done Sent Done false 9770 Acked Write 9 Done Sent Done false 9771 Acked Write 6 Done Sent Done false 9772 Acked Write 8 Done Sent Done false 9773 Acked Write 2 Done Sent Done false 9774 Acked Read 8 Done Sent Done false 9775 Acked Write 9 Done Sent Done false 9776 Acked Read 4 Done Sent Done false 9777 Acked Flush 0 Done Sent Done false 9778 Acked Read 2 Done Sent Done false 9779 Acked Read 4 Done Sent Done false 9780 Acked Write 8 Done Sent Done false 9781 Acked Read 7 Done Sent Done false 9782 Acked Read 8 Done Sent Done false 9783 Acked Flush 0 Done Sent Done false 9784 Acked Flush 0 Done Sent Done false 9785 Acked Write 9 Done Sent Done false 9786 Acked Write 5 Done Sent Done false 9787 Acked Read 8 Done Sent Done false 9788 Acked Write 6 Done Sent Done false 9789 Acked Read 3 Done Sent Done false 9790 Acked Read 1 Done Sent Done false 9791 Acked Read 7 Done Sent Done false 9792 Acked Read 7 Done Sent Done false 9793 Acked Write 7 Done Sent Done false 9794 Acked Read 5 Done Sent Done false 9795 Acked Read 10 Done Sent Done false 9796 Acked Write 2 Done Sent Done false 9797 Acked Write 10 Done Sent Done false 9798 Acked Write 4 Done Sent Done false 9799 Acked Write 2 Done Sent Done false 9800 Acked Write 9 Done Sent Done false 9801 Acked Write 7 Done Sent Done false 9802 Acked Read 7 Done Sent Done false 9803 Acked Read 4 Done Sent Done false 9804 Acked Read 5 Done Sent Done false 9805 Acked Write 7 Done Sent Done false 9806 Acked Read 8 Done Sent Done false 9807 Acked Write 4 Done Sent Done false 9808 Acked Write 9 Done Sent Done false 9809 Acked Flush 0 Done Sent Done false 9810 Acked Flush 0 Done Sent Done false 9811 Acked Write 9 Done Sent Done false 9812 Acked Write 5 Done Sent Done false 9813 Acked Read 5 Done Sent Done false 9814 Acked Read 4 Done Sent Done false 9815 Acked Read 10 Done Sent Done false 9816 Acked Write 2 Done Sent Done false 9817 Acked Write 8 Done Sent Done false 9818 Acked Read 2 Done Sent Done false 9819 Acked Read 1 Done Sent Done false 9820 Acked Read 10 Done Sent Done false 9821 Acked Flush 0 Done Sent Done false 9822 Acked Flush 0 Done Sent Done false 9823 Acked Read 1 Done Sent Done false 9824 Acked Read 5 Done Sent Done false 9825 Acked Read 3 Done Sent Done false 9826 Acked Flush 0 Done Sent Done false 9827 Acked Read 10 Done Sent Done false 9828 Acked Read 8 Done Sent Done false 9829 Acked Read 2 Done Sent Done false 9830 Acked Read 3 Done Sent Done false 9831 Acked Write 1 Done Sent Done false 9832 Acked Read 7 Done Sent Done false 9833 Acked Read 4 Done Sent Done false 9834 Acked Read 8 Done Sent Done false 9835 Acked Read 8 Done Sent Done false 9836 Acked Read 10 Done Sent Done false 9837 Acked Write 6 Done Sent Done false 9838 Acked Write 7 Done Sent Done false 9839 Acked Write 8 Done Sent Done false 9840 Acked Write 5 Done Sent Done false 9841 Acked Read 4 Done Sent Done false 9842 Acked Flush 0 Done Sent Done false 9843 Acked Read 9 Done Sent Done false 9844 Acked Read 4 Done Sent Done false 9845 Acked Flush 0 Done Sent Done false 9846 Acked Read 10 Done Sent Done false 9847 Acked Write 2 Done Sent Done false 9848 Acked Write 7 Done Sent Done false 9849 Acked Write 6 Done Sent Done false 9850 Acked Read 1 Done Sent Done false 9851 Acked Read 10 Done Sent Done false 9852 Acked Read 7 Done Sent Done false 9853 Acked Write 9 Done Sent Done false 9854 Acked Read 8 Done Sent Done false 9855 Acked Write 6 Done Sent Done false 9856 Acked Write 6 Done Sent Done false 9857 Acked Read 7 Done Sent Done false 9858 Acked Write 5 Done Sent Done false 9859 Acked Write 8 Done Sent Done false 9860 Acked Write 8 Done Sent Done false 9861 Acked Read 10 Done Sent Done false 9862 Acked Read 8 Done Sent Done false 9863 Acked Write 9 Done Sent Done false 9864 Acked Write 10 Done Sent Done false 9865 Acked Read 9 Done Sent Done false 9866 Acked Read 7 Done Sent Done false 9867 Acked Read 10 Done Sent Done false 9868 Acked Write 5 Done Sent Done false 9869 Acked Read 9 Done Sent Done false 9870 Acked Write 8 Done Sent Done false 9871 Acked Write 4 Done Sent Done false 9872 Acked Flush 0 Done Sent Done false 9873 Acked Read 9 Done Sent Done false 9874 Acked Read 5 Done Sent Done false 9875 Acked Write 7 Done Sent Done false 9876 Acked Read 7 Done Sent Done false 9877 Acked Flush 0 Done Sent Done false 9878 Acked Write 7 Done Sent Done false 9879 Acked Read 3 Done Sent Done false 9880 Acked Write 9 Done Sent Done false 9881 Acked Write 1 Done Sent Done false 9882 Acked Flush 0 Done Sent Done false 9883 Acked Read 8 Done Sent Done false 9884 Acked Write 5 Done Sent Done false 9885 Acked Write 5 Done Sent Done false 9886 Acked Read 3 Done Sent Done false 9887 Acked Read 1 Done Sent Done false 9888 Acked Write 10 Done Sent Done false 9889 Acked Write 8 Done Sent Done false 9890 Acked Write 9 Done Sent Done false 9891 Acked Read 5 Done Sent Done false 9892 Acked Write 10 Done Sent Done false 9893 Acked Read 2 Done Sent Done false 9894 Acked Read 1 Done Sent Done false 9895 Acked Read 2 Done Sent Done false 9896 Acked Read 5 Done Sent Done false 9897 Acked Write 7 Done Sent Done false 9898 Acked Read 7 Done Sent Done false 9899 Acked Flush 0 Done Sent Done false 9900 Acked Read 4 Done Sent Done false 9901 Acked Read 9 Done Sent Done false 9902 Acked Write 6 Done Sent Done false 9903 Acked Read 4 Done Sent Done false 9904 Acked Flush 0 Done Sent Done false 9905 Acked Read 4 Done Sent Done false 9906 Acked Write 10 Done Sent Done false 9907 Acked Read 8 Done Sent Done false 9908 Acked Read 6 Done Sent Done false 9909 Acked Flush 0 Done Sent Done false 9910 Acked Read 6 Done Sent Done false 9911 Acked Read 7 Done Sent Done false 9912 Acked Read 10 Done Sent Done false 9913 Acked Write 2 Done Sent Done false 9914 Acked Read 4 Done Sent Done false 9915 Acked Read 6 Done Sent Done false 9916 Acked Write 2 Done Sent Done false 9917 Acked Write 1 Done Sent Done false 9918 Acked Write 10 Done Sent Done false 9919 Acked Write 6 Done Sent Done false 9920 Acked Read 5 Done Sent Done false 9921 Acked Read 9 Done Sent Done false 9922 Acked Write 5 Done Sent Done false 9923 Acked Write 10 Done Sent Done false 9924 Acked Write 3 Done Sent Done false 9925 Acked Read 5 Done Sent Done false 9926 Acked Write 6 Done Sent Done false 9927 Acked Read 8 Done Sent Done false 9928 Acked Read 4 Done Sent Done false 9929 Acked Read 6 Done Sent Done false 9930 Acked Read 6 Done Sent Done false 9931 Acked Flush 0 Done Sent Done false 9932 Acked Write 10 Done Sent Done false 9933 Acked Read 9 Done Sent Done false 9934 Acked Write 3 Done Sent Done false 9935 Acked Read 7 Done Sent Done false 9936 Acked Read 6 Done Sent Done false 9937 Acked Write 6 Done Sent Done false 9938 Acked Write 5 Done Sent Done false 9939 Acked Read 8 Done Sent Done false 9940 Acked Write 1 Done Sent Done false 9941 Acked Flush 0 Done Sent Done false 9942 Acked Flush 0 Done Sent Done false 9943 Acked Flush 0 Done Sent Done false 9944 Acked Read 4 Done Sent Done false 9945 Acked Flush 0 Done Sent Done false 9946 Acked Read 6 Done Sent Done false 9947 Acked Read 6 Done Sent Done false 9948 Acked Read 4 Done Sent Done false 9949 Acked Read 3 Done Sent Done false 9950 Acked Read 5 Done Sent Done false 9951 Acked Read 9 Done Sent Done false 9952 Acked Read 4 Done Sent Done false 9953 Acked Flush 0 Done Sent Done false 9954 Acked Write 3 Done Sent Done false 9955 Acked Flush 0 Done Sent Done false 9956 Acked Read 3 Done Sent Done false 9957 Acked Read 6 Done Sent Done false 9958 Acked Read 2 Done Sent Done false 9959 Acked Read 10 Done Sent Done false 9960 Acked Write 4 Done Sent Done false 9961 Acked Write 1 Done Sent Done false 9962 Acked Write 5 Done Sent Done false 9963 Acked Flush 0 Done Sent Done false 9964 Acked Read 1 Done Sent Done false 9965 Acked Write 9 Done Sent Done false 9966 Acked Read 2 Done Sent Done false 9967 Acked Write 5 Done Sent Done false 9968 Acked Write 7 Done Sent Done false 9969 Acked Read 9 Done Sent Done false 9970 Acked Write 2 Done Sent Done false 9971 Acked Read 4 Done Sent Done false 9972 Acked Flush 0 Done Sent Done false 9973 Acked Read 6 Done Sent Done false 9974 Acked Read 1 Done Sent Done false 9975 Acked Read 6 Done Sent Done false 9976 Acked Write 1 Done Sent Done false 9977 Acked Flush 0 Done Sent Done false 9978 Acked Read 4 Done Sent Done false 9979 Acked Write 6 Done Sent Done false 9980 Acked Write 10 Done Sent Done false 9981 Acked Read 10 Done Sent Done false 9982 Acked Write 3 Done Sent Done false 9983 Acked Write 1 Done Sent Done false 9984 Acked Read 3 Done Sent Done false 9985 Acked Flush 0 Done Sent Done false 9986 Acked Read 10 Done Sent Done false 9987 Acked Write 1 Done Sent Done false 9988 Acked Write 1 Done Sent Done false 9989 Acked Read 6 Done Sent Done false 9990 Acked Write 7 Done Sent Done false 9991 Acked Read 4 Done Sent Done false 9992 Acked Read 5 Done Sent Done false 9993 Acked Read 7 Done Sent Done false 9994 Acked Read 9 Done Sent Done false 9995 Acked Write 7 Done Sent Done false 9996 Acked Write 3 Done Sent Done false 9997 Acked Read 3 Done Sent Done false 9998 Acked Write 3 Done Sent Done false 9999 Acked Read 4 Done Sent Done false 10000 Acked Read 5 Done Sent Done false 10001 Acked Read 7 Done Sent Done false 10002 Acked Write 7 Done Sent Done false 10003 Acked Read 3 Done Sent Done false 10004 Acked Write 10 Done Sent Done false 10005 Acked Write 6 Done Sent Done false 10006 Acked Read 8 Done Sent Done false 10007 Acked Write 3 Done Sent Done false 10008 Acked Write 3 Done Sent Done false 10009 Acked Write 9 Done Sent Done false 10010 Acked Write 7 Done Sent Done false 10011 Acked Write 5 Done Sent Done false 10012 Acked Read 6 Done Sent Done false 10013 Acked Flush 0 Done Sent Done false 10014 Acked Write 3 Done Sent Done false 10015 Acked Write 6 Done Sent Done false 10016 Acked Write 6 Done Sent Done false 10017 Acked Read 1 Done Sent Done false 10018 Acked Flush 0 Done Sent Done false 10019 Acked Flush 0 Done Sent Done false 10020 Acked Read 3 Done Sent Done false 10021 Acked Write 2 Done Sent Done false 10022 Acked Write 10 Done Sent Done false 10023 Acked Write 4 Done Sent Done false 10024 Acked Read 7 Done Sent Done false 10025 Acked Write 8 Done Sent Done false 10026 Acked Write 8 Done Sent Done false 10027 Acked Write 4 Done Sent Done false 10028 Acked Read 2 Done Sent Done false 10029 Acked Write 5 Done Sent Done false 10030 Acked Read 4 Done Sent Done false 10031 Acked Flush 0 Done Sent Done false 10032 Acked Read 1 Done Sent Done false 10033 Acked Write 3 Done Sent Done false 10034 Acked Read 5 Done Sent Done false 10035 Acked Read 3 Done Sent Done false 10036 Acked Read 8 Done Sent Done false 10037 Acked Write 6 Done Sent Done false 10038 Acked Write 5 Done Sent Done false 10039 Acked Read 6 Done Sent Done false 10040 Acked Read 5 Done Sent Done false 10041 Acked Read 8 Done Sent Done false 10042 Acked Write 3 Done Sent Done false 10043 Acked Write 2 Done Sent Done false 10044 Acked Write 10 Done Sent Done false 10045 Acked Write 9 Done Sent Done false 10046 Acked Read 1 Done Sent Done false 10047 Acked Read 7 Done Sent Done false 10048 Acked Read 2 Done Sent Done false 10049 Acked Read 4 Done Sent Done false 10050 Acked Write 10 Done Sent Done false 10051 Acked Flush 0 Done Sent Done false 10052 Acked Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 301 0 301 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10051 9751 10051 Downstairs last five completed: 9751 9750 9749 9748 9747 Upstairs last five completed: 10052 10051 10050 10049 10048 CLIENT: Up:0 ds:301 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9752 Acked Read 7 Done Sent Done false 9753 Acked Read 5 Done Sent Done false 9754 Acked Flush 0 Done Sent Done false 9755 Acked Write 6 Done Sent Done false 9756 Acked Write 10 Done Sent Done false 9757 Acked Read 8 Done Sent Done false 9758 Acked Read 5 Done Sent Done false 9759 Acked Flush 0 Done Sent Done false 9760 Acked Read 8 Done Sent Done false 9761 Acked Read 4 Done Sent Done false 9762 Acked Read 4 Done Sent Done false 9763 Acked Read 3 Done Sent Done false 9764 Acked Write 3 Done Sent Done false 9765 Acked Read 5 Done Sent Done false 9766 Acked Read 6 Done Sent Done false 9767 Acked Read 10 Done Sent Done false 9768 Acked Write 6 Done Sent Done false 9769 Acked Flush 0 Done Sent Done false 9770 Acked Write 9 Done Sent Done false 9771 Acked Write 6 Done Sent Done false 9772 Acked Write 8 Done Sent Done false 9773 Acked Write 2 Done Sent Done false 9774 Acked Read 8 Done Sent Done false 9775 Acked Write 9 Done Sent Done false 9776 Acked Read 4 Done Sent Done false 9777 Acked Flush 0 Done Sent Done false 9778 Acked Read 2 Done Sent Done false 9779 Acked Read 4 Done Sent Done false 9780 Acked Write 8 Done Sent Done false 9781 Acked Read 7 Done Sent Done false 9782 Acked Read 8 Done Sent Done false 9783 Acked Flush 0 Done Sent Done false 9784 Acked Flush 0 Done Sent Done false 9785 Acked Write 9 Done Sent Done false 9786 Acked Write 5 Done Sent Done false 9787 Acked Read 8 Done Sent Done false 9788 Acked Write 6 Done Sent Done false 9789 Acked Read 3 Done Sent Done false 9790 Acked Read 1 Done Sent Done false 9791 Acked Read 7 Done Sent Done false 9792 Acked Read 7 Done Sent Done false 9793 Acked Write 7 Done Sent Done false 9794 Acked Read 5 Done Sent Done false 9795 Acked Read 10 Done Sent Done false 9796 Acked Write 2 Done Sent Done false 9797 Acked Write 10 Done Sent Done false 9798 Acked Write 4 Done Sent Done false 9799 Acked Write 2 Done Sent Done false 9800 Acked Write 9 Done Sent Done false 9801 Acked Write 7 Done Sent Done false 9802 Acked Read 7 Done Sent Done false 9803 Acked Read 4 Done Sent Done false 9804 Acked Read 5 Done Sent Done false 9805 Acked Write 7 Done Sent Done false 9806 Acked Read 8 Done Sent Done false 9807 Acked Write 4 Done Sent Done false 9808 Acked Write 9 Done Sent Done false 9809 Acked Flush 0 Done Sent Done false 9810 Acked Flush 0 Done Sent Done false 9811 Acked Write 9 Done Sent Done false 9812 Acked Write 5 Done Sent Done false 9813 Acked Read 5 Done Sent Done false 9814 Acked Read 4 Done Sent Done false 9815 Acked Read 10 Done Sent Done false 9816 Acked Write 2 Done Sent Done false 9817 Acked Write 8 Done Sent Done false 9818 Acked Read 2 Done Sent Done false 9819 Acked Read 1 Done Sent Done false 9820 Acked Read 10 Done Sent Done false 9821 Acked Flush 0 Done Sent Done false 9822 Acked Flush 0 Done Sent Done false 9823 Acked Read 1 Done Sent Done false 9824 Acked Read 5 Done Sent Done false 9825 Acked Read 3 Done Sent Done false 9826 Acked Flush 0 Done Sent Done false 9827 Acked Read 10 Done Sent Done false 9828 Acked Read 8 Done Sent Done false 9829 Acked Read 2 Done Sent Done false 9830 Acked Read 3 Done Sent Done false 9831 Acked Write 1 Done Sent Done false 9832 Acked Read 7 Done Sent Done false 9833 Acked Read 4 Done Sent Done false 9834 Acked Read 8 Done Sent Done false 9835 Acked Read 8 Done Sent Done false 9836 Acked Read 10 Done Sent Done false 9837 Acked Write 6 Done Sent Done false 9838 Acked Write 7 Done Sent Done false 9839 Acked Write 8 Done Sent Done false 9840 Acked Write 5 Done Sent Done false 9841 Acked Read 4 Done Sent Done false 9842 Acked Flush 0 Done Sent Done false 9843 Acked Read 9 Done Sent Done false 9844 Acked Read 4 Done Sent Done false 9845 Acked Flush 0 Done Sent Done false 9846 Acked Read 10 Done Sent Done false 9847 Acked Write 2 Done Sent Done false 9848 Acked Write 7 Done Sent Done false 9849 Acked Write 6 Done Sent Done false 9850 Acked Read 1 Done Sent Done false 9851 Acked Read 10 Done Sent Done false 9852 Acked Read 7 Done Sent Done false 9853 Acked Write 9 Done Sent Done false 9854 Acked Read 8 Done Sent Done false 9855 Acked Write 6 Done Sent Done false 9856 Acked Write 6 Done Sent Done false 9857 Acked Read 7 Done Sent Done false 9858 Acked Write 5 Done Sent Done false 9859 Acked Write 8 Done Sent Done false 9860 Acked Write 8 Done Sent Done false 9861 Acked Read 10 Done Sent Done false 9862 Acked Read 8 Done Sent Done false 9863 Acked Write 9 Done Sent Done false 9864 Acked Write 10 Done Sent Done false 9865 Acked Read 9 Done Sent Done false 9866 Acked Read 7 Done Sent Done false 9867 Acked Read 10 Done Sent Done false 9868 Acked Write 5 Done Sent Done false 9869 Acked Read 9 Done Sent Done false 9870 Acked Write 8 Done Sent Done false 9871 Acked Write 4 Done Sent Done false 9872 Acked Flush 0 Done Sent Done false 9873 Acked Read 9 Done Sent Done false 9874 Acked Read 5 Done Sent Done false 9875 Acked Write 7 Done Sent Done false 9876 Acked Read 7 Done Sent Done false 9877 Acked Flush 0 Done Sent Done false 9878 Acked Write 7 Done Sent Done false 9879 Acked Read 3 Done Sent Done false 9880 Acked Write 9 Done Sent Done false 9881 Acked Write 1 Done Sent Done false 9882 Acked Flush 0 Done Sent Done false 9883 Acked Read 8 Done Sent Done false 9884 Acked Write 5 Done Sent Done false 9885 Acked Write 5 Done Sent Done false 9886 Acked Read 3 Done Sent Done false 9887 Acked Read 1 Done Sent Done false 9888 Acked Write 10 Done Sent Done false 9889 Acked Write 8 Done Sent Done false 9890 Acked Write 9 Done Sent Done false 9891 Acked Read 5 Done Sent Done false 9892 Acked Write 10 Done Sent Done false 9893 Acked Read 2 Done Sent Done false 9894 Acked Read 1 Done Sent Done false 9895 Acked Read 2 Done Sent Done false 9896 Acked Read 5 Done Sent Done false 9897 Acked Write 7 Done Sent Done false 9898 Acked Read 7 Done Sent Done false 9899 Acked Flush 0 Done Sent Done false 9900 Acked Read 4 Done Sent Done false 9901 Acked Read 9 Done Sent Done false 9902 Acked Write 6 Done Sent Done false 9903 Acked Read 4 Done Sent Done false 9904 Acked Flush 0 Done Sent Done false 9905 Acked Read 4 Done Sent Done false 9906 Acked Write 10 Done Sent Done false 9907 Acked Read 8 Done Sent Done false 9908 Acked Read 6 Done Sent Done false 9909 Acked Flush 0 Done Sent Done false 9910 Acked Read 6 Done Sent Done false 9911 Acked Read 7 Done Sent Done false 9912 Acked Read 10 Done Sent Done false 9913 Acked Write 2 Done Sent Done false 9914 Acked Read 4 Done Sent Done false 9915 Acked Read 6 Done Sent Done false 9916 Acked Write 2 Done Sent Done false 9917 Acked Write 1 Done Sent Done false 9918 Acked Write 10 Done Sent Done false 9919 Acked Write 6 Done Sent Done false 9920 Acked Read 5 Done Sent Done false 9921 Acked Read 9 Done Sent Done false 9922 Acked Write 5 Done Sent Done false 9923 Acked Write 10 Done Sent Done false 9924 Acked Write 3 Done Sent Done false 9925 Acked Read 5 Done Sent Done false 9926 Acked Write 6 Done Sent Done false 9927 Acked Read 8 Done Sent Done false 9928 Acked Read 4 Done Sent Done false 9929 Acked Read 6 Done Sent Done false 9930 Acked Read 6 Done Sent Done false 9931 Acked Flush 0 Done Sent Done false 9932 Acked Write 10 Done Sent Done false 9933 Acked Read 9 Done Sent Done false 9934 Acked Write 3 Done Sent Done false 9935 Acked Read 7 Done Sent Done false 9936 Acked Read 6 Done Sent Done false 9937 Acked Write 6 Done Sent Done false 9938 Acked Write 5 Done Sent Done false 9939 Acked Read 8 Done Sent Done false 9940 Acked Write 1 Done Sent Done false 9941 Acked Flush 0 Done Sent Done false 9942 Acked Flush 0 Done Sent Done false 9943 Acked Flush 0 Done Sent Done false 9944 Acked Read 4 Done Sent Done false 9945 Acked Flush 0 Done Sent Done false 9946 Acked Read 6 Done Sent Done false 9947 Acked Read 6 Done Sent Done false 9948 Acked Read 4 Done Sent Done false 9949 Acked Read 3 Done Sent Done false 9950 Acked Read 5 Done Sent Done false 9951 Acked Read 9 Done Sent Done false 9952 Acked Read 4 Done Sent Done false 9953 Acked Flush 0 Done Sent Done false 9954 Acked Write 3 Done Sent Done false 9955 Acked Flush 0 Done Sent Done false 9956 Acked Read 3 Done Sent Done false 9957 Acked Read 6 Done Sent Done false 9958 Acked Read 2 Done Sent Done false 9959 Acked Read 10 Done Sent Done false 9960 Acked Write 4 Done Sent Done false 9961 Acked Write 1 Done Sent Done false 9962 Acked Write 5 Done Sent Done false 9963 Acked Flush 0 Done Sent Done false 9964 Acked Read 1 Done Sent Done false 9965 Acked Write 9 Done Sent Done false 9966 Acked Read 2 Done Sent Done false 9967 Acked Write 5 Done Sent Done false 9968 Acked Write 7 Done Sent Done false 9969 Acked Read 9 Done Sent Done false 9970 Acked Write 2 Done Sent Done false 9971 Acked Read 4 Done Sent Done false 9972 Acked Flush 0 Done Sent Done false 9973 Acked Read 6 Done Sent Done false 9974 Acked Read 1 Done Sent Done false 9975 Acked Read 6 Done Sent Done false 9976 Acked Write 1 Done Sent Done false 9977 Acked Flush 0 Done Sent Done false 9978 Acked Read 4 Done Sent Done false 9979 Acked Write 6 Done Sent Done false 9980 Acked Write 10 Done Sent Done false 9981 Acked Read 10 Done Sent Done false 9982 Acked Write 3 Done Sent Done false 9983 Acked Write 1 Done Sent Done false 9984 Acked Read 3 Done Sent Done false 9985 Acked Flush 0 Done Sent Done false 9986 Acked Read 10 Done Sent Done false 9987 Acked Write 1 Done Sent Done false 9988 Acked Write 1 Done Sent Done false 9989 Acked Read 6 Done Sent Done false 9990 Acked Write 7 Done Sent Done false 9991 Acked Read 4 Done Sent Done false 9992 Acked Read 5 Done Sent Done false 9993 Acked Read 7 Done Sent Done false 9994 Acked Read 9 Done Sent Done false 9995 Acked Write 7 Done Sent Done false 9996 Acked Write 3 Done Sent Done false 9997 Acked Read 3 Done Sent Done false 9998 Acked Write 3 Done Sent Done false 9999 Acked Read 4 Done Sent Done false 10000 Acked Read 5 Done Sent Done false 10001 Acked Read 7 Done Sent Done false 10002 Acked Write 7 Done Sent Done false 10003 Acked Read 3 Done Sent Done false 10004 Acked Write 10 Done Sent Done false 10005 Acked Write 6 Done Sent Done false 10006 Acked Read 8 Done Sent Done false 10007 Acked Write 3 Done Sent Done false 10008 Acked Write 3 Done Sent Done false 10009 Acked Write 9 Done Sent Done false 10010 Acked Write 7 Done Sent Done false 10011 Acked Write 5 Done Sent Done false 10012 Acked Read 6 Done Sent Done false 10013 Acked Flush 0 Done Sent Done false 10014 Acked Write 3 Done Sent Done false 10015 Acked Write 6 Done Sent Done false 10016 Acked Write 6 Done Sent Done false 10017 Acked Read 1 Done Sent Done false 10018 Acked Flush 0 Done Sent Done false 10019 Acked Flush 0 Done Sent Done false 10020 Acked Read 3 Done Sent Done false 10021 Acked Write 2 Done Sent Done false 10022 Acked Write 10 Done Sent Done false 10023 Acked Write 4 Done Sent Done false 10024 Acked Read 7 Done Sent Done false 10025 Acked Write 8 Done Sent Done false 10026 Acked Write 8 Done Sent Done false 10027 Acked Write 4 Done Sent Done false 10028 Acked Read 2 Done Sent Done false 10029 Acked Write 5 Done Sent Done false 10030 Acked Read 4 Done Sent Done false 10031 Acked Flush 0 Done Sent Done false 10032 Acked Read 1 Done Sent Done false 10033 Acked Write 3 Done Sent Done false 10034 Acked Read 5 Done Sent Done false 10035 Acked Read 3 Done Sent Done false 10036 Acked Read 8 Done Sent Done false 10037 Acked Write 6 Done Sent Done false 10038 Acked Write 5 Done Sent Done false 10039 Acked Read 6 Done Sent Done false 10040 Acked Read 5 Done Sent Done false 10041 Acked Read 8 Done Sent Done false 10042 Acked Write 3 Done Sent Done false 10043 Acked Write 2 Done Sent Done false 10044 Acked Write 10 Done Sent Done false 10045 Acked Write 9 Done Sent Done false 10046 Acked Read 1 Done Sent Done false 10047 Acked Read 7 Done Sent Done false 10048 Acked Read 2 Done Sent Done false 10049 Acked Read 4 Done Sent Done false 10050 Acked Write 10 Done Sent Done false 10051 Acked Flush 0 Done Sent Done false 10052 Acked Read 9 Done Sent Done false 10053 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 302 0 302 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10053 9751 10053 Downstairs last five completed: 9751 9750 9749 9748 9747 Upstairs last five completed: 10053 10052 10051 10050 10049 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.691227435Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:56124)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.691289958Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:28.692061731Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692574563Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692605605Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 9751","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.69262192Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 9751","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692828425Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692848869Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692862916Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"1","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] client re-new 302 jobs since flush 9751","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.692879831Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[1] Replayed 302 jobs since flush: 9751","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:28.693103191Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 10053 10052 10051 10050 10049 Upstairs last five completed: 10053 10052 10051 10050 10049 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Main task has work to do, go find it got dsc Stop(2) stop 2 Nov 27 01:49:30.932 INFO [8830] Got stop action so:false kr:false request completed, latency_us: 379, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e2a22d4c-7f71-49b4-9e20-9be7e65666e0, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:30.933094607Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:30.933166808Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:30.933192071Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:30.933212366Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:30.93323358Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} Nov 27 01:49:30.933 INFO request completed{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:30.933278728Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} , latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: 129f8032-0e43-4869-a80b-1e9213c92a65, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit Nov 27 01:49:34.935 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/2, method: GET, req_id: d871daeb-0021-4cb1-bffc-1865d0c3b63a, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 001/300 Read block 4931 len 24576 002/300 Write block 724 len 8192 data: 5 6 003/300 Write block 4409 len 32768 data: 5 6 8 7 8 11 8 6 004/300 Read block 4173 len 24576 005/300 Read block 4918 len 36864 006/300 Write block 2028 len 28672 data: 6 6 6 5 5 5 7 007/300 Read block 2827 len 32768 008/300 Write block 3613 len 12288 data: 8 9 9 009/300 Read block 1420 len 28672 010/300 Flush 011/300 Write block 2081 len 16384 data: 8 6 5 5 012/300 Write block 3122 len 28672 data: 3 3 5 6 5 5 6 013/300 Read block 4130 len 8192 014/300 Write block 1591 len 40960 data: 5 5 4 2 2 3 5 7 8 7 015/300 Write block 1107 len 16384 data: 9 12 10 9 016/300 Write block 2165 len 20480 data: 6 5 5 5 4 017/300 Write block 1965 len 20480 data: 7 7 7 7 6 018/300 Flush 019/300 Write block 1916 len 8192 data: 6 4 020/300 Read block 2726 len 20480 021/300 Write block 3277 len 28672 data: 6 4 4 4 4 4 4 022/300 Read block 923 len 24576 023/300 Read block 4408 len 16384 024/300 Read block 2664 len 40960 025/300 Read block 2095 len 16384 026/300 Read block 1029 len 4096 027/300 Read block 3245 len 4096 028/300 Write block 1081 len 12288 data: 7 8 7 029/300 Write block 2323 len 20480 data: 5 5 6 6 7 030/300 Write block 535 len 8192 data: 6 6 031/300 Read block 3702 len 20480 032/300 Read block 677 len 4096 033/300 Write block 4523 len 20480 data: 6 6 6 5 5 034/300 Write block 4090 len 40960 data: 9 7 7 8 8 8 7 7 8 6 035/300 Flush 036/300 Read block 1668 len 24576 037/300 Write block 4835 len 4096 data: 6 038/300 Read block 4709 len 24576 039/300 Write block 3658 len 16384 data: 11 10 8 8 040/300 Read block 1579 len 12288 041/300 Read block 4762 len 8192 042/300 Write block 4906 len 16384 data: 4 4 4 4 043/300 Read block 3388 len 24576 044/300 Read block 1557 len 16384 045/300 Write block 3488 len 20480 data: 6 7 10 10 8 046/300 Write block 3707 len 16384 data: 7 6 6 7 047/300 Write block 4614 len 28672 data: 3 3 3 3 3 5 4 048/300 Write block 4107 len 20480 data: 3 4 5 5 6 049/300 Read block 1592 len 20480 050/300 Write block 4800 len 8192 data: 10 9 051/300 Write block 2585 len 8192 data: 5 4 052/300 Write block 4128 len 16384 data: 9 9 8 6 053/300 Read block 1059 len 16384 054/300 Write block 3652 len 28672 data: 5 9 9 10 10 10 12 055/300 Read block 3829 len 20480 056/300 Write block 2765 len 4096 data: 6 057/300 Flush 058/300 Read block 2189 len 36864 059/300 Flush 060/300 Write block 2440 len 36864 data: 7 7 7 9 8 9 9 8 7 061/300 Flush 062/300 Flush 063/300 Read block 2687 len 16384 064/300 Write block 2567 len 4096 data: 7 065/300 Read block 4116 len 32768 066/300 Read block 1795 len 4096 067/300 Read block 312 len 12288 068/300 Read block 3873 len 20480 069/300 Write block 4574 len 12288 data: 3 5 4 070/300 Read block 4232 len 28672 071/300 Read block 880 len 40960 072/300 Read block 4368 len 24576 073/300 Read block 447 len 40960 074/300 Read block 1534 len 16384 075/300 Read block 3766 len 28672 076/300 Read block 402 len 36864 077/300 Flush 078/300 Read block 2340 len 36864 079/300 Write block 2150 len 24576 data: 5 5 4 3 4 4 080/300 Read block 654 len 16384 081/300 Read block 3789 len 12288 082/300 Write block 2786 len 36864 data: 5 6 7 8 7 6 7 6 6 083/300 Read block 2992 len 20480 084/300 Read block 798 len 24576 085/300 Write block 2133 len 8192 data: 5 6 086/300 Write block 2875 len 28672 data: 8 10 10 8 6 5 4 087/300 Write block 3941 len 28672 data: 6 6 6 6 5 6 6 088/300 Write block 801 len 4096 data: 8 089/300 Read block 4023 len 28672 090/300 Read block 1249 len 20480 091/300 Read block 3908 len 12288 092/300 Flush 093/300 Read block 4846 len 8192 094/300 Read block 1760 len 24576 095/300 Write block 319 len 12288 data: 3 3 3 096/300 Flush 097/300 Write block 559 len 8192 data: 5 4 098/300 Flush 099/300 Write block 1019 len 12288 data: 6 5 5 100/300 Read block 4673 len 36864 101/300 Read block 682 len 16384 102/300 Read block 1687 len 40960 103/300 Write block 2532 len 28672 data: 4 5 6 4 4 6 6 104/300 Read block 2038 len 32768 105/300 Write block 4354 len 20480 data: 7 7 11 10 9 106/300 Read block 2838 len 20480 107/300 Write block 946 len 4096 data: 7 108/300 Flush 109/300 Flush 110/300 Write block 3996 len 24576 data: 6 6 5 5 5 5 111/300 Read block 1562 len 40960 112/300 Read block 1176 len 16384 113/300 Read block 57 len 16384 114/300 Read block 1851 len 24576 115/300 Write block 4345 len 8192 data: 2 2 116/300 Write block 106 len 12288 data: 6 6 10 117/300 Write block 945 len 8192 data: 6 8 118/300 Read block 4910 len 32768 119/300 Write block 1999 len 16384 data: 12 11 11 9 120/300 Write block 4052 len 8192 data: 6 6 121/300 Write block 4392 len 28672 data: 8 10 9 9 9 8 8 122/300 Read block 4736 len 36864 123/300 Write block 1453 len 8192 data: 6 6 124/300 Write block 2634 len 12288 data: 5 4 6 125/300 Read block 3459 len 32768 126/300 Write block 679 len 28672 data: 10 8 8 7 7 7 7 127/300 Read block 3204 len 12288 128/300 Flush 129/300 Write block 1957 len 32768 data: 11 10 10 7 7 9 9 8 130/300 Read block 850 len 12288 131/300 Read block 2521 len 32768 132/300 Read block 49 len 40960 133/300 Write block 3564 len 32768 data: 5 4 3 3 4 5 5 5 134/300 Read block 2422 len 4096 135/300 Read block 476 len 36864 136/300 Read block 1654 len 16384 137/300 Read block 2686 len 16384 138/300 Read block 4735 len 8192 139/300 Read block 2260 len 28672 140/300 Write block 3530 len 8192 data: 5 5 141/300 Read block 916 len 12288 142/300 Read block 1671 len 40960 143/300 Flush 144/300 Read block 1718 len 16384 145/300 Read block 3754 len 32768 146/300 Read block 2741 len 36864 147/300 Write block 534 len 24576 data: 6 7 7 5 6 5 148/300 Read block 1806 len 12288 149/300 Write block 1189 len 8192 data: 7 9 150/300 Flush 151/300 Read block 4903 len 4096 152/300 Read block 1133 len 40960 153/300 Read block 4872 len 40960 154/300 Read block 3995 len 36864 155/300 Flush 156/300 Read block 2624 len 24576 157/300 Write block 4247 len 8192 data: 5 5 158/300 Write block 3138 len 28672 data: 6 5 5 5 4 4 4 159/300 Read block 828 len 36864 160/300 Write block 2086 len 32768 data: 7 5 5 5 5 4 6 5 161/300 Write block 2409 len 32768 data: 4 3 4 4 4 4 4 4 162/300 Read block 2496 len 8192 163/300 Read block 2226 len 24576 164/300 Write block 2714 len 40960 data: 6 6 7 6 6 7 6 6 6 6 165/300 Read block 4832 len 28672 166/300 Read block 4852 len 24576 167/300 Read block 1327 len 20480 168/300 Read block 450 len 20480 169/300 Write block 2859 len 8192 data: 9 7 170/300 Read block 2872 len 8192 171/300 Write block 1032 len 8192 data: 7 8 172/300 Read block 3469 len 8192 173/300 Write block 2985 len 4096 data: 4 174/300 Read block 2360 len 16384 175/300 Write block 4199 len 4096 data: 6 176/300 Write block 1032 len 16384 data: 8 9 8 4 177/300 Read block 4180 len 40960 178/300 Read block 3409 len 16384 179/300 Read block 4526 len 32768 180/300 Read block 1311 len 40960 181/300 Flush 182/300 Write block 4466 len 24576 data: 6 6 6 3 6 7 183/300 Read block 4542 len 8192 184/300 Flush 185/300 Read block 3301 len 28672 186/300 Write block 35 len 24576 data: 4 3 3 5 4 4 187/300 Write block 1984 len 4096 data: 5 188/300 Write block 3390 len 36864 data: 5 6 5 6 6 6 8 7 6 189/300 Read block 1233 len 40960 190/300 Read block 306 len 24576 191/300 Read block 4210 len 12288 192/300 Flush 193/300 Read block 4037 len 36864 194/300 Read block 211 len 24576 195/300 Write block 2270 len 40960 data: 9 10 7 8 8 8 7 6 6 7 196/300 Read block 189 len 24576 197/300 Write block 4388 len 12288 data: 10 13 12 198/300 Write block 4778 len 12288 data: 5 6 6 199/300 Read block 4003 len 16384 200/300 Flush 201/300 Read block 3932 len 8192 202/300 Write block 151 len 8192 data: 6 5 203/300 Write block 2972 len 4096 data: 9 204/300 Write block 1541 len 40960 data: 11 9 11 9 9 6 6 4 4 4 205/300 Read block 1903 len 12288 206/300 Write block 4640 len 28672 data: 5 5 6 6 6 5 6 207/300 Read block 653 len 28672 208/300 Write block 2907 len 12288 data: 4 3 3 209/300 Write block 2356 len 8192 data: 4 3 210/300 Write block 885 len 28672 data: 6 7 8 7 10 11 9 211/300 Flush 212/300 Write block 4875 len 20480 data: 4 4 3 5 5 213/300 Read block 4759 len 32768 214/300 Write block 4160 len 4096 data: 4 215/300 Read block 2420 len 12288 216/300 Read block 4798 len 28672 217/300 Read block 2152 len 32768 218/300 Write block 3062 len 20480 data: 3 3 3 3 3 219/300 Read block 167 len 24576 220/300 Read block 1977 len 36864 221/300 Read block 4664 len 16384 222/300 Write block 2011 len 4096 data: 4 223/300 Read block 3914 len 28672 224/300 Read block 4206 len 32768 225/300 Write block 3163 len 28672 data: 5 5 6 6 5 5 5 226/300 Read block 2781 len 8192 227/300 Write block 4846 len 16384 data: 9 7 4 4 228/300 Write block 4989 len 36864 data: 7 6 5 6 6 7 6 4 5 229/300 Read block 3338 len 32768 230/300 Read block 2657 len 4096 231/300 Read block 2228 len 36864 232/300 Read block 2866 len 24576 233/300 Read block 407 len 24576 234/300 Read block 4764 len 36864 235/300 Read block 1051 len 28672 236/300 Read block 1655 len 4096 237/300 Read block 4626 len 32768 238/300 Read block 322 len 4096 239/300 Read block 1305 len 40960 240/300 Write block 941 len 24576 data: 6 6 6 6 7 9 241/300 Write block 1112 len 40960 data: 8 8 8 8 8 8 9 8 6 6 242/300 Write block 4786 len 32768 data: 4 5 6 5 6 7 7 9 243/300 Read block 3250 len 28672 244/300 Flush 245/300 Write block 2272 len 12288 data: 8 9 9 246/300 Write block 4090 len 28672 data: 10 8 8 9 9 9 8 247/300 Write block 3385 len 16384 data: 7 6 6 4 248/300 Read block 4486 len 4096 249/300 Read block 3706 len 8192 250/300 Read block 4656 len 20480 251/300 Read block 4652 len 28672 252/300 Write block 2459 len 36864 data: 9 8 7 6 7 8 8 8 8 253/300 Read block 606 len 28672 254/300 Write block 73 len 16384 data: 5 5 5 5 255/300 Read block 3195 len 28672 256/300 Read block 3885 len 4096 257/300 Write block 4511 len 32768 data: 4 6 6 6 6 6 6 5 258/300 Read block 4212 len 16384 259/300 Read block 3152 len 12288 260/300 Write block 2205 len 36864 data: 4 4 4 5 5 6 5 5 5 261/300 Flush 262/300 Read block 501 len 16384 263/300 Write block 1222 len 28672 data: 8 8 7 7 8 6 6 264/300 Write block 2265 len 28672 data: 8 8 8 8 9 10 11 265/300 Write block 1335 len 36864 data: 6 6 7 7 5 6 5 5 6 266/300 Write block 4000 len 40960 data: 6 6 5 4 4 2 2 2 4 4 267/300 Read block 3411 len 40960 268/300 Read block 206 len 20480 269/300 Read block 219 len 28672 270/300 Write block 645 len 36864 data: 3 4 4 5 5 6 7 6 6 271/300 Write block 600 len 24576 data: 3 5 4 4 3 4 272/300 Read block 793 len 16384 273/300 Write block 351 len 16384 data: 5 5 5 5 274/300 Read block 315 len 8192 275/300 Read block 559 len 24576 276/300 Read block 349 len 16384 277/300 Read block 390 len 28672 278/300 Read block 329 len 24576 279/300 Flush 280/300 Read block 3928 len 16384 281/300 Read block 1468 len 40960 282/300 Read block 2956 len 24576 283/300 Write block 37 len 4096 data: 4 284/300 Read block 2907 len 16384 285/300 Write block 1804 len 16384 data: 7 7 6 8 286/300 Read block 905 len 28672 287/300 Read block 4806 len 8192 288/300 Write block 4333 len 20480 data: 8 7 7 7 5 289/300 Write block 4163 len 8192 data: 7 7 290/300 Read block 3069 len 32768 291/300 Write block 2052 len 36864 data: 8 8 7 7 9 7 6 4 4 292/300 Write block 645 len 20480 data: 4 5 5 6 6 293/300 Write block 3391 len 12288 data: 7 6 7 294/300 Write block 3452 len 8192 data: 3 3 295/300 Read block 1915 len 4096 296/300 Read block 3277 len 24576 297/300 Read block 4675 len 20480 298/300 Flush 299/300 Read block 3145 len 16384 300/300 Flush 301/300 Write block 1194 len 28672 data: 8 8 9 7 7 7 10 Main task has work to do, go find it got dsc Start(2) start 2 Nov 27 01:49:35.173 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/2, method: POST, req_id: 03dc3983-6797-416c-83db-dfca713b655e, remote_addr: 127.0.0.1:51788, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 10054 Acked Read 6 Done Done Sent false 10055 Acked Write 2 Done Done Sent false 10056 Acked Write 8 Done Done Sent false 10057 Acked Read 6 Done Done Sent false 10058 Acked Read 9 Done Done Sent false 10059 Acked Write 7 Done Done Sent false 10060 Acked Read 8 Done Done Sent false 10061 Acked Write 3 Done Done Sent false 10062 Acked Read 7 Done Done Sent false 10063 Acked Flush 0 Done Done Sent false 10064 Acked Write 4 Done Done Sent false 10065 Acked Write 7 Done Done Sent false 10066 Acked Read 2 Done Done Sent false 10067 Acked Write 10 Done Done Sent false 10068 Acked Write 4 Done Done Sent false 10069 Acked Write 5 Done Done Sent false 10070 Acked Write 5 Done Done Sent false 10071 Acked Flush 0 Done Done Sent false 10072 Acked Write 2 Done Done Sent false 10073 Acked Read 5 Done Done Sent false 10074 Acked Write 7 Done Done Sent false 10075 Acked Read 6 Done Done Sent false 10076 Acked Read 4 Done Done Sent false 10077 Acked Read 10 Done Done Sent false 10078 Acked Read 4 Done Done Sent false 10079 Acked Read 1 Done Done Sent false 10080 Acked Read 1 Done Done Sent false 10081 Acked Write 3 Done Done Sent false 10082 Acked Write 5 Done Done Sent false 10083 Acked Write 2 Done Done Sent false 10084 Acked Read 5 Done Done Sent false 10085 Acked Read 1 Done Done Sent false 10086 Acked Write 5 Done Done Sent false 10087 Acked Write 10 Done Done Sent false 10088 Acked Flush 0 Done Done Sent false 10089 Acked Read 6 Done Done Sent false 10090 Acked Write 1 Done Done Sent false 10091 Acked Read 6 Done Done Sent false 10092 Acked Write 4 Done Done Sent false 10093 Acked Read 3 Done Done Sent false 10094 Acked Read 2 Done Done Sent false 10095 Acked Write 4 Done Done Sent false 10096 Acked Read 6 Done Done Sent false 10097 Acked Read 4 Done Done Sent false 10098 Acked Write 5 Done Done Sent false 10099 Acked Write 4 Done Done Sent false 10100 Acked Write 7 Done Done Sent false 10101 Acked Write 5 Done Done Sent false 10102 Acked Read 5 Done Done Sent false 10103 Acked Write 2 Done Done Sent false 10104 Acked Write 2 Done Done Sent false 10105 Acked Write 4 Done Done Sent false 10106 Acked Read 4 Done Done Sent false 10107 Acked Write 7 Done Done Sent false 10108 Acked Read 5 Done Done Sent false 10109 Acked Write 1 Done Done Sent false 10110 Acked Flush 0 Done Done Sent false 10111 Acked Read 9 Done Done Sent false 10112 Acked Flush 0 Done Done Sent false 10113 Acked Write 9 Done Done Sent false 10114 Acked Flush 0 Done Done Sent false 10115 Acked Flush 0 Done Done Sent false 10116 Acked Read 4 Done Done Sent false 10117 Acked Write 1 Done Done Sent false 10118 Acked Read 8 Done Done Sent false 10119 Acked Read 1 Done Done Sent false 10120 Acked Read 3 Done Done Sent false 10121 Acked Read 5 Done Done Sent false 10122 Acked Write 3 Done Done Sent false 10123 Acked Read 7 Done Done Sent false 10124 Acked Read 10 Done Done Sent false 10125 Acked Read 6 Done Done Sent false 10126 Acked Read 10 Done Done Sent false 10127 Acked Read 4 Done Done Sent false 10128 Acked Read 7 Done Done Sent false 10129 Acked Read 9 Done Done Sent false 10130 Acked Flush 0 Done Done Sent false 10131 Acked Read 9 Done Done Sent false 10132 Acked Write 6 Done Done Sent false 10133 Acked Read 4 Done Done Sent false 10134 Acked Read 3 Done Done Sent false 10135 Acked Write 9 Done Done Sent false 10136 Acked Read 5 Done Done Sent false 10137 Acked Read 6 Done Done Sent false 10138 Acked Write 2 Done Done Sent false 10139 Acked Write 7 Done Done Sent false 10140 Acked Write 7 Done Done Sent false 10141 Acked Write 1 Done Done Sent false 10142 Acked Read 7 Done Done Sent false 10143 Acked Read 5 Done Done Sent false 10144 Acked Read 3 Done Done Sent false 10145 Acked Flush 0 Done Done Sent false 10146 Acked Read 2 Done Done Sent false 10147 Acked Read 6 Done Done Sent false 10148 Acked Write 3 Done Done Sent false 10149 Acked Flush 0 Done Done Sent false 10150 Acked Write 2 Done Done Sent false 10151 Acked Flush 0 Done Done Sent false 10152 Acked Write 3 Done Done Sent false 10153 Acked Read 9 Done Done Sent false 10154 Acked Read 4 Done Done Sent false 10155 Acked Read 10 Done Done Sent false 10156 Acked Write 7 Done Done Sent false 10157 Acked Read 8 Done Done Sent false 10158 Acked Write 5 Done Done Sent false 10159 Acked Read 5 Done Done Sent false 10160 Acked Write 1 Done Done Sent false 10161 Acked Flush 0 Done Done Sent false 10162 Acked Flush 0 Done Done Sent false 10163 Acked Write 6 Done Done Sent false 10164 Acked Read 10 Done Done Sent false 10165 Acked Read 4 Done Done Sent false 10166 Acked Read 4 Done Done Sent false 10167 Acked Read 6 Done Done Sent false 10168 Acked Write 2 Done Done Sent false 10169 Acked Write 3 Done Done Sent false 10170 Acked Write 2 Done Done Sent false 10171 Acked Read 8 Done Done Sent false 10172 Acked Write 4 Done Done Sent false 10173 Acked Write 2 Done Done Sent false 10174 Acked Write 7 Done Done Sent false 10175 Acked Read 9 Done Done Sent false 10176 Acked Write 2 Done Done Sent false 10177 Acked Write 3 Done Done Sent false 10178 Acked Read 8 Done Done Sent false 10179 Acked Write 7 Done Done Sent false 10180 Acked Read 3 Done Done Sent false 10181 Acked Flush 0 Done Done Sent false 10182 Acked Write 8 Done Done Sent false 10183 Acked Read 3 Done Done Sent false 10184 Acked Read 8 Done Done Sent false 10185 Acked Read 10 Done Done Sent false 10186 Acked Write 8 Done Done Sent false 10187 Acked Read 1 Done Done Sent false 10188 Acked Read 9 Done Done Sent false 10189 Acked Read 4 Done Done Sent false 10190 Acked Read 4 Done Done Sent false 10191 Acked Read 2 Done Done Sent false 10192 Acked Read 7 Done Done Sent false 10193 Acked Write 2 Done Done Sent false 10194 Acked Read 3 Done Done Sent false 10195 Acked Read 10 Done Done Sent false 10196 Acked Flush 0 Done Done Sent false 10197 Acked Read 4 Done Done Sent false 10198 Acked Read 8 Done Done Sent false 10199 Acked Read 9 Done Done Sent false 10200 Acked Write 6 Done Done Sent false 10201 Acked Read 3 Done Done Sent false 10202 Acked Write 2 Done Done Sent false 10203 Acked Flush 0 Done Done Sent false 10204 Acked Read 1 Done Done Sent false 10205 Acked Read 10 Done Done Sent false 10206 Acked Read 10 Done Done Sent false 10207 Acked Read 9 Done Done Sent false 10208 Acked Flush 0 Done Done Sent false 10209 Acked Read 6 Done Done Sent false 10210 Acked Write 2 Done Done Sent false 10211 Acked Write 7 Done Done Sent false 10212 Acked Read 9 Done Done Sent false 10213 Acked Write 8 Done Done Sent false 10214 Acked Write 8 Done Done Sent false 10215 Acked Read 2 Done Done Sent false 10216 Acked Read 6 Done Done Sent false 10217 Acked Write 10 Done Done Sent false 10218 Acked Read 7 Done Done Sent false 10219 Acked Read 6 Done Done Sent false 10220 Acked Read 5 Done Done Sent false 10221 Acked Read 5 Done Done Sent false 10222 Acked Write 2 Done Done Sent false 10223 Acked Read 2 Done Done Sent false 10224 Acked Write 2 Done Done Sent false 10225 Acked Read 2 Done Done Sent false 10226 Acked Write 1 Done Done Sent false 10227 Acked Read 4 Done Done Sent false 10228 Acked Write 1 Done Done Sent false 10229 Acked Write 4 Done Done Sent false 10230 Acked Read 10 Done Done Sent false 10231 Acked Read 4 Done Done Sent false 10232 Acked Read 8 Done Done Sent false 10233 Acked Read 10 Done Done Sent false 10234 Acked Flush 0 Done Done Sent false 10235 Acked Write 6 Done Done Sent false 10236 Acked Read 2 Done Done Sent false 10237 Acked Flush 0 Done Done Sent false 10238 Acked Read 7 Done Done Sent false 10239 Acked Write 6 Done Done Sent false 10240 Acked Write 1 Done Done Sent false 10241 Acked Write 9 Done Done Sent false 10242 Acked Read 10 Done Done Sent false 10243 Acked Read 6 Done Done Sent false 10244 Acked Read 3 Done Done Sent false 10245 Acked Flush 0 Done Done Sent false 10246 Acked Read 9 Done Done Sent false 10247 Acked Read 6 Done Done Sent false 10248 Acked Write 10 Done Done Sent false 10249 Acked Read 6 Done Done Sent false 10250 Acked Write 3 Done Done Sent false 10251 Acked Write 3 Done Done Sent false 10252 Acked Read 4 Done Done Sent false 10253 Acked Flush 0 Done Done Sent false 10254 Acked Read 2 Done Done Sent false 10255 Acked Write 2 Done Done Sent false 10256 Acked Write 1 Done Done Sent false 10257 Acked Write 10 Done Done Sent false 10258 Acked Read 3 Done Done Sent false 10259 Acked Write 7 Done Done Sent false 10260 Acked Read 7 Done Done Sent false 10261 Acked Write 3 Done Done Sent false 10262 Acked Write 2 Done Done Sent false 10263 Acked Write 7 Done Done Sent false 10264 Acked Flush 0 Done Done Sent false 10265 Acked Write 5 Done Done Sent false 10266 Acked Read 8 Done Done Sent false 10267 Acked Write 1 Done Done Sent false 10268 Acked Read 3 Done Done Sent false 10269 Acked Read 7 Done Done Sent false 10270 Acked Read 8 Done Done Sent false 10271 Acked Write 5 Done Done Sent false 10272 Acked Read 6 Done Done Sent false 10273 Acked Read 9 Done Done Sent false 10274 Acked Read 4 Done Done Sent false 10275 Acked Write 1 Done Done Sent false 10276 Acked Read 7 Done Done Sent false 10277 Acked Read 8 Done Done Sent false 10278 Acked Write 7 Done Done Sent false 10279 Acked Read 2 Done Done Sent false 10280 Acked Write 4 Done Done Sent false 10281 Acked Write 9 Done Done Sent false 10282 Acked Read 8 Done Done Sent false 10283 Acked Read 1 Done Done Sent false 10284 Acked Read 9 Done Done Sent false 10285 Acked Read 6 Done Done Sent false 10286 Acked Read 6 Done Done Sent false 10287 Acked Read 9 Done Done Sent false 10288 Acked Read 7 Done Done Sent false 10289 Acked Read 1 Done Done Sent false 10290 Acked Read 8 Done Done Sent false 10291 Acked Read 1 Done Done Sent false 10292 Acked Read 10 Done Done Sent false 10293 Acked Write 6 Done Done Sent false 10294 Acked Write 10 Done Done Sent false 10295 Acked Write 8 Done Done Sent false 10296 Acked Read 7 Done Done Sent false 10297 Acked Flush 0 Done Done Sent false 10298 Acked Write 3 Done Done Sent false 10299 Acked Write 7 Done Done Sent false 10300 Acked Write 4 Done Done Sent false 10301 Acked Read 1 Done Done Sent false 10302 Acked Read 2 Done Done Sent false 10303 Acked Read 5 Done Done Sent false 10304 Acked Read 7 Done Done Sent false 10305 Acked Write 9 Done Done Sent false 10306 Acked Read 7 Done Done Sent false 10307 Acked Write 4 Done Done Sent false 10308 Acked Read 7 Done Done Sent false 10309 Acked Read 1 Done Done Sent false 10310 Acked Write 8 Done Done Sent false 10311 Acked Read 4 Done Done Sent false 10312 Acked Read 3 Done Done Sent false 10313 Acked Write 9 Done Done Sent false 10314 Acked Flush 0 Done Done Sent false 10315 Acked Read 4 Done Done Sent false 10316 Acked Write 7 Done Done Sent false 10317 Acked Write 7 Done Done Sent false 10318 Acked Write 9 Done Done Sent false 10319 Acked Write 10 Done Done Sent false 10320 Acked Read 10 Done Done Sent false 10321 Acked Read 5 Done Done Sent false 10322 Acked Read 7 Done Done Sent false 10323 Acked Write 9 Done Done Sent false 10324 Acked Write 6 Done Done Sent false 10325 Acked Read 4 Done Done Sent false 10326 Acked Write 4 Done Done Sent false 10327 Acked Read 2 Done Done Sent false 10328 Acked Read 6 Done Done Sent false 10329 Acked Read 4 Done Done Sent false 10330 Acked Read 7 Done Done Sent false 10331 Acked Read 6 Done Done Sent false 10332 Acked Flush 0 Done Done Sent false 10333 Acked Read 4 Done Done Sent false 10334 Acked Read 10 Done Done Sent false 10335 Acked Read 6 Done Done Sent false 10336 Acked Write 1 Done Done Sent false 10337 Acked Read 4 Done Done Sent false 10338 Acked Write 4 Done Done Sent false 10339 Acked Read 7 Done Done Sent false 10340 Acked Read 2 Done Done Sent false 10341 Acked Write 5 Done Done Sent false 10342 Acked Write 2 Done Done Sent false 10343 Acked Read 8 Done Done Sent false 10344 Acked Write 9 Done Done Sent false 10345 Acked Write 5 Done Done Sent false 10346 Acked Write 3 Done Done Sent false 10347 Acked Write 2 Done Done Sent false 10348 Acked Read 1 Done Done Sent false 10349 Acked Read 6 Done Done Sent false 10350 Acked Read 5 Done Done Sent false 10351 Acked Flush 0 Done Done Sent false 10352 Acked Read 4 Done Done Sent false 10353 Acked Flush 0 Done Done Sent false 10354 Acked Write 7 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 301 301 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10353 10353 10053 Downstairs last five completed: 10053 10052 10051 10050 10049 Upstairs last five completed: 10354 10353 10352 10351 10350 CLIENT: Up:0 ds:301 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 10054 Acked Read 6 Done Done Sent false 10055 Acked Write 2 Done Done Sent false 10056 Acked Write 8 Done Done Sent false 10057 Acked Read 6 Done Done Sent false 10058 Acked Read 9 Done Done Sent false 10059 Acked Write 7 Done Done Sent false 10060 Acked Read 8 Done Done Sent false 10061 Acked Write 3 Done Done Sent false 10062 Acked Read 7 Done Done Sent false 10063 Acked Flush 0 Done Done Sent false 10064 Acked Write 4 Done Done Sent false 10065 Acked Write 7 Done Done Sent false 10066 Acked Read 2 Done Done Sent false 10067 Acked Write 10 Done Done Sent false 10068 Acked Write 4 Done Done Sent false 10069 Acked Write 5 Done Done Sent false 10070 Acked Write 5 Done Done Sent false 10071 Acked Flush 0 Done Done Sent false 10072 Acked Write 2 Done Done Sent false 10073 Acked Read 5 Done Done Sent false 10074 Acked Write 7 Done Done Sent false 10075 Acked Read 6 Done Done Sent false 10076 Acked Read 4 Done Done Sent false 10077 Acked Read 10 Done Done Sent false 10078 Acked Read 4 Done Done Sent false 10079 Acked Read 1 Done Done Sent false 10080 Acked Read 1 Done Done Sent false 10081 Acked Write 3 Done Done Sent false 10082 Acked Write 5 Done Done Sent false 10083 Acked Write 2 Done Done Sent false 10084 Acked Read 5 Done Done Sent false 10085 Acked Read 1 Done Done Sent false 10086 Acked Write 5 Done Done Sent false 10087 Acked Write 10 Done Done Sent false 10088 Acked Flush 0 Done Done Sent false 10089 Acked Read 6 Done Done Sent false 10090 Acked Write 1 Done Done Sent false 10091 Acked Read 6 Done Done Sent false 10092 Acked Write 4 Done Done Sent false 10093 Acked Read 3 Done Done Sent false 10094 Acked Read 2 Done Done Sent false 10095 Acked Write 4 Done Done Sent false 10096 Acked Read 6 Done Done Sent false 10097 Acked Read 4 Done Done Sent false 10098 Acked Write 5 Done Done Sent false 10099 Acked Write 4 Done Done Sent false 10100 Acked Write 7 Done Done Sent false 10101 Acked Write 5 Done Done Sent false 10102 Acked Read 5 Done Done Sent false 10103 Acked Write 2 Done Done Sent false 10104 Acked Write 2 Done Done Sent false 10105 Acked Write 4 Done Done Sent false 10106 Acked Read 4 Done Done Sent false 10107 Acked Write 7 Done Done Sent false 10108 Acked Read 5 Done Done Sent false 10109 Acked Write 1 Done Done Sent false 10110 Acked Flush 0 Done Done Sent false 10111 Acked Read 9 Done Done Sent false 10112 Acked Flush 0 Done Done Sent false 10113 Acked Write 9 Done Done Sent false 10114 Acked Flush 0 Done Done Sent false 10115 Acked Flush 0 Done Done Sent false 10116 Acked Read 4 Done Done Sent false 10117 Acked Write 1 Done Done Sent false 10118 Acked Read 8 Done Done Sent false 10119 Acked Read 1 Done Done Sent false 10120 Acked Read 3 Done Done Sent false 10121 Acked Read 5 Done Done Sent false 10122 Acked Write 3 Done Done Sent false 10123 Acked Read 7 Done Done Sent false 10124 Acked Read 10 Done Done Sent false 10125 Acked Read 6 Done Done Sent false 10126 Acked Read 10 Done Done Sent false 10127 Acked Read 4 Done Done Sent false 10128 Acked Read 7 Done Done Sent false 10129 Acked Read 9 Done Done Sent false 10130 Acked Flush 0 Done Done Sent false 10131 Acked Read 9 Done Done Sent false 10132 Acked Write 6 Done Done Sent false 10133 Acked Read 4 Done Done Sent false 10134 Acked Read 3 Done Done Sent false 10135 Acked Write 9 Done Done Sent false 10136 Acked Read 5 Done Done Sent false 10137 Acked Read 6 Done Done Sent false 10138 Acked Write 2 Done Done Sent false 10139 Acked Write 7 Done Done Sent false 10140 Acked Write 7 Done Done Sent false 10141 Acked Write 1 Done Done Sent false 10142 Acked Read 7 Done Done Sent false 10143 Acked Read 5 Done Done Sent false 10144 Acked Read 3 Done Done Sent false 10145 Acked Flush 0 Done Done Sent false 10146 Acked Read 2 Done Done Sent false 10147 Acked Read 6 Done Done Sent false 10148 Acked Write 3 Done Done Sent false 10149 Acked Flush 0 Done Done Sent false 10150 Acked Write 2 Done Done Sent false 10151 Acked Flush 0 Done Done Sent false 10152 Acked Write 3 Done Done Sent false 10153 Acked Read 9 Done Done Sent false 10154 Acked Read 4 Done Done Sent false 10155 Acked Read 10 Done Done Sent false 10156 Acked Write 7 Done Done Sent false 10157 Acked Read 8 Done Done Sent false 10158 Acked Write 5 Done Done Sent false 10159 Acked Read 5 Done Done Sent false 10160 Acked Write 1 Done Done Sent false 10161 Acked Flush 0 Done Done Sent false 10162 Acked Flush 0 Done Done Sent false 10163 Acked Write 6 Done Done Sent false 10164 Acked Read 10 Done Done Sent false 10165 Acked Read 4 Done Done Sent false 10166 Acked Read 4 Done Done Sent false 10167 Acked Read 6 Done Done Sent false 10168 Acked Write 2 Done Done Sent false 10169 Acked Write 3 Done Done Sent false 10170 Acked Write 2 Done Done Sent false 10171 Acked Read 8 Done Done Sent false 10172 Acked Write 4 Done Done Sent false 10173 Acked Write 2 Done Done Sent false 10174 Acked Write 7 Done Done Sent false 10175 Acked Read 9 Done Done Sent false 10176 Acked Write 2 Done Done Sent false 10177 Acked Write 3 Done Done Sent false 10178 Acked Read 8 Done Done Sent false 10179 Acked Write 7 Done Done Sent false 10180 Acked Read 3 Done Done Sent false 10181 Acked Flush 0 Done Done Sent false 10182 Acked Write 8 Done Done Sent false 10183 Acked Read 3 Done Done Sent false 10184 Acked Read 8 Done Done Sent false 10185 Acked Read 10 Done Done Sent false 10186 Acked Write 8 Done Done Sent false 10187 Acked Read 1 Done Done Sent false 10188 Acked Read 9 Done Done Sent false 10189 Acked Read 4 Done Done Sent false 10190 Acked Read 4 Done Done Sent false 10191 Acked Read 2 Done Done Sent false 10192 Acked Read 7 Done Done Sent false 10193 Acked Write 2 Done Done Sent false 10194 Acked Read 3 Done Done Sent false 10195 Acked Read 10 Done Done Sent false 10196 Acked Flush 0 Done Done Sent false 10197 Acked Read 4 Done Done Sent false 10198 Acked Read 8 Done Done Sent false 10199 Acked Read 9 Done Done Sent false 10200 Acked Write 6 Done Done Sent false 10201 Acked Read 3 Done Done Sent false 10202 Acked Write 2 Done Done Sent false 10203 Acked Flush 0 Done Done Sent false 10204 Acked Read 1 Done Done Sent false 10205 Acked Read 10 Done Done Sent false 10206 Acked Read 10 Done Done Sent false 10207 Acked Read 9 Done Done Sent false 10208 Acked Flush 0 Done Done Sent false 10209 Acked Read 6 Done Done Sent false 10210 Acked Write 2 Done Done Sent false 10211 Acked Write 7 Done Done Sent false 10212 Acked Read 9 Done Done Sent false 10213 Acked Write 8 Done Done Sent false 10214 Acked Write 8 Done Done Sent false 10215 Acked Read 2 Done Done Sent false 10216 Acked Read 6 Done Done Sent false 10217 Acked Write 10 Done Done Sent false 10218 Acked Read 7 Done Done Sent false 10219 Acked Read 6 Done Done Sent false 10220 Acked Read 5 Done Done Sent false 10221 Acked Read 5 Done Done Sent false 10222 Acked Write 2 Done Done Sent false 10223 Acked Read 2 Done Done Sent false 10224 Acked Write 2 Done Done Sent false 10225 Acked Read 2 Done Done Sent false 10226 Acked Write 1 Done Done Sent false 10227 Acked Read 4 Done Done Sent false 10228 Acked Write 1 Done Done Sent false 10229 Acked Write 4 Done Done Sent false 10230 Acked Read 10 Done Done Sent false 10231 Acked Read 4 Done Done Sent false 10232 Acked Read 8 Done Done Sent false 10233 Acked Read 10 Done Done Sent false 10234 Acked Flush 0 Done Done Sent false 10235 Acked Write 6 Done Done Sent false 10236 Acked Read 2 Done Done Sent false 10237 Acked Flush 0 Done Done Sent false 10238 Acked Read 7 Done Done Sent false 10239 Acked Write 6 Done Done Sent false 10240 Acked Write 1 Done Done Sent false 10241 Acked Write 9 Done Done Sent false 10242 Acked Read 10 Done Done Sent false 10243 Acked Read 6 Done Done Sent false 10244 Acked Read 3 Done Done Sent false 10245 Acked Flush 0 Done Done Sent false 10246 Acked Read 9 Done Done Sent false 10247 Acked Read 6 Done Done Sent false 10248 Acked Write 10 Done Done Sent false 10249 Acked Read 6 Done Done Sent false 10250 Acked Write 3 Done Done Sent false 10251 Acked Write 3 Done Done Sent false 10252 Acked Read 4 Done Done Sent false 10253 Acked Flush 0 Done Done Sent false 10254 Acked Read 2 Done Done Sent false 10255 Acked Write 2 Done Done Sent false 10256 Acked Write 1 Done Done Sent false 10257 Acked Write 10 Done Done Sent false 10258 Acked Read 3 Done Done Sent false 10259 Acked Write 7 Done Done Sent false 10260 Acked Read 7 Done Done Sent false 10261 Acked Write 3 Done Done Sent false 10262 Acked Write 2 Done Done Sent false 10263 Acked Write 7 Done Done Sent false 10264 Acked Flush 0 Done Done Sent false 10265 Acked Write 5 Done Done Sent false 10266 Acked Read 8 Done Done Sent false 10267 Acked Write 1 Done Done Sent false 10268 Acked Read 3 Done Done Sent false 10269 Acked Read 7 Done Done Sent false 10270 Acked Read 8 Done Done Sent false 10271 Acked Write 5 Done Done Sent false 10272 Acked Read 6 Done Done Sent false 10273 Acked Read 9 Done Done Sent false 10274 Acked Read 4 Done Done Sent false 10275 Acked Write 1 Done Done Sent false 10276 Acked Read 7 Done Done Sent false 10277 Acked Read 8 Done Done Sent false 10278 Acked Write 7 Done Done Sent false 10279 Acked Read 2 Done Done Sent false 10280 Acked Write 4 Done Done Sent false 10281 Acked Write 9 Done Done Sent false 10282 Acked Read 8 Done Done Sent false 10283 Acked Read 1 Done Done Sent false 10284 Acked Read 9 Done Done Sent false 10285 Acked Read 6 Done Done Sent false 10286 Acked Read 6 Done Done Sent false 10287 Acked Read 9 Done Done Sent false 10288 Acked Read 7 Done Done Sent false 10289 Acked Read 1 Done Done Sent false 10290 Acked Read 8 Done Done Sent false 10291 Acked Read 1 Done Done Sent false 10292 Acked Read 10 Done Done Sent false 10293 Acked Write 6 Done Done Sent false 10294 Acked Write 10 Done Done Sent false 10295 Acked Write 8 Done Done Sent false 10296 Acked Read 7 Done Done Sent false 10297 Acked Flush 0 Done Done Sent false 10298 Acked Write 3 Done Done Sent false 10299 Acked Write 7 Done Done Sent false 10300 Acked Write 4 Done Done Sent false 10301 Acked Read 1 Done Done Sent false 10302 Acked Read 2 Done Done Sent false 10303 Acked Read 5 Done Done Sent false 10304 Acked Read 7 Done Done Sent false 10305 Acked Write 9 Done Done Sent false 10306 Acked Read 7 Done Done Sent false 10307 Acked Write 4 Done Done Sent false 10308 Acked Read 7 Done Done Sent false 10309 Acked Read 1 Done Done Sent false 10310 Acked Write 8 Done Done Sent false 10311 Acked Read 4 Done Done Sent false 10312 Acked Read 3 Done Done Sent false 10313 Acked Write 9 Done Done Sent false 10314 Acked Flush 0 Done Done Sent false 10315 Acked Read 4 Done Done Sent false 10316 Acked Write 7 Done Done Sent false 10317 Acked Write 7 Done Done Sent false 10318 Acked Write 9 Done Done Sent false 10319 Acked Write 10 Done Done Sent false 10320 Acked Read 10 Done Done Sent false 10321 Acked Read 5 Done Done Sent false 10322 Acked Read 7 Done Done Sent false 10323 Acked Write 9 Done Done Sent false 10324 Acked Write 6 Done Done Sent false 10325 Acked Read 4 Done Done Sent false 10326 Acked Write 4 Done Done Sent false 10327 Acked Read 2 Done Done Sent false 10328 Acked Read 6 Done Done Sent false 10329 Acked Read 4 Done Done Sent false 10330 Acked Read 7 Done Done Sent false 10331 Acked Read 6 Done Done Sent false 10332 Acked Flush 0 Done Done Sent false 10333 Acked Read 4 Done Done Sent false 10334 Acked Read 10 Done Done Sent false 10335 Acked Read 6 Done Done Sent false 10336 Acked Write 1 Done Done Sent false 10337 Acked Read 4 Done Done Sent false 10338 Acked Write 4 Done Done Sent false 10339 Acked Read 7 Done Done Sent false 10340 Acked Read 2 Done Done Sent false 10341 Acked Write 5 Done Done Sent false 10342 Acked Write 2 Done Done Sent false 10343 Acked Read 8 Done Done Sent false 10344 Acked Write 9 Done Done Sent false 10345 Acked Write 5 Done Done Sent false 10346 Acked Write 3 Done Done Sent false 10347 Acked Write 2 Done Done Sent false 10348 Acked Read 1 Done Done Sent false 10349 Acked Read 6 Done Done Sent false 10350 Acked Read 5 Done Done Sent false 10351 Acked Flush 0 Done Done Sent false 10352 Acked Read 4 Done Done Sent false 10353 Acked Flush 0 Done Done Sent false 10354 Acked Write 7 Done Done Sent false 10355 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10355 10355 10053 Downstairs last five completed: 10053 10052 10051 10050 10049 Upstairs last five completed: 10355 10354 10353 10352 10351 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:40.934195342Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:40.934258335Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:40.934281249Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:40.934301703Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:40.934318149Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1233) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 10054 Acked Read 6 Done Done Sent false 10055 Acked Write 2 Done Done Sent false 10056 Acked Write 8 Done Done Sent false 10057 Acked Read 6 Done Done Sent false 10058 Acked Read 9 Done Done Sent false 10059 Acked Write 7 Done Done Sent false 10060 Acked Read 8 Done Done Sent false 10061 Acked Write 3 Done Done Sent false 10062 Acked Read 7 Done Done Sent false 10063 Acked Flush 0 Done Done Sent false 10064 Acked Write 4 Done Done Sent false 10065 Acked Write 7 Done Done Sent false 10066 Acked Read 2 Done Done Sent false 10067 Acked Write 10 Done Done Sent false 10068 Acked Write 4 Done Done Sent false 10069 Acked Write 5 Done Done Sent false 10070 Acked Write 5 Done Done Sent false 10071 Acked Flush 0 Done Done Sent false 10072 Acked Write 2 Done Done Sent false 10073 Acked Read 5 Done Done Sent false 10074 Acked Write 7 Done Done Sent false 10075 Acked Read 6 Done Done Sent false 10076 Acked Read 4 Done Done Sent false 10077 Acked Read 10 Done Done Sent false 10078 Acked Read 4 Done Done Sent false 10079 Acked Read 1 Done Done Sent false 10080 Acked Read 1 Done Done Sent false 10081 Acked Write 3 Done Done Sent false 10082 Acked Write 5 Done Done Sent false 10083 Acked Write 2 Done Done Sent false 10084 Acked Read 5 Done Done Sent false 10085 Acked Read 1 Done Done Sent false 10086 Acked Write 5 Done Done Sent false 10087 Acked Write 10 Done Done Sent false 10088 Acked Flush 0 Done Done Sent false 10089 Acked Read 6 Done Done Sent false 10090 Acked Write 1 Done Done Sent false 10091 Acked Read 6 Done Done Sent false 10092 Acked Write 4 Done Done Sent false 10093 Acked Read 3 Done Done Sent false 10094 Acked Read 2 Done Done Sent false 10095 Acked Write 4 Done Done Sent false 10096 Acked Read 6 Done Done Sent false 10097 Acked Read 4 Done Done Sent false 10098 Acked Write 5 Done Done Sent false 10099 Acked Write 4 Done Done Sent false 10100 Acked Write 7 Done Done Sent false 10101 Acked Write 5 Done Done Sent false 10102 Acked Read 5 Done Done Sent false 10103 Acked Write 2 Done Done Sent false 10104 Acked Write 2 Done Done Sent false 10105 Acked Write 4 Done Done Sent false 10106 Acked Read 4 Done Done Sent false 10107 Acked Write 7 Done Done Sent false 10108 Acked Read 5 Done Done Sent false 10109 Acked Write 1 Done Done Sent false 10110 Acked Flush 0 Done Done Sent false 10111 Acked Read 9 Done Done Sent false 10112 Acked Flush 0 Done Done Sent false 10113 Acked Write 9 Done Done Sent false 10114 Acked Flush 0 Done Done Sent false 10115 Acked Flush 0 Done Done Sent false 10116 Acked Read 4 Done Done Sent false 10117 Acked Write 1 Done Done Sent false 10118 Acked Read 8 Done Done Sent false 10119 Acked Read 1 Done Done Sent false 10120 Acked Read 3 Done Done Sent false 10121 Acked Read 5 Done Done Sent false 10122 Acked Write 3 Done Done Sent false 10123 Acked Read 7 Done Done Sent false 10124 Acked Read 10 Done Done Sent false 10125 Acked Read 6 Done Done Sent false 10126 Acked Read 10 Done Done Sent false 10127 Acked Read 4 Done Done Sent false 10128 Acked Read 7 Done Done Sent false 10129 Acked Read 9 Done Done Sent false 10130 Acked Flush 0 Done Done Sent false 10131 Acked Read 9 Done Done Sent false 10132 Acked Write 6 Done Done Sent false 10133 Acked Read 4 Done Done Sent false 10134 Acked Read 3 Done Done Sent false 10135 Acked Write 9 Done Done Sent false 10136 Acked Read 5 Done Done Sent false 10137 Acked Read 6 Done Done Sent false 10138 Acked Write 2 Done Done Sent false 10139 Acked Write 7 Done Done Sent false 10140 Acked Write 7 Done Done Sent false 10141 Acked Write 1 Done Done Sent false 10142 Acked Read 7 Done Done Sent false 10143 Acked Read 5 Done Done Sent false 10144 Acked Read 3 Done Done Sent false 10145 Acked Flush 0 Done Done Sent false 10146 Acked Read 2 Done Done Sent false 10147 Acked Read 6 Done Done Sent false 10148 Acked Write 3 Done Done Sent false 10149 Acked Flush 0 Done Done Sent false 10150 Acked Write 2 Done Done Sent false 10151 Acked Flush 0 Done Done Sent false 10152 Acked Write 3 Done Done Sent false 10153 Acked Read 9 Done Done Sent false 10154 Acked Read 4 Done Done Sent false 10155 Acked Read 10 Done Done Sent false 10156 Acked Write 7 Done Done Sent false 10157 Acked Read 8 Done Done Sent false 10158 Acked Write 5 Done Done Sent false 10159 Acked Read 5 Done Done Sent false 10160 Acked Write 1 Done Done Sent false 10161 Acked Flush 0 Done Done Sent false 10162 Acked Flush 0 Done Done Sent false 10163 Acked Write 6 Done Done Sent false 10164 Acked Read 10 Done Done Sent false 10165 Acked Read 4 Done Done Sent false 10166 Acked Read 4 Done Done Sent false 10167 Acked Read 6 Done Done Sent false 10168 Acked Write 2 Done Done Sent false 10169 Acked Write 3 Done Done Sent false 10170 Acked Write 2 Done Done Sent false 10171 Acked Read 8 Done Done Sent false 10172 Acked Write 4 Done Done Sent false 10173 Acked Write 2 Done Done Sent false 10174 Acked Write 7 Done Done Sent false 10175 Acked Read 9 Done Done Sent false 10176 Acked Write 2 Done Done Sent false 10177 Acked Write 3 Done Done Sent false 10178 Acked Read 8 Done Done Sent false 10179 Acked Write 7 Done Done Sent false 10180 Acked Read 3 Done Done Sent false 10181 Acked Flush 0 Done Done Sent false 10182 Acked Write 8 Done Done Sent false 10183 Acked Read 3 Done Done Sent false 10184 Acked Read 8 Done Done Sent false 10185 Acked Read 10 Done Done Sent false 10186 Acked Write 8 Done Done Sent false 10187 Acked Read 1 Done Done Sent false 10188 Acked Read 9 Done Done Sent false 10189 Acked Read 4 Done Done Sent false 10190 Acked Read 4 Done Done Sent false 10191 Acked Read 2 Done Done Sent false 10192 Acked Read 7 Done Done Sent false 10193 Acked Write 2 Done Done Sent false 10194 Acked Read 3 Done Done Sent false 10195 Acked Read 10 Done Done Sent false 10196 Acked Flush 0 Done Done Sent false 10197 Acked Read 4 Done Done Sent false 10198 Acked Read 8 Done Done Sent false 10199 Acked Read 9 Done Done Sent false 10200 Acked Write 6 Done Done Sent false 10201 Acked Read 3 Done Done Sent false 10202 Acked Write 2 Done Done Sent false 10203 Acked Flush 0 Done Done Sent false 10204 Acked Read 1 Done Done Sent false 10205 Acked Read 10 Done Done Sent false 10206 Acked Read 10 Done Done Sent false 10207 Acked Read 9 Done Done Sent false 10208 Acked Flush 0 Done Done Sent false 10209 Acked Read 6 Done Done Sent false 10210 Acked Write 2 Done Done Sent false 10211 Acked Write 7 Done Done Sent false 10212 Acked Read 9 Done Done Sent false 10213 Acked Write 8 Done Done Sent false 10214 Acked Write 8 Done Done Sent false 10215 Acked Read 2 Done Done Sent false 10216 Acked Read 6 Done Done Sent false 10217 Acked Write 10 Done Done Sent false 10218 Acked Read 7 Done Done Sent false 10219 Acked Read 6 Done Done Sent false 10220 Acked Read 5 Done Done Sent false 10221 Acked Read 5 Done Done Sent false 10222 Acked Write 2 Done Done Sent false 10223 Acked Read 2 Done Done Sent false 10224 Acked Write 2 Done Done Sent false 10225 Acked Read 2 Done Done Sent false 10226 Acked Write 1 Done Done Sent false 10227 Acked Read 4 Done Done Sent false 10228 Acked Write 1 Done Done Sent false 10229 Acked Write 4 Done Done Sent false 10230 Acked Read 10 Done Done Sent false 10231 Acked Read 4 Done Done Sent false 10232 Acked Read 8 Done Done Sent false 10233 Acked Read 10 Done Done Sent false 10234 Acked Flush 0 Done Done Sent false 10235 Acked Write 6 Done Done Sent false 10236 Acked Read 2 Done Done Sent false 10237 Acked Flush 0 Done Done Sent false 10238 Acked Read 7 Done Done Sent false 10239 Acked Write 6 Done Done Sent false 10240 Acked Write 1 Done Done Sent false 10241 Acked Write 9 Done Done Sent false 10242 Acked Read 10 Done Done Sent false 10243 Acked Read 6 Done Done Sent false 10244 Acked Read 3 Done Done Sent false 10245 Acked Flush 0 Done Done Sent false 10246 Acked Read 9 Done Done Sent false 10247 Acked Read 6 Done Done Sent false 10248 Acked Write 10 Done Done Sent false 10249 Acked Read 6 Done Done Sent false 10250 Acked Write 3 Done Done Sent false 10251 Acked Write 3 Done Done Sent false 10252 Acked Read 4 Done Done Sent false 10253 Acked Flush 0 Done Done Sent false 10254 Acked Read 2 Done Done Sent false 10255 Acked Write 2 Done Done Sent false 10256 Acked Write 1 Done Done Sent false 10257 Acked Write 10 Done Done Sent false 10258 Acked Read 3 Done Done Sent false 10259 Acked Write 7 Done Done Sent false 10260 Acked Read 7 Done Done Sent false 10261 Acked Write 3 Done Done Sent false 10262 Acked Write 2 Done Done Sent false 10263 Acked Write 7 Done Done Sent false 10264 Acked Flush 0 Done Done Sent false 10265 Acked Write 5 Done Done Sent false 10266 Acked Read 8 Done Done Sent false 10267 Acked Write 1 Done Done Sent false 10268 Acked Read 3 Done Done Sent false 10269 Acked Read 7 Done Done Sent false 10270 Acked Read 8 Done Done Sent false 10271 Acked Write 5 Done Done Sent false 10272 Acked Read 6 Done Done Sent false 10273 Acked Read 9 Done Done Sent false 10274 Acked Read 4 Done Done Sent false 10275 Acked Write 1 Done Done Sent false 10276 Acked Read 7 Done Done Sent false 10277 Acked Read 8 Done Done Sent false 10278 Acked Write 7 Done Done Sent false 10279 Acked Read 2 Done Done Sent false 10280 Acked Write 4 Done Done Sent false 10281 Acked Write 9 Done Done Sent false 10282 Acked Read 8 Done Done Sent false 10283 Acked Read 1 Done Done Sent false 10284 Acked Read 9 Done Done Sent false 10285 Acked Read 6 Done Done Sent false 10286 Acked Read 6 Done Done Sent false 10287 Acked Read 9 Done Done Sent false 10288 Acked Read 7 Done Done Sent false 10289 Acked Read 1 Done Done Sent false 10290 Acked Read 8 Done Done Sent false 10291 Acked Read 1 Done Done Sent false 10292 Acked Read 10 Done Done Sent false 10293 Acked Write 6 Done Done Sent false 10294 Acked Write 10 Done Done Sent false 10295 Acked Write 8 Done Done Sent false 10296 Acked Read 7 Done Done Sent false 10297 Acked Flush 0 Done Done Sent false 10298 Acked Write 3 Done Done Sent false 10299 Acked Write 7 Done Done Sent false 10300 Acked Write 4 Done Done Sent false 10301 Acked Read 1 Done Done Sent false 10302 Acked Read 2 Done Done Sent false 10303 Acked Read 5 Done Done Sent false 10304 Acked Read 7 Done Done Sent false 10305 Acked Write 9 Done Done Sent false 10306 Acked Read 7 Done Done Sent false 10307 Acked Write 4 Done Done Sent false 10308 Acked Read 7 Done Done Sent false 10309 Acked Read 1 Done Done Sent false 10310 Acked Write 8 Done Done Sent false 10311 Acked Read 4 Done Done Sent false 10312 Acked Read 3 Done Done Sent false 10313 Acked Write 9 Done Done Sent false 10314 Acked Flush 0 Done Done Sent false 10315 Acked Read 4 Done Done Sent false 10316 Acked Write 7 Done Done Sent false 10317 Acked Write 7 Done Done Sent false 10318 Acked Write 9 Done Done Sent false 10319 Acked Write 10 Done Done Sent false 10320 Acked Read 10 Done Done Sent false 10321 Acked Read 5 Done Done Sent false 10322 Acked Read 7 Done Done Sent false 10323 Acked Write 9 Done Done Sent false 10324 Acked Write 6 Done Done Sent false 10325 Acked Read 4 Done Done Sent false 10326 Acked Write 4 Done Done Sent false 10327 Acked Read 2 Done Done Sent false 10328 Acked Read 6 Done Done Sent false 10329 Acked Read 4 Done Done Sent false 10330 Acked Read 7 Done Done Sent false 10331 Acked Read 6 Done Done Sent false 10332 Acked Flush 0 Done Done Sent false 10333 Acked Read 4 Done Done Sent false 10334 Acked Read 10 Done Done Sent false 10335 Acked Read 6 Done Done Sent false 10336 Acked Write 1 Done Done Sent false 10337 Acked Read 4 Done Done Sent false 10338 Acked Write 4 Done Done Sent false 10339 Acked Read 7 Done Done Sent false 10340 Acked Read 2 Done Done Sent false 10341 Acked Write 5 Done Done Sent false 10342 Acked Write 2 Done Done Sent false 10343 Acked Read 8 Done Done Sent false 10344 Acked Write 9 Done Done Sent false 10345 Acked Write 5 Done Done Sent false 10346 Acked Write 3 Done Done Sent false 10347 Acked Write 2 Done Done Sent false 10348 Acked Read 1 Done Done Sent false 10349 Acked Read 6 Done Done Sent false 10350 Acked Read 5 Done Done Sent false 10351 Acked Flush 0 Done Done Sent false 10352 Acked Read 4 Done Done Sent false 10353 Acked Flush 0 Done Done Sent false 10354 Acked Write 7 Done Done Sent false 10355 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10355 10355 10053 Downstairs last five completed: 10053 10052 10051 10050 10049 Upstairs last five completed: 10355 10354 10353 10352 10351 CLIENT: Up:0 ds:302 act:2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 10054 Acked Read 6 Done Done Sent false 10055 Acked Write 2 Done Done Sent false 10056 Acked Write 8 Done Done Sent false 10057 Acked Read 6 Done Done Sent false 10058 Acked Read 9 Done Done Sent false 10059 Acked Write 7 Done Done Sent false 10060 Acked Read 8 Done Done Sent false 10061 Acked Write 3 Done Done Sent false 10062 Acked Read 7 Done Done Sent false 10063 Acked Flush 0 Done Done Sent false 10064 Acked Write 4 Done Done Sent false 10065 Acked Write 7 Done Done Sent false 10066 Acked Read 2 Done Done Sent false 10067 Acked Write 10 Done Done Sent false 10068 Acked Write 4 Done Done Sent false 10069 Acked Write 5 Done Done Sent false 10070 Acked Write 5 Done Done Sent false 10071 Acked Flush 0 Done Done Sent false 10072 Acked Write 2 Done Done Sent false 10073 Acked Read 5 Done Done Sent false 10074 Acked Write 7 Done Done Sent false 10075 Acked Read 6 Done Done Sent false 10076 Acked Read 4 Done Done Sent false 10077 Acked Read 10 Done Done Sent false 10078 Acked Read 4 Done Done Sent false 10079 Acked Read 1 Done Done Sent false 10080 Acked Read 1 Done Done Sent false 10081 Acked Write 3 Done Done Sent false 10082 Acked Write 5 Done Done Sent false 10083 Acked Write 2 Done Done Sent false 10084 Acked Read 5 Done Done Sent false 10085 Acked Read 1 Done Done Sent false 10086 Acked Write 5 Done Done Sent false 10087 Acked Write 10 Done Done Sent false 10088 Acked Flush 0 Done Done Sent false 10089 Acked Read 6 Done Done Sent false 10090 Acked Write 1 Done Done Sent false 10091 Acked Read 6 Done Done Sent false 10092 Acked Write 4 Done Done Sent false 10093 Acked Read 3 Done Done Sent false 10094 Acked Read 2 Done Done Sent false 10095 Acked Write 4 Done Done Sent false 10096 Acked Read 6 Done Done Sent false 10097 Acked Read 4 Done Done Sent false 10098 Acked Write 5 Done Done Sent false 10099 Acked Write 4 Done Done Sent false 10100 Acked Write 7 Done Done Sent false 10101 Acked Write 5 Done Done Sent false 10102 Acked Read 5 Done Done Sent false 10103 Acked Write 2 Done Done Sent false 10104 Acked Write 2 Done Done Sent false 10105 Acked Write 4 Done Done Sent false 10106 Acked Read 4 Done Done Sent false 10107 Acked Write 7 Done Done Sent false 10108 Acked Read 5 Done Done Sent false 10109 Acked Write 1 Done Done Sent false 10110 Acked Flush 0 Done Done Sent false 10111 Acked Read 9 Done Done Sent false 10112 Acked Flush 0 Done Done Sent false 10113 Acked Write 9 Done Done Sent false 10114 Acked Flush 0 Done Done Sent false 10115 Acked Flush 0 Done Done Sent false 10116 Acked Read 4 Done Done Sent false 10117 Acked Write 1 Done Done Sent false 10118 Acked Read 8 Done Done Sent false 10119 Acked Read 1 Done Done Sent false 10120 Acked Read 3 Done Done Sent false 10121 Acked Read 5 Done Done Sent false 10122 Acked Write 3 Done Done Sent false 10123 Acked Read 7 Done Done Sent false 10124 Acked Read 10 Done Done Sent false 10125 Acked Read 6 Done Done Sent false 10126 Acked Read 10 Done Done Sent false 10127 Acked Read 4 Done Done Sent false 10128 Acked Read 7 Done Done Sent false 10129 Acked Read 9 Done Done Sent false 10130 Acked Flush 0 Done Done Sent false 10131 Acked Read 9 Done Done Sent false 10132 Acked Write 6 Done Done Sent false 10133 Acked Read 4 Done Done Sent false 10134 Acked Read 3 Done Done Sent false 10135 Acked Write 9 Done Done Sent false 10136 Acked Read 5 Done Done Sent false 10137 Acked Read 6 Done Done Sent false 10138 Acked Write 2 Done Done Sent false 10139 Acked Write 7 Done Done Sent false 10140 Acked Write 7 Done Done Sent false 10141 Acked Write 1 Done Done Sent false 10142 Acked Read 7 Done Done Sent false 10143 Acked Read 5 Done Done Sent false 10144 Acked Read 3 Done Done Sent false 10145 Acked Flush 0 Done Done Sent false 10146 Acked Read 2 Done Done Sent false 10147 Acked Read 6 Done Done Sent false 10148 Acked Write 3 Done Done Sent false 10149 Acked Flush 0 Done Done Sent false 10150 Acked Write 2 Done Done Sent false 10151 Acked Flush 0 Done Done Sent false 10152 Acked Write 3 Done Done Sent false 10153 Acked Read 9 Done Done Sent false 10154 Acked Read 4 Done Done Sent false 10155 Acked Read 10 Done Done Sent false 10156 Acked Write 7 Done Done Sent false 10157 Acked Read 8 Done Done Sent false 10158 Acked Write 5 Done Done Sent false 10159 Acked Read 5 Done Done Sent false 10160 Acked Write 1 Done Done Sent false 10161 Acked Flush 0 Done Done Sent false 10162 Acked Flush 0 Done Done Sent false 10163 Acked Write 6 Done Done Sent false 10164 Acked Read 10 Done Done Sent false 10165 Acked Read 4 Done Done Sent false 10166 Acked Read 4 Done Done Sent false 10167 Acked Read 6 Done Done Sent false 10168 Acked Write 2 Done Done Sent false 10169 Acked Write 3 Done Done Sent false 10170 Acked Write 2 Done Done Sent false 10171 Acked Read 8 Done Done Sent false 10172 Acked Write 4 Done Done Sent false 10173 Acked Write 2 Done Done Sent false 10174 Acked Write 7 Done Done Sent false 10175 Acked Read 9 Done Done Sent false 10176 Acked Write 2 Done Done Sent false 10177 Acked Write 3 Done Done Sent false 10178 Acked Read 8 Done Done Sent false 10179 Acked Write 7 Done Done Sent false 10180 Acked Read 3 Done Done Sent false 10181 Acked Flush 0 Done Done Sent false 10182 Acked Write 8 Done Done Sent false 10183 Acked Read 3 Done Done Sent false 10184 Acked Read 8 Done Done Sent false 10185 Acked Read 10 Done Done Sent false 10186 Acked Write 8 Done Done Sent false 10187 Acked Read 1 Done Done Sent false 10188 Acked Read 9 Done Done Sent false 10189 Acked Read 4 Done Done Sent false 10190 Acked Read 4 Done Done Sent false 10191 Acked Read 2 Done Done Sent false 10192 Acked Read 7 Done Done Sent false 10193 Acked Write 2 Done Done Sent false 10194 Acked Read 3 Done Done Sent false 10195 Acked Read 10 Done Done Sent false 10196 Acked Flush 0 Done Done Sent false 10197 Acked Read 4 Done Done Sent false 10198 Acked Read 8 Done Done Sent false 10199 Acked Read 9 Done Done Sent false 10200 Acked Write 6 Done Done Sent false 10201 Acked Read 3 Done Done Sent false 10202 Acked Write 2 Done Done Sent false 10203 Acked Flush 0 Done Done Sent false 10204 Acked Read 1 Done Done Sent false 10205 Acked Read 10 Done Done Sent false 10206 Acked Read 10 Done Done Sent false 10207 Acked Read 9 Done Done Sent false 10208 Acked Flush 0 Done Done Sent false 10209 Acked Read 6 Done Done Sent false 10210 Acked Write 2 Done Done Sent false 10211 Acked Write 7 Done Done Sent false 10212 Acked Read 9 Done Done Sent false 10213 Acked Write 8 Done Done Sent false 10214 Acked Write 8 Done Done Sent false 10215 Acked Read 2 Done Done Sent false 10216 Acked Read 6 Done Done Sent false 10217 Acked Write 10 Done Done Sent false 10218 Acked Read 7 Done Done Sent false 10219 Acked Read 6 Done Done Sent false 10220 Acked Read 5 Done Done Sent false 10221 Acked Read 5 Done Done Sent false 10222 Acked Write 2 Done Done Sent false 10223 Acked Read 2 Done Done Sent false 10224 Acked Write 2 Done Done Sent false 10225 Acked Read 2 Done Done Sent false 10226 Acked Write 1 Done Done Sent false 10227 Acked Read 4 Done Done Sent false 10228 Acked Write 1 Done Done Sent false 10229 Acked Write 4 Done Done Sent false 10230 Acked Read 10 Done Done Sent false 10231 Acked Read 4 Done Done Sent false 10232 Acked Read 8 Done Done Sent false 10233 Acked Read 10 Done Done Sent false 10234 Acked Flush 0 Done Done Sent false 10235 Acked Write 6 Done Done Sent false 10236 Acked Read 2 Done Done Sent false 10237 Acked Flush 0 Done Done Sent false 10238 Acked Read 7 Done Done Sent false 10239 Acked Write 6 Done Done Sent false 10240 Acked Write 1 Done Done Sent false 10241 Acked Write 9 Done Done Sent false 10242 Acked Read 10 Done Done Sent false 10243 Acked Read 6 Done Done Sent false 10244 Acked Read 3 Done Done Sent false 10245 Acked Flush 0 Done Done Sent false 10246 Acked Read 9 Done Done Sent false 10247 Acked Read 6 Done Done Sent false 10248 Acked Write 10 Done Done Sent false 10249 Acked Read 6 Done Done Sent false 10250 Acked Write 3 Done Done Sent false 10251 Acked Write 3 Done Done Sent false 10252 Acked Read 4 Done Done Sent false 10253 Acked Flush 0 Done Done Sent false 10254 Acked Read 2 Done Done Sent false 10255 Acked Write 2 Done Done Sent false 10256 Acked Write 1 Done Done Sent false 10257 Acked Write 10 Done Done Sent false 10258 Acked Read 3 Done Done Sent false 10259 Acked Write 7 Done Done Sent false 10260 Acked Read 7 Done Done Sent false 10261 Acked Write 3 Done Done Sent false 10262 Acked Write 2 Done Done Sent false 10263 Acked Write 7 Done Done Sent false 10264 Acked Flush 0 Done Done Sent false 10265 Acked Write 5 Done Done Sent false 10266 Acked Read 8 Done Done Sent false 10267 Acked Write 1 Done Done Sent false 10268 Acked Read 3 Done Done Sent false 10269 Acked Read 7 Done Done Sent false 10270 Acked Read 8 Done Done Sent false 10271 Acked Write 5 Done Done Sent false 10272 Acked Read 6 Done Done Sent false 10273 Acked Read 9 Done Done Sent false 10274 Acked Read 4 Done Done Sent false 10275 Acked Write 1 Done Done Sent false 10276 Acked Read 7 Done Done Sent false 10277 Acked Read 8 Done Done Sent false 10278 Acked Write 7 Done Done Sent false 10279 Acked Read 2 Done Done Sent false 10280 Acked Write 4 Done Done Sent false 10281 Acked Write 9 Done Done Sent false 10282 Acked Read 8 Done Done Sent false 10283 Acked Read 1 Done Done Sent false 10284 Acked Read 9 Done Done Sent false 10285 Acked Read 6 Done Done Sent false 10286 Acked Read 6 Done Done Sent false 10287 Acked Read 9 Done Done Sent false 10288 Acked Read 7 Done Done Sent false 10289 Acked Read 1 Done Done Sent false 10290 Acked Read 8 Done Done Sent false 10291 Acked Read 1 Done Done Sent false 10292 Acked Read 10 Done Done Sent false 10293 Acked Write 6 Done Done Sent false 10294 Acked Write 10 Done Done Sent false 10295 Acked Write 8 Done Done Sent false 10296 Acked Read 7 Done Done Sent false 10297 Acked Flush 0 Done Done Sent false 10298 Acked Write 3 Done Done Sent false 10299 Acked Write 7 Done Done Sent false 10300 Acked Write 4 Done Done Sent false 10301 Acked Read 1 Done Done Sent false 10302 Acked Read 2 Done Done Sent false 10303 Acked Read 5 Done Done Sent false 10304 Acked Read 7 Done Done Sent false 10305 Acked Write 9 Done Done Sent false 10306 Acked Read 7 Done Done Sent false 10307 Acked Write 4 Done Done Sent false 10308 Acked Read 7 Done Done Sent false 10309 Acked Read 1 Done Done Sent false 10310 Acked Write 8 Done Done Sent false 10311 Acked Read 4 Done Done Sent false 10312 Acked Read 3 Done Done Sent false 10313 Acked Write 9 Done Done Sent false 10314 Acked Flush 0 Done Done Sent false 10315 Acked Read 4 Done Done Sent false 10316 Acked Write 7 Done Done Sent false 10317 Acked Write 7 Done Done Sent false 10318 Acked Write 9 Done Done Sent false 10319 Acked Write 10 Done Done Sent false 10320 Acked Read 10 Done Done Sent false 10321 Acked Read 5 Done Done Sent false 10322 Acked Read 7 Done Done Sent false 10323 Acked Write 9 Done Done Sent false 10324 Acked Write 6 Done Done Sent false 10325 Acked Read 4 Done Done Sent false 10326 Acked Write 4 Done Done Sent false 10327 Acked Read 2 Done Done Sent false 10328 Acked Read 6 Done Done Sent false 10329 Acked Read 4 Done Done Sent false 10330 Acked Read 7 Done Done Sent false 10331 Acked Read 6 Done Done Sent false 10332 Acked Flush 0 Done Done Sent false 10333 Acked Read 4 Done Done Sent false 10334 Acked Read 10 Done Done Sent false 10335 Acked Read 6 Done Done Sent false 10336 Acked Write 1 Done Done Sent false 10337 Acked Read 4 Done Done Sent false 10338 Acked Write 4 Done Done Sent false 10339 Acked Read 7 Done Done Sent false 10340 Acked Read 2 Done Done Sent false 10341 Acked Write 5 Done Done Sent false 10342 Acked Write 2 Done Done Sent false 10343 Acked Read 8 Done Done Sent false 10344 Acked Write 9 Done Done Sent false 10345 Acked Write 5 Done Done Sent false 10346 Acked Write 3 Done Done Sent false 10347 Acked Write 2 Done Done Sent false 10348 Acked Read 1 Done Done Sent false 10349 Acked Read 6 Done Done Sent false 10350 Acked Read 5 Done Done Sent false 10351 Acked Flush 0 Done Done Sent false 10352 Acked Read 4 Done Done Sent false 10353 Acked Flush 0 Done Done Sent false 10354 Acked Write 7 Done Done Sent false 10355 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 302 302 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 10355 10355 10053 Downstairs last five completed: 10053 10052 10051 10050 10049 Upstairs last five completed: 10355 10354 10353 10352 10351 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.935253797Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_connection connected from Ok(127.0.0.1:51081)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.93531619Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"version negotiation from state Offline","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:50.936379234Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.936933536Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.93695423Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"send last flush ID to this DS: 10053","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.936969156Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"Replied this last flush ID: 10053","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.937241583Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"ds_transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.937262477Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Transition from Offline to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.937277293Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"client":"2","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] client re-new 302 jobs since flush 10053","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.937291929Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} {"msg":"[2] Replayed 302 jobs since flush: 10053","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:50.937537694Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1202,"":"downstairs","":"downstairs","session_id":"94dfdc19-74c7-449c-a147-e4ca5570ea17"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 10355 10354 10353 10352 10351 Upstairs last five completed: 10355 10354 10353 10352 10351 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished Test replay has completed Wrote out file "/tmp/test_replay_verify.log" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 10355 10354 10353 10352 10351 Upstairs last five completed: 10355 10354 10353 10352 10351 CLIENT: Up:0 ds:0 act:3 CLIENT: All jobs finished, all DS active. Replays:30 time: 8:20 Do final verify Nov 27 01:49:51.261 INFO accepted connection, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 Nov 27 01:49:51.262 INFO request completed, latency_us: 196, response_code: 200, uri: /regioninfo, method: GET, req_id: 1c9a8d56-cc1a-44ae-a4c2-3456c2584aa6, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 {"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.26270584Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"task":"crutest"} Nov 27 01:49:51.262 INFO request completed, latency_us: 93, response_code: 200, uri: /regioncount, method: GET, req_id: a851bc0d-0fbf-46a9-8e6c-efbd592d9b63, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 {"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.263213184Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"task":"crutest"} Nov 27 01:49:51.263 INFO request completed, latency_us: 121, response_code: 200, uri: /port/cid/0, method: GET, req_id: 1ca70e50-df6b-4679-8dde-e5fefd4a0306, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 Nov 27 01:49:51.264 INFO request completed, latency_us: 94, response_code: 200, uri: /port/cid/1, method: GET, req_id: b7b8f356-d031-4b07-bbe4-64f0b01d21bd, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 Nov 27 01:49:51.264 INFO request completed, latency_us: 113, response_code: 200, uri: /port/cid/2, method: GET, req_id: 1c8b189d-8eb4-4955-8072-6e78f0d26f88, remote_addr: 127.0.0.1:60547, local_addr: 127.0.0.1:9998 {"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.264851494Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265036704Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"7dc0be16221dc0ce43144ff676801ae1255fa766\",\n git_commit_timestamp: \"2024-11-27T01:26:03.000000000Z\",\n git_branch: \"mkeeter/deactivate-need-flush\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265067306Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265080982Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265096868Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} {"msg":"Crucible 4250e9cc-5d83-46f1-9b02-e82ae96d9e67 has session id: 740b7315-6aac-4675-80f2-c8a89153a5ce","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265548037Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Upstairs opts: Upstairs UUID: 4250e9cc-5d83-46f1-9b02-e82ae96d9e67, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265595834Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Crucible stats registered with UUID: 4250e9cc-5d83-46f1-9b02-e82ae96d9e67","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.265621697Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.299898895Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"task":"crutest"} {"msg":"The guest has requested activation with gen:3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.299936315Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} {"msg":"4250e9cc-5d83-46f1-9b02-e82ae96d9e67 active request set","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300052854Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300076297Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300100661Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300115997Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300130813Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300148158Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300171112Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300199984Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300227047Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300254249Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:43569)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300359601Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:65059)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300673117Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_connection connected from Ok(127.0.0.1:60640)","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.300719265Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301095744Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301126425Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301244784Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301280324Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301364371Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.30140704Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301792916Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.301833896Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.30196528Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302519312Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.3025608Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302579425Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302604389Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302618895Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302632291Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302713229Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302732944Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[0]R flush_numbers[0..12]: [954, 950, 883, 953, 908, 944, 954, 929, 950, 951, 946, 956]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.3027475Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302761407Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302775533Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[1]R flush_numbers[0..12]: [954, 950, 883, 953, 908, 944, 954, 929, 950, 951, 946, 956]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302789349Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302802885Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302816842Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[2]R flush_numbers[0..12]: [954, 950, 883, 953, 908, 944, 954, 929, 950, 951, 946, 956]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302833837Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302847693Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.30286176Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302875136Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302889222Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Next flush: 957","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302902359Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.302915515Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"":"downstairs","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303013209Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303032004Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"0","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.30304641Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"1","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303059876Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"client":"2","":"downstairs","session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303073263Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.30308572Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"4250e9cc-5d83-46f1-9b02-e82ae96d9e67 is now active with session: 740b7315-6aac-4675-80f2-c8a89153a5ce","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303098756Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303111013Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238,"session_id":"740b7315-6aac-4675-80f2-c8a89153a5ce"} {"msg":"The guest has finished waiting for activation with:3","v":0,"name":"crucible","level":30,"time":"2024-11-27T01:49:51.303125829Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Loading write count information from file "/tmp/test_replay_verify.log" Read and Verify all blocks (0..5000 range:false) Wrote out file "/tmp/test_replay_verify.log" Verify test completed Wrote out file "/tmp/test_replay_verify.log" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 256 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false 1003 Acked Read 256 Done Done Done false 1004 Acked Read 256 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false 1007 Acked Read 256 Done Done Done false 1008 Acked Read 256 Done Done Done false 1009 Acked Read 256 Done Done Done false 1010 Acked Read 256 Done Done Done false 1011 Acked Read 256 Done Done Done false 1012 Acked Read 256 Done Done Done false 1013 Acked Read 256 Done Done Done false 1014 Acked Read 256 Done Done Done false 1015 Acked Read 256 Done Done Done false 1016 Acked Read 256 Done Done Done false 1017 Acked Read 256 Done Done Done false 1018 Acked Read 256 Done Done Done false 1019 Acked Read 136 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 20 20 20 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1019 1018 1017 1016 1015 CLIENT: Up:0 ds:20 act:3 ---------------------------------------------------------------- Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 1020 1019 1018 1017 1016 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-11-27T01:49:55.707586936Z","hostname":"w-01JDNMCXATVP1SS7F9N73EV8XP","pid":1238} Nov 27 01:49:55.773 INFO accepted connection, remote_addr: 127.0.0.1:35212, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Nov 27 01:49:55.774 INFO request completed, latency_usShut it down :[8810] Disable keep_running [8810] Got stop action so:false kr:false 153[8830] Disable keep_running , [8830] Got stop action so:false kr:false response_code: 204, uri: /shutdown, method: POST, req_id: c7c86298-e4b3-47cd-b6db-d7db6013c03a, remote_addr: 127.0.0.1:35212, local_addr: 127.0.0.1:9998 [8820] Disable keep_running [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit November 27, 2024 at 01:50:00 AM UTC Test ends with 0