Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] initial start wait reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1507) Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1508) [8810][0] initial start wait reports Running [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1509) [8820][1] initial start wait reports Running All downstairs are running Mar 17 22:21:50.350 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Mar 17 22:21:55.396 INFO accepted connection, remote_addr: 127.0.0.1:61115, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Mar 17 22:21:55.398 INFO request completed, latency_us: 138, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 83071730-db23-49d4-a405-6a1e6e0de798, remote_addr: 127.0.0.1:61115, local_addr: 127.0.0.1:9998 [8820] Disable keep_running [8830] Disable keep_running [8810] Disable keep_running Mar 17 22:23:24.094 INFO accepted connection, remote_addr: 127.0.0.1:39970, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Mar 17 22:23:24.094 INFO request completed, latency_us: Shut it down 69, response_code: 204[8810] Disable keep_running , [8810] Got stop action so:false kr:false uri[8820] Disable keep_running :[8820] Got stop action so:false kr:false [8830] Disable keep_running /shutdown[8830] Got stop action so:false kr:false , method: POST, req_id: 6b5e4922-980f-4024-81e4-f941c4af4ebd, remote_addr: 127.0.0.1:39970, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit "Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"f72bfdb36aa576722b377a737f26a9b5ef54faf2\",\n git_commit_timestamp: \"2025-03-17T21:58:36.000000000Z\",\n git_branch: \"renovate/tempfile-3.x-lockfile\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.468575271Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.468603378Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.468610212Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} {"msg":"Crucible 91f25286-e420-4cbe-8762-377272c83f75 has session id: 2f84bb49-f5b1-4f16-ac0c-5d624d4d4702","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469265029Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Upstairs opts: Upstairs UUID: 91f25286-e420-4cbe-8762-377272c83f75, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469288617Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Crucible stats registered with UUID: 91f25286-e420-4cbe-8762-377272c83f75","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469302084Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:55.469307936Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469709199Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469725803Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} {"msg":"91f25286-e420-4cbe-8762-377272c83f75 active request set","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469799812Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469806877Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469822198Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469828561Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469839573Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469845665Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.469851708Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470068717Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470096754Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.47010468Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"ds_connection connected from Ok(127.0.0.1:37437)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470126194Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"ds_connection connected from Ok(127.0.0.1:52080)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470140914Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"ds_connection connected from Ok(127.0.0.1:51670)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.47031707Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470961566Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470976687Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.470983942Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.471136952Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.471187494Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.471327648Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.482237162Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.482940216Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485312182Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485352083Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485365049Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485375941Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485386462Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485407214Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485418477Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485429149Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.48543976Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485450873Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485461304Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485471966Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485482507Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"":"downstairs","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485492948Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485501956Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"0","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485513059Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"1","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.48552364Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"client":"2","":"downstairs","session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485534172Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.48554312Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"91f25286-e420-4cbe-8762-377272c83f75 is now active with session: 2f84bb49-f5b1-4f16-ac0c-5d624d4d4702","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485552469Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485565535Z","hostname":"bmat-EVT22200007-0000568f","pid":1512,"session_id":"2f84bb49-f5b1-4f16-ac0c-5d624d4d4702"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485575495Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} {"msg":"Activated sub_volume 91f25286-e420-4cbe-8762-377272c83f75","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:55.485733515Z","hostname":"bmat-EVT22200007-0000568f","pid":1512} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:60 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 104447 Acked Write 256 Done Done Done false 104448 Acked Write 256 Done Done Done false 104449 Acked Write 256 Done Done Done false 104450 Acked Write 256 Done Done Done false 104451 Acked Write 256 Done Done Done false 104452 Acked Write 256 Done Done Done false 104453 Acked Write 256 Done Done Done false 104454 Acked Write 256 Done Done Done false 104455 Acked Write 256 Done Done Done false 104456 Acked Write 256 Done Done Done false 104457 Acked Write 256 Done Done Done false 104458 Acked Write 256 Done Done Done false 104459 Acked Write 256 Done Done Done false 104460 Acked Write 256 Done Done Done false 104461 Acked Write 256 Done Done Done false 104462 Acked Write 256 Done Done Done false 104463 Acked Write 256 Done Done Done false 104464 Acked Write 256 Done Done Done false 104465 Acked Write 256 Done Done Done false 104466 Acked Write 256 Done Done Done false 104467 Acked Write 256 Done Done Done false 104468 Acked Write 256 Done Done Done false 104469 Acked Write 256 Done Done Done false 104470 Acked Write 256 Done Done Done false 104471 Acked Write 256 Done Done Done false 104472 Acked Write 256 Done Done Done false 104473 Acked Write 256 Done Done Done false 104474 Acked Write 256 Done Done Done false 104475 Acked Write 256 Done Done Done false 104476 Acked Write 256 Done Done Done false 104477 Acked Write 256 Done Done Done false 104478 Acked Write 256 Done Done Done false 104479 Acked Write 256 Done Done Done false 104480 Acked Write 256 Done Done Done false 104481 Acked Write 256 Done Done Done false 104482 Acked Write 256 Done Done Done false 104483 Acked Write 256 Done Done Done false 104484 Acked Write 256 Done Done Done false 104485 Acked Write 256 Done Done Done false 104486 Acked Write 256 Done Done Done false 104487 Acked Write 256 Done Done Done false 104488 Acked Write 256 Done Done Done false 104489 Acked Write 256 Done Done Done false 104490 Acked Write 256 Done Done Done false 104491 Acked Write 256 Done Done Done false 104492 Acked Write 256 Done Done Done false 104493 Acked Write 256 Done Done Done false 104494 Acked Write 256 Done Done Done false 104495 Acked Write 256 Done Done Done false 104496 Acked Write 256 Done Done Done false 104497 Acked Write 256 Done Done Done false 104498 Acked Write 256 Done Done Done false 104499 Acked Write 256 Done Done Done false 104500 Acked Write 256 Done Done Done false 104501 Acked Write 256 Done Done Done false 104502 Acked Write 256 Done Done Done false 104503 Acked Write 256 Done Done Done false 104504 Acked Write 256 Done Done Done false 104505 Acked Write 256 Done Done Done false 104506 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 60 60 59 179 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(104506)) Some(JobId(104506)) None Downstairs last five completed: 104446 104445 104444 104443 104442 Upstairs last five completed: 104506 104505 104504 104503 104502 CLIENT: Up:0 ds:60 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 104506 104505 104504 104503 104502 Upstairs last five completed: 104506 104505 104504 104503 104502 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program