Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1504) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1505) [8820][1] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1506) [8830][2] initial start wait reports Running All downstairs are running Mar 17 22:38:10.151 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Mar 17 22:38:15.195 INFO accepted connection, remote_addr: 127.0.0.1:61857, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Mar 17 22:38:15.196 INFO request completed, latency_us: [8820] Disable keep_running 122[8830] Disable keep_running , response_code: 204, uri: /disablerestart/all, method:[8810] Disable keep_running POST, req_id: 921b79aa-2474-4f15-a66f-3ee0b52a83ac, remote_addr: 127.0.0.1:61857, local_addr: 127.0.0.1:9998 Mar 17 22:39:43.800 INFO accepted connection, remote_addr: 127.0.0.1:32910, local_addr: 127.0.0.1:9998 Mar 17 22:39:43.800 INFO request completed, latency_us: 71, response_code: 204Main task has work to do, go find it , got dsc Shutdown uriShutdown : /shutdown, method: POST, req_id:Shut it down 16fd7e1f-5952-4c29-84f4-8cc8c2e8fac0, remote_addr: 127.0.0.1:[8810] Disable keep_running 32910[8810] Got stop action so:false kr:false , [8820] Disable keep_running local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit :"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"45feeae3dd1bdcd88dce11a80acb5415f57b38bc\",\n git_commit_timestamp: \"2025-03-17T22:11:58.000000000Z\",\n git_branch: \"renovate/opentelemetry-0.x\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.258826507Z","hostname":"bmat-EVT22200007-00005697","pid":1509} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.258838581Z","hostname":"bmat-EVT22200007-00005697","pid":1509} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.25884836Z","hostname":"bmat-EVT22200007-00005697","pid":1509} {"msg":"Crucible 25b29037-8ec2-481f-b217-2c8dfd8511f8 has session id: ae5b9c5a-99c4-4a5f-8a77-b98776012255","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.259691429Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Upstairs opts: Upstairs UUID: 25b29037-8ec2-481f-b217-2c8dfd8511f8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.259720617Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Crucible stats registered with UUID: 25b29037-8ec2-481f-b217-2c8dfd8511f8","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.259734946Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:38:15.259740076Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.259966166Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.259978521Z","hostname":"bmat-EVT22200007-00005697","pid":1509} {"msg":"25b29037-8ec2-481f-b217-2c8dfd8511f8 active request set","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260072338Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260079772Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260092778Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.26009857Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260104401Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260109482Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260114922Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260121305Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260172227Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.2601842Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"ds_connection connected from Ok(127.0.0.1:41577)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260545981Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"ds_connection connected from Ok(127.0.0.1:57365)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.260563476Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"ds_connection connected from Ok(127.0.0.1:33928)","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.26068099Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261040135Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261396765Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261407877Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261497065Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261683536Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.261748986Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.271253884Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.271502629Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272740724Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272750845Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272756837Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272762408Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272767808Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272779071Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272785083Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272790423Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272795904Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272801335Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272806596Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272812187Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272817457Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"":"downstairs","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272822638Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272827297Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"0","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272833299Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"1","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.27283884Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"client":"2","":"downstairs","session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272844842Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272849752Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"25b29037-8ec2-481f-b217-2c8dfd8511f8 is now active with session: ae5b9c5a-99c4-4a5f-8a77-b98776012255","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272854571Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272861715Z","hostname":"bmat-EVT22200007-00005697","pid":1509,"session_id":"ae5b9c5a-99c4-4a5f-8a77-b98776012255"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.272866775Z","hostname":"bmat-EVT22200007-00005697","pid":1509} {"msg":"Activated sub_volume 25b29037-8ec2-481f-b217-2c8dfd8511f8","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:38:15.273021273Z","hostname":"bmat-EVT22200007-00005697","pid":1509} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:90 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 104383 Acked Write 256 Done Done Done false 104384 Acked Write 256 Done Done Done false 104385 Acked Write 256 Done Done Done false 104386 Acked Write 256 Done Done Done false 104387 Acked Write 256 Done Done Done false 104388 Acked Write 256 Done Done Done false 104389 Acked Write 256 Done Done Done false 104390 Acked Write 256 Done Done Done false 104391 Acked Write 256 Done Done Done false 104392 Acked Write 256 Done Done Done false 104393 Acked Write 256 Done Done Done false 104394 Acked Write 256 Done Done Done false 104395 Acked Write 256 Done Done Done false 104396 Acked Write 256 Done Done Done false 104397 Acked Write 256 Done Done Done false 104398 Acked Write 256 Done Done Done false 104399 Acked Write 256 Done Done Done false 104400 Acked Write 256 Done Done Done false 104401 Acked Write 256 Done Done Done false 104402 Acked Write 256 Done Done Done false 104403 Acked Write 256 Done Done Done false 104404 Acked Write 256 Done Done Done false 104405 Acked Write 256 Done Done Done false 104406 Acked Write 256 Done Done Done false 104407 Acked Write 256 Done Done Done false 104408 Acked Write 256 Done Done Done false 104409 Acked Write 256 Done Done Done false 104410 Acked Write 256 Done Done Done false 104411 Acked Write 256 Done Done Done false 104412 Acked Write 256 Done Done Done false 104413 Acked Write 256 Done Done Done false 104414 Acked Write 256 Done Done Done false 104415 Acked Write 256 Done Done Done false 104416 Acked Write 256 Done Done Done false 104417 Acked Write 256 Done Done Done false 104418 Acked Write 256 Done Done Done false 104419 Acked Write 256 Done Done Done false 104420 Acked Write 256 Done Done Done false 104421 Acked Write 256 Done Done Done false 104422 Acked Write 256 Done Done Done false 104423 Acked Write 256 Done Done Done false 104424 Acked Write 256 Done Done Done false 104425 Acked Write 256 Done Done Done false 104426 Acked Write 256 Done Done Done false 104427 Acked Write 256 Done Done Done false 104428 Acked Write 256 Done Done Done false 104429 Acked Write 256 Done Done Done false 104430 Acked Write 256 Done Done Done false 104431 Acked Write 256 Done Done Done false 104432 Acked Write 256 Done Done Done false 104433 Acked Write 256 Done Done Done false 104434 Acked Write 256 Done Done Done false 104435 Acked Write 256 Done Done Done false 104436 Acked Write 256 Done Done Done false 104437 Acked Write 256 Done Done Done false 104438 Acked Write 256 Done Done Done false 104439 Acked Write 256 Done Done Done false 104440 Acked Write 256 Done Done Done false 104441 Acked Write 256 Done Done Done false 104442 Acked Write 256 Done Done Done false 104443 Acked Write 256 Done Done Done false 104444 Acked Write 256 Done Done Done false 104445 Acked Write 256 Done Done Done false 104446 Acked Write 256 Done Done Done false 104447 Acked Write 256 Done Done Done false 104448 Acked Write 256 Done Done Done false 104449 Acked Write 256 Done Done Done false 104450 Acked Write 256 Done Done Done false 104451 Acked Write 256 Done Done Done false 104452 Acked Write 256 Done Done Done false 104453 Acked Write 256 Done Done Done false 104454 Acked Write 256 Done Done Done false 104455 Acked Write 256 Done Done Done false 104456 Acked Write 256 Done Done Done false 104457 Acked Write 256 Done Done Done false 104458 Acked Write 256 Done Done Done false 104459 Acked Write 256 Done Done Done false 104460 Acked Write 256 Done Done Done false 104461 Acked Write 256 Done Done Done false 104462 Acked Write 256 Done Done Done false 104463 Acked Write 256 Done Done Done false 104464 Acked Write 256 Done Done Done false 104465 Acked Write 256 Done Done Done false 104466 Acked Write 256 Done Done Done false 104467 Acked Write 256 Done Done Done false 104468 Acked Write 256 Done Done Done false 104469 Acked Write 256 Done Done Done false 104470 Acked Write 256 Done Done Done false 104471 Acked Write 256 Done Done Done false 104472 Acked Flush 0 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 0 1 Done 90 89 90 269 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(104472)) None Some(JobId(104472)) Downstairs last five completed: 104382 104381 104380 104379 104378 Upstairs last five completed: 104472 104471 104470 104469 104468 CLIENT: Up:0 ds:90 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 104472 104471 104470 104469 104468 Upstairs last five completed: 104472 104471 104470 104469 104468 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program