Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 [8810][0] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8810.txt" Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1498) [8820][1] initial start wait reports Running Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1499) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1500) [8830][2] initial start wait reports Running All downstairs are running Jan 28 21:58:36.347 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Jan 28 21:58:41.389 INFO accepted connection, remote_addr: 127.0.0.1:37522, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll Jan 28 21:58:41.389 INFO request completed, latency_us: 97, response_code: 204disable restart on all: 3 , uri: /disablerestart/all, method: POST, req_id: 58b545c4-d0ec-42ec-b3a7-d84d4023698e, remote_addr: 127.0.0.[8810] Disable keep_running 1[8820] Disable keep_running :[8830] Disable keep_running 37522, local_addr: 127.0.0.1:9998 Jan 28 22:00:18.095 INFO accepted connection, remote_addr: 127.0.0.1:46487, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Jan 28 22:00:18.096 Shut it down INFO[8810] Disable keep_running [8830] Disable keep_running request completed[8820] Disable keep_running , [8820] Got stop action so:false kr:false latency_us: 59, response_code: 204, uri: /shutdown, method: POST, req_id: [8830] Got stop action so:false kr:false e9d2a098-5c39-406d-8215-f02ddf621349, remote_addr: 127.0.0.1:46487, local_addr: 127.0.0.1:9998 [8810] Got stop action so:false kr:false [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit 3} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"7817632486f84ab6865c5ce71e16196ec0c710fc\",\n git_commit_timestamp: \"2025-01-28T21:38:04.000000000Z\",\n git_branch: \"mkeeter/early-io-rejection\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445324169Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445328485Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445336657Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503} {"msg":"Crucible 18cb8e0b-213c-463e-b103-6f374d4a616d has session id: 4c7e0a38-28e2-420e-911c-b8d06d5daf0e","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445750971Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Upstairs opts: Upstairs UUID: 18cb8e0b-213c-463e-b103-6f374d4a616d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445801704Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Crucible stats registered with UUID: 18cb8e0b-213c-463e-b103-6f374d4a616d","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.44581611Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:58:41.445822964Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445986512Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.445997242Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503} {"msg":"18cb8e0b-213c-463e-b103-6f374d4a616d active request set","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.44605203Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446057026Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446065728Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446070293Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446074949Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446079105Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446083301Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.44609595Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446151558Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446164376Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:56223)","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446199073Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:36454)","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446278389Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58171)","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446294584Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"io task","client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446592926Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446616194Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446657166Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.44683604Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446848168Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.446934737Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.454220345Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455033867Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455263003Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455276361Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455283634Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455289988Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455300429Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455307002Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455313377Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455319411Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455325585Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455331639Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455337594Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455343728Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455350392Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"":"downstairs","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455356886Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455362191Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"0","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455368615Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"1","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455374719Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"client":"2","":"downstairs","session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455380634Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455386029Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"18cb8e0b-213c-463e-b103-6f374d4a616d is now active with session: 4c7e0a38-28e2-420e-911c-b8d06d5daf0e","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455391793Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455399596Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGWWait for a query_work_queue command to finish before sending IO ","pid":1503,"session_id":"4c7e0a38-28e2-420e-911c-b8d06d5daf0e"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:41.455412734Z","hostname":"w-01JJQEP02WPTZPZCGE1J29RTGW","pid":1503} Disk: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:72 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 104710 Acked Write 256 Done Done Done false 104711 Acked Write 256 Done Done Done false 104712 Acked Write 256 Done Done Done false 104713 Acked Write 256 Done Done Done false 104714 Acked Write 256 Done Done Done false 104715 Acked Write 256 Done Done Done false 104716 Acked Write 256 Done Done Done false 104717 Acked Write 256 Done Done Done false 104718 Acked Write 256 Done Done Done false 104719 Acked Write 256 Done Done Done false 104720 Acked Write 256 Done Done Done false 104721 Acked Write 256 Done Done Done false 104722 Acked Write 256 Done Done Done false 104723 Acked Write 256 Done Done Done false 104724 Acked Write 256 Done Done Done false 104725 Acked Write 256 Done Done Done false 104726 Acked Write 256 Done Done Done false 104727 Acked Write 256 Done Done Done false 104728 Acked Write 256 Done Done Done false 104729 Acked Write 256 Done Done Done false 104730 Acked Write 256 Done Done Done false 104731 Acked Write 256 Done Done Done false 104732 Acked Write 256 Done Done Done false 104733 Acked Write 256 Done Done Done false 104734 Acked Write 256 Done Done Done false 104735 Acked Write 256 Done Done Done false 104736 Acked Write 256 Done Done Done false 104737 Acked Write 256 Done Done Done false 104738 Acked Write 256 Done Done Done false 104739 Acked Write 256 Done Done Done false 104740 Acked Write 256 Done Done Done false 104741 Acked Write 256 Done Done Done false 104742 Acked Write 256 Done Done Done false 104743 Acked Write 256 Done Done Done false 104744 Acked Write 256 Done Done Done false 104745 Acked Write 256 Done Done Done false 104746 Acked Write 256 Done Done Done false 104747 Acked Write 256 Done Done Done false 104748 Acked Write 256 Done Done Done false 104749 Acked Write 256 Done Done Done false 104750 Acked Write 256 Done Done Done false 104751 Acked Write 256 Done Done Done false 104752 Acked Write 256 Done Done Done false 104753 Acked Write 256 Done Done Done false 104754 Acked Write 256 Done Done Done false 104755 Acked Write 256 Done Done Done false 104756 Acked Write 256 Done Done Done false 104757 Acked Write 256 Done Done Done false 104758 Acked Write 256 Done Done Done false 104759 Acked Write 256 Done Done Done false 104760 Acked Write 256 Done Done Done false 104761 Acked Write 256 Done Done Done false 104762 Acked Write 256 Done Done Done false 104763 Acked Write 256 Done Done Done false 104764 Acked Write 256 Done Done Done false 104765 Acked Write 256 Done Done Done false 104766 Acked Write 256 Done Done Done false 104767 Acked Write 256 Done Done Done false 104768 Acked Write 256 Done Done Done false 104769 Acked Write 256 Done Done Done false 104770 Acked Write 256 Done Done Done false 104771 Acked Write 256 Done Done Done false 104772 Acked Write 256 Done Done Done false 104773 Acked Write 256 Done Done Done false 104774 Acked Write 256 Done Done Done false 104775 Acked Write 256 Done Done Done false 104776 Acked Write 256 Done Done Done false 104777 Acked Write 256 Done Done Done false 104778 Acked Write 256 Done Done Done false 104779 Acked Write 256 Done Done Done false 104780 Acked Write 256 Done Done Done false 104781 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 72 72 71 215 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(104781)) Some(JobId(104781)) None Downstairs last five completed: 104709 104708 104707 104706 104705 Upstairs last five completed: 104781 104709 104780 104779 104778 CLIENT: Up:0 ds:72 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 104781 104780 104779 104778 104777 Upstairs last five completed: 104781 104709 104780 104779 104778 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program