Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 [8810][0] reports Starting Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1455) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1456) [8830][2] reports Running Mar 06 21:02:07.160 INFO listening, local_addr: 127.0.0.1:9998 Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1457) [8820][1] reports Running Control access at:127.0.0.1:9998 Mar 06 21:02:12.196 INFO accepted connection, remote_addr: 127.0.0.1:49329, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Mar 06 21:02:12.197 INFO request completed, latency_us: 112, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 47799412-e7e1-40fe-b488-fd24c334e292, remote_addr[8820] Disable keep_running :[8810] Disable keep_running 127.0.0.1:49329, local_addr: 127.0.0.1:[8830] Disable keep_running 9998 Mar 06 21:07:35.771 INFO accepted connection, remote_addr: 127.0.0.1:59213, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Mar 06 21:07:35.772 INFO request completed, latency_us: 46, response_code[8810] Disable keep_running : 204, uri: /shutdown[8830] Disable keep_running , method[8810] Got stop action so:false kr:false : POST, req_id: ea8a7c14-a500-4928-8312-da6d6ef9230d, remote_addr: 127.0.0.1:59213, local_addr[8820] Disable keep_running : 127.0.0.1:9998 [8830] Got stop action so:false kr:false [8820] Got stop action so:false kr:false [8810] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820][1] reports Exit [8830][2] reports Exit T21:02:12.208033218Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208046067Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460} {"msg":"Crucible f2b8902b-06c0-4616-b163-2f8e6587413d has session id: e713dca9-e740-4e21-ad84-56a4aa85d424","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208050197Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Upstairs opts: Upstairs UUID: f2b8902b-06c0-4616-b163-2f8e6587413d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208056806Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Crucible stats registered with UUID: f2b8902b-06c0-4616-b163-2f8e6587413d","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208060986Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208064825Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460} {"msg":"f2b8902b-06c0-4616-b163-2f8e6587413d active request set","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208078474Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208082333Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208092062Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208096932Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208101381Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208105621Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.20810979Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208163355Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208186972Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208198651Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208278432Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208292721Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.20830303Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"io task","client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208597509Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208607377Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208612847Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208617896Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208622616Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208629655Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.20886401Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208873479Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.208878789Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21508032Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215092308Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215099408Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215344652Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21535474Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21535995Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215444491Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21545097Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21545555Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215460139Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215464669Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215471558Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215476018Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215480477Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215484857Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215489166Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215493496Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215498115Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215502805Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215507154Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215511564Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"":"downstairs","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215515963Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215519733Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"0","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215524102Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"1","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215528452Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"client":"2","":"downstairs","session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215532772Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215536601Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"f2b8902b-06c0-4616-b163-2f8e6587413d is now active with session: e713dca9-e740-4e21-ad84-56a4aa85d424","Wait for a query_work_queue command to finish before sending IO v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21554467Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.21554881Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460,"session_id":"e713dca9-e740-4e21-ad84-56a4aa85d424"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-03-06T21:02:12.215552639Z","hostname":"w-01HRARZ93N9BGQW4HCBGR72YKX","pid":1460} Region: es:16384 ec:1600 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:88 States: Active Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 262694 Acked 263693 Write 100 Done Done Done false 262695 Acked 263694 Write 100 Done Done Done false 262696 Acked 263695 Write 100 Done Done Done false 262697 Acked 263696 Write 100 Done Done Done false 262698 Acked 263697 Write 100 Done Done Done false 262699 Acked 263698 Write 100 Done Done Done false 262700 Acked 263699 Write 100 Done Done Done false 262701 Acked 263700 Write 100 Done Done Done false 262702 Acked 263701 Write 100 Done Done Done false 262703 Acked 263702 Write 100 Done Done Done false 262704 Acked 263703 Write 100 Done Done Done false 262705 Acked 263704 Write 100 Done Done Done false 262706 Acked 263705 Write 100 Done Done Done false 262707 Acked 263706 Write 100 Done Done Done false 262708 Acked 263707 Write 100 Done Done Done false 262709 Acked 263708 Write 100 Done Done Done false 262710 Acked 263709 Write 100 Done Done Done false 262711 Acked 263710 Write 100 Done Done Done false 262712 Acked 263711 Write 100 Done Done Done false 262713 Acked 263712 Write 100 Done Done Done false 262714 Acked 263713 Write 100 Done Done Done false 262715 Acked 263714 Write 100 Done Done Done false 262716 Acked 263715 Write 100 Done Done Done false 262717 Acked 263716 Write 100 Done Done Done false 262718 Acked 263717 Write 100 Done Done Done false 262719 Acked 263718 Write 100 Done Done Done false 262720 Acked 263719 Write 100 Done Done Done false 262721 Acked 263720 Write 100 Done Done Done false 262722 Acked 263721 Write 100 Done Done Done false 262723 Acked 263722 Write 100 Done Done Done false 262724 Acked 263723 Write 100 Done Done Done false 262725 Acked 263724 Write 100 Done Done Done false 262726 Acked 263725 Write 100 Done Done Done false 262727 Acked 263726 Write 100 Done Done Done false 262728 Acked 263727 Write 100 Done Done Done false 262729 Acked 263728 Write 100 Done Done Done false 262730 Acked 263729 Write 100 Done Done Done false 262731 Acked 263730 Write 100 Done Done Done false 262732 Acked 263731 Write 100 Done Done Done false 262733 Acked 263732 Write 100 Done Done Done false 262734 Acked 263733 Write 100 Done Done Done false 262735 Acked 263734 Write 100 Done Done Done false 262736 Acked 263735 Write 100 Done Done Done false 262737 Acked 263736 Write 100 Done Done Done false 262738 Acked 263737 Write 100 Done Done Done false 262739 Acked 263738 Write 100 Done Done Done false 262740 Acked 263739 Write 100 Done Done Done false 262741 Acked 263740 Write 100 Done Done Done false 262742 Acked 263741 Write 100 Done Done Done false 262743 Acked 263742 Write 100 Done Done Done false 262744 Acked 263743 Write 100 Done Done Done false 262745 Acked 263744 Write 100 Done Done Done false 262746 Acked 263745 Write 100 Done Done Done false 262747 Acked 263746 Write 100 Done Done Done false 262748 Acked 263747 Write 100 Done Done Done false 262749 Acked 263748 Write 100 Done Done Done false 262750 Acked 263749 Write 100 Done Done Done false 262751 Acked 263750 Write 100 Done Done Done false 262752 Acked 263751 Write 100 Done Done Done false 262753 Acked 263752 Write 100 Done Done Done false 262754 Acked 263753 Write 100 Done Done Done false 262755 Acked 263754 Write 100 Done Done Done false 262756 Acked 263755 Write 100 Done Done Done false 262757 Acked 263756 Write 100 Done Done Done false 262758 Acked 263757 Write 100 Done Done Done false 262759 Acked 263758 Write 100 Done Done Done false 262760 Acked 263759 Write 100 Done Done Done false 262761 Acked 263760 Write 100 Done Done Done false 262762 Acked 263761 Write 100 Done Done Done false 262763 Acked 263762 Write 100 Done Done Done false 262764 Acked 263763 Write 100 Done Done Done false 262765 Acked 263764 Write 100 Done Done Done false 262766 Acked 263765 Write 100 Done Done Done false 262767 Acked 263766 Write 100 Done Done Done false 262768 Acked 263767 Write 100 Done Done Done false 262769 Acked 263768 Write 100 Done Done Done false 262770 Acked 263769 Write 100 Done Done Done false 262771 Acked 263770 Write 100 Done Done Done false 262772 Acked 263771 Write 100 Done Done Done false 262773 Acked 263772 Write 100 Done Done Done false 262774 Acked 263773 Write 100 Done Done Done false 262775 Acked 263774 Write 100 Done Done Done false 262776 Acked 263775 Write 100 Done Done Done false 262777 Acked 263776 Write 100 Done Done Done false 262778 Acked 263777 Write 100 Done Done Done false 262779 Acked 263778 Write 100 Done Done Done false 262780 Acked 263779 Write 100 Done Done Done false 262781 Acked 263780 Flush 0 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 87 88 88 263 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 263692 263780 263780 Downstairs last five completed: 263692 263691 263690 263689 263688 Upstairs last five completed: 262781 262780 262779 262778 262777 CLIENT: Up:0 ds:88 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 263780 263779 263778 263777 263776 Upstairs last five completed: 262781 262780 262779 262778 262777 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program