Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1451) [8810][0] reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1452) [8830][2] reports Running Feb 07 18:57:06.665 INFO listening, local_addr: 127.0.0.1:9998 Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1453) [8820][1] reports Running Control access at:127.0.0.1:9998 Feb 07 18:57:11.705 INFO accepted connection, remote_addr: 127.0.0.1:33707, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Feb 07 18:57:11.706 INFO request completed, latency_us: 120, response_code: 204[8810] Disable keep_running , [8820] Disable keep_running uri[8830] Disable keep_running : /disablerestart/all, method: POST, req_id: 805b0fa2-cdbd-43d7-8514-1efc6fc3ec36, remote_addr: 127.0.0.1:33707, local_addr: 127.0.0.1:9998 Feb 07 19:01:20.322 INFO accepted connection, remote_addr: 127.0.0.1:65071, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Feb 07 19:01:20.323 [8810] Disable keep_running INFOShut it down [8830] Disable keep_running request completed[8820] Disable keep_running , [8830] Got stop action so:false kr:false latency_us: 47, response_code: 204, uri: /shutdown, method: POST, req_id[8810] Got stop action so:false kr:false : a5b7b947-ae68-4f02-8c83-dd8ae83c7d7f, remote_addr: 127.0.0.[8820] Got stop action so:false kr:false 1:65071, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8820] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit e":"2024-02-07T18:57:11.717396Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.71740598Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456} {"msg":"Crucible 82b2a06f-7e39-40d2-8ef1-fb7defdbffe2 has session id: dd400f1a-7cd7-4044-bcf5-df2343a18ccd","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.71741081Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Upstairs opts: Upstairs UUID: 82b2a06f-7e39-40d2-8ef1-fb7defdbffe2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717420029Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Crucible stats registered with UUID: 82b2a06f-7e39-40d2-8ef1-fb7defdbffe2","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717424829Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717428779Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456} {"msg":"82b2a06f-7e39-40d2-8ef1-fb7defdbffe2 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717432389Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717436149Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717446689Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717451658Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717456068Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717460358Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717464698Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717469148Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717475668Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717480268Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717570375Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717595474Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717617233Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717872866Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717881666Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717917265Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717928694Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717941074Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.717949094Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.718311653Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.718325623Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.718335972Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.725860332Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.725899471Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.725910111Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.726603621Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.72661293Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.72661796Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.7269666Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.7269739Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.72697869Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.726986599Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.726991329Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.726995699Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727000249Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727004669Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727008909Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727013149Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727017418Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727021629Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727026248Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727030888Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727035238Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"":"downstairs","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727039488Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727043238Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"0","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727047578Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"1","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727053337Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"client":"2","":"downstairs","session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727057887Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727061927Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"82b2a06f-7e39-40d2-8ef1-fb7defdbffe2 is now active with session: dd400f1a-7cd7-4044-bcf5-df2343a18ccd","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-02-07T18:57:11.727065877Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727073577Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456,"session_id":"dd400f1a-7cd7-4044-bcf5-df2343a18ccd"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-07T18:57:11.727077597Z","hostname":"ip-10-150-1-149.us-west-2.compute.internal","pid":1456} Region: es:16384 ec:1600 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 262608 Acked 263607 Write 100 Done Done Done false 262609 Acked 263608 Write 100 Done Done Done false 262610 Acked 263609 Write 100 Done Done Done false 262611 Acked 263610 Write 100 Done Done Done false 262612 Acked 263611 Write 100 Done Done Done false 262613 Acked 263612 Write 100 Done Done Done false 262614 Acked 263613 Write 100 Done Done Done false 262615 Acked 263614 Write 100 Done Done Done false 262616 Acked 263615 Write 100 Done Done Done false 262617 Acked 263616 Write 100 Done Done Done false 262618 Acked 263617 Write 100 Done Done Done false 262619 Acked 263618 Write 100 Done Done Done false 262620 Acked 263619 Write 100 Done Done Done false 262621 Acked 263620 Write 100 Done Done Done false 262622 Acked 263621 Write 100 Done Done Done false 262623 Acked 263622 Write 100 Done Done Done false 262624 Acked 263623 Write 100 Done Done Done false 262625 Acked 263624 Write 100 Done Done Done false 262626 Acked 263625 Write 100 Done Done Done false 262627 Acked 263626 Write 100 Done Done Done false 262628 Acked 263627 Write 100 Done Done Done false 262629 Acked 263628 Write 100 Done Done Done false 262630 Acked 263629 Write 100 Done Done Done false 262631 Acked 263630 Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 1 1 Done 24 24 23 71 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 263630 263630 263606 Downstairs last five completed: 263606 263605 263604 263603 263602 Upstairs last five completed: 262631 262607 262630 262629 262628 CLIENT: Up:0 ds:24 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 263630 263629 263628 263627 263626 Upstairs last five completed: 262631 262607 262630 262629 262628 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program