Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 [8810][0] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1490) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1491) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1492) [8820][1] initial start wait reports Running All downstairs are running Nov 01 16:50:17.157 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Nov 01 16:50:22.195 INFO accepted connection, remote_addr: 127.0.0.1:48766, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll Nov 01 16:50:22.196 INFO request completed, latency_us: 108, response_code: 204, uri: /disablerestart/all, method: POST, req_id: 9f052564-fc50-46d5-89ff-ef23e1edf639, remote_addr: 127.0.0.1:disable restart on all: 3 48766, local_addr: 127.0.0.1:9998 [8810] Disable keep_running [8820] Disable keep_running [8830] Disable keep_running Nov 01 16:52:11.394 INFO accepted connection, remote_addr: 127.0.0.1:48389, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Nov 01 16:52:11.395 INFO request completed, latency_us: 58, response_code: 204, Shut it down uri: /shutdown, method: POST, req_id: da8d6738-230e-4253-9a6f-9fcd840f4ba7, remote_addr: 127.0.0.1:48389[8810] Disable keep_running , [8810] Got stop action so:false kr:false local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Disable keep_running [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit } {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3818f0fc76fe787c6cc412af30fd59b930203885\",\n git_commit_timestamp: \"2024-11-01T16:08:30.000000000Z\",\n git_branch: \"renovate/hyper-1.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.247078537Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.247086888Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.247094188Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495} {"msg":"Crucible 3e26af76-c9b6-490d-abaf-5938a6a5ae2a has session id: 743e3245-b998-48f4-bdbb-a7db20c0757f","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.247429598Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Upstairs opts: Upstairs UUID: 3e26af76-c9b6-490d-abaf-5938a6a5ae2a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.247448219Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Crucible stats registered with UUID: 3e26af76-c9b6-490d-abaf-5938a6a5ae2a","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.24745708Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277082276Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277094146Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495} {"msg":"3e26af76-c9b6-490d-abaf-5938a6a5ae2a active request set","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277105217Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277111387Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277126068Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277131928Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277137219Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277142409Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.2771477Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.27715323Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.27716234Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277172541Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_connection connected from Ok(127.0.0.1:64545)","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277251936Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_connection connected from Ok(127.0.0.1:47389)","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277265807Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43998)","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277274107Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277555984Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277600156Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277636059Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277641849Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277646049Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277651079Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.27783012Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277840971Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.277907015Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28436615Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28437442Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28437966Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285681698Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285691129Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285696159Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285836687Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285846378Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285851478Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285856008Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285860509Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285864789Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285868939Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285873189Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28587741Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28588171Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28588597Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.28589017Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285894561Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285898871Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285903081Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"":"downstairs","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285910052Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285913852Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"0","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285918562Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"1","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285922943Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"client":"2","":"downstairs","session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285927203Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285930943Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"3e26af76-c9b6-490d-abaf-5938a6a5ae2a is now active with session: 743e3245-b998-48f4-bdbb-a7db20c0757f","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285934733Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285938404Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495,"session_id":"743e3245-b998-48f4-bdbb-a7db20c0757f"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:22.285942104Z","hostname":"w-01JBMA1HE651A32K3750KQ19XV","pid":1495} Wait for a query_work_queue command to finish before sending IO Region: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:282 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 103325 Acked Write 256 Done Done Done false 103326 Acked Write 256 Done Done Done false 103327 Acked Write 256 Done Done Done false 103328 Acked Write 256 Done Done Done false 103329 Acked Write 256 Done Done Done false 103330 Acked Write 256 Done Done Done false 103331 Acked Write 256 Done Done Done false 103332 Acked Write 256 Done Done Done false 103333 Acked Write 256 Done Done Done false 103334 Acked Write 256 Done Done Done false 103335 Acked Write 256 Done Done Done false 103336 Acked Write 256 Done Done Done false 103337 Acked Write 256 Done Done Done false 103338 Acked Write 256 Done Done Done false 103339 Acked Write 256 Done Done Done false 103340 Acked Write 256 Done Done Done false 103341 Acked Write 256 Done Done Done false 103342 Acked Write 256 Done Done Done false 103343 Acked Write 256 Done Done Done false 103344 Acked Write 256 Done Done Done false 103345 Acked Write 256 Done Done Done false 103346 Acked Write 256 Done Done Done false 103347 Acked Write 256 Done Done Done false 103348 Acked Write 256 Done Done Done false 103349 Acked Write 256 Done Done Done false 103350 Acked Write 256 Done Done Done false 103351 Acked Write 256 Done Done Done false 103352 Acked Write 256 Done Done Done false 103353 Acked Write 256 Done Done Done false 103354 Acked Write 256 Done Done Done false 103355 Acked Write 256 Done Done Done false 103356 Acked Write 256 Done Done Done false 103357 Acked Write 256 Done Done Done false 103358 Acked Write 256 Done Done Done false 103359 Acked Write 256 Done Done Done false 103360 Acked Write 256 Done Done Done false 103361 Acked Write 256 Done Done Done false 103362 Acked Write 256 Done Done Done false 103363 Acked Write 256 Done Done Done false 103364 Acked Write 256 Done Done Done false 103365 Acked Write 256 Done Done Done false 103366 Acked Write 256 Done Done Done false 103367 Acked Write 256 Done Done Done false 103368 Acked Write 256 Done Done Done false 103369 Acked Write 256 Done Done Done false 103370 Acked Write 256 Done Done Done false 103371 Acked Write 256 Done Done Done false 103372 Acked Write 256 Done Done Done false 103373 Acked Write 256 Done Done Done false 103374 Acked Write 256 Done Done Done false 103375 Acked Write 256 Done Done Done false 103376 Acked Write 256 Done Done Done false 103377 Acked Write 256 Done Done Done false 103378 Acked Write 256 Done Done Done false 103379 Acked Write 256 Done Done Done false 103380 Acked Write 256 Done Done Done false 103381 Acked Write 256 Done Done Done false 103382 Acked Write 256 Done Done Done false 103383 Acked Write 256 Done Done Done false 103384 Acked Write 256 Done Done Done false 103385 Acked Write 256 Done Done Done false 103386 Acked Write 256 Done Done Done false 103387 Acked Write 256 Done Done Done false 103388 Acked Write 256 Done Done Done false 103389 Acked Write 256 Done Done Done false 103390 Acked Write 256 Done Done Done false 103391 Acked Write 256 Done Done Done false 103392 Acked Write 256 Done Done Done false 103393 Acked Write 256 Done Done Done false 103394 Acked Write 256 Done Done Done false 103395 Acked Write 256 Done Done Done false 103396 Acked Write 256 Done Done Done false 103397 Acked Write 256 Done Done Done false 103398 Acked Write 256 Done Done Done false 103399 Acked Write 256 Done Done Done false 103400 Acked Write 256 Done Done Done false 103401 Acked Write 256 Done Done Done false 103402 Acked Write 256 Done Done Done false 103403 Acked Write 256 Done Done Done false 103404 Acked Write 256 Done Done Done false 103405 Acked Write 256 Done Done Done false 103406 Acked Write 256 Done Done Done false 103407 Acked Write 256 Done Done Done false 103408 Acked Write 256 Done Done Done false 103409 Acked Write 256 Done Done Done false 103410 Acked Write 256 Done Done Done false 103411 Acked Write 256 Done Done Done false 103412 Acked Write 256 Done Done Done false 103413 Acked Write 256 Done Done Done false 103414 Acked Write 256 Done Done Done false 103415 Acked Write 256 Done Done Done false 103416 Acked Write 256 Done Done Done false 103417 Acked Write 256 Done Done Done false 103418 Acked Write 256 Done Done Done false 103419 Acked Write 256 Done Done Done false 103420 Acked Write 256 Done Done Done false 103421 Acked Write 256 Done Done Done false 103422 Acked Write 256 Done Done Done false 103423 Acked Write 256 Done Done Done false 103424 Acked Write 256 Done Done Done false 103425 Acked Write 256 Done Done Done false 103426 Acked Write 256 Done Done Done false 103427 Acked Write 256 Done Done Done false 103428 Acked Write 256 Done Done Done false 103429 Acked Write 256 Done Done Done false 103430 Acked Write 256 Done Done Done false 103431 Acked Write 256 Done Done Done false 103432 Acked Write 256 Done Done Done false 103433 Acked Write 256 Done Done Done false 103434 Acked Write 256 Done Done Done false 103435 Acked Write 256 Done Done Done false 103436 Acked Write 256 Done Done Done false 103437 Acked Write 256 Done Done Done false 103438 Acked Write 256 Done Done Done false 103439 Acked Write 256 Done Done Done false 103440 Acked Write 256 Done Done Done false 103441 Acked Write 256 Done Done Done false 103442 Acked Write 256 Done Done Done false 103443 Acked Write 256 Done Done Done false 103444 Acked Write 256 Done Done Done false 103445 Acked Write 256 Done Done Done false 103446 Acked Write 256 Done Done Done false 103447 Acked Write 256 Done Done Done false 103448 Acked Write 256 Done Done Done false 103449 Acked Write 256 Done Done Done false 103450 Acked Write 256 Done Done Done false 103451 Acked Write 256 Done Done Done false 103452 Acked Write 256 Done Done Done false 103453 Acked Write 256 Done Done Done false 103454 Acked Write 256 Done Done Done false 103455 Acked Write 256 Done Done Done false 103456 Acked Write 256 Done Done Done false 103457 Acked Write 256 Done Done Done false 103458 Acked Write 256 Done Done Done false 103459 Acked Write 256 Done Done Done false 103460 Acked Write 256 Done Done Done false 103461 Acked Write 256 Done Done Done false 103462 Acked Write 256 Done Done Done false 103463 Acked Write 256 Done Done Done false 103464 Acked Write 256 Done Done Done false 103465 Acked Write 256 Done Done Done false 103466 Acked Write 256 Done Done Done false 103467 Acked Write 256 Done Done Done false 103468 Acked Write 256 Done Done Done false 103469 Acked Write 256 Done Done Done false 103470 Acked Write 256 Done Done Done false 103471 Acked Write 256 Done Done Done false 103472 Acked Write 256 Done Done Done false 103473 Acked Write 256 Done Done Done false 103474 Acked Write 256 Done Done Done false 103475 Acked Write 256 Done Done Done false 103476 Acked Write 256 Done Done Done false 103477 Acked Write 256 Done Done Done false 103478 Acked Write 256 Done Done Done false 103479 Acked Write 256 Done Done Done false 103480 Acked Write 256 Done Done Done false 103481 Acked Write 256 Done Done Done false 103482 Acked Write 256 Done Done Done false 103483 Acked Write 256 Done Done Done false 103484 Acked Write 256 Done Done Done false 103485 Acked Write 256 Done Done Done false 103486 Acked Write 256 Done Done Done false 103487 Acked Write 256 Done Done Done false 103488 Acked Write 256 Done Done Done false 103489 Acked Write 256 Done Done Done false 103490 Acked Write 256 Done Done Done false 103491 Acked Write 256 Done Done Done false 103492 Acked Write 256 Done Done Done false 103493 Acked Write 256 Done Done Done false 103494 Acked Write 256 Done Done Done false 103495 Acked Write 256 Done Done Done false 103496 Acked Write 256 Done Done Done false 103497 Acked Write 256 Done Done Done false 103498 Acked Write 256 Done Done Done false 103499 Acked Write 256 Done Done Done false 103500 Acked Write 256 Done Done Done false 103501 Acked Write 256 Done Done Done false 103502 Acked Write 256 Done Done Done false 103503 Acked Write 256 Done Done Done false 103504 Acked Write 256 Done Done Done false 103505 Acked Write 256 Done Done Done false 103506 Acked Write 256 Done Done Done false 103507 Acked Write 256 Done Done Done false 103508 Acked Write 256 Done Done Done false 103509 Acked Write 256 Done Done Done false 103510 Acked Write 256 Done Done Done false 103511 Acked Write 256 Done Done Done false 103512 Acked Write 256 Done Done Done false 103513 Acked Write 256 Done Done Done false 103514 Acked Write 256 Done Done Done false 103515 Acked Write 256 Done Done Done false 103516 Acked Write 256 Done Done Done false 103517 Acked Write 256 Done Done Done false 103518 Acked Write 256 Done Done Done false 103519 Acked Write 256 Done Done Done false 103520 Acked Write 256 Done Done Done false 103521 Acked Write 256 Done Done Done false 103522 Acked Write 256 Done Done Done false 103523 Acked Write 256 Done Done Done false 103524 Acked Write 256 Done Done Done false 103525 Acked Write 256 Done Done Done false 103526 Acked Write 256 Done Done Done false 103527 Acked Write 256 Done Done Done false 103528 Acked Write 256 Done Done Done false 103529 Acked Write 256 Done Done Done false 103530 Acked Write 256 Done Done Done false 103531 Acked Write 256 Done Done Done false 103532 Acked Write 256 Done Done Done false 103533 Acked Write 256 Done Done Done false 103534 Acked Write 256 Done Done Done false 103535 Acked Write 256 Done Done Done false 103536 Acked Write 256 Done Done Done false 103537 Acked Write 256 Done Done Done false 103538 Acked Write 256 Done Done Done false 103539 Acked Write 256 Done Done Done false 103540 Acked Write 256 Done Done Done false 103541 Acked Write 256 Done Done Done false 103542 Acked Write 256 Done Done Done false 103543 Acked Write 256 Done Done Done false 103544 Acked Write 256 Done Done Done false 103545 Acked Write 256 Done Done Done false 103546 Acked Write 256 Done Done Done false 103547 Acked Write 256 Done Done Done false 103548 Acked Write 256 Done Done Done false 103549 Acked Write 256 Done Done Done false 103550 Acked Write 256 Done Done Done false 103551 Acked Write 256 Done Done Done false 103552 Acked Write 256 Done Done Done false 103553 Acked Write 256 Done Done Done false 103554 Acked Write 256 Done Done Done false 103555 Acked Write 256 Done Done Done false 103556 Acked Write 256 Done Done Done false 103557 Acked Write 256 Done Done Done false 103558 Acked Write 256 Done Done Done false 103559 Acked Write 256 Done Done Done false 103560 Acked Write 256 Done Done Done false 103561 Acked Write 256 Done Done Done false 103562 Acked Write 256 Done Done Done false 103563 Acked Write 256 Done Done Done false 103564 Acked Write 256 Done Done Done false 103565 Acked Write 256 Done Done Done false 103566 Acked Write 256 Done Done Sent false 103567 Acked Write 256 Done Done Sent false 103568 Acked Write 256 Done Done Sent false 103569 Acked Write 256 Done Done Sent false 103570 Acked Write 256 Done Done Sent false 103571 Acked Write 256 Done Done Sent false 103572 Acked Write 256 Done Done Sent false 103573 Acked Write 256 Done Done Sent false 103574 Acked Write 256 Done Done Sent false 103575 Acked Write 256 Done Done Sent false 103576 Acked Write 256 Done Done Sent false 103577 Acked Write 256 Done Done Sent false 103578 Acked Write 256 Done Done Sent false 103579 Acked Write 256 Done Done Sent false 103580 Acked Write 256 Done Done Sent false 103581 Acked Write 256 Done Done Sent false 103582 Acked Write 256 Done Done Sent false 103583 Acked Write 256 Done Done Sent false 103584 Acked Write 256 Done Done Sent false 103585 Acked Write 256 Done Done Sent false 103586 Acked Write 256 Done Done Sent false 103587 Acked Write 256 Done Done Sent false 103588 Acked Write 256 Done Done Sent false 103589 Acked Write 256 Done Done Sent false 103590 Acked Write 256 Done Done Sent false 103591 Acked Write 256 Done Done Sent false 103592 Acked Write 256 Done Done Sent false 103593 Acked Write 256 Done Done Sent false 103594 Acked Write 256 Done Done Sent false 103595 Acked Write 256 Done Done Sent false 103596 Acked Write 256 Done Done Sent false 103597 Acked Write 256 Done Done Sent false 103598 Acked Write 256 Done Done Sent false 103599 Acked Write 256 Done Done Sent false 103600 Acked Write 256 Done Done Sent false 103601 Acked Write 256 Done Done Sent false 103602 Acked Write 256 Done Done Sent false 103603 Acked Write 256 Done Done Sent false 103604 Acked Write 256 Done Done Sent false 103605 Acked Write 256 Done Done Sent false 103606 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 41 41 Done 282 282 241 805 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 103606 103606 103324 Downstairs last five completed: 103324 103323 103322 103321 103320 Upstairs last five completed: 103606 103605 103604 103603 103602 CLIENT: Up:0 ds:282 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 103606 103605 103604 103603 103602 Upstairs last five completed: 103606 103605 103604 103603 103602 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program