Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1489) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1490) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1491) [8820][1] initial start wait reports Running All downstairs are running Nov 01 19:55:26.704 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Nov 01 19:55:31.743 INFO accepted connection, remote_addr: 127.0.0.1:63280, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Nov 01 19:55:31.744 INFO request completed, latency_us: 110, response_code: 204, uri: /disablerestart/all, method: POST, req_id: f011f1a0-da36-4d0d-8bde-2f4fe142dd00, remote_addr: 127.0.0[8820] Disable keep_running .[8810] Disable keep_running 1:63280, local_addr: 127.0.0.1:[8830] Disable keep_running 9998 Nov 01 19:57:08.181 INFO accepted connection, remote_addr: 127.0.0.1:52483, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Nov 01 19:57:08.182 INFO request completed, latency_usShut it down :[8810] Disable keep_running [8830] Disable keep_running 62, response_code: 204[8830] Got stop action so:false kr:false , uri:[8810] Got stop action so:false kr:false [8820] Disable keep_running /shutdown, method: POST, req_id: 1d3dcf98-bf3e-40dd-a53a-810cbb5cd960, remote_addr: 127.0.0.1:52483, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit 4} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"bcad54297a74678c586d050f6f6ca6a7fdd7e549\",\n git_commit_timestamp: \"2024-11-01T18:47:47.000000000Z\",\n git_branch: \"mkeeter/no-auto-flush\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.795927185Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.795933064Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.795938084Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494} {"msg":"Crucible b85a5487-d8f3-4418-aef6-988dab9ec6c5 has session id: 7588bf81-f6b8-4a46-be65-e7b5eda099a9","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.796256306Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Upstairs opts: Upstairs UUID: b85a5487-d8f3-4418-aef6-988dab9ec6c5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.796276444Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Crucible stats registered with UUID: b85a5487-d8f3-4418-aef6-988dab9ec6c5","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.796287044Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826076979Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826086678Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494} {"msg":"b85a5487-d8f3-4418-aef6-988dab9ec6c5 active request set","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826134826Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826142155Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826158224Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826168274Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826178113Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826187663Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826197372Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826207371Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826217191Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.82622711Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:49797)","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826281677Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:40433)","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826299776Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:58025)","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826365082Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826729151Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.82674332Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826753359Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826762199Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826769109Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826774358Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.826941939Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.827014054Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.827023854Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.833545472Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.833559601Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.833565611Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.834978438Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.834988917Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.834994517Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835200535Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835209254Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835216104Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835221723Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835227203Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835232533Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835237762Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835243102Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835248422Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835253692Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835259021Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835264321Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835269661Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.83527496Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.83528041Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"":"downstairs","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835288Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835292869Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"0","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835298219Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"1","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835303479Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"client":"2","":"downstairs","session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835309028Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835314158Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"b85a5487-d8f3-4418-aef6-988dab9ec6c5 is now active with session: 7588bf81-f6b8-4a46-be65-e7b5eda099a9","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835318918Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835323568Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494,"session_id":"7588bf81-f6b8-4a46-be65-e7b5eda099a9"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-11-01T19:55:31.835328177Z","hostname":"w-01JBMMMYZRJ2P6DPSFPKYE5AAK","pid":1494} Wait for a query_work_queue command to finish before sending IO Region: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:322 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 103178 Acked Write 256 Done Done Done false 103179 Acked Write 256 Done Done Done false 103180 Acked Write 256 Done Done Done false 103181 Acked Write 256 Done Done Done false 103182 Acked Write 256 Done Done Done false 103183 Acked Write 256 Done Done Done false 103184 Acked Write 256 Done Done Done false 103185 Acked Write 256 Done Done Done false 103186 Acked Write 256 Done Done Done false 103187 Acked Write 256 Done Done Done false 103188 Acked Write 256 Done Done Done false 103189 Acked Write 256 Done Done Done false 103190 Acked Write 256 Done Done Done false 103191 Acked Write 256 Done Done Done false 103192 Acked Write 256 Done Done Done false 103193 Acked Write 256 Done Done Done false 103194 Acked Write 256 Done Done Done false 103195 Acked Write 256 Done Done Done false 103196 Acked Write 256 Done Done Done false 103197 Acked Write 256 Done Done Done false 103198 Acked Write 256 Done Done Done false 103199 Acked Write 256 Done Done Done false 103200 Acked Write 256 Done Done Done false 103201 Acked Write 256 Done Done Done false 103202 Acked Write 256 Done Done Done false 103203 Acked Write 256 Done Done Done false 103204 Acked Write 256 Done Done Done false 103205 Acked Write 256 Done Done Done false 103206 Acked Write 256 Done Done Done false 103207 Acked Write 256 Done Done Done false 103208 Acked Write 256 Done Done Done false 103209 Acked Write 256 Done Done Done false 103210 Acked Write 256 Done Done Done false 103211 Acked Write 256 Done Done Done false 103212 Acked Write 256 Done Done Done false 103213 Acked Write 256 Done Done Done false 103214 Acked Write 256 Done Done Done false 103215 Acked Write 256 Done Done Done false 103216 Acked Write 256 Done Done Done false 103217 Acked Write 256 Done Done Done false 103218 Acked Write 256 Done Done Done false 103219 Acked Write 256 Done Done Done false 103220 Acked Write 256 Done Done Done false 103221 Acked Write 256 Done Done Done false 103222 Acked Write 256 Done Done Done false 103223 Acked Write 256 Done Done Done false 103224 Acked Write 256 Done Done Done false 103225 Acked Write 256 Done Done Done false 103226 Acked Write 256 Done Done Done false 103227 Acked Write 256 Done Done Done false 103228 Acked Write 256 Done Done Done false 103229 Acked Write 256 Done Done Done false 103230 Acked Write 256 Done Done Done false 103231 Acked Write 256 Done Done Done false 103232 Acked Write 256 Done Done Done false 103233 Acked Write 256 Done Done Done false 103234 Acked Write 256 Done Done Done false 103235 Acked Write 256 Done Done Done false 103236 Acked Write 256 Done Done Done false 103237 Acked Write 256 Done Done Done false 103238 Acked Write 256 Done Done Done false 103239 Acked Write 256 Done Done Done false 103240 Acked Write 256 Done Done Done false 103241 Acked Write 256 Done Done Done false 103242 Acked Write 256 Done Done Done false 103243 Acked Write 256 Done Done Done false 103244 Acked Write 256 Done Done Done false 103245 Acked Write 256 Done Done Done false 103246 Acked Write 256 Done Done Done false 103247 Acked Write 256 Done Done Done false 103248 Acked Write 256 Done Done Done false 103249 Acked Write 256 Done Done Done false 103250 Acked Write 256 Done Done Done false 103251 Acked Write 256 Done Done Done false 103252 Acked Write 256 Done Done Done false 103253 Acked Write 256 Done Done Done false 103254 Acked Write 256 Done Done Done false 103255 Acked Write 256 Done Done Done false 103256 Acked Write 256 Done Done Done false 103257 Acked Write 256 Done Done Done false 103258 Acked Write 256 Done Done Done false 103259 Acked Write 256 Done Done Done false 103260 Acked Write 256 Done Done Done false 103261 Acked Write 256 Done Done Done false 103262 Acked Write 256 Done Done Done false 103263 Acked Write 256 Done Done Done false 103264 Acked Write 256 Done Done Done false 103265 Acked Write 256 Done Done Done false 103266 Acked Write 256 Done Done Done false 103267 Acked Write 256 Done Done Done false 103268 Acked Write 256 Done Done Done false 103269 Acked Write 256 Done Done Done false 103270 Acked Write 256 Done Done Done false 103271 Acked Write 256 Done Done Done false 103272 Acked Write 256 Done Done Done false 103273 Acked Write 256 Done Done Done false 103274 Acked Write 256 Done Done Done false 103275 Acked Write 256 Done Done Done false 103276 Acked Write 256 Done Done Done false 103277 Acked Write 256 Done Done Done false 103278 Acked Write 256 Done Done Done false 103279 Acked Write 256 Done Done Done false 103280 Acked Write 256 Done Done Done false 103281 Acked Write 256 Done Done Done false 103282 Acked Write 256 Done Done Done false 103283 Acked Write 256 Done Done Done false 103284 Acked Write 256 Done Done Done false 103285 Acked Write 256 Done Done Done false 103286 Acked Write 256 Done Done Done false 103287 Acked Write 256 Done Done Done false 103288 Acked Write 256 Done Done Done false 103289 Acked Write 256 Done Done Done false 103290 Acked Write 256 Done Done Done false 103291 Acked Write 256 Done Done Done false 103292 Acked Write 256 Done Done Done false 103293 Acked Write 256 Done Done Done false 103294 Acked Write 256 Done Done Done false 103295 Acked Write 256 Done Done Done false 103296 Acked Write 256 Done Done Done false 103297 Acked Write 256 Done Done Done false 103298 Acked Write 256 Done Done Done false 103299 Acked Write 256 Done Done Done false 103300 Acked Write 256 Done Done Done false 103301 Acked Write 256 Done Done Done false 103302 Acked Write 256 Done Done Done false 103303 Acked Write 256 Done Done Done false 103304 Acked Write 256 Done Done Done false 103305 Acked Write 256 Done Done Done false 103306 Acked Write 256 Done Done Done false 103307 Acked Write 256 Done Done Done false 103308 Acked Write 256 Done Done Done false 103309 Acked Write 256 Done Done Done false 103310 Acked Write 256 Done Done Done false 103311 Acked Write 256 Done Done Done false 103312 Acked Write 256 Done Done Done false 103313 Acked Write 256 Done Done Done false 103314 Acked Write 256 Done Done Done false 103315 Acked Write 256 Done Done Done false 103316 Acked Write 256 Done Done Done false 103317 Acked Write 256 Done Done Done false 103318 Acked Write 256 Done Done Done false 103319 Acked Write 256 Done Done Done false 103320 Acked Write 256 Done Done Done false 103321 Acked Write 256 Done Done Done false 103322 Acked Write 256 Done Done Done false 103323 Acked Write 256 Done Done Done false 103324 Acked Write 256 Done Done Done false 103325 Acked Write 256 Done Done Done false 103326 Acked Write 256 Done Done Done false 103327 Acked Write 256 Done Done Done false 103328 Acked Write 256 Done Done Done false 103329 Acked Write 256 Done Done Done false 103330 Acked Write 256 Done Done Done false 103331 Acked Write 256 Done Done Done false 103332 Acked Write 256 Done Done Done false 103333 Acked Write 256 Done Done Done false 103334 Acked Write 256 Done Done Done false 103335 Acked Write 256 Done Done Done false 103336 Acked Write 256 Done Done Done false 103337 Acked Write 256 Done Done Done false 103338 Acked Write 256 Done Done Done false 103339 Acked Write 256 Done Done Done false 103340 Acked Write 256 Done Done Done false 103341 Acked Write 256 Done Done Done false 103342 Acked Write 256 Done Done Done false 103343 Acked Write 256 Done Done Done false 103344 Acked Write 256 Done Done Done false 103345 Acked Write 256 Done Done Done false 103346 Acked Write 256 Done Done Done false 103347 Acked Write 256 Done Done Done false 103348 Acked Write 256 Done Done Done false 103349 Acked Write 256 Done Done Done false 103350 Acked Write 256 Done Done Done false 103351 Acked Write 256 Done Done Done false 103352 Acked Write 256 Done Done Done false 103353 Acked Write 256 Done Done Done false 103354 Acked Write 256 Done Done Done false 103355 Acked Write 256 Done Done Done false 103356 Acked Write 256 Done Done Done false 103357 Acked Write 256 Done Done Done false 103358 Acked Write 256 Done Done Done false 103359 Acked Write 256 Done Done Done false 103360 Acked Write 256 Done Done Done false 103361 Acked Write 256 Done Done Done false 103362 Acked Write 256 Done Done Done false 103363 Acked Write 256 Done Done Done false 103364 Acked Write 256 Done Done Done false 103365 Acked Write 256 Done Done Done false 103366 Acked Write 256 Done Done Done false 103367 Acked Write 256 Done Done Done false 103368 Acked Write 256 Done Done Done false 103369 Acked Write 256 Done Done Done false 103370 Acked Write 256 Done Done Done false 103371 Acked Write 256 Done Done Done false 103372 Acked Write 256 Done Done Done false 103373 Acked Write 256 Done Done Done false 103374 Acked Write 256 Done Done Done false 103375 Acked Write 256 Done Done Done false 103376 Acked Write 256 Done Done Done false 103377 Acked Write 256 Done Done Done false 103378 Acked Write 256 Done Done Done false 103379 Acked Write 256 Done Done Done false 103380 Acked Write 256 Done Done Done false 103381 Acked Write 256 Done Done Done false 103382 Acked Write 256 Done Done Done false 103383 Acked Write 256 Done Done Done false 103384 Acked Write 256 Done Done Done false 103385 Acked Write 256 Done Done Done false 103386 Acked Write 256 Done Done Done false 103387 Acked Write 256 Done Done Done false 103388 Acked Write 256 Done Done Done false 103389 Acked Write 256 Done Done Done false 103390 Acked Write 256 Done Done Done false 103391 Acked Write 256 Done Done Done false 103392 Acked Write 256 Done Done Done false 103393 Acked Write 256 Done Done Done false 103394 Acked Write 256 Done Done Done false 103395 Acked Write 256 Done Done Done false 103396 Acked Write 256 Done Done Done false 103397 Acked Write 256 Done Done Done false 103398 Acked Write 256 Done Done Done false 103399 Acked Write 256 Done Done Done false 103400 Acked Write 256 Done Done Done false 103401 Acked Write 256 Done Done Done false 103402 Acked Write 256 Done Done Done false 103403 Acked Write 256 Done Done Done false 103404 Acked Write 256 Done Done Done false 103405 Acked Write 256 Done Done Done false 103406 Acked Write 256 Done Done Done false 103407 Acked Write 256 Done Done Done false 103408 Acked Write 256 Done Done Done false 103409 Acked Write 256 Done Done Done false 103410 Acked Write 256 Done Done Done false 103411 Acked Write 256 Done Done Done false 103412 Acked Write 256 Done Done Done false 103413 Acked Write 256 Done Done Done false 103414 Acked Write 256 Done Done Done false 103415 Acked Write 256 Done Done Done false 103416 Acked Write 256 Done Done Done false 103417 Acked Write 256 Done Done Done false 103418 Acked Write 256 Done Done Done false 103419 Acked Write 256 Done Done Done false 103420 Acked Write 256 Done Done Done false 103421 Acked Write 256 Done Done Done false 103422 Acked Write 256 Done Done Done false 103423 Acked Write 256 Done Done Done false 103424 Acked Write 256 Done Done Done false 103425 Acked Write 256 Done Done Done false 103426 Acked Write 256 Done Done Done false 103427 Acked Write 256 Done Done Done false 103428 Acked Write 256 Done Done Done false 103429 Acked Write 256 Done Done Done false 103430 Acked Write 256 Done Done Done false 103431 Acked Write 256 Done Done Done false 103432 Acked Write 256 Done Done Done false 103433 Acked Write 256 Done Done Done false 103434 Acked Write 256 Done Done Done false 103435 Acked Write 256 Done Done Done false 103436 Acked Write 256 Done Done Done false 103437 Acked Write 256 Done Done Done false 103438 Acked Write 256 Done Done Done false 103439 Acked Write 256 Done Done Done false 103440 Acked Write 256 Done Done Done false 103441 Acked Write 256 Done Done Done false 103442 Acked Write 256 Done Done Done false 103443 Acked Write 256 Done Done Done false 103444 Acked Write 256 Done Done Done false 103445 Acked Write 256 Done Done Done false 103446 Acked Write 256 Done Done Done false 103447 Acked Write 256 Done Done Done false 103448 Acked Write 256 Done Done Done false 103449 Acked Write 256 Done Done Done false 103450 Acked Write 256 Done Done Done false 103451 Acked Write 256 Done Done Done false 103452 Acked Write 256 Done Done Done false 103453 Acked Write 256 Done Done Done false 103454 Acked Write 256 Done Done Done false 103455 Acked Write 256 Done Done Done false 103456 Acked Write 256 Done Done Done false 103457 Acked Write 256 Done Done Done false 103458 Acked Write 256 Done Done Done false 103459 Acked Write 256 Done Done Done false 103460 Acked Write 256 Done Done Done false 103461 Acked Write 256 Done Done Done false 103462 Acked Write 256 Done Done Done false 103463 Acked Write 256 Done Done Done false 103464 Acked Write 256 Done Done Done false 103465 Acked Write 256 Done Done Done false 103466 Acked Write 256 Done Done Done false 103467 Acked Write 256 Done Done Done false 103468 Acked Write 256 Done Done Done false 103469 Acked Write 256 Done Done Done false 103470 Acked Write 256 Done Done Done false 103471 Acked Write 256 Done Done Done false 103472 Acked Write 256 Done Done Done false 103473 Acked Write 256 Done Done Done false 103474 Acked Write 256 Done Done Done false 103475 Acked Write 256 Done Done Done false 103476 Acked Write 256 Done Done Done false 103477 Acked Write 256 Done Done Done false 103478 Acked Write 256 Done Done Done false 103479 Acked Write 256 Done Done Done false 103480 Acked Write 256 Done Done Done false 103481 Acked Write 256 Done Done Done false 103482 Acked Write 256 Done Done Done false 103483 Acked Write 256 Done Done Done false 103484 Acked Write 256 Done Done Done false 103485 Acked Write 256 Done Done Done false 103486 Acked Write 256 Done Done Done false 103487 Acked Write 256 Done Done Done false 103488 Acked Write 256 Done Done Done false 103489 Acked Write 256 Done Done Done false 103490 Acked Write 256 Done Done Done false 103491 Acked Write 256 Done Done Done false 103492 Acked Write 256 Done Done Done false 103493 Acked Write 256 Done Done Done false 103494 Acked Write 256 Done Done Done false 103495 Acked Write 256 Done Done Done false 103496 Acked Write 256 Done Done Done false 103497 Acked Write 256 Done Done Done false 103498 Acked Write 256 Done Done Done false 103499 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 322 322 321 965 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 103499 103499 0 Downstairs last five completed: 103177 103176 103175 103174 103173 Upstairs last five completed: 103499 103498 103497 103496 103495 CLIENT: Up:0 ds:322 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 103499 103498 103497 103496 103495 Upstairs last five completed: 103499 103498 103497 103496 103495 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program