Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8820.txt" Starting downstairs at port 8830 [8820][1] initial start wait reports Starting [8830][2] initial start wait reports Starting Make output file at "/tmp/dsc/downstairs-8830.txt" Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1490) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1491) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1492) [8820][1] initial start wait reports Running All downstairs are running Sep 27 23:07:24.235 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Sep 27 23:07:29.274 INFO accepted connection, remote_addr: 127.0.0.1:64115, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Sep 27 23:07:29.275 INFO request completed, latency_us: 111, response_code: 204, uri: /disablerestart/all, method: POST, req_id:[8810] Disable keep_running [8830] Disable keep_running 809321e9-18c4-46c5-96d7-021fd099f9fd, remote_addr: 127.0.0.1:64115, local_addr: 127.0.0.[8820] Disable keep_running 1:9998 Sep 27 23:09:14.638 INFO accepted connection, remote_addr: 127.0.0.1:46646, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Sep 27 23:09:14.639 INFO request completed[8810] Disable keep_running , [8810] Got stop action so:false kr:false latency_us[8820] Disable keep_running :[8820] Got stop action so:false kr:false 60, response_code: 204, uri: /shutdown, method: POST, req_id: a3c6a611-db13-418f-972f-3aafeb42c3d5, remote_addr: 127.0.0.1:46646, local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit 5} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"60f78286ad49cec90cdcc3e028e5e9c76d4fdc1d\",\n git_commit_timestamp: \"2024-09-27T22:46:32.000000000Z\",\n git_branch: \"alan/tests-use-dsc\",\n rustc_semver: \"1.80.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"051478957371ee0084a7c0913941d2a8c4757bb9\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.326887704Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.326893129Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.326898005Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495} {"msg":"Crucible 9dffd57f-b91a-43f3-8661-a7906396d4de has session id: 07e031cd-dd89-441a-a95a-6d11a5bd90ca","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.327232058Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Upstairs opts: Upstairs UUID: 9dffd57f-b91a-43f3-8661-a7906396d4de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.327248353Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Crucible stats registered with UUID: 9dffd57f-b91a-43f3-8661-a7906396d4de","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.327258355Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.35724869Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495} {"msg":"9dffd57f-b91a-43f3-8661-a7906396d4de active request set","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357261409Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357273159Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357287457Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357297228Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.35730707Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357317001Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357326773Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357335645Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357345477Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357355648Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:41118)","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357405285Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:35011)","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357478253Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:43927)","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357512223Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357753436Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357765665Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357786088Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357791073Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357883334Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.35789962Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.357978482Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.358092824Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.358102895Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364332935Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364345903Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364351738Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364539517Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364550347Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364558021Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364642888Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364655297Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364660333Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364664969Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364669505Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364673741Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364677848Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364682014Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364686191Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364690287Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364694413Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.36469859Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364702876Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364707113Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364711259Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"":"downstairs","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364717204Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364720941Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"0","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364725077Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"1","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364729274Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"client":"2","":"downstairs","session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.36473344Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364737127Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"9dffd57f-b91a-43f3-8661-a7906396d4de is now active with session: 07e031cd-dd89-441a-a95a-6d11a5bd90ca","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364740864Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.36474448Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495,"session_id":"07e031cd-dd89-441a-a95a-6d11a5bd90ca"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T23:07:29.364748137Z","hostname":"w-01J8TVQEQJJVQVDW2XTBJ46Z9S","pid":1495} Wait for a query_work_queue command to finish before sending IO Region: es:16384 ec:1600 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:306 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 102294 Acked 103293 Write 256 Done Done Done false 102295 Acked 103294 Write 256 Done Done Done false 102296 Acked 103295 Write 256 Done Done Done false 102297 Acked 103296 Write 256 Done Done Done false 102298 Acked 103297 Write 256 Done Done Done false 102299 Acked 103298 Write 256 Done Done Done false 102300 Acked 103299 Write 256 Done Done Done false 102301 Acked 103300 Write 256 Done Done Done false 102302 Acked 103301 Write 256 Done Done Done false 102303 Acked 103302 Write 256 Done Done Done false 102304 Acked 103303 Write 256 Done Done Done false 102305 Acked 103304 Write 256 Done Done Done false 102306 Acked 103305 Write 256 Done Done Done false 102307 Acked 103306 Write 256 Done Done Done false 102308 Acked 103307 Write 256 Done Done Done false 102309 Acked 103308 Write 256 Done Done Done false 102310 Acked 103309 Write 256 Done Done Done false 102311 Acked 103310 Write 256 Done Done Done false 102312 Acked 103311 Write 256 Done Done Done false 102313 Acked 103312 Write 256 Done Done Done false 102314 Acked 103313 Write 256 Done Done Done false 102315 Acked 103314 Write 256 Done Done Done false 102316 Acked 103315 Write 256 Done Done Done false 102317 Acked 103316 Write 256 Done Done Done false 102318 Acked 103317 Write 256 Done Done Done false 102319 Acked 103318 Write 256 Done Done Done false 102320 Acked 103319 Write 256 Done Done Done false 102321 Acked 103320 Write 256 Done Done Done false 102322 Acked 103321 Write 256 Done Done Done false 102323 Acked 103322 Write 256 Done Done Done false 102324 Acked 103323 Write 256 Done Done Done false 102325 Acked 103324 Write 256 Done Done Done false 102326 Acked 103325 Write 256 Done Done Done false 102327 Acked 103326 Write 256 Done Done Done false 102328 Acked 103327 Write 256 Done Done Done false 102329 Acked 103328 Write 256 Done Done Done false 102330 Acked 103329 Write 256 Done Done Done false 102331 Acked 103330 Write 256 Done Done Done false 102332 Acked 103331 Write 256 Done Done Done false 102333 Acked 103332 Write 256 Done Done Done false 102334 Acked 103333 Write 256 Done Done Done false 102335 Acked 103334 Write 256 Done Done Done false 102336 Acked 103335 Write 256 Done Done Done false 102337 Acked 103336 Write 256 Done Done Done false 102338 Acked 103337 Write 256 Done Done Done false 102339 Acked 103338 Write 256 Done Done Done false 102340 Acked 103339 Write 256 Done Done Done false 102341 Acked 103340 Write 256 Done Done Done false 102342 Acked 103341 Write 256 Done Done Done false 102343 Acked 103342 Write 256 Done Done Done false 102344 Acked 103343 Write 256 Done Done Done false 102345 Acked 103344 Write 256 Done Done Done false 102346 Acked 103345 Write 256 Done Done Done false 102347 Acked 103346 Write 256 Done Done Done false 102348 Acked 103347 Write 256 Done Done Done false 102349 Acked 103348 Write 256 Done Done Done false 102350 Acked 103349 Write 256 Done Done Done false 102351 Acked 103350 Write 256 Done Done Done false 102352 Acked 103351 Write 256 Done Done Done false 102353 Acked 103352 Write 256 Done Done Done false 102354 Acked 103353 Write 256 Done Done Done false 102355 Acked 103354 Write 256 Done Done Done false 102356 Acked 103355 Write 256 Done Done Done false 102357 Acked 103356 Write 256 Done Done Done false 102358 Acked 103357 Write 256 Done Done Done false 102359 Acked 103358 Write 256 Done Done Done false 102360 Acked 103359 Write 256 Done Done Done false 102361 Acked 103360 Write 256 Done Done Done false 102362 Acked 103361 Write 256 Done Done Done false 102363 Acked 103362 Write 256 Done Done Done false 102364 Acked 103363 Write 256 Done Done Done false 102365 Acked 103364 Write 256 Done Done Done false 102366 Acked 103365 Write 256 Done Done Done false 102367 Acked 103366 Write 256 Done Done Done false 102368 Acked 103367 Write 256 Done Done Done false 102369 Acked 103368 Write 256 Done Done Done false 102370 Acked 103369 Write 256 Done Done Done false 102371 Acked 103370 Write 256 Done Done Done false 102372 Acked 103371 Write 256 Done Done Done false 102373 Acked 103372 Write 256 Done Done Done false 102374 Acked 103373 Write 256 Done Done Done false 102375 Acked 103374 Write 256 Done Done Done false 102376 Acked 103375 Write 256 Done Done Done false 102377 Acked 103376 Write 256 Done Done Done false 102378 Acked 103377 Write 256 Done Done Done false 102379 Acked 103378 Write 256 Done Done Done false 102380 Acked 103379 Write 256 Done Done Done false 102381 Acked 103380 Write 256 Done Done Done false 102382 Acked 103381 Write 256 Done Done Done false 102383 Acked 103382 Write 256 Done Done Done false 102384 Acked 103383 Write 256 Done Done Done false 102385 Acked 103384 Write 256 Done Done Done false 102386 Acked 103385 Write 256 Done Done Done false 102387 Acked 103386 Write 256 Done Done Done false 102388 Acked 103387 Write 256 Done Done Done false 102389 Acked 103388 Write 256 Done Done Done false 102390 Acked 103389 Write 256 Done Done Done false 102391 Acked 103390 Write 256 Done Done Done false 102392 Acked 103391 Write 256 Done Done Done false 102393 Acked 103392 Write 256 Done Done Done false 102394 Acked 103393 Write 256 Done Done Done false 102395 Acked 103394 Write 256 Done Done Done false 102396 Acked 103395 Write 256 Done Done Done false 102397 Acked 103396 Write 256 Done Done Done false 102398 Acked 103397 Write 256 Done Done Done false 102399 Acked 103398 Write 256 Done Done Done false 102400 Acked 103399 Write 256 Done Done Done false 102401 Acked 103400 Write 256 Done Done Done false 102402 Acked 103401 Write 256 Done Done Done false 102403 Acked 103402 Write 256 Done Done Done false 102404 Acked 103403 Write 256 Done Done Done false 102405 Acked 103404 Write 256 Done Done Done false 102406 Acked 103405 Write 256 Done Done Done false 102407 Acked 103406 Write 256 Done Done Done false 102408 Acked 103407 Write 256 Done Done Done false 102409 Acked 103408 Write 256 Done Done Done false 102410 Acked 103409 Write 256 Done Done Done false 102411 Acked 103410 Write 256 Done Done Done false 102412 Acked 103411 Write 256 Done Done Done false 102413 Acked 103412 Write 256 Done Done Done false 102414 Acked 103413 Write 256 Done Done Done false 102415 Acked 103414 Write 256 Done Done Done false 102416 Acked 103415 Write 256 Done Done Done false 102417 Acked 103416 Write 256 Done Done Done false 102418 Acked 103417 Write 256 Done Done Done false 102419 Acked 103418 Write 256 Done Done Done false 102420 Acked 103419 Write 256 Done Done Done false 102421 Acked 103420 Write 256 Done Done Done false 102422 Acked 103421 Write 256 Done Done Done false 102423 Acked 103422 Write 256 Done Done Done false 102424 Acked 103423 Write 256 Done Done Done false 102425 Acked 103424 Write 256 Done Done Done false 102426 Acked 103425 Write 256 Done Done Done false 102427 Acked 103426 Write 256 Done Done Done false 102428 Acked 103427 Write 256 Done Done Done false 102429 Acked 103428 Write 256 Done Done Done false 102430 Acked 103429 Write 256 Done Done Done false 102431 Acked 103430 Write 256 Done Done Done false 102432 Acked 103431 Write 256 Done Done Done false 102433 Acked 103432 Write 256 Done Done Done false 102434 Acked 103433 Write 256 Done Done Done false 102435 Acked 103434 Write 256 Done Done Done false 102436 Acked 103435 Write 256 Done Done Done false 102437 Acked 103436 Write 256 Done Done Done false 102438 Acked 103437 Write 256 Done Done Done false 102439 Acked 103438 Write 256 Done Done Done false 102440 Acked 103439 Write 256 Done Done Done false 102441 Acked 103440 Write 256 Done Done Done false 102442 Acked 103441 Write 256 Done Done Done false 102443 Acked 103442 Write 256 Done Done Done false 102444 Acked 103443 Write 256 Done Done Done false 102445 Acked 103444 Write 256 Done Done Done false 102446 Acked 103445 Write 256 Done Done Done false 102447 Acked 103446 Write 256 Done Done Done false 102448 Acked 103447 Write 256 Done Done Done false 102449 Acked 103448 Write 256 Done Done Done false 102450 Acked 103449 Write 256 Done Done Done false 102451 Acked 103450 Write 256 Done Done Done false 102452 Acked 103451 Write 256 Done Done Done false 102453 Acked 103452 Write 256 Done Done Done false 102454 Acked 103453 Write 256 Done Done Done false 102455 Acked 103454 Write 256 Done Done Done false 102456 Acked 103455 Write 256 Done Done Done false 102457 Acked 103456 Write 256 Done Done Done false 102458 Acked 103457 Write 256 Done Done Done false 102459 Acked 103458 Write 256 Done Done Done false 102460 Acked 103459 Write 256 Done Done Done false 102461 Acked 103460 Write 256 Done Done Done false 102462 Acked 103461 Write 256 Done Done Done false 102463 Acked 103462 Write 256 Done Done Done false 102464 Acked 103463 Write 256 Done Done Done false 102465 Acked 103464 Write 256 Done Done Done false 102466 Acked 103465 Write 256 Done Done Done false 102467 Acked 103466 Write 256 Done Done Done false 102468 Acked 103467 Write 256 Done Done Done false 102469 Acked 103468 Write 256 Done Done Done false 102470 Acked 103469 Write 256 Done Done Done false 102471 Acked 103470 Write 256 Done Done Done false 102472 Acked 103471 Write 256 Done Done Done false 102473 Acked 103472 Write 256 Done Done Done false 102474 Acked 103473 Write 256 Done Done Done false 102475 Acked 103474 Write 256 Done Done Done false 102476 Acked 103475 Write 256 Done Done Done false 102477 Acked 103476 Write 256 Done Done Done false 102478 Acked 103477 Write 256 Done Done Done false 102479 Acked 103478 Write 256 Done Done Done false 102480 Acked 103479 Write 256 Done Done Done false 102481 Acked 103480 Write 256 Done Done Done false 102482 Acked 103481 Write 256 Done Done Done false 102483 Acked 103482 Write 256 Done Done Done false 102484 Acked 103483 Write 256 Done Done Done false 102485 Acked 103484 Write 256 Done Done Done false 102486 Acked 103485 Write 256 Done Done Done false 102487 Acked 103486 Write 256 Done Done Done false 102488 Acked 103487 Write 256 Done Done Done false 102489 Acked 103488 Write 256 Done Done Done false 102490 Acked 103489 Write 256 Done Done Done false 102491 Acked 103490 Write 256 Done Done Done false 102492 Acked 103491 Write 256 Done Done Done false 102493 Acked 103492 Write 256 Done Done Done false 102494 Acked 103493 Write 256 Done Done Done false 102495 Acked 103494 Write 256 Done Done Done false 102496 Acked 103495 Write 256 Done Done Done false 102497 Acked 103496 Write 256 Done Done Done false 102498 Acked 103497 Write 256 Done Done Done false 102499 Acked 103498 Write 256 Done Done Done false 102500 Acked 103499 Write 256 Done Done Done false 102501 Acked 103500 Write 256 Done Done Done false 102502 Acked 103501 Write 256 Done Done Done false 102503 Acked 103502 Write 256 Done Done Done false 102504 Acked 103503 Write 256 Done Done Done false 102505 Acked 103504 Write 256 Done Done Done false 102506 Acked 103505 Write 256 Done Done Done false 102507 Acked 103506 Write 256 Done Done Done false 102508 Acked 103507 Write 256 Done Done Done false 102509 Acked 103508 Write 256 Done Done Done false 102510 Acked 103509 Write 256 Done Done Done false 102511 Acked 103510 Write 256 Done Done Done false 102512 Acked 103511 Write 256 Done Done Done false 102513 Acked 103512 Write 256 Done Done Done false 102514 Acked 103513 Write 256 Done Done Done false 102515 Acked 103514 Write 256 Done Done Done false 102516 Acked 103515 Write 256 Done Done Done false 102517 Acked 103516 Write 256 Done Done Done false 102518 Acked 103517 Write 256 Done Done Done false 102519 Acked 103518 Write 256 Done Done Done false 102520 Acked 103519 Write 256 Done Done Done false 102521 Acked 103520 Write 256 Done Done Done false 102522 Acked 103521 Write 256 Done Done Done false 102523 Acked 103522 Write 256 Done Done Done false 102524 Acked 103523 Write 256 Done Done Done false 102525 Acked 103524 Write 256 Done Done Done false 102526 Acked 103525 Write 256 Done Done Done false 102527 Acked 103526 Write 256 Done Done Done false 102528 Acked 103527 Write 256 Done Done Done false 102529 Acked 103528 Write 256 Done Done Done false 102530 Acked 103529 Write 256 Done Done Done false 102531 Acked 103530 Write 256 Done Done Done false 102532 Acked 103531 Write 256 Done Done Done false 102533 Acked 103532 Write 256 Done Done Done false 102534 Acked 103533 Write 256 Done Done Done false 102535 Acked 103534 Write 256 Done Done Done false 102536 Acked 103535 Write 256 Done Done Done false 102537 Acked 103536 Write 256 Done Done Done false 102538 Acked 103537 Write 256 Done Done Done false 102539 Acked 103538 Write 256 Done Done Done false 102540 Acked 103539 Write 256 Done Done Done false 102541 Acked 103540 Write 256 Done Done Done false 102542 Acked 103541 Write 256 Done Done Done false 102543 Acked 103542 Write 256 Done Done Done false 102544 Acked 103543 Write 256 Done Done Done false 102545 Acked 103544 Write 256 Done Done Done false 102546 Acked 103545 Write 256 Done Done Done false 102547 Acked 103546 Write 256 Done Done Done false 102548 Acked 103547 Write 256 Done Done Done false 102549 Acked 103548 Write 256 Done Done Done false 102550 Acked 103549 Write 256 Done Done Done false 102551 Acked 103550 Write 256 Done Done Done false 102552 Acked 103551 Write 256 Done Done Done false 102553 Acked 103552 Write 256 Done Done Done false 102554 Acked 103553 Write 256 Done Done Done false 102555 Acked 103554 Write 256 Done Done Done false 102556 Acked 103555 Write 256 Done Done Done false 102557 Acked 103556 Write 256 Done Done Done false 102558 Acked 103557 Write 256 Done Done Done false 102559 Acked 103558 Write 256 Done Done Done false 102560 Acked 103559 Write 256 Done Done Done false 102561 Acked 103560 Write 256 Done Done Done false 102562 Acked 103561 Write 256 Done Done Done false 102563 Acked 103562 Write 256 Done Done Done false 102564 Acked 103563 Write 256 Done Done Done false 102565 Acked 103564 Write 256 Done Done Done false 102566 Acked 103565 Write 256 Done Done Done false 102567 Acked 103566 Write 256 Done Done Done false 102568 Acked 103567 Write 256 Done Done Done false 102569 Acked 103568 Write 256 Done Done Done false 102570 Acked 103569 Write 256 Done Done Done false 102571 Acked 103570 Write 256 Done Done Done false 102572 Acked 103571 Write 256 Done Done Done false 102573 Acked 103572 Write 256 Done Done Done false 102574 Acked 103573 Write 256 Done Done Done false 102575 Acked 103574 Write 256 Done Done Done false 102576 Acked 103575 Write 256 Done Done Done false 102577 Acked 103576 Write 256 Done Done Done false 102578 Acked 103577 Write 256 Done Done Done false 102579 Acked 103578 Write 256 Done Done Done false 102580 Acked 103579 Write 256 Done Done Done false 102581 Acked 103580 Write 256 Done Done Done false 102582 Acked 103581 Write 256 Done Done Done false 102583 Acked 103582 Write 256 Done Done Done false 102584 Acked 103583 Write 256 Done Done Done false 102585 Acked 103584 Write 256 Done Done Done false 102586 Acked 103585 Write 256 Done Done Done false 102587 Acked 103586 Write 256 Done Done Done false 102588 Acked 103587 Write 256 Done Done Done false 102589 Acked 103588 Write 256 Done Done Done false 102590 Acked 103589 Write 256 Done Done Done false 102591 Acked 103590 Write 256 Done Done Done false 102592 Acked 103591 Write 256 Done Done Done false 102593 Acked 103592 Write 256 Done Done Done false 102594 Acked 103593 Write 256 Done Done Done false 102595 Acked 103594 Write 256 Done Done Done false 102596 Acked 103595 Write 256 Done Done Done false 102597 Acked 103596 Write 256 Done Done Done false 102598 Acked 103597 Write 256 Done Done Done false 102599 Acked 103598 Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 306 306 305 917 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 103598 103598 103292 Downstairs last five completed: 103292 103291 103290 103289 103288 Upstairs last five completed: 102599 102598 102597 102596 102595 CLIENT: Up:0 ds:306 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 103598 103597 103596 103595 103594 Upstairs last five completed: 102599 102598 102597 102596 102595 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program