Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1488) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1489) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1490) [8820][1] initial start wait reports Running All downstairs are running Sep 27 03:52:31.322 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Sep 27 03:52:36.362 INFO accepted connection, remote_addr: 127.0.0.1:49745, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Sep 27 03:52:36.363 INFO request completed, latency_us: 143, [8810] Disable keep_running response_code[8820] Disable keep_running : 204, uri: /disablerestart/all, method: POST, req_id: d63adf94-0d9e-4588-8202-f7ec0e86912f, remote_addr: 127[8830] Disable keep_running .0.0.1:49745, local_addr: 127.0.0.1:9998 Sep 27 03:54:24.267 INFO accepted connection, remote_addr: 127.0.0.1:58690, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Sep 27 03:54:24.268 Shut it down INFO[8810] Disable keep_running [8810] Got stop action so:false kr:false request completed, latency_us: 63, response_code: 204, uri: /shutdown, method: [8830] Disable keep_running POST[8830] Got stop action so:false kr:false , req_id: ab47cfdc-b21f-4d04-ad1b-ee4c7b94d257, remote_addr[8820] Disable keep_running : 127.0.0.1:58690, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit 93} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6901b9ee7172d672adc2780a2da758752eb0a2a4\",\n git_commit_timestamp: \"2024-09-27T03:30:33.000000000Z\",\n git_branch: \"alan/more-dsc-for-tests\",\n rustc_semver: \"1.80.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"051478957371ee0084a7c0913941d2a8c4757bb9\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415435094Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415442393Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415448932Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493} {"msg":"Crucible 6c27f373-031a-43c1-ba84-c923ec15461d has session id: 1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415712091Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Upstairs opts: Upstairs UUID: 6c27f373-031a-43c1-ba84-c923ec15461d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415728509Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Crucible stats registered with UUID: 6c27f373-031a-43c1-ba84-c923ec15461d","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.415736758Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445821944Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493} {"msg":"6c27f373-031a-43c1-ba84-c923ec15461d active request set","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445835671Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.44584461Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445859818Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445869216Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445878515Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445887903Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445897592Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445907151Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445916549Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.445925628Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_connection connected from Ok(127.0.0.1:36153)","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.44597294Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_connection connected from Ok(127.0.0.1:38321)","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446045639Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_connection connected from Ok(127.0.0.1:53142)","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446059807Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446366959Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446382846Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446458775Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446471313Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446480151Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446541542Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446600223Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446659434Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.446760088Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.453020415Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.453029064Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.453033443Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454161298Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454169647Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454174816Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.45459787Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454608739Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454614368Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454619737Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454625026Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454630175Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454635304Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454640464Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454645603Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454650732Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454655881Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454661011Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.45466619Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454671689Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454676978Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"":"downstairs","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454683807Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454688366Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"0","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454693765Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"1","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454699125Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"client":"2","":"downstairs","session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454704564Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454709203Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"6c27f373-031a-43c1-ba84-c923ec15461d is now active with session: 1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454713872Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454718451Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493,"session_id":"1ddf7e1e-e59c-4883-bbaf-d458f6ee8f82"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-09-27T03:52:36.454723051Z","hostname":"w-01J8RSMRHMNQ8ZBV367MYP4QPM","pid":1493} Wait for a query_work_queue command to finish before sending IO region info: block size: 4096 bytes sub_volume 0 blocks / extent: 16384 sub_volume 0 extent size: 64 MiB sub_volume 0 extent count: 1600 total blocks: 26214400 total size: 100 GiB encryption: no Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:154 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 102452 Acked 103451 Write 256 Done Done Done false 102453 Acked 103452 Write 256 Done Done Done false 102454 Acked 103453 Write 256 Done Done Done false 102455 Acked 103454 Write 256 Done Done Done false 102456 Acked 103455 Write 256 Done Done Done false 102457 Acked 103456 Write 256 Done Done Done false 102458 Acked 103457 Write 256 Done Done Done false 102459 Acked 103458 Write 256 Done Done Done false 102460 Acked 103459 Write 256 Done Done Done false 102461 Acked 103460 Write 256 Done Done Done false 102462 Acked 103461 Write 256 Done Done Done false 102463 Acked 103462 Write 256 Done Done Done false 102464 Acked 103463 Write 256 Done Done Done false 102465 Acked 103464 Write 256 Done Done Done false 102466 Acked 103465 Write 256 Done Done Done false 102467 Acked 103466 Write 256 Done Done Done false 102468 Acked 103467 Write 256 Done Done Done false 102469 Acked 103468 Write 256 Done Done Done false 102470 Acked 103469 Write 256 Done Done Done false 102471 Acked 103470 Write 256 Done Done Done false 102472 Acked 103471 Write 256 Done Done Done false 102473 Acked 103472 Write 256 Done Done Done false 102474 Acked 103473 Write 256 Done Done Done false 102475 Acked 103474 Write 256 Done Done Done false 102476 Acked 103475 Write 256 Done Done Done false 102477 Acked 103476 Write 256 Done Done Done false 102478 Acked 103477 Write 256 Done Done Done false 102479 Acked 103478 Write 256 Done Done Done false 102480 Acked 103479 Write 256 Done Done Done false 102481 Acked 103480 Write 256 Done Done Done false 102482 Acked 103481 Write 256 Done Done Done false 102483 Acked 103482 Write 256 Done Done Done false 102484 Acked 103483 Write 256 Done Done Done false 102485 Acked 103484 Write 256 Done Done Done false 102486 Acked 103485 Write 256 Done Done Done false 102487 Acked 103486 Write 256 Done Done Done false 102488 Acked 103487 Write 256 Done Done Done false 102489 Acked 103488 Write 256 Done Done Done false 102490 Acked 103489 Write 256 Done Done Done false 102491 Acked 103490 Write 256 Done Done Done false 102492 Acked 103491 Write 256 Done Done Done false 102493 Acked 103492 Write 256 Done Done Done false 102494 Acked 103493 Write 256 Done Done Done false 102495 Acked 103494 Write 256 Done Done Done false 102496 Acked 103495 Write 256 Done Done Done false 102497 Acked 103496 Write 256 Done Done Done false 102498 Acked 103497 Write 256 Done Done Done false 102499 Acked 103498 Write 256 Done Done Done false 102500 Acked 103499 Write 256 Done Done Done false 102501 Acked 103500 Write 256 Done Done Done false 102502 Acked 103501 Write 256 Done Done Done false 102503 Acked 103502 Write 256 Done Done Done false 102504 Acked 103503 Write 256 Done Done Done false 102505 Acked 103504 Write 256 Done Done Done false 102506 Acked 103505 Write 256 Done Done Done false 102507 Acked 103506 Write 256 Done Done Done false 102508 Acked 103507 Write 256 Done Done Done false 102509 Acked 103508 Write 256 Done Done Done false 102510 Acked 103509 Write 256 Done Done Done false 102511 Acked 103510 Write 256 Done Done Done false 102512 Acked 103511 Write 256 Done Done Done false 102513 Acked 103512 Write 256 Done Done Done false 102514 Acked 103513 Write 256 Done Done Done false 102515 Acked 103514 Write 256 Done Done Done false 102516 Acked 103515 Write 256 Done Done Done false 102517 Acked 103516 Write 256 Done Done Done false 102518 Acked 103517 Write 256 Done Done Done false 102519 Acked 103518 Write 256 Done Done Done false 102520 Acked 103519 Write 256 Done Done Done false 102521 Acked 103520 Write 256 Done Done Done false 102522 Acked 103521 Write 256 Done Done Done false 102523 Acked 103522 Write 256 Done Done Done false 102524 Acked 103523 Write 256 Done Done Done false 102525 Acked 103524 Write 256 Done Done Done false 102526 Acked 103525 Write 256 Done Done Done false 102527 Acked 103526 Write 256 Done Done Done false 102528 Acked 103527 Write 256 Done Done Done false 102529 Acked 103528 Write 256 Done Done Done false 102530 Acked 103529 Write 256 Done Done Done false 102531 Acked 103530 Write 256 Done Done Done false 102532 Acked 103531 Write 256 Done Done Done false 102533 Acked 103532 Write 256 Done Done Done false 102534 Acked 103533 Write 256 Done Done Done false 102535 Acked 103534 Write 256 Done Done Done false 102536 Acked 103535 Write 256 Done Done Done false 102537 Acked 103536 Write 256 Done Done Done false 102538 Acked 103537 Write 256 Done Done Done false 102539 Acked 103538 Write 256 Done Done Done false 102540 Acked 103539 Write 256 Done Done Done false 102541 Acked 103540 Write 256 Done Done Done false 102542 Acked 103541 Write 256 Done Done Done false 102543 Acked 103542 Write 256 Done Done Done false 102544 Acked 103543 Write 256 Done Done Done false 102545 Acked 103544 Write 256 Done Done Done false 102546 Acked 103545 Write 256 Done Done Done false 102547 Acked 103546 Write 256 Done Done Done false 102548 Acked 103547 Write 256 Done Done Done false 102549 Acked 103548 Write 256 Done Done Done false 102550 Acked 103549 Write 256 Done Done Done false 102551 Acked 103550 Write 256 Done Done Done false 102552 Acked 103551 Write 256 Done Done Done false 102553 Acked 103552 Write 256 Done Done Done false 102554 Acked 103553 Write 256 Done Done Done false 102555 Acked 103554 Write 256 Done Done Done false 102556 Acked 103555 Write 256 Done Done Done false 102557 Acked 103556 Write 256 Done Done Done false 102558 Acked 103557 Write 256 Done Done Done false 102559 Acked 103558 Write 256 Done Done Done false 102560 Acked 103559 Write 256 Done Done Done false 102561 Acked 103560 Write 256 Done Done Done false 102562 Acked 103561 Write 256 Done Done Done false 102563 Acked 103562 Write 256 Done Done Done false 102564 Acked 103563 Write 256 Done Done Done false 102565 Acked 103564 Write 256 Done Done Done false 102566 Acked 103565 Write 256 Done Done Done false 102567 Acked 103566 Write 256 Done Done Done false 102568 Acked 103567 Write 256 Done Done Done false 102569 Acked 103568 Write 256 Done Done Done false 102570 Acked 103569 Write 256 Done Done Done false 102571 Acked 103570 Write 256 Done Done Done false 102572 Acked 103571 Write 256 Done Done Done false 102573 Acked 103572 Write 256 Done Done Done false 102574 Acked 103573 Write 256 Done Done Done false 102575 Acked 103574 Write 256 Done Done Done false 102576 Acked 103575 Write 256 Done Done Done false 102577 Acked 103576 Write 256 Done Done Done false 102578 Acked 103577 Write 256 Done Done Done false 102579 Acked 103578 Write 256 Done Done Done false 102580 Acked 103579 Write 256 Done Done Done false 102581 Acked 103580 Write 256 Done Done Done false 102582 Acked 103581 Write 256 Done Done Done false 102583 Acked 103582 Write 256 Done Done Done false 102584 Acked 103583 Write 256 Done Done Done false 102585 Acked 103584 Write 256 Done Done Done false 102586 Acked 103585 Write 256 Done Done Done false 102587 Acked 103586 Write 256 Done Done Done false 102588 Acked 103587 Write 256 Done Done Done false 102589 Acked 103588 Write 256 Done Done Done false 102590 Acked 103589 Write 256 Done Done Done false 102591 Acked 103590 Write 256 Done Done Done false 102592 Acked 103591 Write 256 Done Done Done false 102593 Acked 103592 Write 256 Done Done Done false 102594 Acked 103593 Write 256 Done Done Done false 102595 Acked 103594 Write 256 Done Done Done false 102596 Acked 103595 Write 256 Done Done Done false 102597 Acked 103596 Write 256 Done Done Done false 102598 Acked 103597 Write 256 Done Done Done false 102599 Acked 103598 Write 256 Done Done Done false 102600 Acked 103599 Write 256 Done Done Done false 102601 Acked 103600 Write 256 Done Done Done false 102602 Acked 103601 Write 256 Done Done Done false 102603 Acked 103602 Write 256 Done Done Done false 102604 Acked 103603 Write 256 Done Done Done false 102605 Acked 103604 Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 154 154 153 461 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 103604 103604 103450 Downstairs last five completed: 103450 103449 103448 103447 103446 Upstairs last five completed: 102605 102604 102603 102602 102601 CLIENT: Up:0 ds:154 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 103604 103603 103602 103601 103600 Upstairs last five completed: 102605 102604 102603 102602 102601 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program